Nov 01 14:16:08.541468 ip-10-0-134-37 systemd[1]: Starting Kubernetes Kubelet...
Nov 01 14:16:08.936305 ip-10-0-134-37 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 14:16:08.936305 ip-10-0-134-37 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 01 14:16:08.936305 ip-10-0-134-37 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 14:16:08.936305 ip-10-0-134-37 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 14:16:08.944080 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.943995    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 14:16:08.944080 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944076    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 14:16:08.944080 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944080    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 14:16:08.944080 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944083    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 14:16:08.944208 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944086    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 14:16:08.944208 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944090    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 14:16:08.944208 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944094    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 14:16:08.944208 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944097    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 14:16:08.944208 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944099    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 14:16:08.944208 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944102    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 14:16:08.944208 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944105    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 14:16:08.944208 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944111    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 14:16:08.944208 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944114    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 14:16:08.944208 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944117    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 14:16:08.944208 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944119    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 14:16:08.944208 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944122    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 14:16:08.944208 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944124    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 14:16:08.944208 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944127    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 14:16:08.944208 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944130    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 14:16:08.944208 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944132    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 14:16:08.944208 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944135    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 14:16:08.944208 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944137    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 14:16:08.944208 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944141    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 14:16:08.944208 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944143    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 14:16:08.944671 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944146    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 14:16:08.944671 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944148    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 14:16:08.944671 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944151    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 14:16:08.944671 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944154    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 14:16:08.944671 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944156    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 14:16:08.944671 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944172    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 14:16:08.944671 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944176    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 14:16:08.944671 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944181    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 14:16:08.944671 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944184    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 14:16:08.944671 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944187    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 14:16:08.944671 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944190    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 14:16:08.944671 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944193    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 14:16:08.944671 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944196    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 14:16:08.944671 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944199    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 14:16:08.944671 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944202    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 14:16:08.944671 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944205    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 14:16:08.944671 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944207    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 14:16:08.944671 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944211    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 14:16:08.944671 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944213    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 14:16:08.945111 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944216    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 14:16:08.945111 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944219    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 14:16:08.945111 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944222    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 14:16:08.945111 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944224    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 14:16:08.945111 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944227    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 14:16:08.945111 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944230    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 14:16:08.945111 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944232    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 14:16:08.945111 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944235    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 14:16:08.945111 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944238    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 14:16:08.945111 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944240    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 14:16:08.945111 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944243    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 14:16:08.945111 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944245    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 14:16:08.945111 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944248    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 14:16:08.945111 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944251    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 14:16:08.945111 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944254    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 14:16:08.945111 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944256    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 14:16:08.945111 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944259    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 14:16:08.945111 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944261    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 14:16:08.945111 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944264    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 14:16:08.945111 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944267    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 14:16:08.945586 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944269    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 14:16:08.945586 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944272    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 14:16:08.945586 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944274    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 14:16:08.945586 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944277    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 14:16:08.945586 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944280    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 14:16:08.945586 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944282    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 14:16:08.945586 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944285    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 14:16:08.945586 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944288    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 14:16:08.945586 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944291    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 14:16:08.945586 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944295    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 14:16:08.945586 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944298    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 14:16:08.945586 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944301    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 14:16:08.945586 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944304    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 14:16:08.945586 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944306    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 14:16:08.945586 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944309    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 14:16:08.945586 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944319    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 14:16:08.945586 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944323    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 14:16:08.945586 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944326    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 14:16:08.945586 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944329    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 14:16:08.946042 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944332    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 14:16:08.946042 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944334    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 14:16:08.946042 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944337    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 14:16:08.946042 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944340    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 14:16:08.946042 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944342    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 14:16:08.946042 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944344    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 14:16:08.946042 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944347    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 14:16:08.946042 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944350    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 14:16:08.946042 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944352    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 14:16:08.946042 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944355    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 14:16:08.946042 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944358    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 14:16:08.946042 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944360    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 14:16:08.946042 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944363    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 14:16:08.946042 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944365    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 14:16:08.946042 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944368    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 14:16:08.946042 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944371    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 14:16:08.946042 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944373    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 14:16:08.946042 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944376    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 14:16:08.946042 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944379    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 14:16:08.946042 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944382    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 14:16:08.946510 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944385    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 14:16:08.946510 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944387    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 14:16:08.946510 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944390    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 14:16:08.946510 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.944393    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 14:16:08.946510 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945202    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 14:16:08.946510 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945209    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 14:16:08.946510 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945212    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 14:16:08.946510 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945215    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 14:16:08.946510 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945217    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 14:16:08.946510 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945220    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 14:16:08.946510 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945223    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 14:16:08.946510 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945226    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 14:16:08.946510 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945228    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 14:16:08.946510 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945231    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 14:16:08.946510 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945234    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 14:16:08.946510 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945236    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 14:16:08.946510 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945239    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 14:16:08.946510 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945242    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 14:16:08.946510 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945244    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 14:16:08.946966 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945247    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 14:16:08.946966 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945249    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 14:16:08.946966 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945252    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 14:16:08.946966 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945255    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 14:16:08.946966 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945259    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 14:16:08.946966 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945263    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 14:16:08.946966 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945266    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 14:16:08.946966 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945269    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 14:16:08.946966 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945271    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 14:16:08.946966 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945274    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 14:16:08.946966 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945277    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 14:16:08.946966 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945280    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 14:16:08.946966 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945282    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 14:16:08.946966 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945285    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 14:16:08.946966 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945288    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 14:16:08.946966 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945291    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 14:16:08.946966 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945294    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 14:16:08.946966 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945302    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 14:16:08.946966 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945305    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 14:16:08.947423 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945308    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 14:16:08.947423 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945310    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 14:16:08.947423 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945313    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 14:16:08.947423 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945316    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 14:16:08.947423 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945318    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 14:16:08.947423 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945321    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 14:16:08.947423 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945325    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 14:16:08.947423 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945327    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 14:16:08.947423 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945331    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 14:16:08.947423 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945335    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 14:16:08.947423 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945338    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 14:16:08.947423 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945341    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 14:16:08.947423 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945343    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 14:16:08.947423 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945346    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 14:16:08.947423 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945349    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 14:16:08.947423 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945351    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 14:16:08.947423 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945354    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 14:16:08.947423 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945357    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 14:16:08.947423 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945359    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 14:16:08.947863 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945362    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 14:16:08.947863 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945364    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 14:16:08.947863 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945367    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 14:16:08.947863 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945369    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 14:16:08.947863 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945373    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 14:16:08.947863 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945376    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 14:16:08.947863 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945379    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 14:16:08.947863 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945382    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 14:16:08.947863 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945384    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 14:16:08.947863 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945387    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 14:16:08.947863 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945390    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 14:16:08.947863 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945392    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 14:16:08.947863 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945400    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 14:16:08.947863 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945403    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 14:16:08.947863 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945406    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 14:16:08.947863 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945408    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 14:16:08.947863 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945411    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 14:16:08.947863 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945413    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 14:16:08.947863 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945416    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 14:16:08.947863 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945418    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 14:16:08.948352 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945421    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 14:16:08.948352 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945423    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 14:16:08.948352 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945426    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 14:16:08.948352 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945428    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 14:16:08.948352 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945431    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 14:16:08.948352 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945433    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 14:16:08.948352 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945436    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 14:16:08.948352 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945439    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 14:16:08.948352 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945442    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 14:16:08.948352 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945444    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 14:16:08.948352 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945447    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 14:16:08.948352 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945449    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 14:16:08.948352 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945451    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 14:16:08.948352 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945454    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 14:16:08.948352 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945456    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 14:16:08.948352 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945459    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 14:16:08.948352 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945461    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 14:16:08.948352 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945464    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 14:16:08.948352 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945467    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 14:16:08.948831 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945469    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 14:16:08.948831 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945472    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 14:16:08.948831 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945474    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 14:16:08.948831 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945477    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 14:16:08.948831 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945479    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 14:16:08.948831 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945482    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 14:16:08.948831 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945489    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 14:16:08.948831 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945492    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 14:16:08.948831 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945494    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 14:16:08.948831 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945497    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 14:16:08.948831 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945500    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 14:16:08.948831 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945502    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 14:16:08.948831 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945504    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 14:16:08.948831 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.945507    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 14:16:08.948831 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946063    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 14:16:08.948831 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946070    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 14:16:08.948831 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946074    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 14:16:08.948831 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946077    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 14:16:08.948831 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946080    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 14:16:08.948831 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946082    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 14:16:08.949324 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946085    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 14:16:08.949324 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946088    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 14:16:08.949324 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946091    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 14:16:08.949324 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946093    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 14:16:08.949324 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946096    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 14:16:08.949324 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946099    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 14:16:08.949324 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946102    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 14:16:08.949324 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946104    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 14:16:08.949324 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946107    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 14:16:08.949324 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946110    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 14:16:08.949324 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946112    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 14:16:08.949324 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946116    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 14:16:08.949324 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946120    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 14:16:08.949324 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946123    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 14:16:08.949324 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946126    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 14:16:08.949324 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946129    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 14:16:08.949324 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946131    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 14:16:08.949324 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946134    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 14:16:08.949324 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946137    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 14:16:08.949778 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946140    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 14:16:08.949778 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946142    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 14:16:08.949778 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946145    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 14:16:08.949778 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946148    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 14:16:08.949778 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946150    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 14:16:08.949778 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946153    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 14:16:08.949778 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946155    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 14:16:08.949778 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946158    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 14:16:08.949778 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946175    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 14:16:08.949778 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946178    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 14:16:08.949778 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946181    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 14:16:08.949778 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946184    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 14:16:08.949778 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946187    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 14:16:08.949778 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946189    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 14:16:08.949778 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946192    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 14:16:08.949778 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946194    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 14:16:08.949778 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946197    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 14:16:08.949778 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946199    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 14:16:08.949778 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946202    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 14:16:08.950271 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946205    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 14:16:08.950271 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946207    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 14:16:08.950271 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946210    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 14:16:08.950271 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946212    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 14:16:08.950271 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946215    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 14:16:08.950271 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946218    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 14:16:08.950271 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946220    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 14:16:08.950271 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946223    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 14:16:08.950271 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946225    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 14:16:08.950271 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946228    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 14:16:08.950271 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946230    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 14:16:08.950271 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946233    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 14:16:08.950271 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946236    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 14:16:08.950271 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946239    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 14:16:08.950271 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946242    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 14:16:08.950271 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946244    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 14:16:08.950271 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946247    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 14:16:08.950271 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946249    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 14:16:08.950271 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946252    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 14:16:08.950271 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946255    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 14:16:08.950754 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946258    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 14:16:08.950754 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946261    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 14:16:08.950754 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946263    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 14:16:08.950754 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946266    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 14:16:08.950754 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946269    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 14:16:08.950754 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946272    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 14:16:08.950754 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946274    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 14:16:08.950754 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946277    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 14:16:08.950754 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946279    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 14:16:08.950754 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946282    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 14:16:08.950754 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946284    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 14:16:08.950754 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946287    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 14:16:08.950754 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946289    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 14:16:08.950754 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946292    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 14:16:08.950754 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946294    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 14:16:08.950754 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946297    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 14:16:08.950754 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946299    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 14:16:08.950754 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946302    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 14:16:08.950754 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946304    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 14:16:08.951216 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946307    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 14:16:08.951216 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946309    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 14:16:08.951216 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946312    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 14:16:08.951216 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946314    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 14:16:08.951216 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946317    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 14:16:08.951216 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946319    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 14:16:08.951216 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946322    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 14:16:08.951216 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946325    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 14:16:08.951216 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946328    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 14:16:08.951216 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946330    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 14:16:08.951216 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946333    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 14:16:08.951216 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946335    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 14:16:08.951216 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946337    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 14:16:08.951216 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946340    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 14:16:08.951216 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946342    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 14:16:08.951216 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946345    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 14:16:08.951216 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946347    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 14:16:08.951216 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946350    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 14:16:08.951216 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946353    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 14:16:08.951216 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946356    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 14:16:08.951714 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946358    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 14:16:08.951714 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946361    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 14:16:08.951714 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.946363    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 14:16:08.951714 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947180    2576 flags.go:64] FLAG: --address="0.0.0.0"
Nov 01 14:16:08.951714 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947189    2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 01 14:16:08.951714 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947195    2576 flags.go:64] FLAG: --anonymous-auth="true"
Nov 01 14:16:08.951714 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947200    2576 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 01 14:16:08.951714 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947204    2576 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 01 14:16:08.951714 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947208    2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 01 14:16:08.951714 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947212    2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 01 14:16:08.951714 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947216    2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 01 14:16:08.951714 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947219    2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 01 14:16:08.951714 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947222    2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 01 14:16:08.951714 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947226    2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 01 14:16:08.951714 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947229    2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 01 14:16:08.951714 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947232    2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 01 14:16:08.951714 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947235    2576 flags.go:64] FLAG: --cgroup-root=""
Nov 01 14:16:08.951714 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947238    2576 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 01 14:16:08.951714 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947241    2576 flags.go:64] FLAG: --client-ca-file=""
Nov 01 14:16:08.951714 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947244    2576 flags.go:64] FLAG: --cloud-provider="external"
Nov 01 14:16:08.951714 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947248    2576 flags.go:64] FLAG: --cluster-dns="[]"
Nov 01 14:16:08.951714 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947253    2576 flags.go:64] FLAG: --cluster-domain=""
Nov 01 14:16:08.951714 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947256    2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 01 14:16:08.951714 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947259    2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 01 14:16:08.952300 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947262    2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 01 14:16:08.952300 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947265    2576 flags.go:64] FLAG: --container-log-max-files="5"
Nov 01 14:16:08.952300 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947269    2576 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 01 14:16:08.952300 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947272    2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 01 14:16:08.952300 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947275    2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 01 14:16:08.952300 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947278    2576 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 01 14:16:08.952300 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947281    2576 flags.go:64] FLAG: --contention-profiling="false"
Nov 01 14:16:08.952300 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947284    2576 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 01 14:16:08.952300 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947288    2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 01 14:16:08.952300 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947291    2576 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 01 14:16:08.952300 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947294    2576 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 01 14:16:08.952300 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947298    2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 01 14:16:08.952300 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947301    2576 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 01 14:16:08.952300 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947304    2576 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 01 14:16:08.952300 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947307    2576 flags.go:64] FLAG: --enable-load-reader="false"
Nov 01 14:16:08.952300 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947310    2576 flags.go:64] FLAG: --enable-server="true"
Nov 01 14:16:08.952300 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947313    2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 01 14:16:08.952300 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947318    2576 flags.go:64] FLAG: --event-burst="100"
Nov 01 14:16:08.952300 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947321    2576 flags.go:64] FLAG: --event-qps="50"
Nov 01 14:16:08.952300 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947324    2576 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 01 14:16:08.952300 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947327    2576 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 01 14:16:08.952300 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947330    2576 flags.go:64] FLAG: --eviction-hard=""
Nov 01 14:16:08.952300 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947334    2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 01 14:16:08.952300 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947337    2576 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 01 14:16:08.952890 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947340    2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 01 14:16:08.952890 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947343    2576 flags.go:64] FLAG: --eviction-soft=""
Nov 01 14:16:08.952890 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947346    2576 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 01 14:16:08.952890 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947349    2576 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 01 14:16:08.952890 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947352    2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 01 14:16:08.952890 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947356    2576 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 01 14:16:08.952890 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947359    2576 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 01 14:16:08.952890 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947361    2576 flags.go:64] FLAG: --fail-swap-on="true"
Nov 01 14:16:08.952890 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947364    2576 flags.go:64] FLAG: --feature-gates=""
Nov 01 14:16:08.952890 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947368    2576 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 01 14:16:08.952890 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947371    2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 01 14:16:08.952890 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947375    2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 01 14:16:08.952890 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947378    2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 01 14:16:08.952890 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947381    2576 flags.go:64] FLAG: --healthz-port="10248"
Nov 01 14:16:08.952890 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947384    2576 flags.go:64] FLAG: --help="false"
Nov 01 14:16:08.952890 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947387    2576 flags.go:64] FLAG: --hostname-override="ip-10-0-134-37.us-west-2.compute.internal"
Nov 01 14:16:08.952890 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947390    2576 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 01 14:16:08.952890 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947393    2576 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 01 14:16:08.952890 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947396    2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 01 14:16:08.952890 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947399    2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 01 14:16:08.952890 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947403    2576 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 01 14:16:08.952890 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947406    2576 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 01 14:16:08.952890 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947408    2576 flags.go:64] FLAG: --image-service-endpoint=""
Nov 01 14:16:08.953456 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947412    2576 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 01 14:16:08.953456 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947415    2576 flags.go:64] FLAG: --kube-api-burst="100"
Nov 01 14:16:08.953456 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947417    2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 01 14:16:08.953456 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947421    2576 flags.go:64] FLAG: --kube-api-qps="50"
Nov 01 14:16:08.953456 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947423    2576 flags.go:64] FLAG: --kube-reserved=""
Nov 01 14:16:08.953456 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947426    2576 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 01 14:16:08.953456 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947429    2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 01 14:16:08.953456 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947433    2576 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 01 14:16:08.953456 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947436    2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 01 14:16:08.953456 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947439    2576 flags.go:64] FLAG: --lock-file=""
Nov 01 14:16:08.953456 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947442    2576 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 01 14:16:08.953456 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947445    2576 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 01 14:16:08.953456 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947448    2576 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 01 14:16:08.953456 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947454    2576 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 01 14:16:08.953456 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947456    2576 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 01 14:16:08.953456 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947460    2576 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 01 14:16:08.953456 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947463    2576 flags.go:64] FLAG: --logging-format="text"
Nov 01 14:16:08.953456 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947466    2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 01 14:16:08.953456 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947470    2576 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 01 14:16:08.953456 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947472    2576 flags.go:64] FLAG: --manifest-url=""
Nov 01 14:16:08.953456 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947476    2576 flags.go:64] FLAG: --manifest-url-header=""
Nov 01 14:16:08.953456 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947480    2576 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 01 14:16:08.953456 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947483    2576 flags.go:64] FLAG: --max-open-files="1000000"
Nov 01 14:16:08.953456 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947487    2576 flags.go:64] FLAG: --max-pods="110"
Nov 01 14:16:08.953456 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947491    2576 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 01 14:16:08.954074 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947494    2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 01 14:16:08.954074 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947497    2576 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 01 14:16:08.954074 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947501    2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 01 14:16:08.954074 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947504    2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 01 14:16:08.954074 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947507    2576 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 01 14:16:08.954074 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947510    2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 01 14:16:08.954074 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947516    2576 flags.go:64] FLAG: --node-status-max-images="50"
Nov 01 14:16:08.954074 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947520    2576 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 01 14:16:08.954074 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947523    2576 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 01 14:16:08.954074 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947526    2576 flags.go:64] FLAG: --pod-cidr=""
Nov 01 14:16:08.954074 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947529    2576 flags.go:64] FLAG: --pod-manifest-path=""
Nov 01 14:16:08.954074 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947532    2576 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 01 14:16:08.954074 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947535    2576 flags.go:64] FLAG: --pods-per-core="0"
Nov 01 14:16:08.954074 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947539    2576 flags.go:64] FLAG: --port="10250"
Nov 01 14:16:08.954074 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947543    2576 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 01 14:16:08.954074 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947546    2576 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-06bb2c418a73ad5ed"
Nov 01 14:16:08.954074 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947549    2576 flags.go:64] FLAG: --qos-reserved=""
Nov 01 14:16:08.954074 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947552    2576 flags.go:64] FLAG: --read-only-port="10255"
Nov 01 14:16:08.954074 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947555    2576 flags.go:64] FLAG: --register-node="true"
Nov 01 14:16:08.954074 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947558    2576 flags.go:64] FLAG: --register-with-taints=""
Nov 01 14:16:08.954074 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947562    2576 flags.go:64] FLAG: --registry-burst="10"
Nov 01 14:16:08.954074 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947565    2576 flags.go:64] FLAG: --registry-qps="5"
Nov 01 14:16:08.954074 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947568    2576 flags.go:64] FLAG: --reserved-cpus=""
Nov 01 14:16:08.954074 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947571    2576 flags.go:64] FLAG: --reserved-memory=""
Nov 01 14:16:08.954074 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947575    2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 01 14:16:08.954074 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947578    2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 01 14:16:08.954877 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947582    2576 flags.go:64] FLAG: --rotate-certificates="false"
Nov 01 14:16:08.954877 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947586    2576 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 01 14:16:08.954877 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947589    2576 flags.go:64] FLAG: --runonce="false"
Nov 01 14:16:08.954877 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947592    2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 01 14:16:08.954877 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947595    2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 01 14:16:08.954877 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947599    2576 flags.go:64] FLAG: --seccomp-default="false"
Nov 01 14:16:08.954877 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947602    2576 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 01 14:16:08.954877 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947605    2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 01 14:16:08.954877 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947608    2576 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 01 14:16:08.954877 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947611    2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 01 14:16:08.954877 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947614    2576 flags.go:64] FLAG: --storage-driver-password="root"
Nov 01 14:16:08.954877 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947617    2576 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 01 14:16:08.954877 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947620    2576 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 01 14:16:08.954877 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947623    2576 flags.go:64] FLAG: --storage-driver-user="root"
Nov 01 14:16:08.954877 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947629    2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 01 14:16:08.954877 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947633    2576 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 01 14:16:08.954877 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947636    2576 flags.go:64] FLAG: --system-cgroups=""
Nov 01 14:16:08.954877 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947638    2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 01 14:16:08.954877 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947644    2576 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 01 14:16:08.954877 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947649    2576 flags.go:64] FLAG: --tls-cert-file=""
Nov 01 14:16:08.954877 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947651    2576 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 01 14:16:08.954877 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947657    2576 flags.go:64] FLAG: --tls-min-version=""
Nov 01 14:16:08.954877 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947660    2576 flags.go:64] FLAG: --tls-private-key-file=""
Nov 01 14:16:08.954877 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947663    2576 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 01 14:16:08.954877 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947666    2576 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 01 14:16:08.955776 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947669    2576 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 01 14:16:08.955776 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947672    2576 flags.go:64] FLAG: --v="2"
Nov 01 14:16:08.955776 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947676    2576 flags.go:64] FLAG: --version="false"
Nov 01 14:16:08.955776 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947680    2576 flags.go:64] FLAG: --vmodule=""
Nov 01 14:16:08.955776 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947684    2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 01 14:16:08.955776 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.947688    2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 01 14:16:08.955776 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.947832    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 14:16:08.955776 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.947835    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 14:16:08.955776 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.947840    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 14:16:08.955776 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.947842    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 14:16:08.955776 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.947845    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 14:16:08.955776 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.947848    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 14:16:08.955776 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.947850    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 14:16:08.955776 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.947853    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 14:16:08.955776 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.947855    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 14:16:08.955776 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.947858    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 14:16:08.955776 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.947861    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 14:16:08.955776 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.947863    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 14:16:08.955776 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.947868    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 14:16:08.955776 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.947872    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 14:16:08.956321 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.947876    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 14:16:08.956321 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.947879    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 14:16:08.956321 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.947881    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 14:16:08.956321 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.947884    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 14:16:08.956321 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.947887    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 14:16:08.956321 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.947889    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 14:16:08.956321 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.947893    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 14:16:08.956321 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.947896    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 14:16:08.956321 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.947898    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 14:16:08.956321 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.947901    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 14:16:08.956321 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.947904    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 14:16:08.956321 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.947907    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 14:16:08.956321 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.947910    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 14:16:08.956321 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.947912    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 14:16:08.956321 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.947915    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 14:16:08.956321 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.947917    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 14:16:08.956321 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.947920    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 14:16:08.956321 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.947922    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 14:16:08.956321 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.947925    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 14:16:08.956321 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.947928    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 14:16:08.956798 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.947933    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 14:16:08.956798 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.947935    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 14:16:08.956798 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.947938    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 14:16:08.956798 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.947941    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 14:16:08.956798 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.947943    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 14:16:08.956798 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.947946    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 14:16:08.956798 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.947948    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 14:16:08.956798 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.947951    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 14:16:08.956798 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.947954    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 14:16:08.956798 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.947956    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 14:16:08.956798 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.947959    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 14:16:08.956798 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.947961    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 14:16:08.956798 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.947964    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 14:16:08.956798 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.947966    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 14:16:08.956798 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.947969    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 14:16:08.956798 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.947971    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 14:16:08.956798 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.947974    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 14:16:08.956798 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.947976    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 14:16:08.956798 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.947980    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 14:16:08.957331 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.947983    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 14:16:08.957331 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.947985    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 14:16:08.957331 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.947988    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 14:16:08.957331 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.947991    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 14:16:08.957331 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.947993    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 14:16:08.957331 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.947996    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 14:16:08.957331 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.947998    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 14:16:08.957331 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948001    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 14:16:08.957331 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948003    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 14:16:08.957331 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948005    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 14:16:08.957331 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948008    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 14:16:08.957331 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948010    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 14:16:08.957331 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948013    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 14:16:08.957331 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948017    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 14:16:08.957331 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948020    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 14:16:08.957331 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948022    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 14:16:08.957331 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948025    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 14:16:08.957331 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948028    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 14:16:08.957331 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948030    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 14:16:08.957331 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948033    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 14:16:08.957796 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948036    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 14:16:08.957796 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948038    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 14:16:08.957796 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948042    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 14:16:08.957796 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948046    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 14:16:08.957796 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948050    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 14:16:08.957796 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948052    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 14:16:08.957796 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948055    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 14:16:08.957796 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948057    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 14:16:08.957796 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948060    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 14:16:08.957796 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948062    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 14:16:08.957796 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948065    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 14:16:08.957796 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948068    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 14:16:08.957796 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948071    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 14:16:08.957796 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948074    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 14:16:08.957796 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948076    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 14:16:08.957796 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948079    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 14:16:08.957796 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948081    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 14:16:08.957796 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948084    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 14:16:08.957796 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948087    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 14:16:08.958257 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948089    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 14:16:08.958257 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948092    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 14:16:08.958257 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948094    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 14:16:08.958257 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948097    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 14:16:08.958257 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948100    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 14:16:08.958257 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948102    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 14:16:08.958257 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948107    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 14:16:08.958257 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948109    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 14:16:08.958257 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948112    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 14:16:08.958257 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948115    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 14:16:08.958257 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948117    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 14:16:08.958257 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948120    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 14:16:08.958257 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948122    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 14:16:08.958257 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948125    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 14:16:08.958257 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.948151    2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 14:16:08.958257 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948203    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 14:16:08.958645 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948207    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 14:16:08.958645 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948211    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 14:16:08.958645 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948214    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 14:16:08.958645 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948217    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 14:16:08.958645 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948219    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 14:16:08.958645 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948222    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 14:16:08.958645 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948225    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 14:16:08.958645 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948227    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 14:16:08.958645 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948231    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 14:16:08.958645 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948234    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 14:16:08.958645 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948237    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 14:16:08.958645 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948240    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 14:16:08.958645 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948242    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 14:16:08.958645 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948245    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 14:16:08.958645 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948247    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 14:16:08.958645 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948250    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 14:16:08.958645 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948252    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 14:16:08.958645 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948255    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 14:16:08.958645 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948257    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 14:16:08.958645 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948260    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 14:16:08.959109 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948263    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 14:16:08.959109 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948265    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 14:16:08.959109 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948268    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 14:16:08.959109 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948272    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 14:16:08.959109 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948275    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 14:16:08.959109 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948277    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 14:16:08.959109 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948280    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 14:16:08.959109 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948282    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 14:16:08.959109 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948285    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 14:16:08.959109 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948288    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 14:16:08.959109 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948290    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 14:16:08.959109 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948293    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 14:16:08.959109 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948295    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 14:16:08.959109 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948298    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 14:16:08.959109 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948301    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 14:16:08.959109 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948303    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 14:16:08.959109 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948306    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 14:16:08.959109 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948308    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 14:16:08.959109 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948311    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 14:16:08.959562 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948313    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 14:16:08.959562 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948317    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 14:16:08.959562 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948320    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 14:16:08.959562 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948322    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 14:16:08.959562 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948325    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 14:16:08.959562 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948327    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 14:16:08.959562 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948330    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 14:16:08.959562 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948332    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 14:16:08.959562 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948335    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 14:16:08.959562 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948337    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 14:16:08.959562 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948340    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 14:16:08.959562 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948342    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 14:16:08.959562 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948345    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 14:16:08.959562 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948347    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 14:16:08.959562 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948350    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 14:16:08.959562 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948352    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 14:16:08.959562 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948356    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 14:16:08.959562 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948360    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 14:16:08.959562 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948363    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 14:16:08.959562 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948365    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 14:16:08.960027 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948370    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 14:16:08.960027 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948374    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 14:16:08.960027 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948377    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 14:16:08.960027 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948380    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 14:16:08.960027 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948383    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 14:16:08.960027 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948385    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 14:16:08.960027 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948388    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 14:16:08.960027 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948391    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 14:16:08.960027 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948393    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 14:16:08.960027 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948396    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 14:16:08.960027 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948398    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 14:16:08.960027 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948401    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 14:16:08.960027 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948403    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 14:16:08.960027 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948407    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 14:16:08.960027 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948409    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 14:16:08.960027 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948412    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 14:16:08.960027 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948415    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 14:16:08.960027 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948417    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 14:16:08.960027 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948420    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 14:16:08.960491 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948422    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 14:16:08.960491 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948425    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 14:16:08.960491 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948428    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 14:16:08.960491 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948430    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 14:16:08.960491 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948433    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 14:16:08.960491 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948435    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 14:16:08.960491 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948438    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 14:16:08.960491 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948440    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 14:16:08.960491 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948443    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 14:16:08.960491 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948448    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 14:16:08.960491 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948450    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 14:16:08.960491 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948454    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 14:16:08.960491 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948456    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 14:16:08.960491 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948459    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 14:16:08.960491 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948462    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 14:16:08.960491 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948464    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 14:16:08.960491 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948467    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 14:16:08.960491 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948469    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 14:16:08.960491 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948472    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 14:16:08.960491 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948474    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 14:16:08.961038 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948477    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 14:16:08.961038 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948480    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 14:16:08.961038 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948482    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 14:16:08.961038 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948485    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 14:16:08.961038 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948489    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 14:16:08.961038 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948493    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 14:16:08.961038 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.948496    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 14:16:08.961038 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.957098    2576 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 01 14:16:08.961038 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.957114    2576 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 01 14:16:08.961038 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.957130    2576 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 01 14:16:08.961038 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.957135    2576 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 01 14:16:08.961038 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957211    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 14:16:08.961038 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957217    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 14:16:08.961038 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957221    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 14:16:08.961038 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957224    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 14:16:08.961038 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957226    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 14:16:08.961038 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957229    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 14:16:08.961038 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957232    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 14:16:08.961038 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957236    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 14:16:08.961549 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957240    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 14:16:08.961549 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957244    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 14:16:08.961549 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957248    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 14:16:08.961549 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957251    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 14:16:08.961549 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957253    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 14:16:08.961549 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957256    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 14:16:08.961549 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957258    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 14:16:08.961549 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957261    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 14:16:08.961549 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957264    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 14:16:08.961549 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957266    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 14:16:08.961549 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957273    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 14:16:08.961549 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957276    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 14:16:08.961549 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957279    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 14:16:08.961549 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957282    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 14:16:08.961549 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957285    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 14:16:08.961549 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957287    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 14:16:08.961549 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957290    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 14:16:08.961549 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957292    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 14:16:08.961549 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957295    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 14:16:08.962021 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957298    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 14:16:08.962021 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957300    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 14:16:08.962021 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957303    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 14:16:08.962021 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957305    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 14:16:08.962021 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957308    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 14:16:08.962021 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957310    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 14:16:08.962021 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957313    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 14:16:08.962021 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957316    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 14:16:08.962021 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957318    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 14:16:08.962021 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957321    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 14:16:08.962021 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957324    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 14:16:08.962021 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957326    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 14:16:08.962021 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957329    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 14:16:08.962021 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957332    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 14:16:08.962021 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957334    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 14:16:08.962021 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957337    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 14:16:08.962021 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957340    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 14:16:08.962021 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957343    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 14:16:08.962021 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957346    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 14:16:08.962021 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957348    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 14:16:08.962619 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957351    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 14:16:08.962619 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957354    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 14:16:08.962619 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957356    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 14:16:08.962619 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957359    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 14:16:08.962619 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957361    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 14:16:08.962619 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957364    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 14:16:08.962619 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957366    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 14:16:08.962619 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957369    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 14:16:08.962619 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957371    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 14:16:08.962619 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957374    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 14:16:08.962619 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957376    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 14:16:08.962619 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957379    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 14:16:08.962619 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957382    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 14:16:08.962619 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957385    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 14:16:08.962619 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957387    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 14:16:08.962619 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957389    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 14:16:08.962619 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957392    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 14:16:08.962619 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957395    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 14:16:08.962619 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957397    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 14:16:08.962619 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957400    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 14:16:08.963087 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957402    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 14:16:08.963087 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957405    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 14:16:08.963087 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957407    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 14:16:08.963087 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957410    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 14:16:08.963087 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957413    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 14:16:08.963087 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957415    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 14:16:08.963087 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957418    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 14:16:08.963087 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957421    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 14:16:08.963087 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957425    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 14:16:08.963087 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957428    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 14:16:08.963087 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957431    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 14:16:08.963087 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957433    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 14:16:08.963087 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957436    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 14:16:08.963087 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957439    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 14:16:08.963087 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957442    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 14:16:08.963087 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957444    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 14:16:08.963087 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957447    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 14:16:08.963087 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957449    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 14:16:08.963087 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957452    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 14:16:08.963645 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957455    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 14:16:08.963645 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957457    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 14:16:08.963645 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957460    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 14:16:08.963645 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957462    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 14:16:08.963645 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957465    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 14:16:08.963645 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957468    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 14:16:08.963645 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957470    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 14:16:08.963645 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957473    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 14:16:08.963645 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957475    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 14:16:08.963645 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957478    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 14:16:08.963645 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957481    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 14:16:08.963645 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957483    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 14:16:08.963645 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957486    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 14:16:08.963645 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957488    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 14:16:08.963645 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957492    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 14:16:08.963645 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957494    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 14:16:08.963645 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957497    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 14:16:08.963645 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957499    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 14:16:08.963645 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957502    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 14:16:08.964289 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957504    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 14:16:08.964289 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.957533    2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 14:16:08.964289 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957672    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 14:16:08.964289 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957677    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 14:16:08.964289 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957680    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 14:16:08.964289 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957683    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 14:16:08.964289 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957685    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 14:16:08.964289 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957688    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 14:16:08.964289 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957691    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 14:16:08.964289 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957695    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 14:16:08.964289 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957699    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 14:16:08.964289 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957702    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 14:16:08.964289 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957705    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 14:16:08.964289 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957708    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 14:16:08.964289 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957711    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 14:16:08.964889 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957713    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 14:16:08.964889 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957716    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 14:16:08.964889 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957718    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 14:16:08.964889 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957721    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 14:16:08.964889 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957724    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 14:16:08.964889 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957726    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 14:16:08.964889 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957729    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 14:16:08.964889 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957731    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 14:16:08.964889 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957734    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 14:16:08.964889 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957736    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 14:16:08.964889 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957739    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 14:16:08.964889 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957742    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 14:16:08.964889 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957744    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 14:16:08.964889 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957747    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 14:16:08.964889 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957749    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 14:16:08.964889 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957752    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 14:16:08.964889 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957755    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 14:16:08.964889 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957757    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 14:16:08.964889 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957759    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 14:16:08.965446 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957762    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 14:16:08.965446 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957764    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 14:16:08.965446 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957767    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 14:16:08.965446 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957770    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 14:16:08.965446 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957772    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 14:16:08.965446 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957775    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 14:16:08.965446 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957778    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 14:16:08.965446 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957781    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 14:16:08.965446 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957783    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 14:16:08.965446 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957786    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 14:16:08.965446 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957789    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 14:16:08.965446 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957792    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 14:16:08.965446 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957794    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 14:16:08.965446 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957797    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 14:16:08.965446 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957800    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 14:16:08.965446 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957802    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 14:16:08.965446 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957805    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 14:16:08.965446 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957808    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 14:16:08.965446 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957810    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 14:16:08.965446 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957813    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 14:16:08.965919 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957815    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 14:16:08.965919 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957818    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 14:16:08.965919 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957821    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 14:16:08.965919 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957823    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 14:16:08.965919 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957826    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 14:16:08.965919 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957829    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 14:16:08.965919 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957831    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 14:16:08.965919 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957834    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 14:16:08.965919 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957836    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 14:16:08.965919 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957839    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 14:16:08.965919 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957841    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 14:16:08.965919 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957844    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 14:16:08.965919 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957846    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 14:16:08.965919 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957849    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 14:16:08.965919 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957851    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 14:16:08.965919 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957854    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 14:16:08.965919 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957856    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 14:16:08.965919 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957859    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 14:16:08.965919 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957861    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 14:16:08.966403 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957864    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 14:16:08.966403 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957867    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 14:16:08.966403 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957870    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 14:16:08.966403 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957872    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 14:16:08.966403 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957875    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 14:16:08.966403 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957877    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 14:16:08.966403 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957880    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 14:16:08.966403 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957882    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 14:16:08.966403 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957885    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 14:16:08.966403 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957887    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 14:16:08.966403 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957890    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 14:16:08.966403 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957896    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 14:16:08.966403 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957899    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 14:16:08.966403 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957902    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 14:16:08.966403 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957905    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 14:16:08.966403 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957907    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 14:16:08.966403 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957910    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 14:16:08.966403 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957914    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 14:16:08.966403 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957916    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 14:16:08.967106 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957919    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 14:16:08.967106 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957922    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 14:16:08.967106 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957924    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 14:16:08.967106 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957927    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 14:16:08.967106 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957929    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 14:16:08.967106 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957932    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 14:16:08.967106 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957935    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 14:16:08.967106 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957937    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 14:16:08.967106 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957940    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 14:16:08.967106 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957942    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 14:16:08.967106 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957945    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 14:16:08.967106 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957948    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 14:16:08.967106 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957950    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 14:16:08.967106 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957952    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 14:16:08.967106 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957956    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 14:16:08.967106 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.957959    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 14:16:08.967541 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.957982    2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 14:16:08.967541 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958015    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 14:16:08.967541 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958019    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 14:16:08.967541 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958022    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 14:16:08.967541 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958025    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 14:16:08.967541 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958028    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 14:16:08.967541 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958031    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 14:16:08.967541 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958033    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 14:16:08.967541 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958036    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 14:16:08.967541 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958040    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 14:16:08.967541 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958042    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 14:16:08.967541 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958045    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 14:16:08.967541 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958048    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 14:16:08.967541 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958051    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 14:16:08.967541 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958053    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 14:16:08.967541 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958056    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 14:16:08.967918 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958059    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 14:16:08.967918 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958061    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 14:16:08.967918 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958064    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 14:16:08.967918 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958066    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 14:16:08.967918 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958069    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 14:16:08.967918 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958072    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 14:16:08.967918 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958074    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 14:16:08.967918 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958077    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 14:16:08.967918 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958080    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 14:16:08.967918 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958082    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 14:16:08.967918 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958085    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 14:16:08.967918 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958087    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 14:16:08.967918 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958089    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 14:16:08.967918 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958092    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 14:16:08.967918 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958095    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 14:16:08.967918 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958098    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 14:16:08.967918 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958101    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 14:16:08.967918 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958104    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 14:16:08.967918 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958106    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 14:16:08.967918 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958109    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 14:16:08.968415 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958111    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 14:16:08.968415 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958114    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 14:16:08.968415 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958117    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 14:16:08.968415 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958119    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 14:16:08.968415 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958121    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 14:16:08.968415 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958125    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 14:16:08.968415 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958128    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 14:16:08.968415 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958130    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 14:16:08.968415 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958133    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 14:16:08.968415 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958136    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 14:16:08.968415 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958138    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 14:16:08.968415 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958141    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 14:16:08.968415 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958143    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 14:16:08.968415 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958146    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 14:16:08.968415 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958149    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 14:16:08.968415 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958151    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 14:16:08.968415 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958153    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 14:16:08.968415 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958156    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 14:16:08.968415 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958171    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 14:16:08.968930 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958174    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 14:16:08.968930 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958176    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 14:16:08.968930 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958179    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 14:16:08.968930 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958182    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 14:16:08.968930 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958184    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 14:16:08.968930 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958187    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 14:16:08.968930 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958189    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 14:16:08.968930 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958192    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 14:16:08.968930 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958194    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 14:16:08.968930 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958198    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 14:16:08.968930 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958200    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 14:16:08.968930 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958203    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 14:16:08.968930 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958206    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 14:16:08.968930 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958208    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 14:16:08.968930 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958211    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 14:16:08.968930 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958213    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 14:16:08.968930 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958216    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 14:16:08.968930 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958219    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 14:16:08.968930 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958225    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 14:16:08.969411 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958228    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 14:16:08.969411 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958231    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 14:16:08.969411 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958234    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 14:16:08.969411 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958236    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 14:16:08.969411 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958239    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 14:16:08.969411 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958242    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 14:16:08.969411 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958244    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 14:16:08.969411 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958247    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 14:16:08.969411 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958249    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 14:16:08.969411 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958252    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 14:16:08.969411 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958255    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 14:16:08.969411 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958257    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 14:16:08.969411 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958260    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 14:16:08.969411 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958262    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 14:16:08.969411 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958265    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 14:16:08.969411 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958268    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 14:16:08.969411 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958271    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 14:16:08.969411 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958273    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 14:16:08.969411 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958276    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 14:16:08.969411 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958279    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 14:16:08.970148 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958281    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 14:16:08.970148 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958284    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 14:16:08.970148 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958286    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 14:16:08.970148 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958290    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 14:16:08.970148 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958292    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 14:16:08.970148 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958295    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 14:16:08.970148 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958297    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 14:16:08.970148 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958300    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 14:16:08.970148 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958302    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 14:16:08.970148 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958307    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 14:16:08.970148 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958311    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 14:16:08.970148 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958316    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 14:16:08.970148 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:08.958318    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 14:16:08.970148 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.958376    2576 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 01 14:16:08.970148 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.961210    2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 01 14:16:08.970148 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.961992    2576 server.go:1014] "Starting client certificate rotation"
Nov 01 14:16:08.970148 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.962099    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 01 14:16:08.970148 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.962156    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 01 14:16:08.970611 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.963813    2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 14:16:08.970611 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.964931    2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 14:16:08.976551 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.976537    2576 log.go:25] "Validated CRI v1 runtime API"
Nov 01 14:16:08.980077 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.980062    2576 log.go:25] "Validated CRI v1 image API"
Nov 01 14:16:08.980904 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.980880    2576 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 01 14:16:08.985297 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.985276    2576 fs.go:135] Filesystem UUIDs: map[0a1d85e9-1934-4ffa-8bb0-39d9aa0fedbc:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 b246a694-e8c5-41c0-b060-ed5c8f313806:/dev/nvme0n1p4]
Nov 01 14:16:08.985372 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.985295    2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 01 14:16:08.991565 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.991450    2576 manager.go:217] Machine: {Timestamp:2026-11-01 14:16:08.990037443 +0000 UTC m=+0.335984294 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101932 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b76fc81e3d383e99c32ec3afca649 SystemUUID:ec2b76fc-81e3-d383-e99c-32ec3afca649 BootID:f3985537-feee-4310-b198-e5c2a3ce5604 Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:ea:0f:98:35 Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:ea:0f:98:35 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:b2:46:ac:74:d9:06 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 01 14:16:08.992321 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.992311    2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 01 14:16:08.992407 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.992395    2576 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 01 14:16:08.995455 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.995426    2576 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 01 14:16:08.995582 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.995458    2576 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-37.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 01 14:16:08.995623 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.995584    2576 topology_manager.go:143] "Creating topology manager with none policy"
Nov 01 14:16:08.995623 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.995593    2576 container_manager_linux.go:311] "Creating device plugin manager"
Nov 01 14:16:08.995623 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.995619    2576 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 14:16:08.995694 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.995631    2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 14:16:08.996447 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:08.996437    2576 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 01 14:16:09.000142 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.000130    2576 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 01 14:16:09.000278 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.000268    2576 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 01 14:16:09.001662 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.001653    2576 kubelet.go:493] "Attempting to sync node with API server"
Nov 01 14:16:09.001694 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.001666    2576 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 01 14:16:09.001694 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.001675    2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 01 14:16:09.001694 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.001684    2576 kubelet.go:405] "Adding apiserver pod source"
Nov 01 14:16:09.001694 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.001692    2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 01 14:16:09.002035 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.002022    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 14:16:09.004227 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.004207    2576 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 01 14:16:09.008417 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.008404    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 14:16:09.009707 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.009694    2576 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 01 14:16:09.009768 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.009723    2576 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 01 14:16:09.009900 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.009889    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 01 14:16:09.009929 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.009905    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 01 14:16:09.009929 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.009912    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 01 14:16:09.009929 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.009918    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 01 14:16:09.009929 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.009923    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 01 14:16:09.009929 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.009928    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 01 14:16:09.010427 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.010418    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 01 14:16:09.010463 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.010427    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 01 14:16:09.010463 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.010435    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 01 14:16:09.010463 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.010441    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 01 14:16:09.010463 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.010450    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 01 14:16:09.010463 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.010458    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 01 14:16:09.012476 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.012467    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 01 14:16:09.012518 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.012479    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 01 14:16:09.016225 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.016213    2576 server.go:1263] "Started kubelet"
Nov 01 14:16:09.016310 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.016286    2576 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 01 14:16:09.016362 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.016309    2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 01 14:16:09.016395 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.016380    2576 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 01 14:16:09.016888 ip-10-0-134-37 systemd[1]: Started Kubernetes Kubelet.
Nov 01 14:16:09.017615 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.017599    2576 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 01 14:16:09.021679 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.021665    2576 server.go:317] "Adding debug handlers to kubelet server"
Nov 01 14:16:09.023114 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:09.023047    2576 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 01 14:16:09.024229 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.024196    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 01 14:16:09.024918 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.024859    2576 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 01 14:16:09.025139 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:09.024993    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-37.us-west-2.compute.internal\" not found"
Nov 01 14:16:09.025738 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.025723    2576 factory.go:153] Registering CRI-O factory
Nov 01 14:16:09.025738 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.025740    2576 factory.go:223] Registration of the crio container factory successfully
Nov 01 14:16:09.025851 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.025786    2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 01 14:16:09.025851 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.025794    2576 factory.go:55] Registering systemd factory
Nov 01 14:16:09.025851 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.025801    2576 factory.go:223] Registration of the systemd container factory successfully
Nov 01 14:16:09.025851 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.025819    2576 factory.go:103] Registering Raw factory
Nov 01 14:16:09.025851 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.025831    2576 manager.go:1199] Started watching for new ooms in manager
Nov 01 14:16:09.026078 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.025877    2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 01 14:16:09.026078 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.025877    2576 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 01 14:16:09.026078 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.025902    2576 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 01 14:16:09.026078 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.025994    2576 reconstruct.go:97] "Volume reconstruction finished"
Nov 01 14:16:09.026078 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.026003    2576 reconciler.go:29] "Reconciler: start to sync state"
Nov 01 14:16:09.027116 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.027095    2576 manager.go:319] Starting recovery of all containers
Nov 01 14:16:09.037911 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.037894    2576 manager.go:324] Recovery completed
Nov 01 14:16:09.042638 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.042622    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 14:16:09.044440 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.044418    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-37.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 14:16:09.044502 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.044453    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-37.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 14:16:09.044502 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.044466    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-37.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 14:16:09.044974 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.044961    2576 cpu_manager.go:226] "Starting" policy="none"
Nov 01 14:16:09.045017 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.044974    2576 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 01 14:16:09.045017 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.044962    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 14:16:09.045084 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.044995    2576 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 01 14:16:09.045648 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.045634    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-37.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 14:16:09.045705 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.045658    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-37.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 14:16:09.045705 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.045668    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-37.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 14:16:09.047432 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.047418    2576 policy_none.go:50] "Start"
Nov 01 14:16:09.047506 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.047437    2576 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 01 14:16:09.047506 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.047456    2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 01 14:16:09.049572 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.049559    2576 policy_none.go:44] "Start"
Nov 01 14:16:09.077996 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.077946    2576 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 01 14:16:09.106148 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.081392    2576 manager.go:341] "Starting Device Plugin manager"
Nov 01 14:16:09.106148 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:09.081439    2576 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 01 14:16:09.106148 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.081451    2576 server.go:87] "Starting device plugin registration server"
Nov 01 14:16:09.106148 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.081685    2576 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 01 14:16:09.106148 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.081719    2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 01 14:16:09.106148 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.081796    2576 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 01 14:16:09.106148 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.081868    2576 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 01 14:16:09.106148 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.081875    2576 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 01 14:16:09.106148 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:09.082331    2576 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 01 14:16:09.106148 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:09.082369    2576 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-37.us-west-2.compute.internal\" not found"
Nov 01 14:16:09.106148 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.092197    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 01 14:16:09.106148 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.093381    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 01 14:16:09.106148 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.093404    2576 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 01 14:16:09.106148 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.093426    2576 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 01 14:16:09.106148 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:09.093468    2576 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Nov 01 14:16:09.163409 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.163177    2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-37.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 01 14:16:09.163409 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:09.163183    2576 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-37.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 01 14:16:09.164121 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:09.163236    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-37.us-west-2.compute.internal.18e3f176c3c05b3d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-37.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-37.us-west-2.compute.internal,},FirstTimestamp:2026-11-01 14:16:09.016187709 +0000 UTC m=+0.362134560,LastTimestamp:2026-11-01 14:16:09.016187709 +0000 UTC m=+0.362134560,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-37.us-west-2.compute.internal,}"
Nov 01 14:16:09.182046 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.182026    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 14:16:09.182850 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.182835    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-37.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 14:16:09.182929 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.182865    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-37.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 14:16:09.182929 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.182877    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-37.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 14:16:09.182929 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.182900    2576 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-37.us-west-2.compute.internal"
Nov 01 14:16:09.194565 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.194505    2576 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-37.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-37.us-west-2.compute.internal"]
Nov 01 14:16:09.194631 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.194568    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 14:16:09.195276 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.195261    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-37.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 14:16:09.195328 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.195287    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-37.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 14:16:09.195328 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.195297    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-37.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 14:16:09.197481 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.197470    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 14:16:09.198085 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.198057    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-37.us-west-2.compute.internal"
Nov 01 14:16:09.198182 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.198096    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 14:16:09.198182 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.198111    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-37.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 14:16:09.198182 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.198135    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-37.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 14:16:09.198182 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.198149    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-37.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 14:16:09.198936 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.198922    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-37.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 14:16:09.198984 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.198948    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-37.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 14:16:09.198984 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.198958    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-37.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 14:16:09.200243 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.200231    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-37.us-west-2.compute.internal"
Nov 01 14:16:09.200284 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.200255    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 14:16:09.200931 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.200916    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-37.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 14:16:09.201009 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.200944    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-37.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 14:16:09.201009 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.200960    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-37.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 14:16:09.226590 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:09.226565    2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-37.us-west-2.compute.internal\" not found" node="ip-10-0-134-37.us-west-2.compute.internal"
Nov 01 14:16:09.226817 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.226799    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4d681fe8617362d2405aa92b260bf206-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-37.us-west-2.compute.internal\" (UID: \"4d681fe8617362d2405aa92b260bf206\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-37.us-west-2.compute.internal"
Nov 01 14:16:09.228246 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.228228    2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 01 14:16:09.228309 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.228292    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-n6t5r"
Nov 01 14:16:09.229936 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:09.229921    2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-37.us-west-2.compute.internal\" not found" node="ip-10-0-134-37.us-west-2.compute.internal"
Nov 01 14:16:09.252784 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:09.252749    2576 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass"
Nov 01 14:16:09.252879 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:09.252745    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-37.us-west-2.compute.internal.18e3f176c56f7349  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-37.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-37.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-37.us-west-2.compute.internal,},FirstTimestamp:2026-11-01 14:16:09.044439881 +0000 UTC m=+0.390386732,LastTimestamp:2026-11-01 14:16:09.044439881 +0000 UTC m=+0.390386732,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-37.us-west-2.compute.internal,}"
Nov 01 14:16:09.252940 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:09.252877    2576 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Nov 01 14:16:09.312756 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.312730    2576 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 14:16:09.312875 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:09.312849    2576 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-134-37.us-west-2.compute.internal\" not found" node="ip-10-0-134-37.us-west-2.compute.internal"
Nov 01 14:16:09.313821 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.313805    2576 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 14:16:09.327634 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.327614    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f4211ca6cf8dc522c889bb47e17f738b-config\") pod \"kube-apiserver-proxy-ip-10-0-134-37.us-west-2.compute.internal\" (UID: \"f4211ca6cf8dc522c889bb47e17f738b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-37.us-west-2.compute.internal"
Nov 01 14:16:09.327706 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.327647    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4d681fe8617362d2405aa92b260bf206-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-37.us-west-2.compute.internal\" (UID: \"4d681fe8617362d2405aa92b260bf206\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-37.us-west-2.compute.internal"
Nov 01 14:16:09.327751 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.327707    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4d681fe8617362d2405aa92b260bf206-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-37.us-west-2.compute.internal\" (UID: \"4d681fe8617362d2405aa92b260bf206\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-37.us-west-2.compute.internal"
Nov 01 14:16:09.327751 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.327727    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d681fe8617362d2405aa92b260bf206-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-37.us-west-2.compute.internal\" (UID: \"4d681fe8617362d2405aa92b260bf206\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-37.us-west-2.compute.internal"
Nov 01 14:16:09.428738 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.428706    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d681fe8617362d2405aa92b260bf206-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-37.us-west-2.compute.internal\" (UID: \"4d681fe8617362d2405aa92b260bf206\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-37.us-west-2.compute.internal"
Nov 01 14:16:09.428738 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.428739    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f4211ca6cf8dc522c889bb47e17f738b-config\") pod \"kube-apiserver-proxy-ip-10-0-134-37.us-west-2.compute.internal\" (UID: \"f4211ca6cf8dc522c889bb47e17f738b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-37.us-west-2.compute.internal"
Nov 01 14:16:09.428913 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.428780    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f4211ca6cf8dc522c889bb47e17f738b-config\") pod \"kube-apiserver-proxy-ip-10-0-134-37.us-west-2.compute.internal\" (UID: \"f4211ca6cf8dc522c889bb47e17f738b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-37.us-west-2.compute.internal"
Nov 01 14:16:09.428913 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.428839    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d681fe8617362d2405aa92b260bf206-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-37.us-west-2.compute.internal\" (UID: \"4d681fe8617362d2405aa92b260bf206\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-37.us-west-2.compute.internal"
Nov 01 14:16:09.460117 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:09.460064    2576 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-134-37.us-west-2.compute.internal\" not found" node="ip-10-0-134-37.us-west-2.compute.internal"
Nov 01 14:16:09.513948 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.513921    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 14:16:09.514978 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.514961    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-37.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 14:16:09.515067 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.514992    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-37.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 14:16:09.515067 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.515001    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-37.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 14:16:09.515067 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.515036    2576 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-37.us-west-2.compute.internal"
Nov 01 14:16:09.529111 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.529094    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-37.us-west-2.compute.internal"
Nov 01 14:16:09.532333 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.532320    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-37.us-west-2.compute.internal"
Nov 01 14:16:09.566284 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.566265    2576 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-37.us-west-2.compute.internal"
Nov 01 14:16:09.625222 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.625196    2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-37.us-west-2.compute.internal"
Nov 01 14:16:09.780616 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.780542    2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 14:16:09.782790 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:09.782776    2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-37.us-west-2.compute.internal"
Nov 01 14:16:11.064314 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.064291    2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 14:16:11.147990 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.147965    2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 01 14:16:11.148483 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.148128    2576 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 01 14:16:11.148483 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:11.148088    2576 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a2ba49ef185a5444dbb27d29e6baf327-c3ad9244eeee9bec.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-134-37.us-west-2.compute.internal.18e3f176c56fe89c\": read tcp 10.0.134.37:57908->18.220.90.157:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-134-37.us-west-2.compute.internal.18e3f176c56fe89c  default   4811 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-37.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-134-37.us-west-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-134-37.us-west-2.compute.internal,},FirstTimestamp:2026-11-01 14:16:09 +0000 UTC,LastTimestamp:2026-11-01 14:16:09.195300668 +0000 UTC m=+0.541247519,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-37.us-west-2.compute.internal,}"
Nov 01 14:16:11.188034 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.188012    2576 apiserver.go:52] "Watching apiserver"
Nov 01 14:16:11.210889 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.210868    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 01 14:16:11.419769 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.419687    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 14:11:10 +0000 UTC" deadline="2028-06-13 08:14:44.755915461 +0000 UTC"
Nov 01 14:16:11.419769 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.419714    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14153h58m33.336204655s"
Nov 01 14:16:11.453802 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.453778    2576 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 01 14:16:11.454112 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.454095    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdqx4","openshift-dns/node-resolver-pwgqr","kube-system/konnectivity-agent-qzf26","openshift-cluster-node-tuning-operator/tuned-m5m8p","openshift-image-registry/node-ca-d7z97","openshift-multus/multus-additional-cni-plugins-9vt8t","openshift-network-diagnostics/network-check-target-8mqxj","openshift-network-operator/iptables-alerter-8jxks","kube-system/global-pull-secret-syncer-sc9jq","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-37.us-west-2.compute.internal","openshift-multus/multus-cwzp8","openshift-multus/network-metrics-daemon-b6rtr","openshift-ovn-kubernetes/ovnkube-node-cxjpl","kube-system/kube-apiserver-proxy-ip-10-0-134-37.us-west-2.compute.internal"]
Nov 01 14:16:11.454802 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.454785    2576 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 14:16:11.456576 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.456559    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8mqxj"
Nov 01 14:16:11.456672 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:11.456639    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8mqxj" podUID="0d4352db-d2e7-43ba-97ba-9e344abbac8b"
Nov 01 14:16:11.458617 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.458601    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8jxks"
Nov 01 14:16:11.460704 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.460688    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-m5m8p"
Nov 01 14:16:11.463049 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.463027    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdqx4"
Nov 01 14:16:11.465472 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.465453    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9vt8t"
Nov 01 14:16:11.467583 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.467565    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b6rtr"
Nov 01 14:16:11.467656 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:11.467625    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b6rtr" podUID="ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1"
Nov 01 14:16:11.469716 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.469699    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:11.473154 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.473137    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pwgqr"
Nov 01 14:16:11.475325 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.475305    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-d7z97"
Nov 01 14:16:11.477379 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.477364    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:11.479504 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.479484    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sc9jq"
Nov 01 14:16:11.479579 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:11.479535    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sc9jq" podUID="8b1eb53c-8c9a-46c4-b654-2aef42491b14"
Nov 01 14:16:11.481583 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.481565    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qzf26"
Nov 01 14:16:11.510719 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.510690    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 01 14:16:11.510806 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.510777    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 01 14:16:11.510806 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.510793    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-m46fn\""
Nov 01 14:16:11.510869 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.510853    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 01 14:16:11.511798 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.511784    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 01 14:16:11.513429 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.513415    2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 01 14:16:11.520428 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.520410    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b73fabd2-0d2c-4988-bbcf-7564d9331390-system-cni-dir\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:11.520513 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.520433    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2e013e38-9840-45c2-b4a0-80df0f319109-sys-fs\") pod \"aws-ebs-csi-driver-node-vdqx4\" (UID: \"2e013e38-9840-45c2-b4a0-80df0f319109\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdqx4"
Nov 01 14:16:11.520513 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.520451    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdsp8\" (UniqueName: \"kubernetes.io/projected/bcc6412c-2003-49de-9817-a3713e28afef-kube-api-access-vdsp8\") pod \"multus-additional-cni-plugins-9vt8t\" (UID: \"bcc6412c-2003-49de-9817-a3713e28afef\") " pod="openshift-multus/multus-additional-cni-plugins-9vt8t"
Nov 01 14:16:11.520513 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.520467    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00216deb-5dd0-4b4f-8940-c17dc329303c-tmp-dir\") pod \"node-resolver-pwgqr\" (UID: \"00216deb-5dd0-4b4f-8940-c17dc329303c\") " pod="openshift-dns/node-resolver-pwgqr"
Nov 01 14:16:11.520513 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.520482    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-node-log\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:11.520722 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.520531    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bcc6412c-2003-49de-9817-a3713e28afef-system-cni-dir\") pod \"multus-additional-cni-plugins-9vt8t\" (UID: \"bcc6412c-2003-49de-9817-a3713e28afef\") " pod="openshift-multus/multus-additional-cni-plugins-9vt8t"
Nov 01 14:16:11.520722 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.520570    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-host-slash\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:11.520722 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.520595    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5e15a59f-6aac-45e7-8812-ae97d0559052-etc-modprobe-d\") pod \"tuned-m5m8p\" (UID: \"5e15a59f-6aac-45e7-8812-ae97d0559052\") " pod="openshift-cluster-node-tuning-operator/tuned-m5m8p"
Nov 01 14:16:11.520722 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.520619    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b73fabd2-0d2c-4988-bbcf-7564d9331390-host-run-multus-certs\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:11.520722 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.520660    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8b1eb53c-8c9a-46c4-b654-2aef42491b14-original-pull-secret\") pod \"global-pull-secret-syncer-sc9jq\" (UID: \"8b1eb53c-8c9a-46c4-b654-2aef42491b14\") " pod="kube-system/global-pull-secret-syncer-sc9jq"
Nov 01 14:16:11.520722 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.520692    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvgpz\" (UniqueName: \"kubernetes.io/projected/00216deb-5dd0-4b4f-8940-c17dc329303c-kube-api-access-cvgpz\") pod \"node-resolver-pwgqr\" (UID: \"00216deb-5dd0-4b4f-8940-c17dc329303c\") " pod="openshift-dns/node-resolver-pwgqr"
Nov 01 14:16:11.520943 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.520738    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-host-run-ovn-kubernetes\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:11.520943 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.520823    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpm8k\" (UniqueName: \"kubernetes.io/projected/ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1-kube-api-access-hpm8k\") pod \"network-metrics-daemon-b6rtr\" (UID: \"ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1\") " pod="openshift-multus/network-metrics-daemon-b6rtr"
Nov 01 14:16:11.520943 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.520864    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b73fabd2-0d2c-4988-bbcf-7564d9331390-host-var-lib-cni-bin\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:11.520943 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.520890    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b73fabd2-0d2c-4988-bbcf-7564d9331390-host-var-lib-cni-multus\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:11.520943 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.520911    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8b1eb53c-8c9a-46c4-b654-2aef42491b14-kubelet-config\") pod \"global-pull-secret-syncer-sc9jq\" (UID: \"8b1eb53c-8c9a-46c4-b654-2aef42491b14\") " pod="kube-system/global-pull-secret-syncer-sc9jq"
Nov 01 14:16:11.520943 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.520926    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8b1eb53c-8c9a-46c4-b654-2aef42491b14-dbus\") pod \"global-pull-secret-syncer-sc9jq\" (UID: \"8b1eb53c-8c9a-46c4-b654-2aef42491b14\") " pod="kube-system/global-pull-secret-syncer-sc9jq"
Nov 01 14:16:11.521241 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.520958    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1b941fae-3c27-4d68-8205-38e492179ee9-ovnkube-script-lib\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:11.521241 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.520986    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ccc5b32b-07bc-4b3c-91ef-b184b0a9afe3-konnectivity-ca\") pod \"konnectivity-agent-qzf26\" (UID: \"ccc5b32b-07bc-4b3c-91ef-b184b0a9afe3\") " pod="kube-system/konnectivity-agent-qzf26"
Nov 01 14:16:11.521241 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.521013    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-var-lib-openvswitch\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:11.521241 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.521060    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2e013e38-9840-45c2-b4a0-80df0f319109-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vdqx4\" (UID: \"2e013e38-9840-45c2-b4a0-80df0f319109\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdqx4"
Nov 01 14:16:11.521241 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.521095    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzjlr\" (UniqueName: \"kubernetes.io/projected/2e013e38-9840-45c2-b4a0-80df0f319109-kube-api-access-wzjlr\") pod \"aws-ebs-csi-driver-node-vdqx4\" (UID: \"2e013e38-9840-45c2-b4a0-80df0f319109\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdqx4"
Nov 01 14:16:11.521241 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.521123    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/00216deb-5dd0-4b4f-8940-c17dc329303c-hosts-file\") pod \"node-resolver-pwgqr\" (UID: \"00216deb-5dd0-4b4f-8940-c17dc329303c\") " pod="openshift-dns/node-resolver-pwgqr"
Nov 01 14:16:11.521241 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.521148    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnzsm\" (UniqueName: \"kubernetes.io/projected/a12d30a7-8564-43c1-9742-3194fe8b975f-kube-api-access-gnzsm\") pod \"node-ca-d7z97\" (UID: \"a12d30a7-8564-43c1-9742-3194fe8b975f\") " pod="openshift-image-registry/node-ca-d7z97"
Nov 01 14:16:11.521241 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.521194    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-run-systemd\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:11.521241 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.521222    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zcj8\" (UniqueName: \"kubernetes.io/projected/5e15a59f-6aac-45e7-8812-ae97d0559052-kube-api-access-8zcj8\") pod \"tuned-m5m8p\" (UID: \"5e15a59f-6aac-45e7-8812-ae97d0559052\") " pod="openshift-cluster-node-tuning-operator/tuned-m5m8p"
Nov 01 14:16:11.521572 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.521254    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b73fabd2-0d2c-4988-bbcf-7564d9331390-multus-cni-dir\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:11.521572 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.521292    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a12d30a7-8564-43c1-9742-3194fe8b975f-serviceca\") pod \"node-ca-d7z97\" (UID: \"a12d30a7-8564-43c1-9742-3194fe8b975f\") " pod="openshift-image-registry/node-ca-d7z97"
Nov 01 14:16:11.521572 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.521317    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-host-run-netns\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:11.521572 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.521349    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-run-ovn\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:11.521572 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.521387    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-log-socket\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:11.521572 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.521414    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5e15a59f-6aac-45e7-8812-ae97d0559052-etc-tuned\") pod \"tuned-m5m8p\" (UID: \"5e15a59f-6aac-45e7-8812-ae97d0559052\") " pod="openshift-cluster-node-tuning-operator/tuned-m5m8p"
Nov 01 14:16:11.521572 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.521436    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b73fabd2-0d2c-4988-bbcf-7564d9331390-host-var-lib-kubelet\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:11.521572 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.521451    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-systemd-units\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:11.521572 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.521466    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-etc-openvswitch\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:11.521572 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.521499    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ccc5b32b-07bc-4b3c-91ef-b184b0a9afe3-agent-certs\") pod \"konnectivity-agent-qzf26\" (UID: \"ccc5b32b-07bc-4b3c-91ef-b184b0a9afe3\") " pod="kube-system/konnectivity-agent-qzf26"
Nov 01 14:16:11.521572 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.521543    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp6rs\" (UniqueName: \"kubernetes.io/projected/0d4352db-d2e7-43ba-97ba-9e344abbac8b-kube-api-access-vp6rs\") pod \"network-check-target-8mqxj\" (UID: \"0d4352db-d2e7-43ba-97ba-9e344abbac8b\") " pod="openshift-network-diagnostics/network-check-target-8mqxj"
Nov 01 14:16:11.521572 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.521573    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5e15a59f-6aac-45e7-8812-ae97d0559052-etc-sysconfig\") pod \"tuned-m5m8p\" (UID: \"5e15a59f-6aac-45e7-8812-ae97d0559052\") " pod="openshift-cluster-node-tuning-operator/tuned-m5m8p"
Nov 01 14:16:11.522019 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.521597    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5e15a59f-6aac-45e7-8812-ae97d0559052-etc-kubernetes\") pod \"tuned-m5m8p\" (UID: \"5e15a59f-6aac-45e7-8812-ae97d0559052\") " pod="openshift-cluster-node-tuning-operator/tuned-m5m8p"
Nov 01 14:16:11.522019 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.521643    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5e15a59f-6aac-45e7-8812-ae97d0559052-host\") pod \"tuned-m5m8p\" (UID: \"5e15a59f-6aac-45e7-8812-ae97d0559052\") " pod="openshift-cluster-node-tuning-operator/tuned-m5m8p"
Nov 01 14:16:11.522019 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.521672    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b73fabd2-0d2c-4988-bbcf-7564d9331390-multus-conf-dir\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:11.522019 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.521694    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1b941fae-3c27-4d68-8205-38e492179ee9-ovnkube-config\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:11.522019 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.521709    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1b941fae-3c27-4d68-8205-38e492179ee9-ovn-node-metrics-cert\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:11.522019 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.521734    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5e15a59f-6aac-45e7-8812-ae97d0559052-run\") pod \"tuned-m5m8p\" (UID: \"5e15a59f-6aac-45e7-8812-ae97d0559052\") " pod="openshift-cluster-node-tuning-operator/tuned-m5m8p"
Nov 01 14:16:11.522019 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.521758    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b73fabd2-0d2c-4988-bbcf-7564d9331390-multus-socket-dir-parent\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:11.522019 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.521776    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddmmn\" (UniqueName: \"kubernetes.io/projected/b73fabd2-0d2c-4988-bbcf-7564d9331390-kube-api-access-ddmmn\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:11.522019 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.521812    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8a6e7f39-7853-4fec-9ea2-30099983e5bd-iptables-alerter-script\") pod \"iptables-alerter-8jxks\" (UID: \"8a6e7f39-7853-4fec-9ea2-30099983e5bd\") " pod="openshift-network-operator/iptables-alerter-8jxks"
Nov 01 14:16:11.522019 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.521843    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbnlg\" (UniqueName: \"kubernetes.io/projected/8a6e7f39-7853-4fec-9ea2-30099983e5bd-kube-api-access-hbnlg\") pod \"iptables-alerter-8jxks\" (UID: \"8a6e7f39-7853-4fec-9ea2-30099983e5bd\") " pod="openshift-network-operator/iptables-alerter-8jxks"
Nov 01 14:16:11.522019 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.521869    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a12d30a7-8564-43c1-9742-3194fe8b975f-host\") pod \"node-ca-d7z97\" (UID: \"a12d30a7-8564-43c1-9742-3194fe8b975f\") " pod="openshift-image-registry/node-ca-d7z97"
Nov 01 14:16:11.522019 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.521935    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5e15a59f-6aac-45e7-8812-ae97d0559052-lib-modules\") pod \"tuned-m5m8p\" (UID: \"5e15a59f-6aac-45e7-8812-ae97d0559052\") " pod="openshift-cluster-node-tuning-operator/tuned-m5m8p"
Nov 01 14:16:11.522019 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.521969    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e15a59f-6aac-45e7-8812-ae97d0559052-tmp\") pod \"tuned-m5m8p\" (UID: \"5e15a59f-6aac-45e7-8812-ae97d0559052\") " pod="openshift-cluster-node-tuning-operator/tuned-m5m8p"
Nov 01 14:16:11.522019 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.522003    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1-metrics-certs\") pod \"network-metrics-daemon-b6rtr\" (UID: \"ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1\") " pod="openshift-multus/network-metrics-daemon-b6rtr"
Nov 01 14:16:11.522511 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.522051    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-host-kubelet\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:11.522511 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.522087    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-host-cni-netd\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:11.522511 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.522131    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5e15a59f-6aac-45e7-8812-ae97d0559052-etc-sysctl-d\") pod \"tuned-m5m8p\" (UID: \"5e15a59f-6aac-45e7-8812-ae97d0559052\") " pod="openshift-cluster-node-tuning-operator/tuned-m5m8p"
Nov 01 14:16:11.522511 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.522191    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e15a59f-6aac-45e7-8812-ae97d0559052-var-lib-kubelet\") pod \"tuned-m5m8p\" (UID: \"5e15a59f-6aac-45e7-8812-ae97d0559052\") " pod="openshift-cluster-node-tuning-operator/tuned-m5m8p"
Nov 01 14:16:11.522511 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.522225    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b73fabd2-0d2c-4988-bbcf-7564d9331390-host-run-k8s-cni-cncf-io\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:11.522511 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.522257    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bcc6412c-2003-49de-9817-a3713e28afef-cnibin\") pod \"multus-additional-cni-plugins-9vt8t\" (UID: \"bcc6412c-2003-49de-9817-a3713e28afef\") " pod="openshift-multus/multus-additional-cni-plugins-9vt8t"
Nov 01 14:16:11.522511 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.522288    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:11.522511 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.522310    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bcc6412c-2003-49de-9817-a3713e28afef-cni-binary-copy\") pod \"multus-additional-cni-plugins-9vt8t\" (UID: \"bcc6412c-2003-49de-9817-a3713e28afef\") " pod="openshift-multus/multus-additional-cni-plugins-9vt8t"
Nov 01 14:16:11.522511 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.522351    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-run-openvswitch\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:11.522511 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.522386    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5e15a59f-6aac-45e7-8812-ae97d0559052-etc-systemd\") pod \"tuned-m5m8p\" (UID: \"5e15a59f-6aac-45e7-8812-ae97d0559052\") " pod="openshift-cluster-node-tuning-operator/tuned-m5m8p"
Nov 01 14:16:11.522511 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.522409    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5e15a59f-6aac-45e7-8812-ae97d0559052-sys\") pod \"tuned-m5m8p\" (UID: \"5e15a59f-6aac-45e7-8812-ae97d0559052\") " pod="openshift-cluster-node-tuning-operator/tuned-m5m8p"
Nov 01 14:16:11.522511 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.522446    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b73fabd2-0d2c-4988-bbcf-7564d9331390-cni-binary-copy\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:11.522511 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.522475    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b73fabd2-0d2c-4988-bbcf-7564d9331390-hostroot\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:11.522511 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.522500    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8a6e7f39-7853-4fec-9ea2-30099983e5bd-host-slash\") pod \"iptables-alerter-8jxks\" (UID: \"8a6e7f39-7853-4fec-9ea2-30099983e5bd\") " pod="openshift-network-operator/iptables-alerter-8jxks"
Nov 01 14:16:11.522511 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.522516    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1b941fae-3c27-4d68-8205-38e492179ee9-env-overrides\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:11.523197 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.522537    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8mfv\" (UniqueName: \"kubernetes.io/projected/1b941fae-3c27-4d68-8205-38e492179ee9-kube-api-access-p8mfv\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:11.523197 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.522592    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b73fabd2-0d2c-4988-bbcf-7564d9331390-multus-daemon-config\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:11.523197 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.522627    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2e013e38-9840-45c2-b4a0-80df0f319109-registration-dir\") pod \"aws-ebs-csi-driver-node-vdqx4\" (UID: \"2e013e38-9840-45c2-b4a0-80df0f319109\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdqx4"
Nov 01 14:16:11.523197 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.522685    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2e013e38-9840-45c2-b4a0-80df0f319109-device-dir\") pod \"aws-ebs-csi-driver-node-vdqx4\" (UID: \"2e013e38-9840-45c2-b4a0-80df0f319109\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdqx4"
Nov 01 14:16:11.523197 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.522712    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2e013e38-9840-45c2-b4a0-80df0f319109-etc-selinux\") pod \"aws-ebs-csi-driver-node-vdqx4\" (UID: \"2e013e38-9840-45c2-b4a0-80df0f319109\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdqx4"
Nov 01 14:16:11.523197 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.522731    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bcc6412c-2003-49de-9817-a3713e28afef-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9vt8t\" (UID: \"bcc6412c-2003-49de-9817-a3713e28afef\") " pod="openshift-multus/multus-additional-cni-plugins-9vt8t"
Nov 01 14:16:11.523197 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.522746    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bcc6412c-2003-49de-9817-a3713e28afef-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9vt8t\" (UID: \"bcc6412c-2003-49de-9817-a3713e28afef\") " pod="openshift-multus/multus-additional-cni-plugins-9vt8t"
Nov 01 14:16:11.523197 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.522764    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-host-cni-bin\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:11.523197 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.522793    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5e15a59f-6aac-45e7-8812-ae97d0559052-etc-sysctl-conf\") pod \"tuned-m5m8p\" (UID: \"5e15a59f-6aac-45e7-8812-ae97d0559052\") " pod="openshift-cluster-node-tuning-operator/tuned-m5m8p"
Nov 01 14:16:11.523197 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.522825    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b73fabd2-0d2c-4988-bbcf-7564d9331390-etc-kubernetes\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:11.523197 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.522865    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bcc6412c-2003-49de-9817-a3713e28afef-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9vt8t\" (UID: \"bcc6412c-2003-49de-9817-a3713e28afef\") " pod="openshift-multus/multus-additional-cni-plugins-9vt8t"
Nov 01 14:16:11.523197 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.522888    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b73fabd2-0d2c-4988-bbcf-7564d9331390-cnibin\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:11.523197 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.522904    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b73fabd2-0d2c-4988-bbcf-7564d9331390-host-run-netns\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:11.523197 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.522939    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2e013e38-9840-45c2-b4a0-80df0f319109-socket-dir\") pod \"aws-ebs-csi-driver-node-vdqx4\" (UID: \"2e013e38-9840-45c2-b4a0-80df0f319109\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdqx4"
Nov 01 14:16:11.556400 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.556379    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-24rb9\""
Nov 01 14:16:11.556510 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.556446    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-smpqh\""
Nov 01 14:16:11.556770 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.556752    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 01 14:16:11.556770 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.556763    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 01 14:16:11.557020 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.557007    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 01 14:16:11.557084 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.557023    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 01 14:16:11.557084 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.557060    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 01 14:16:11.557084 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.557071    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-cj7jb\""
Nov 01 14:16:11.557334 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.557321    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 01 14:16:11.557510 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.557480    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 01 14:16:11.557510 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.557499    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 01 14:16:11.557654 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.557532    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 01 14:16:11.557654 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.557541    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 01 14:16:11.557654 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.557543    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 01 14:16:11.557654 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.557563    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 01 14:16:11.557654 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.557589    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-lp9km\""
Nov 01 14:16:11.557654 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.557590    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 01 14:16:11.557654 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.557597    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 01 14:16:11.558006 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.557729    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 01 14:16:11.558006 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.557758    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 01 14:16:11.558006 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.557776    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-9p5x8\""
Nov 01 14:16:11.558006 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.557786    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 01 14:16:11.558006 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.557788    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 01 14:16:11.558254 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.558231    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 01 14:16:11.560493 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.558458    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 01 14:16:11.560493 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.558580    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 01 14:16:11.560493 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.558568    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-v4966\""
Nov 01 14:16:11.560493 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.558813    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7pw86\""
Nov 01 14:16:11.560493 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.558853    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 01 14:16:11.560493 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.558864    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 01 14:16:11.560493 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.558899    2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 01 14:16:11.560493 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.558915    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4fnkb\""
Nov 01 14:16:11.560493 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.559070    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-dg8mq"
Nov 01 14:16:11.623229 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.623206    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a12d30a7-8564-43c1-9742-3194fe8b975f-host\") pod \"node-ca-d7z97\" (UID: \"a12d30a7-8564-43c1-9742-3194fe8b975f\") " pod="openshift-image-registry/node-ca-d7z97"
Nov 01 14:16:11.623352 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.623233    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5e15a59f-6aac-45e7-8812-ae97d0559052-lib-modules\") pod \"tuned-m5m8p\" (UID: \"5e15a59f-6aac-45e7-8812-ae97d0559052\") " pod="openshift-cluster-node-tuning-operator/tuned-m5m8p"
Nov 01 14:16:11.623352 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.623249    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e15a59f-6aac-45e7-8812-ae97d0559052-tmp\") pod \"tuned-m5m8p\" (UID: \"5e15a59f-6aac-45e7-8812-ae97d0559052\") " pod="openshift-cluster-node-tuning-operator/tuned-m5m8p"
Nov 01 14:16:11.623352 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.623266    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1-metrics-certs\") pod \"network-metrics-daemon-b6rtr\" (UID: \"ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1\") " pod="openshift-multus/network-metrics-daemon-b6rtr"
Nov 01 14:16:11.623352 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.623279    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-host-kubelet\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:11.623352 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.623318    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-host-kubelet\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:11.623352 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.623346    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a12d30a7-8564-43c1-9742-3194fe8b975f-host\") pod \"node-ca-d7z97\" (UID: \"a12d30a7-8564-43c1-9742-3194fe8b975f\") " pod="openshift-image-registry/node-ca-d7z97"
Nov 01 14:16:11.623579 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.623367    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5e15a59f-6aac-45e7-8812-ae97d0559052-lib-modules\") pod \"tuned-m5m8p\" (UID: \"5e15a59f-6aac-45e7-8812-ae97d0559052\") " pod="openshift-cluster-node-tuning-operator/tuned-m5m8p"
Nov 01 14:16:11.623579 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.623392    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-host-cni-netd\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:11.623579 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:11.623417    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 14:16:11.623579 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.623423    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5e15a59f-6aac-45e7-8812-ae97d0559052-etc-sysctl-d\") pod \"tuned-m5m8p\" (UID: \"5e15a59f-6aac-45e7-8812-ae97d0559052\") " pod="openshift-cluster-node-tuning-operator/tuned-m5m8p"
Nov 01 14:16:11.623579 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.623446    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e15a59f-6aac-45e7-8812-ae97d0559052-var-lib-kubelet\") pod \"tuned-m5m8p\" (UID: \"5e15a59f-6aac-45e7-8812-ae97d0559052\") " pod="openshift-cluster-node-tuning-operator/tuned-m5m8p"
Nov 01 14:16:11.623579 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.623481    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-host-cni-netd\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:11.623579 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:11.623501    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1-metrics-certs podName:ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1 nodeName:}" failed. No retries permitted until 2026-11-01 14:16:12.123460781 +0000 UTC m=+2.283401443 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1-metrics-certs") pod "network-metrics-daemon-b6rtr" (UID: "ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 14:16:11.623579 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.623505    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e15a59f-6aac-45e7-8812-ae97d0559052-var-lib-kubelet\") pod \"tuned-m5m8p\" (UID: \"5e15a59f-6aac-45e7-8812-ae97d0559052\") " pod="openshift-cluster-node-tuning-operator/tuned-m5m8p"
Nov 01 14:16:11.623579 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.623520    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b73fabd2-0d2c-4988-bbcf-7564d9331390-host-run-k8s-cni-cncf-io\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:11.623579 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.623551    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bcc6412c-2003-49de-9817-a3713e28afef-cnibin\") pod \"multus-additional-cni-plugins-9vt8t\" (UID: \"bcc6412c-2003-49de-9817-a3713e28afef\") " pod="openshift-multus/multus-additional-cni-plugins-9vt8t"
Nov 01 14:16:11.623579 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.623564    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5e15a59f-6aac-45e7-8812-ae97d0559052-etc-sysctl-d\") pod \"tuned-m5m8p\" (UID: \"5e15a59f-6aac-45e7-8812-ae97d0559052\") " pod="openshift-cluster-node-tuning-operator/tuned-m5m8p"
Nov 01 14:16:11.623933 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.623583    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b73fabd2-0d2c-4988-bbcf-7564d9331390-host-run-k8s-cni-cncf-io\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:11.623933 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.623606    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bcc6412c-2003-49de-9817-a3713e28afef-cnibin\") pod \"multus-additional-cni-plugins-9vt8t\" (UID: \"bcc6412c-2003-49de-9817-a3713e28afef\") " pod="openshift-multus/multus-additional-cni-plugins-9vt8t"
Nov 01 14:16:11.623933 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.623574    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:11.623933 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.623640    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bcc6412c-2003-49de-9817-a3713e28afef-cni-binary-copy\") pod \"multus-additional-cni-plugins-9vt8t\" (UID: \"bcc6412c-2003-49de-9817-a3713e28afef\") " pod="openshift-multus/multus-additional-cni-plugins-9vt8t"
Nov 01 14:16:11.623933 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.623640    2576 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 01 14:16:11.623933 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.623663    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-run-openvswitch\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:11.623933 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.623685    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5e15a59f-6aac-45e7-8812-ae97d0559052-etc-systemd\") pod \"tuned-m5m8p\" (UID: \"5e15a59f-6aac-45e7-8812-ae97d0559052\") " pod="openshift-cluster-node-tuning-operator/tuned-m5m8p"
Nov 01 14:16:11.623933 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.623610    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:11.623933 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.623703    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-run-openvswitch\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:11.623933 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.623706    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5e15a59f-6aac-45e7-8812-ae97d0559052-sys\") pod \"tuned-m5m8p\" (UID: \"5e15a59f-6aac-45e7-8812-ae97d0559052\") " pod="openshift-cluster-node-tuning-operator/tuned-m5m8p"
Nov 01 14:16:11.623933 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.623745    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5e15a59f-6aac-45e7-8812-ae97d0559052-sys\") pod \"tuned-m5m8p\" (UID: \"5e15a59f-6aac-45e7-8812-ae97d0559052\") " pod="openshift-cluster-node-tuning-operator/tuned-m5m8p"
Nov 01 14:16:11.623933 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.623746    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5e15a59f-6aac-45e7-8812-ae97d0559052-etc-systemd\") pod \"tuned-m5m8p\" (UID: \"5e15a59f-6aac-45e7-8812-ae97d0559052\") " pod="openshift-cluster-node-tuning-operator/tuned-m5m8p"
Nov 01 14:16:11.623933 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.623745    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b73fabd2-0d2c-4988-bbcf-7564d9331390-cni-binary-copy\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:11.623933 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.623776    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b73fabd2-0d2c-4988-bbcf-7564d9331390-hostroot\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:11.623933 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.623796    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8a6e7f39-7853-4fec-9ea2-30099983e5bd-host-slash\") pod \"iptables-alerter-8jxks\" (UID: \"8a6e7f39-7853-4fec-9ea2-30099983e5bd\") " pod="openshift-network-operator/iptables-alerter-8jxks"
Nov 01 14:16:11.623933 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.623815    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1b941fae-3c27-4d68-8205-38e492179ee9-env-overrides\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:11.623933 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.623829    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b73fabd2-0d2c-4988-bbcf-7564d9331390-hostroot\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:11.623933 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.623836    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8mfv\" (UniqueName: \"kubernetes.io/projected/1b941fae-3c27-4d68-8205-38e492179ee9-kube-api-access-p8mfv\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:11.624737 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.623877    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8a6e7f39-7853-4fec-9ea2-30099983e5bd-host-slash\") pod \"iptables-alerter-8jxks\" (UID: \"8a6e7f39-7853-4fec-9ea2-30099983e5bd\") " pod="openshift-network-operator/iptables-alerter-8jxks"
Nov 01 14:16:11.624737 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.623879    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b73fabd2-0d2c-4988-bbcf-7564d9331390-multus-daemon-config\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:11.624737 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.623920    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2e013e38-9840-45c2-b4a0-80df0f319109-registration-dir\") pod \"aws-ebs-csi-driver-node-vdqx4\" (UID: \"2e013e38-9840-45c2-b4a0-80df0f319109\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdqx4"
Nov 01 14:16:11.624737 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.623945    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2e013e38-9840-45c2-b4a0-80df0f319109-device-dir\") pod \"aws-ebs-csi-driver-node-vdqx4\" (UID: \"2e013e38-9840-45c2-b4a0-80df0f319109\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdqx4"
Nov 01 14:16:11.624737 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.623968    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2e013e38-9840-45c2-b4a0-80df0f319109-etc-selinux\") pod \"aws-ebs-csi-driver-node-vdqx4\" (UID: \"2e013e38-9840-45c2-b4a0-80df0f319109\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdqx4"
Nov 01 14:16:11.624737 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.623993    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bcc6412c-2003-49de-9817-a3713e28afef-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9vt8t\" (UID: \"bcc6412c-2003-49de-9817-a3713e28afef\") " pod="openshift-multus/multus-additional-cni-plugins-9vt8t"
Nov 01 14:16:11.624737 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.624027    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bcc6412c-2003-49de-9817-a3713e28afef-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9vt8t\" (UID: \"bcc6412c-2003-49de-9817-a3713e28afef\") " pod="openshift-multus/multus-additional-cni-plugins-9vt8t"
Nov 01 14:16:11.624737 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.624054    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-host-cni-bin\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:11.624737 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.624075    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5e15a59f-6aac-45e7-8812-ae97d0559052-etc-sysctl-conf\") pod \"tuned-m5m8p\" (UID: \"5e15a59f-6aac-45e7-8812-ae97d0559052\") " pod="openshift-cluster-node-tuning-operator/tuned-m5m8p"
Nov 01 14:16:11.624737 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.624099    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b73fabd2-0d2c-4988-bbcf-7564d9331390-etc-kubernetes\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:11.624737 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.624122    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bcc6412c-2003-49de-9817-a3713e28afef-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9vt8t\" (UID: \"bcc6412c-2003-49de-9817-a3713e28afef\") " pod="openshift-multus/multus-additional-cni-plugins-9vt8t"
Nov 01 14:16:11.624737 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.624148    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b73fabd2-0d2c-4988-bbcf-7564d9331390-cnibin\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:11.624737 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.624187    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b73fabd2-0d2c-4988-bbcf-7564d9331390-host-run-netns\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:11.624737 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.624212    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2e013e38-9840-45c2-b4a0-80df0f319109-socket-dir\") pod \"aws-ebs-csi-driver-node-vdqx4\" (UID: \"2e013e38-9840-45c2-b4a0-80df0f319109\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdqx4"
Nov 01 14:16:11.624737 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.624216    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bcc6412c-2003-49de-9817-a3713e28afef-cni-binary-copy\") pod \"multus-additional-cni-plugins-9vt8t\" (UID: \"bcc6412c-2003-49de-9817-a3713e28afef\") " pod="openshift-multus/multus-additional-cni-plugins-9vt8t"
Nov 01 14:16:11.624737 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.624234    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b73fabd2-0d2c-4988-bbcf-7564d9331390-system-cni-dir\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:11.624737 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.624258    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1b941fae-3c27-4d68-8205-38e492179ee9-env-overrides\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:11.625503 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.624262    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2e013e38-9840-45c2-b4a0-80df0f319109-sys-fs\") pod \"aws-ebs-csi-driver-node-vdqx4\" (UID: \"2e013e38-9840-45c2-b4a0-80df0f319109\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdqx4"
Nov 01 14:16:11.625503 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.624285    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2e013e38-9840-45c2-b4a0-80df0f319109-registration-dir\") pod \"aws-ebs-csi-driver-node-vdqx4\" (UID: \"2e013e38-9840-45c2-b4a0-80df0f319109\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdqx4"
Nov 01 14:16:11.625503 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.624293    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b73fabd2-0d2c-4988-bbcf-7564d9331390-cni-binary-copy\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:11.625503 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.624303    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdsp8\" (UniqueName: \"kubernetes.io/projected/bcc6412c-2003-49de-9817-a3713e28afef-kube-api-access-vdsp8\") pod \"multus-additional-cni-plugins-9vt8t\" (UID: \"bcc6412c-2003-49de-9817-a3713e28afef\") " pod="openshift-multus/multus-additional-cni-plugins-9vt8t"
Nov 01 14:16:11.625503 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.624329    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00216deb-5dd0-4b4f-8940-c17dc329303c-tmp-dir\") pod \"node-resolver-pwgqr\" (UID: \"00216deb-5dd0-4b4f-8940-c17dc329303c\") " pod="openshift-dns/node-resolver-pwgqr"
Nov 01 14:16:11.625503 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.624348    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2e013e38-9840-45c2-b4a0-80df0f319109-device-dir\") pod \"aws-ebs-csi-driver-node-vdqx4\" (UID: \"2e013e38-9840-45c2-b4a0-80df0f319109\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdqx4"
Nov 01 14:16:11.625503 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.624353    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-node-log\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:11.625503 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.624380    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bcc6412c-2003-49de-9817-a3713e28afef-system-cni-dir\") pod \"multus-additional-cni-plugins-9vt8t\" (UID: \"bcc6412c-2003-49de-9817-a3713e28afef\") " pod="openshift-multus/multus-additional-cni-plugins-9vt8t"
Nov 01 14:16:11.625503 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.624387    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b73fabd2-0d2c-4988-bbcf-7564d9331390-etc-kubernetes\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:11.625503 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.624386    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2e013e38-9840-45c2-b4a0-80df0f319109-etc-selinux\") pod \"aws-ebs-csi-driver-node-vdqx4\" (UID: \"2e013e38-9840-45c2-b4a0-80df0f319109\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdqx4"
Nov 01 14:16:11.625503 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.624405    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-host-slash\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:11.625503 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.624428    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5e15a59f-6aac-45e7-8812-ae97d0559052-etc-modprobe-d\") pod \"tuned-m5m8p\" (UID: \"5e15a59f-6aac-45e7-8812-ae97d0559052\") " pod="openshift-cluster-node-tuning-operator/tuned-m5m8p"
Nov 01 14:16:11.625503 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.624451    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b73fabd2-0d2c-4988-bbcf-7564d9331390-host-run-multus-certs\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:11.625503 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.624476    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8b1eb53c-8c9a-46c4-b654-2aef42491b14-original-pull-secret\") pod \"global-pull-secret-syncer-sc9jq\" (UID: \"8b1eb53c-8c9a-46c4-b654-2aef42491b14\") " pod="kube-system/global-pull-secret-syncer-sc9jq"
Nov 01 14:16:11.625503 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.624497    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5e15a59f-6aac-45e7-8812-ae97d0559052-etc-sysctl-conf\") pod \"tuned-m5m8p\" (UID: \"5e15a59f-6aac-45e7-8812-ae97d0559052\") " pod="openshift-cluster-node-tuning-operator/tuned-m5m8p"
Nov 01 14:16:11.625503 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.624500    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvgpz\" (UniqueName: \"kubernetes.io/projected/00216deb-5dd0-4b4f-8940-c17dc329303c-kube-api-access-cvgpz\") pod \"node-resolver-pwgqr\" (UID: \"00216deb-5dd0-4b4f-8940-c17dc329303c\") " pod="openshift-dns/node-resolver-pwgqr"
Nov 01 14:16:11.625503 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.624544    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-host-run-ovn-kubernetes\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:11.626273 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.624573    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpm8k\" (UniqueName: \"kubernetes.io/projected/ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1-kube-api-access-hpm8k\") pod \"network-metrics-daemon-b6rtr\" (UID: \"ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1\") " pod="openshift-multus/network-metrics-daemon-b6rtr"
Nov 01 14:16:11.626273 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.624598    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b73fabd2-0d2c-4988-bbcf-7564d9331390-host-var-lib-cni-bin\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:11.626273 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.624623    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b73fabd2-0d2c-4988-bbcf-7564d9331390-host-var-lib-cni-multus\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:11.626273 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.624649    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8b1eb53c-8c9a-46c4-b654-2aef42491b14-kubelet-config\") pod \"global-pull-secret-syncer-sc9jq\" (UID: \"8b1eb53c-8c9a-46c4-b654-2aef42491b14\") " pod="kube-system/global-pull-secret-syncer-sc9jq"
Nov 01 14:16:11.626273 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.624658    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00216deb-5dd0-4b4f-8940-c17dc329303c-tmp-dir\") pod \"node-resolver-pwgqr\" (UID: \"00216deb-5dd0-4b4f-8940-c17dc329303c\") " pod="openshift-dns/node-resolver-pwgqr"
Nov 01 14:16:11.626273 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.624671    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8b1eb53c-8c9a-46c4-b654-2aef42491b14-dbus\") pod \"global-pull-secret-syncer-sc9jq\" (UID: \"8b1eb53c-8c9a-46c4-b654-2aef42491b14\") " pod="kube-system/global-pull-secret-syncer-sc9jq"
Nov 01 14:16:11.626273 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.624694    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bcc6412c-2003-49de-9817-a3713e28afef-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9vt8t\" (UID: \"bcc6412c-2003-49de-9817-a3713e28afef\") " pod="openshift-multus/multus-additional-cni-plugins-9vt8t"
Nov 01 14:16:11.626273 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.624717    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-node-log\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:11.626273 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.624748    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1b941fae-3c27-4d68-8205-38e492179ee9-ovnkube-script-lib\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:11.626273 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.624757    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2e013e38-9840-45c2-b4a0-80df0f319109-sys-fs\") pod \"aws-ebs-csi-driver-node-vdqx4\" (UID: \"2e013e38-9840-45c2-b4a0-80df0f319109\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdqx4"
Nov 01 14:16:11.626273 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.624776    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ccc5b32b-07bc-4b3c-91ef-b184b0a9afe3-konnectivity-ca\") pod \"konnectivity-agent-qzf26\" (UID: \"ccc5b32b-07bc-4b3c-91ef-b184b0a9afe3\") " pod="kube-system/konnectivity-agent-qzf26"
Nov 01 14:16:11.626273 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.624802    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-var-lib-openvswitch\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:11.626273 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.624802    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b73fabd2-0d2c-4988-bbcf-7564d9331390-multus-daemon-config\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:11.626273 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.624829    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8b1eb53c-8c9a-46c4-b654-2aef42491b14-dbus\") pod \"global-pull-secret-syncer-sc9jq\" (UID: \"8b1eb53c-8c9a-46c4-b654-2aef42491b14\") " pod="kube-system/global-pull-secret-syncer-sc9jq"
Nov 01 14:16:11.626273 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.624839    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5e15a59f-6aac-45e7-8812-ae97d0559052-etc-modprobe-d\") pod \"tuned-m5m8p\" (UID: \"5e15a59f-6aac-45e7-8812-ae97d0559052\") " pod="openshift-cluster-node-tuning-operator/tuned-m5m8p"
Nov 01 14:16:11.626273 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.624859    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bcc6412c-2003-49de-9817-a3713e28afef-system-cni-dir\") pod \"multus-additional-cni-plugins-9vt8t\" (UID: \"bcc6412c-2003-49de-9817-a3713e28afef\") " pod="openshift-multus/multus-additional-cni-plugins-9vt8t"
Nov 01 14:16:11.626273 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.624868    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b73fabd2-0d2c-4988-bbcf-7564d9331390-host-run-multus-certs\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:11.626834 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.624305    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-host-cni-bin\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:11.626834 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.624880    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2e013e38-9840-45c2-b4a0-80df0f319109-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vdqx4\" (UID: \"2e013e38-9840-45c2-b4a0-80df0f319109\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdqx4"
Nov 01 14:16:11.626834 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.624930    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b73fabd2-0d2c-4988-bbcf-7564d9331390-host-run-netns\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:11.626834 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:11.624939    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 14:16:11.626834 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.624940    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-host-run-ovn-kubernetes\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:11.626834 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.624975    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b73fabd2-0d2c-4988-bbcf-7564d9331390-cnibin\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:11.626834 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:11.624984    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b1eb53c-8c9a-46c4-b654-2aef42491b14-original-pull-secret podName:8b1eb53c-8c9a-46c4-b654-2aef42491b14 nodeName:}" failed. No retries permitted until 2026-11-01 14:16:12.124969112 +0000 UTC m=+2.284909780 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8b1eb53c-8c9a-46c4-b654-2aef42491b14-original-pull-secret") pod "global-pull-secret-syncer-sc9jq" (UID: "8b1eb53c-8c9a-46c4-b654-2aef42491b14") : object "kube-system"/"original-pull-secret" not registered
Nov 01 14:16:11.626834 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.624984    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b73fabd2-0d2c-4988-bbcf-7564d9331390-host-var-lib-cni-bin\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:11.626834 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.624997    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8b1eb53c-8c9a-46c4-b654-2aef42491b14-kubelet-config\") pod \"global-pull-secret-syncer-sc9jq\" (UID: \"8b1eb53c-8c9a-46c4-b654-2aef42491b14\") " pod="kube-system/global-pull-secret-syncer-sc9jq"
Nov 01 14:16:11.626834 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.624875    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-host-slash\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:11.626834 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.624829    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2e013e38-9840-45c2-b4a0-80df0f319109-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vdqx4\" (UID: \"2e013e38-9840-45c2-b4a0-80df0f319109\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdqx4"
Nov 01 14:16:11.626834 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.625026    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-var-lib-openvswitch\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:11.626834 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.625098    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b73fabd2-0d2c-4988-bbcf-7564d9331390-system-cni-dir\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:11.626834 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.625117    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzjlr\" (UniqueName: \"kubernetes.io/projected/2e013e38-9840-45c2-b4a0-80df0f319109-kube-api-access-wzjlr\") pod \"aws-ebs-csi-driver-node-vdqx4\" (UID: \"2e013e38-9840-45c2-b4a0-80df0f319109\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdqx4"
Nov 01 14:16:11.626834 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.625143    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b73fabd2-0d2c-4988-bbcf-7564d9331390-host-var-lib-cni-multus\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:11.626834 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.625148    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/00216deb-5dd0-4b4f-8940-c17dc329303c-hosts-file\") pod \"node-resolver-pwgqr\" (UID: \"00216deb-5dd0-4b4f-8940-c17dc329303c\") " pod="openshift-dns/node-resolver-pwgqr"
Nov 01 14:16:11.626834 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.625232    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/00216deb-5dd0-4b4f-8940-c17dc329303c-hosts-file\") pod \"node-resolver-pwgqr\" (UID: \"00216deb-5dd0-4b4f-8940-c17dc329303c\") " pod="openshift-dns/node-resolver-pwgqr"
Nov 01 14:16:11.627438 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.625267    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnzsm\" (UniqueName: \"kubernetes.io/projected/a12d30a7-8564-43c1-9742-3194fe8b975f-kube-api-access-gnzsm\") pod \"node-ca-d7z97\" (UID: \"a12d30a7-8564-43c1-9742-3194fe8b975f\") " pod="openshift-image-registry/node-ca-d7z97"
Nov 01 14:16:11.627438 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.625273    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bcc6412c-2003-49de-9817-a3713e28afef-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9vt8t\" (UID: \"bcc6412c-2003-49de-9817-a3713e28afef\") " pod="openshift-multus/multus-additional-cni-plugins-9vt8t"
Nov 01 14:16:11.627438 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.625293    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-run-systemd\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:11.627438 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.625313    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2e013e38-9840-45c2-b4a0-80df0f319109-socket-dir\") pod \"aws-ebs-csi-driver-node-vdqx4\" (UID: \"2e013e38-9840-45c2-b4a0-80df0f319109\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdqx4"
Nov 01 14:16:11.627438 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.625328    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-run-systemd\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:11.627438 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.625332    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zcj8\" (UniqueName: \"kubernetes.io/projected/5e15a59f-6aac-45e7-8812-ae97d0559052-kube-api-access-8zcj8\") pod \"tuned-m5m8p\" (UID: \"5e15a59f-6aac-45e7-8812-ae97d0559052\") " pod="openshift-cluster-node-tuning-operator/tuned-m5m8p"
Nov 01 14:16:11.627438 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.625366    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b73fabd2-0d2c-4988-bbcf-7564d9331390-multus-cni-dir\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:11.627438 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.625389    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a12d30a7-8564-43c1-9742-3194fe8b975f-serviceca\") pod \"node-ca-d7z97\" (UID: \"a12d30a7-8564-43c1-9742-3194fe8b975f\") " pod="openshift-image-registry/node-ca-d7z97"
Nov 01 14:16:11.627438 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.625402    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bcc6412c-2003-49de-9817-a3713e28afef-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9vt8t\" (UID: \"bcc6412c-2003-49de-9817-a3713e28afef\") " pod="openshift-multus/multus-additional-cni-plugins-9vt8t"
Nov 01 14:16:11.627438 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.625413    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-host-run-netns\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:11.627438 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.625453    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b73fabd2-0d2c-4988-bbcf-7564d9331390-multus-cni-dir\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:11.627438 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.625462    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-host-run-netns\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:11.627438 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.625455    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-run-ovn\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:11.627438 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.625485    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-run-ovn\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:11.627438 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.625499    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-log-socket\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:11.627438 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.625527    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5e15a59f-6aac-45e7-8812-ae97d0559052-etc-tuned\") pod \"tuned-m5m8p\" (UID: \"5e15a59f-6aac-45e7-8812-ae97d0559052\") " pod="openshift-cluster-node-tuning-operator/tuned-m5m8p"
Nov 01 14:16:11.627438 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.625549    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b73fabd2-0d2c-4988-bbcf-7564d9331390-host-var-lib-kubelet\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:11.627438 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.625590    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-log-socket\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:11.628306 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.625596    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b73fabd2-0d2c-4988-bbcf-7564d9331390-host-var-lib-kubelet\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:11.628306 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.625627    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-systemd-units\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:11.628306 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.625687    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-etc-openvswitch\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:11.628306 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.625710    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1b941fae-3c27-4d68-8205-38e492179ee9-ovnkube-script-lib\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:11.628306 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.625721    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ccc5b32b-07bc-4b3c-91ef-b184b0a9afe3-agent-certs\") pod \"konnectivity-agent-qzf26\" (UID: \"ccc5b32b-07bc-4b3c-91ef-b184b0a9afe3\") " pod="kube-system/konnectivity-agent-qzf26"
Nov 01 14:16:11.628306 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.625767    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp6rs\" (UniqueName: \"kubernetes.io/projected/0d4352db-d2e7-43ba-97ba-9e344abbac8b-kube-api-access-vp6rs\") pod \"network-check-target-8mqxj\" (UID: \"0d4352db-d2e7-43ba-97ba-9e344abbac8b\") " pod="openshift-network-diagnostics/network-check-target-8mqxj"
Nov 01 14:16:11.628306 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.625793    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5e15a59f-6aac-45e7-8812-ae97d0559052-etc-sysconfig\") pod \"tuned-m5m8p\" (UID: \"5e15a59f-6aac-45e7-8812-ae97d0559052\") " pod="openshift-cluster-node-tuning-operator/tuned-m5m8p"
Nov 01 14:16:11.628306 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.625768    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-etc-openvswitch\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:11.628306 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.625816    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5e15a59f-6aac-45e7-8812-ae97d0559052-etc-kubernetes\") pod \"tuned-m5m8p\" (UID: \"5e15a59f-6aac-45e7-8812-ae97d0559052\") " pod="openshift-cluster-node-tuning-operator/tuned-m5m8p"
Nov 01 14:16:11.628306 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.625844    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a12d30a7-8564-43c1-9742-3194fe8b975f-serviceca\") pod \"node-ca-d7z97\" (UID: \"a12d30a7-8564-43c1-9742-3194fe8b975f\") " pod="openshift-image-registry/node-ca-d7z97"
Nov 01 14:16:11.628306 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.625856    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5e15a59f-6aac-45e7-8812-ae97d0559052-host\") pod \"tuned-m5m8p\" (UID: \"5e15a59f-6aac-45e7-8812-ae97d0559052\") " pod="openshift-cluster-node-tuning-operator/tuned-m5m8p"
Nov 01 14:16:11.628306 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.625899    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5e15a59f-6aac-45e7-8812-ae97d0559052-host\") pod \"tuned-m5m8p\" (UID: \"5e15a59f-6aac-45e7-8812-ae97d0559052\") " pod="openshift-cluster-node-tuning-operator/tuned-m5m8p"
Nov 01 14:16:11.628306 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.625886    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b73fabd2-0d2c-4988-bbcf-7564d9331390-multus-conf-dir\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:11.628306 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.625862    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5e15a59f-6aac-45e7-8812-ae97d0559052-etc-kubernetes\") pod \"tuned-m5m8p\" (UID: \"5e15a59f-6aac-45e7-8812-ae97d0559052\") " pod="openshift-cluster-node-tuning-operator/tuned-m5m8p"
Nov 01 14:16:11.628306 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.625907    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5e15a59f-6aac-45e7-8812-ae97d0559052-etc-sysconfig\") pod \"tuned-m5m8p\" (UID: \"5e15a59f-6aac-45e7-8812-ae97d0559052\") " pod="openshift-cluster-node-tuning-operator/tuned-m5m8p"
Nov 01 14:16:11.628306 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.625940    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1b941fae-3c27-4d68-8205-38e492179ee9-ovnkube-config\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:11.628306 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.625966    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-systemd-units\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:11.628306 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.625969    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1b941fae-3c27-4d68-8205-38e492179ee9-ovn-node-metrics-cert\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:11.629149 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.625994    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5e15a59f-6aac-45e7-8812-ae97d0559052-run\") pod \"tuned-m5m8p\" (UID: \"5e15a59f-6aac-45e7-8812-ae97d0559052\") " pod="openshift-cluster-node-tuning-operator/tuned-m5m8p"
Nov 01 14:16:11.629149 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.626017    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b73fabd2-0d2c-4988-bbcf-7564d9331390-multus-socket-dir-parent\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:11.629149 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.626044    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddmmn\" (UniqueName: \"kubernetes.io/projected/b73fabd2-0d2c-4988-bbcf-7564d9331390-kube-api-access-ddmmn\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:11.629149 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.626060    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5e15a59f-6aac-45e7-8812-ae97d0559052-run\") pod \"tuned-m5m8p\" (UID: \"5e15a59f-6aac-45e7-8812-ae97d0559052\") " pod="openshift-cluster-node-tuning-operator/tuned-m5m8p"
Nov 01 14:16:11.629149 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.626069    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8a6e7f39-7853-4fec-9ea2-30099983e5bd-iptables-alerter-script\") pod \"iptables-alerter-8jxks\" (UID: \"8a6e7f39-7853-4fec-9ea2-30099983e5bd\") " pod="openshift-network-operator/iptables-alerter-8jxks"
Nov 01 14:16:11.629149 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.625942    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b73fabd2-0d2c-4988-bbcf-7564d9331390-multus-conf-dir\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:11.629149 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.626089    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b73fabd2-0d2c-4988-bbcf-7564d9331390-multus-socket-dir-parent\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:11.629149 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.626116    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbnlg\" (UniqueName: \"kubernetes.io/projected/8a6e7f39-7853-4fec-9ea2-30099983e5bd-kube-api-access-hbnlg\") pod \"iptables-alerter-8jxks\" (UID: \"8a6e7f39-7853-4fec-9ea2-30099983e5bd\") " pod="openshift-network-operator/iptables-alerter-8jxks"
Nov 01 14:16:11.629149 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.626206    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ccc5b32b-07bc-4b3c-91ef-b184b0a9afe3-konnectivity-ca\") pod \"konnectivity-agent-qzf26\" (UID: \"ccc5b32b-07bc-4b3c-91ef-b184b0a9afe3\") " pod="kube-system/konnectivity-agent-qzf26"
Nov 01 14:16:11.629149 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.626414    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1b941fae-3c27-4d68-8205-38e492179ee9-ovnkube-config\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:11.629149 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.627111    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8a6e7f39-7853-4fec-9ea2-30099983e5bd-iptables-alerter-script\") pod \"iptables-alerter-8jxks\" (UID: \"8a6e7f39-7853-4fec-9ea2-30099983e5bd\") " pod="openshift-network-operator/iptables-alerter-8jxks"
Nov 01 14:16:11.629149 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.627496    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e15a59f-6aac-45e7-8812-ae97d0559052-tmp\") pod \"tuned-m5m8p\" (UID: \"5e15a59f-6aac-45e7-8812-ae97d0559052\") " pod="openshift-cluster-node-tuning-operator/tuned-m5m8p"
Nov 01 14:16:11.629149 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.628037    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5e15a59f-6aac-45e7-8812-ae97d0559052-etc-tuned\") pod \"tuned-m5m8p\" (UID: \"5e15a59f-6aac-45e7-8812-ae97d0559052\") " pod="openshift-cluster-node-tuning-operator/tuned-m5m8p"
Nov 01 14:16:11.629149 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.628269    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ccc5b32b-07bc-4b3c-91ef-b184b0a9afe3-agent-certs\") pod \"konnectivity-agent-qzf26\" (UID: \"ccc5b32b-07bc-4b3c-91ef-b184b0a9afe3\") " pod="kube-system/konnectivity-agent-qzf26"
Nov 01 14:16:11.629149 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.628891    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1b941fae-3c27-4d68-8205-38e492179ee9-ovn-node-metrics-cert\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:11.680028 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:11.679954    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 14:16:11.680028 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:11.679980    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 14:16:11.680028 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:11.679994    2576 projected.go:196] Error preparing data for projected volume kube-api-access-vp6rs for pod openshift-network-diagnostics/network-check-target-8mqxj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 14:16:11.680398 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:11.680054    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0d4352db-d2e7-43ba-97ba-9e344abbac8b-kube-api-access-vp6rs podName:0d4352db-d2e7-43ba-97ba-9e344abbac8b nodeName:}" failed. No retries permitted until 2026-11-01 14:16:12.180036225 +0000 UTC m=+2.339976882 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-vp6rs" (UniqueName: "kubernetes.io/projected/0d4352db-d2e7-43ba-97ba-9e344abbac8b-kube-api-access-vp6rs") pod "network-check-target-8mqxj" (UID: "0d4352db-d2e7-43ba-97ba-9e344abbac8b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 14:16:11.681921 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.681889    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvgpz\" (UniqueName: \"kubernetes.io/projected/00216deb-5dd0-4b4f-8940-c17dc329303c-kube-api-access-cvgpz\") pod \"node-resolver-pwgqr\" (UID: \"00216deb-5dd0-4b4f-8940-c17dc329303c\") " pod="openshift-dns/node-resolver-pwgqr"
Nov 01 14:16:11.682390 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.682367    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzjlr\" (UniqueName: \"kubernetes.io/projected/2e013e38-9840-45c2-b4a0-80df0f319109-kube-api-access-wzjlr\") pod \"aws-ebs-csi-driver-node-vdqx4\" (UID: \"2e013e38-9840-45c2-b4a0-80df0f319109\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdqx4"
Nov 01 14:16:11.682491 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.682383    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdsp8\" (UniqueName: \"kubernetes.io/projected/bcc6412c-2003-49de-9817-a3713e28afef-kube-api-access-vdsp8\") pod \"multus-additional-cni-plugins-9vt8t\" (UID: \"bcc6412c-2003-49de-9817-a3713e28afef\") " pod="openshift-multus/multus-additional-cni-plugins-9vt8t"
Nov 01 14:16:11.682910 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.682594    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpm8k\" (UniqueName: \"kubernetes.io/projected/ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1-kube-api-access-hpm8k\") pod \"network-metrics-daemon-b6rtr\" (UID: \"ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1\") " pod="openshift-multus/network-metrics-daemon-b6rtr"
Nov 01 14:16:11.683023 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.682997    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zcj8\" (UniqueName: \"kubernetes.io/projected/5e15a59f-6aac-45e7-8812-ae97d0559052-kube-api-access-8zcj8\") pod \"tuned-m5m8p\" (UID: \"5e15a59f-6aac-45e7-8812-ae97d0559052\") " pod="openshift-cluster-node-tuning-operator/tuned-m5m8p"
Nov 01 14:16:11.683155 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.683137    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddmmn\" (UniqueName: \"kubernetes.io/projected/b73fabd2-0d2c-4988-bbcf-7564d9331390-kube-api-access-ddmmn\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:11.683451 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.683430    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnzsm\" (UniqueName: \"kubernetes.io/projected/a12d30a7-8564-43c1-9742-3194fe8b975f-kube-api-access-gnzsm\") pod \"node-ca-d7z97\" (UID: \"a12d30a7-8564-43c1-9742-3194fe8b975f\") " pod="openshift-image-registry/node-ca-d7z97"
Nov 01 14:16:11.683936 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.683916    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbnlg\" (UniqueName: \"kubernetes.io/projected/8a6e7f39-7853-4fec-9ea2-30099983e5bd-kube-api-access-hbnlg\") pod \"iptables-alerter-8jxks\" (UID: \"8a6e7f39-7853-4fec-9ea2-30099983e5bd\") " pod="openshift-network-operator/iptables-alerter-8jxks"
Nov 01 14:16:11.684411 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.684395    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8mfv\" (UniqueName: \"kubernetes.io/projected/1b941fae-3c27-4d68-8205-38e492179ee9-kube-api-access-p8mfv\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:11.780422 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.780392    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8jxks"
Nov 01 14:16:11.786179 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.786152    2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 14:16:11.802794 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.802770    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-m5m8p"
Nov 01 14:16:11.813786 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.813770    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdqx4"
Nov 01 14:16:11.837565 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.837545    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9vt8t"
Nov 01 14:16:11.837707 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.837564    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:11.848319 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.848291    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pwgqr"
Nov 01 14:16:11.868046 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.868023    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-d7z97"
Nov 01 14:16:11.891869 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.891840    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qzf26"
Nov 01 14:16:11.897467 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.897448    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:11.963914 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:11.963881    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4211ca6cf8dc522c889bb47e17f738b.slice/crio-4730fc3bf0ae3cee7a9949aff05aa788af46b4d0acdf21c3911d6bc1b09a3d6c WatchSource:0}: Error finding container 4730fc3bf0ae3cee7a9949aff05aa788af46b4d0acdf21c3911d6bc1b09a3d6c: Status 404 returned error can't find the container with id 4730fc3bf0ae3cee7a9949aff05aa788af46b4d0acdf21c3911d6bc1b09a3d6c
Nov 01 14:16:11.964489 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:11.964468    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d681fe8617362d2405aa92b260bf206.slice/crio-b91c92e51399848fcf2fe6b04cc993ae0e73f896451e011d2df11db69684f74d WatchSource:0}: Error finding container b91c92e51399848fcf2fe6b04cc993ae0e73f896451e011d2df11db69684f74d: Status 404 returned error can't find the container with id b91c92e51399848fcf2fe6b04cc993ae0e73f896451e011d2df11db69684f74d
Nov 01 14:16:11.969588 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:11.969571    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 14:16:12.129874 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:12.129842    2576 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 14:16:12.130031 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:12.129904    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8b1eb53c-8c9a-46c4-b654-2aef42491b14-original-pull-secret\") pod \"global-pull-secret-syncer-sc9jq\" (UID: \"8b1eb53c-8c9a-46c4-b654-2aef42491b14\") " pod="kube-system/global-pull-secret-syncer-sc9jq"
Nov 01 14:16:12.130031 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:12.129951    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1-metrics-certs\") pod \"network-metrics-daemon-b6rtr\" (UID: \"ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1\") " pod="openshift-multus/network-metrics-daemon-b6rtr"
Nov 01 14:16:12.130099 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:12.130056    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 14:16:12.130099 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:12.130063    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 14:16:12.130158 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:12.130113    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b1eb53c-8c9a-46c4-b654-2aef42491b14-original-pull-secret podName:8b1eb53c-8c9a-46c4-b654-2aef42491b14 nodeName:}" failed. No retries permitted until 2026-11-01 14:16:13.130100477 +0000 UTC m=+3.290041120 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8b1eb53c-8c9a-46c4-b654-2aef42491b14-original-pull-secret") pod "global-pull-secret-syncer-sc9jq" (UID: "8b1eb53c-8c9a-46c4-b654-2aef42491b14") : object "kube-system"/"original-pull-secret" not registered
Nov 01 14:16:12.130158 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:12.130126    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1-metrics-certs podName:ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1 nodeName:}" failed. No retries permitted until 2026-11-01 14:16:13.130120638 +0000 UTC m=+3.290061277 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1-metrics-certs") pod "network-metrics-daemon-b6rtr" (UID: "ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 14:16:12.230795 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:12.230722    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp6rs\" (UniqueName: \"kubernetes.io/projected/0d4352db-d2e7-43ba-97ba-9e344abbac8b-kube-api-access-vp6rs\") pod \"network-check-target-8mqxj\" (UID: \"0d4352db-d2e7-43ba-97ba-9e344abbac8b\") " pod="openshift-network-diagnostics/network-check-target-8mqxj"
Nov 01 14:16:12.231302 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:12.230867    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 14:16:12.231302 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:12.230883    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 14:16:12.231302 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:12.230891    2576 projected.go:196] Error preparing data for projected volume kube-api-access-vp6rs for pod openshift-network-diagnostics/network-check-target-8mqxj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 14:16:12.231302 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:12.230935    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0d4352db-d2e7-43ba-97ba-9e344abbac8b-kube-api-access-vp6rs podName:0d4352db-d2e7-43ba-97ba-9e344abbac8b nodeName:}" failed. No retries permitted until 2026-11-01 14:16:13.230920748 +0000 UTC m=+3.390861402 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-vp6rs" (UniqueName: "kubernetes.io/projected/0d4352db-d2e7-43ba-97ba-9e344abbac8b-kube-api-access-vp6rs") pod "network-check-target-8mqxj" (UID: "0d4352db-d2e7-43ba-97ba-9e344abbac8b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 14:16:12.284986 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:12.284932    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-37.us-west-2.compute.internal" event={"ID":"4d681fe8617362d2405aa92b260bf206","Type":"ContainerStarted","Data":"b91c92e51399848fcf2fe6b04cc993ae0e73f896451e011d2df11db69684f74d"}
Nov 01 14:16:12.285832 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:12.285811    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-37.us-west-2.compute.internal" event={"ID":"f4211ca6cf8dc522c889bb47e17f738b","Type":"ContainerStarted","Data":"4730fc3bf0ae3cee7a9949aff05aa788af46b4d0acdf21c3911d6bc1b09a3d6c"}
Nov 01 14:16:12.562023 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:12.561929    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 14:11:11 +0000 UTC" deadline="2028-04-20 16:07:54.517125738 +0000 UTC"
Nov 01 14:16:12.562023 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:12.561969    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12865h51m41.955159434s"
Nov 01 14:16:12.655453 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:12.655424    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a6e7f39_7853_4fec_9ea2_30099983e5bd.slice/crio-ea2f2bba3fe50a234f88c90b77277001e023ccadb96e49135160bda809bc704b WatchSource:0}: Error finding container ea2f2bba3fe50a234f88c90b77277001e023ccadb96e49135160bda809bc704b: Status 404 returned error can't find the container with id ea2f2bba3fe50a234f88c90b77277001e023ccadb96e49135160bda809bc704b
Nov 01 14:16:12.705630 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:12.705597    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcc6412c_2003_49de_9817_a3713e28afef.slice/crio-f29e33b678b841c2e1e6bcf172b77aaacda8e2522ad70f90727388289f22053d WatchSource:0}: Error finding container f29e33b678b841c2e1e6bcf172b77aaacda8e2522ad70f90727388289f22053d: Status 404 returned error can't find the container with id f29e33b678b841c2e1e6bcf172b77aaacda8e2522ad70f90727388289f22053d
Nov 01 14:16:12.714444 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:12.714421    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e15a59f_6aac_45e7_8812_ae97d0559052.slice/crio-4ba47e39b82c11f96167d1ea027cf86f50b163f3bfc2bfacb073d7429fc76477 WatchSource:0}: Error finding container 4ba47e39b82c11f96167d1ea027cf86f50b163f3bfc2bfacb073d7429fc76477: Status 404 returned error can't find the container with id 4ba47e39b82c11f96167d1ea027cf86f50b163f3bfc2bfacb073d7429fc76477
Nov 01 14:16:12.769395 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:12.769369    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00216deb_5dd0_4b4f_8940_c17dc329303c.slice/crio-a4620bc86b004bf248b53e4ab03c70adcebaba74a374200d2fdd272ee9eb8eb3 WatchSource:0}: Error finding container a4620bc86b004bf248b53e4ab03c70adcebaba74a374200d2fdd272ee9eb8eb3: Status 404 returned error can't find the container with id a4620bc86b004bf248b53e4ab03c70adcebaba74a374200d2fdd272ee9eb8eb3
Nov 01 14:16:12.769935 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:12.769915    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b941fae_3c27_4d68_8205_38e492179ee9.slice/crio-0baeba702a835a54788bc5346bb21d6246e4a408ed299e692f0a26a125c169cc WatchSource:0}: Error finding container 0baeba702a835a54788bc5346bb21d6246e4a408ed299e692f0a26a125c169cc: Status 404 returned error can't find the container with id 0baeba702a835a54788bc5346bb21d6246e4a408ed299e692f0a26a125c169cc
Nov 01 14:16:12.770814 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:12.770787    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb73fabd2_0d2c_4988_bbcf_7564d9331390.slice/crio-15e947b3c1de139876bcd194085602f71b94f52d655de40ca926f9773c1ffbea WatchSource:0}: Error finding container 15e947b3c1de139876bcd194085602f71b94f52d655de40ca926f9773c1ffbea: Status 404 returned error can't find the container with id 15e947b3c1de139876bcd194085602f71b94f52d655de40ca926f9773c1ffbea
Nov 01 14:16:12.793852 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:12.793821    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda12d30a7_8564_43c1_9742_3194fe8b975f.slice/crio-177bc7f098ae5cba274d95f4b592129d6acb9976a463ba085b6d43898bfabd73 WatchSource:0}: Error finding container 177bc7f098ae5cba274d95f4b592129d6acb9976a463ba085b6d43898bfabd73: Status 404 returned error can't find the container with id 177bc7f098ae5cba274d95f4b592129d6acb9976a463ba085b6d43898bfabd73
Nov 01 14:16:12.809434 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:12.809407    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e013e38_9840_45c2_b4a0_80df0f319109.slice/crio-a5dc0434840b345873d399db22c8ae5810e840e4bc25115b32f7425fe76d5423 WatchSource:0}: Error finding container a5dc0434840b345873d399db22c8ae5810e840e4bc25115b32f7425fe76d5423: Status 404 returned error can't find the container with id a5dc0434840b345873d399db22c8ae5810e840e4bc25115b32f7425fe76d5423
Nov 01 14:16:12.914328 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:12.914299    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccc5b32b_07bc_4b3c_91ef_b184b0a9afe3.slice/crio-adbd46f9567ea58f2a324a7a72d0fa034b66bae6f1f22ba13011c8ecf111444e WatchSource:0}: Error finding container adbd46f9567ea58f2a324a7a72d0fa034b66bae6f1f22ba13011c8ecf111444e: Status 404 returned error can't find the container with id adbd46f9567ea58f2a324a7a72d0fa034b66bae6f1f22ba13011c8ecf111444e
Nov 01 14:16:13.136250 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:13.136214    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8b1eb53c-8c9a-46c4-b654-2aef42491b14-original-pull-secret\") pod \"global-pull-secret-syncer-sc9jq\" (UID: \"8b1eb53c-8c9a-46c4-b654-2aef42491b14\") " pod="kube-system/global-pull-secret-syncer-sc9jq"
Nov 01 14:16:13.136424 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:13.136272    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1-metrics-certs\") pod \"network-metrics-daemon-b6rtr\" (UID: \"ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1\") " pod="openshift-multus/network-metrics-daemon-b6rtr"
Nov 01 14:16:13.136424 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:13.136357    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 14:16:13.136424 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:13.136359    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 14:16:13.136424 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:13.136409    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1-metrics-certs podName:ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1 nodeName:}" failed. No retries permitted until 2026-11-01 14:16:15.136397322 +0000 UTC m=+5.296337961 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1-metrics-certs") pod "network-metrics-daemon-b6rtr" (UID: "ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 14:16:13.136424 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:13.136423    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b1eb53c-8c9a-46c4-b654-2aef42491b14-original-pull-secret podName:8b1eb53c-8c9a-46c4-b654-2aef42491b14 nodeName:}" failed. No retries permitted until 2026-11-01 14:16:15.136416726 +0000 UTC m=+5.296357366 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8b1eb53c-8c9a-46c4-b654-2aef42491b14-original-pull-secret") pod "global-pull-secret-syncer-sc9jq" (UID: "8b1eb53c-8c9a-46c4-b654-2aef42491b14") : object "kube-system"/"original-pull-secret" not registered
Nov 01 14:16:13.236849 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:13.236752    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp6rs\" (UniqueName: \"kubernetes.io/projected/0d4352db-d2e7-43ba-97ba-9e344abbac8b-kube-api-access-vp6rs\") pod \"network-check-target-8mqxj\" (UID: \"0d4352db-d2e7-43ba-97ba-9e344abbac8b\") " pod="openshift-network-diagnostics/network-check-target-8mqxj"
Nov 01 14:16:13.237550 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:13.236899    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 14:16:13.237550 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:13.236913    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 14:16:13.237550 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:13.236923    2576 projected.go:196] Error preparing data for projected volume kube-api-access-vp6rs for pod openshift-network-diagnostics/network-check-target-8mqxj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 14:16:13.237550 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:13.236970    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0d4352db-d2e7-43ba-97ba-9e344abbac8b-kube-api-access-vp6rs podName:0d4352db-d2e7-43ba-97ba-9e344abbac8b nodeName:}" failed. No retries permitted until 2026-11-01 14:16:15.236957633 +0000 UTC m=+5.396898285 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-vp6rs" (UniqueName: "kubernetes.io/projected/0d4352db-d2e7-43ba-97ba-9e344abbac8b-kube-api-access-vp6rs") pod "network-check-target-8mqxj" (UID: "0d4352db-d2e7-43ba-97ba-9e344abbac8b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 14:16:13.279845 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:13.279812    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sc9jq"
Nov 01 14:16:13.280018 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:13.279939    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sc9jq" podUID="8b1eb53c-8c9a-46c4-b654-2aef42491b14"
Nov 01 14:16:13.280089 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:13.280028    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8mqxj"
Nov 01 14:16:13.280144 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:13.280103    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8mqxj" podUID="0d4352db-d2e7-43ba-97ba-9e344abbac8b"
Nov 01 14:16:13.280235 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:13.280189    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b6rtr"
Nov 01 14:16:13.280294 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:13.280274    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b6rtr" podUID="ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1"
Nov 01 14:16:13.291649 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:13.291618    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8jxks" event={"ID":"8a6e7f39-7853-4fec-9ea2-30099983e5bd","Type":"ContainerStarted","Data":"ea2f2bba3fe50a234f88c90b77277001e023ccadb96e49135160bda809bc704b"}
Nov 01 14:16:13.293160 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:13.293132    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdqx4" event={"ID":"2e013e38-9840-45c2-b4a0-80df0f319109","Type":"ContainerStarted","Data":"a5dc0434840b345873d399db22c8ae5810e840e4bc25115b32f7425fe76d5423"}
Nov 01 14:16:13.294686 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:13.294658    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-d7z97" event={"ID":"a12d30a7-8564-43c1-9742-3194fe8b975f","Type":"ContainerStarted","Data":"177bc7f098ae5cba274d95f4b592129d6acb9976a463ba085b6d43898bfabd73"}
Nov 01 14:16:13.298015 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:13.297952    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cwzp8" event={"ID":"b73fabd2-0d2c-4988-bbcf-7564d9331390","Type":"ContainerStarted","Data":"15e947b3c1de139876bcd194085602f71b94f52d655de40ca926f9773c1ffbea"}
Nov 01 14:16:13.299357 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:13.299329    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pwgqr" event={"ID":"00216deb-5dd0-4b4f-8940-c17dc329303c","Type":"ContainerStarted","Data":"a4620bc86b004bf248b53e4ab03c70adcebaba74a374200d2fdd272ee9eb8eb3"}
Nov 01 14:16:13.301047 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:13.301021    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qzf26" event={"ID":"ccc5b32b-07bc-4b3c-91ef-b184b0a9afe3","Type":"ContainerStarted","Data":"adbd46f9567ea58f2a324a7a72d0fa034b66bae6f1f22ba13011c8ecf111444e"}
Nov 01 14:16:13.302678 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:13.302631    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl" event={"ID":"1b941fae-3c27-4d68-8205-38e492179ee9","Type":"ContainerStarted","Data":"0baeba702a835a54788bc5346bb21d6246e4a408ed299e692f0a26a125c169cc"}
Nov 01 14:16:13.305000 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:13.304979    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-m5m8p" event={"ID":"5e15a59f-6aac-45e7-8812-ae97d0559052","Type":"ContainerStarted","Data":"4ba47e39b82c11f96167d1ea027cf86f50b163f3bfc2bfacb073d7429fc76477"}
Nov 01 14:16:13.314632 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:13.314603    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9vt8t" event={"ID":"bcc6412c-2003-49de-9817-a3713e28afef","Type":"ContainerStarted","Data":"f29e33b678b841c2e1e6bcf172b77aaacda8e2522ad70f90727388289f22053d"}
Nov 01 14:16:13.563110 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:13.563011    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 14:11:11 +0000 UTC" deadline="2028-05-04 13:38:21.937235545 +0000 UTC"
Nov 01 14:16:13.563110 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:13.563056    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13199h22m8.374183218s"
Nov 01 14:16:15.154445 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:15.154369    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1-metrics-certs\") pod \"network-metrics-daemon-b6rtr\" (UID: \"ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1\") " pod="openshift-multus/network-metrics-daemon-b6rtr"
Nov 01 14:16:15.154445 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:15.154440    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8b1eb53c-8c9a-46c4-b654-2aef42491b14-original-pull-secret\") pod \"global-pull-secret-syncer-sc9jq\" (UID: \"8b1eb53c-8c9a-46c4-b654-2aef42491b14\") " pod="kube-system/global-pull-secret-syncer-sc9jq"
Nov 01 14:16:15.155216 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:15.154572    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 14:16:15.155216 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:15.154631    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b1eb53c-8c9a-46c4-b654-2aef42491b14-original-pull-secret podName:8b1eb53c-8c9a-46c4-b654-2aef42491b14 nodeName:}" failed. No retries permitted until 2026-11-01 14:16:19.154612935 +0000 UTC m=+9.314553578 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8b1eb53c-8c9a-46c4-b654-2aef42491b14-original-pull-secret") pod "global-pull-secret-syncer-sc9jq" (UID: "8b1eb53c-8c9a-46c4-b654-2aef42491b14") : object "kube-system"/"original-pull-secret" not registered
Nov 01 14:16:15.155216 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:15.155001    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 14:16:15.155216 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:15.155123    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1-metrics-certs podName:ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1 nodeName:}" failed. No retries permitted until 2026-11-01 14:16:19.155079337 +0000 UTC m=+9.315019980 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1-metrics-certs") pod "network-metrics-daemon-b6rtr" (UID: "ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 14:16:15.255973 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:15.255757    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp6rs\" (UniqueName: \"kubernetes.io/projected/0d4352db-d2e7-43ba-97ba-9e344abbac8b-kube-api-access-vp6rs\") pod \"network-check-target-8mqxj\" (UID: \"0d4352db-d2e7-43ba-97ba-9e344abbac8b\") " pod="openshift-network-diagnostics/network-check-target-8mqxj"
Nov 01 14:16:15.256151 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:15.256134    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 14:16:15.256245 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:15.256162    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 14:16:15.256245 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:15.256189    2576 projected.go:196] Error preparing data for projected volume kube-api-access-vp6rs for pod openshift-network-diagnostics/network-check-target-8mqxj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 14:16:15.256347 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:15.256260    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0d4352db-d2e7-43ba-97ba-9e344abbac8b-kube-api-access-vp6rs podName:0d4352db-d2e7-43ba-97ba-9e344abbac8b nodeName:}" failed. No retries permitted until 2026-11-01 14:16:19.256242315 +0000 UTC m=+9.416182962 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-vp6rs" (UniqueName: "kubernetes.io/projected/0d4352db-d2e7-43ba-97ba-9e344abbac8b-kube-api-access-vp6rs") pod "network-check-target-8mqxj" (UID: "0d4352db-d2e7-43ba-97ba-9e344abbac8b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 14:16:15.280423 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:15.280395    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sc9jq"
Nov 01 14:16:15.280553 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:15.280523    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sc9jq" podUID="8b1eb53c-8c9a-46c4-b654-2aef42491b14"
Nov 01 14:16:15.280610 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:15.280586    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8mqxj"
Nov 01 14:16:15.280663 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:15.280653    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8mqxj" podUID="0d4352db-d2e7-43ba-97ba-9e344abbac8b"
Nov 01 14:16:15.280722 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:15.280707    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b6rtr"
Nov 01 14:16:15.280798 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:15.280780    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b6rtr" podUID="ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1"
Nov 01 14:16:15.332529 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:15.332487    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-37.us-west-2.compute.internal" event={"ID":"4d681fe8617362d2405aa92b260bf206","Type":"ContainerStarted","Data":"4cfff79920975d1bc452be97e1e84bbbcb57c55587987af90f476708a7dff340"}
Nov 01 14:16:16.342866 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:16.342592    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-37.us-west-2.compute.internal" event={"ID":"f4211ca6cf8dc522c889bb47e17f738b","Type":"ContainerStarted","Data":"4803321d47f7ec635ae01afbb5a7446e3efc65ac900182cfe7d948ede6c14287"}
Nov 01 14:16:16.358549 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:16.358509    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9vt8t" event={"ID":"bcc6412c-2003-49de-9817-a3713e28afef","Type":"ContainerStarted","Data":"752cbe43539da3516c58551b06f593c22d6f7d257f4af36aeb85556ed092bb1b"}
Nov 01 14:16:16.361792 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:16.361764    2576 generic.go:356] "Generic (PLEG): container finished" podID="4d681fe8617362d2405aa92b260bf206" containerID="4cfff79920975d1bc452be97e1e84bbbcb57c55587987af90f476708a7dff340" exitCode=0
Nov 01 14:16:16.361899 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:16.361808    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-37.us-west-2.compute.internal" event={"ID":"4d681fe8617362d2405aa92b260bf206","Type":"ContainerDied","Data":"4cfff79920975d1bc452be97e1e84bbbcb57c55587987af90f476708a7dff340"}
Nov 01 14:16:16.574041 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:16.573990    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-37.us-west-2.compute.internal" podStartSLOduration=5.573977858 podStartE2EDuration="5.573977858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 14:16:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:16:16.460410043 +0000 UTC m=+6.620350705" watchObservedRunningTime="2026-11-01 14:16:16.573977858 +0000 UTC m=+6.733918519"
Nov 01 14:16:17.280633 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:17.280597    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sc9jq"
Nov 01 14:16:17.280811 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:17.280744    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sc9jq" podUID="8b1eb53c-8c9a-46c4-b654-2aef42491b14"
Nov 01 14:16:17.281122 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:17.281065    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b6rtr"
Nov 01 14:16:17.281262 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:17.281236    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b6rtr" podUID="ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1"
Nov 01 14:16:17.281329 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:17.281317    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8mqxj"
Nov 01 14:16:17.281428 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:17.281409    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8mqxj" podUID="0d4352db-d2e7-43ba-97ba-9e344abbac8b"
Nov 01 14:16:17.370814 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:17.370779    2576 generic.go:356] "Generic (PLEG): container finished" podID="bcc6412c-2003-49de-9817-a3713e28afef" containerID="752cbe43539da3516c58551b06f593c22d6f7d257f4af36aeb85556ed092bb1b" exitCode=0
Nov 01 14:16:17.371314 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:17.370867    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9vt8t" event={"ID":"bcc6412c-2003-49de-9817-a3713e28afef","Type":"ContainerDied","Data":"752cbe43539da3516c58551b06f593c22d6f7d257f4af36aeb85556ed092bb1b"}
Nov 01 14:16:17.377655 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:17.377555    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-37.us-west-2.compute.internal" event={"ID":"4d681fe8617362d2405aa92b260bf206","Type":"ContainerStarted","Data":"c98e7849d6416234bacb0dc2b0cf8d26091985d54a9f636410834d339744de52"}
Nov 01 14:16:17.617861 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:17.617532    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-37.us-west-2.compute.internal" podStartSLOduration=7.61751521 podStartE2EDuration="7.61751521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 14:16:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:16:17.6170674 +0000 UTC m=+7.777008063" watchObservedRunningTime="2026-11-01 14:16:17.61751521 +0000 UTC m=+7.777455874"
Nov 01 14:16:18.383617 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:18.383587    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qzf26" event={"ID":"ccc5b32b-07bc-4b3c-91ef-b184b0a9afe3","Type":"ContainerStarted","Data":"9571900dba09ab280b48ef43a526d06205f79527aafc0c4922fecbeacf12e2ec"}
Nov 01 14:16:18.496576 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:18.496519    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-qzf26" podStartSLOduration=2.912113311 podStartE2EDuration="7.49650408s" totalImagesPullingTime="4.584390769s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:16:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:16:18.496045009 +0000 UTC m=+8.655985670" watchObservedRunningTime="2026-11-01 14:16:18.49650408 +0000 UTC m=+8.656444744"
Nov 01 14:16:19.188271 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:19.188163    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1-metrics-certs\") pod \"network-metrics-daemon-b6rtr\" (UID: \"ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1\") " pod="openshift-multus/network-metrics-daemon-b6rtr"
Nov 01 14:16:19.188271 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:19.188255    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8b1eb53c-8c9a-46c4-b654-2aef42491b14-original-pull-secret\") pod \"global-pull-secret-syncer-sc9jq\" (UID: \"8b1eb53c-8c9a-46c4-b654-2aef42491b14\") " pod="kube-system/global-pull-secret-syncer-sc9jq"
Nov 01 14:16:19.188492 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:19.188409    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 14:16:19.188492 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:19.188472    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1-metrics-certs podName:ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1 nodeName:}" failed. No retries permitted until 2026-11-01 14:16:27.188451357 +0000 UTC m=+17.348392001 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1-metrics-certs") pod "network-metrics-daemon-b6rtr" (UID: "ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 14:16:19.188961 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:19.188850    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 14:16:19.188961 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:19.188906    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b1eb53c-8c9a-46c4-b654-2aef42491b14-original-pull-secret podName:8b1eb53c-8c9a-46c4-b654-2aef42491b14 nodeName:}" failed. No retries permitted until 2026-11-01 14:16:27.188889086 +0000 UTC m=+17.348829726 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8b1eb53c-8c9a-46c4-b654-2aef42491b14-original-pull-secret") pod "global-pull-secret-syncer-sc9jq" (UID: "8b1eb53c-8c9a-46c4-b654-2aef42491b14") : object "kube-system"/"original-pull-secret" not registered
Nov 01 14:16:19.284900 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:19.284873    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8mqxj"
Nov 01 14:16:19.285045 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:19.284898    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b6rtr"
Nov 01 14:16:19.285045 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:19.285002    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8mqxj" podUID="0d4352db-d2e7-43ba-97ba-9e344abbac8b"
Nov 01 14:16:19.285162 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:19.285093    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b6rtr" podUID="ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1"
Nov 01 14:16:19.285162 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:19.285134    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sc9jq"
Nov 01 14:16:19.285279 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:19.285230    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sc9jq" podUID="8b1eb53c-8c9a-46c4-b654-2aef42491b14"
Nov 01 14:16:19.288762 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:19.288496    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp6rs\" (UniqueName: \"kubernetes.io/projected/0d4352db-d2e7-43ba-97ba-9e344abbac8b-kube-api-access-vp6rs\") pod \"network-check-target-8mqxj\" (UID: \"0d4352db-d2e7-43ba-97ba-9e344abbac8b\") " pod="openshift-network-diagnostics/network-check-target-8mqxj"
Nov 01 14:16:19.288762 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:19.288621    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 14:16:19.288762 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:19.288633    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 14:16:19.288762 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:19.288644    2576 projected.go:196] Error preparing data for projected volume kube-api-access-vp6rs for pod openshift-network-diagnostics/network-check-target-8mqxj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 14:16:19.288762 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:19.288685    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0d4352db-d2e7-43ba-97ba-9e344abbac8b-kube-api-access-vp6rs podName:0d4352db-d2e7-43ba-97ba-9e344abbac8b nodeName:}" failed. No retries permitted until 2026-11-01 14:16:27.28867275 +0000 UTC m=+17.448613402 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-vp6rs" (UniqueName: "kubernetes.io/projected/0d4352db-d2e7-43ba-97ba-9e344abbac8b-kube-api-access-vp6rs") pod "network-check-target-8mqxj" (UID: "0d4352db-d2e7-43ba-97ba-9e344abbac8b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 14:16:19.387945 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:19.387837    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdqx4" event={"ID":"2e013e38-9840-45c2-b4a0-80df0f319109","Type":"ContainerStarted","Data":"b0474deaf5bf1006df8a849db2e4eff1ad9ac5fdb3c80e099ee9229ca669794f"}
Nov 01 14:16:19.392836 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:19.392812    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-d7z97" event={"ID":"a12d30a7-8564-43c1-9742-3194fe8b975f","Type":"ContainerStarted","Data":"b8db8e5c1c07baee96d93bb51f77c84b47b6a5b52f2011c806609016d15826b8"}
Nov 01 14:16:21.280365 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:21.279904    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sc9jq"
Nov 01 14:16:21.280365 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:21.280016    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sc9jq" podUID="8b1eb53c-8c9a-46c4-b654-2aef42491b14"
Nov 01 14:16:21.280365 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:21.280078    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8mqxj"
Nov 01 14:16:21.280365 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:21.280148    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8mqxj" podUID="0d4352db-d2e7-43ba-97ba-9e344abbac8b"
Nov 01 14:16:21.280365 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:21.280217    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b6rtr"
Nov 01 14:16:21.280365 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:21.280288    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b6rtr" podUID="ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1"
Nov 01 14:16:21.402069 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:21.401992    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cwzp8" event={"ID":"b73fabd2-0d2c-4988-bbcf-7564d9331390","Type":"ContainerStarted","Data":"5291817a3bdd3d2f8327a01999ec8339ca762c1b4e1bf08a749df226bc8f5e13"}
Nov 01 14:16:21.518633 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:21.518531    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-cwzp8" podStartSLOduration=3.256483071 podStartE2EDuration="11.518513934s" totalImagesPullingTime="8.262030863s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 14:16:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:16:21.516009592 +0000 UTC m=+11.675950254" watchObservedRunningTime="2026-11-01 14:16:21.518513934 +0000 UTC m=+11.678454596"
Nov 01 14:16:21.519530 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:21.519482    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-d7z97" podStartSLOduration=4.984823952 podStartE2EDuration="10.519468581s" totalImagesPullingTime="5.534644629s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 14:16:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:16:19.504851698 +0000 UTC m=+9.664792360" watchObservedRunningTime="2026-11-01 14:16:21.519468581 +0000 UTC m=+11.679409243"
Nov 01 14:16:21.700535 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:21.700507    2576 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 01 14:16:22.275711 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:22.275318    2576 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T14:16:21.700529159Z","UUID":"c90d3722-6bb2-4695-8d6c-051693bfc015","Handler":null,"Name":"","Endpoint":""}
Nov 01 14:16:22.276984 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:22.276961    2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 01 14:16:22.277108 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:22.276993    2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 01 14:16:22.407017 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:22.406975    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pwgqr" event={"ID":"00216deb-5dd0-4b4f-8940-c17dc329303c","Type":"ContainerStarted","Data":"595eb3acedb438217dcf7b753c294e38f13f0d898523f54f3556288ab835a991"}
Nov 01 14:16:22.410194 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:22.410151    2576 generic.go:356] "Generic (PLEG): container finished" podID="bcc6412c-2003-49de-9817-a3713e28afef" containerID="02ca50c2f61f7e649a9fbbe54e49525427915c8948a281e26f6bd17f137475ed" exitCode=0
Nov 01 14:16:22.410335 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:22.410261    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9vt8t" event={"ID":"bcc6412c-2003-49de-9817-a3713e28afef","Type":"ContainerDied","Data":"02ca50c2f61f7e649a9fbbe54e49525427915c8948a281e26f6bd17f137475ed"}
Nov 01 14:16:22.413052 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:22.412987    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdqx4" event={"ID":"2e013e38-9840-45c2-b4a0-80df0f319109","Type":"ContainerStarted","Data":"a90f6c72c5e779626f2fc5a43849233fe711ea3ecffc1103bac31a99bdcda079"}
Nov 01 14:16:22.518547 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:22.518479    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-pwgqr" podStartSLOduration=3.72368224 podStartE2EDuration="12.518463012s" totalImagesPullingTime="8.794780772s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 14:16:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:16:22.517727045 +0000 UTC m=+12.677667707" watchObservedRunningTime="2026-11-01 14:16:22.518463012 +0000 UTC m=+12.678403671"
Nov 01 14:16:22.566215 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:22.566133    2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-qzf26"
Nov 01 14:16:22.566725 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:22.566701    2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-qzf26"
Nov 01 14:16:23.280355 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:23.280322    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sc9jq"
Nov 01 14:16:23.280527 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:23.280375    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b6rtr"
Nov 01 14:16:23.280527 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:23.280322    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8mqxj"
Nov 01 14:16:23.280527 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:23.280444    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sc9jq" podUID="8b1eb53c-8c9a-46c4-b654-2aef42491b14"
Nov 01 14:16:23.280684 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:23.280568    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8mqxj" podUID="0d4352db-d2e7-43ba-97ba-9e344abbac8b"
Nov 01 14:16:23.280684 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:23.280659    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b6rtr" podUID="ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1"
Nov 01 14:16:23.416521 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:23.416443    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-m5m8p" event={"ID":"5e15a59f-6aac-45e7-8812-ae97d0559052","Type":"ContainerStarted","Data":"aebbbfacca90a28125994e722a5146f2a0dc6a93ea74acdc5e451f16a1df7942"}
Nov 01 14:16:23.418216 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:23.418191    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8jxks" event={"ID":"8a6e7f39-7853-4fec-9ea2-30099983e5bd","Type":"ContainerStarted","Data":"1629a3950df7f987eb71408346bd0c5987f640da861e62d52ab426b17849787a"}
Nov 01 14:16:23.531389 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:23.531332    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-m5m8p" podStartSLOduration=3.095012149 podStartE2EDuration="13.531316274s" totalImagesPullingTime="10.436304125s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 14:16:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:16:23.531200658 +0000 UTC m=+13.691141319" watchObservedRunningTime="2026-11-01 14:16:23.531316274 +0000 UTC m=+13.691256938"
Nov 01 14:16:24.423755 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:24.423722    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdqx4" event={"ID":"2e013e38-9840-45c2-b4a0-80df0f319109","Type":"ContainerStarted","Data":"0e6073a6d18f621988f389ac8ec8846faf15a66a22b89498a652d56ca4e021a7"}
Nov 01 14:16:24.560060 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:24.560010    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdqx4" podStartSLOduration=4.062485477 podStartE2EDuration="14.55999625s" totalImagesPullingTime="10.497510773s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 14:16:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:16:24.55958992 +0000 UTC m=+14.719530581" watchObservedRunningTime="2026-11-01 14:16:24.55999625 +0000 UTC m=+14.719936905"
Nov 01 14:16:24.560486 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:24.560446    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-8jxks" podStartSLOduration=5.97284449 podStartE2EDuration="14.560439397s" totalImagesPullingTime="8.587594907s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 14:16:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:16:23.643134631 +0000 UTC m=+13.803075293" watchObservedRunningTime="2026-11-01 14:16:24.560439397 +0000 UTC m=+14.720380059"
Nov 01 14:16:25.280595 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:25.280566    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sc9jq"
Nov 01 14:16:25.280749 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:25.280642    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b6rtr"
Nov 01 14:16:25.280749 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:25.280739    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8mqxj"
Nov 01 14:16:25.280873 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:25.280762    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sc9jq" podUID="8b1eb53c-8c9a-46c4-b654-2aef42491b14"
Nov 01 14:16:25.280873 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:25.280858    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8mqxj" podUID="0d4352db-d2e7-43ba-97ba-9e344abbac8b"
Nov 01 14:16:25.280977 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:25.280955    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b6rtr" podUID="ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1"
Nov 01 14:16:25.428876 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:25.428839    2576 generic.go:356] "Generic (PLEG): container finished" podID="bcc6412c-2003-49de-9817-a3713e28afef" containerID="e0dbf291ebc64df129cf689c155ac541c396ab67b535413524ec389de84fe542" exitCode=0
Nov 01 14:16:25.429622 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:25.428971    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9vt8t" event={"ID":"bcc6412c-2003-49de-9817-a3713e28afef","Type":"ContainerDied","Data":"e0dbf291ebc64df129cf689c155ac541c396ab67b535413524ec389de84fe542"}
Nov 01 14:16:27.247867 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:27.247656    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8b1eb53c-8c9a-46c4-b654-2aef42491b14-original-pull-secret\") pod \"global-pull-secret-syncer-sc9jq\" (UID: \"8b1eb53c-8c9a-46c4-b654-2aef42491b14\") " pod="kube-system/global-pull-secret-syncer-sc9jq"
Nov 01 14:16:27.248333 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:27.247926    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1-metrics-certs\") pod \"network-metrics-daemon-b6rtr\" (UID: \"ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1\") " pod="openshift-multus/network-metrics-daemon-b6rtr"
Nov 01 14:16:27.248333 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:27.247806    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 14:16:27.248333 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:27.248004    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b1eb53c-8c9a-46c4-b654-2aef42491b14-original-pull-secret podName:8b1eb53c-8c9a-46c4-b654-2aef42491b14 nodeName:}" failed. No retries permitted until 2026-11-01 14:16:43.247979479 +0000 UTC m=+33.407920132 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8b1eb53c-8c9a-46c4-b654-2aef42491b14-original-pull-secret") pod "global-pull-secret-syncer-sc9jq" (UID: "8b1eb53c-8c9a-46c4-b654-2aef42491b14") : object "kube-system"/"original-pull-secret" not registered
Nov 01 14:16:27.248333 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:27.248046    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 14:16:27.248333 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:27.248092    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1-metrics-certs podName:ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1 nodeName:}" failed. No retries permitted until 2026-11-01 14:16:43.248075625 +0000 UTC m=+33.408016283 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1-metrics-certs") pod "network-metrics-daemon-b6rtr" (UID: "ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 14:16:27.280429 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:27.280361    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b6rtr"
Nov 01 14:16:27.280429 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:27.280394    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8mqxj"
Nov 01 14:16:27.280429 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:27.280407    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sc9jq"
Nov 01 14:16:27.280647 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:27.280539    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b6rtr" podUID="ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1"
Nov 01 14:16:27.280647 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:27.280631    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8mqxj" podUID="0d4352db-d2e7-43ba-97ba-9e344abbac8b"
Nov 01 14:16:27.280755 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:27.280702    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sc9jq" podUID="8b1eb53c-8c9a-46c4-b654-2aef42491b14"
Nov 01 14:16:27.349131 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:27.349099    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp6rs\" (UniqueName: \"kubernetes.io/projected/0d4352db-d2e7-43ba-97ba-9e344abbac8b-kube-api-access-vp6rs\") pod \"network-check-target-8mqxj\" (UID: \"0d4352db-d2e7-43ba-97ba-9e344abbac8b\") " pod="openshift-network-diagnostics/network-check-target-8mqxj"
Nov 01 14:16:27.349292 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:27.349275    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 14:16:27.349328 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:27.349296    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 14:16:27.349328 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:27.349306    2576 projected.go:196] Error preparing data for projected volume kube-api-access-vp6rs for pod openshift-network-diagnostics/network-check-target-8mqxj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 14:16:27.349393 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:27.349355    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0d4352db-d2e7-43ba-97ba-9e344abbac8b-kube-api-access-vp6rs podName:0d4352db-d2e7-43ba-97ba-9e344abbac8b nodeName:}" failed. No retries permitted until 2026-11-01 14:16:43.349341623 +0000 UTC m=+33.509282277 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-vp6rs" (UniqueName: "kubernetes.io/projected/0d4352db-d2e7-43ba-97ba-9e344abbac8b-kube-api-access-vp6rs") pod "network-check-target-8mqxj" (UID: "0d4352db-d2e7-43ba-97ba-9e344abbac8b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 14:16:27.434535 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:27.434499    2576 generic.go:356] "Generic (PLEG): container finished" podID="bcc6412c-2003-49de-9817-a3713e28afef" containerID="5126218ef0fab3767c5e0b17a270a071f3da376d0abd638a12cea24cfb10e6e4" exitCode=0
Nov 01 14:16:27.434661 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:27.434566    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9vt8t" event={"ID":"bcc6412c-2003-49de-9817-a3713e28afef","Type":"ContainerDied","Data":"5126218ef0fab3767c5e0b17a270a071f3da376d0abd638a12cea24cfb10e6e4"}
Nov 01 14:16:28.440297 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:28.440086    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl" event={"ID":"1b941fae-3c27-4d68-8205-38e492179ee9","Type":"ContainerStarted","Data":"f2f3d9759d8000fb7d6060d2a1eeb49dddc9dd66b6e78287f208d6ed53659f6c"}
Nov 01 14:16:28.440641 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:28.440328    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl" event={"ID":"1b941fae-3c27-4d68-8205-38e492179ee9","Type":"ContainerStarted","Data":"6c77f9e076904b4b329de7e58774771167f513598e6cd67b8435f33cbac0e445"}
Nov 01 14:16:28.440641 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:28.440387    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl" event={"ID":"1b941fae-3c27-4d68-8205-38e492179ee9","Type":"ContainerStarted","Data":"7efd0e64368d7bdf040c808f1edbae5ab436863e1874c430da5b978ebe3729b5"}
Nov 01 14:16:28.440641 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:28.440438    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl" event={"ID":"1b941fae-3c27-4d68-8205-38e492179ee9","Type":"ContainerStarted","Data":"b9d1f1f8486b8bc50193614d0e0d32a53c595d9e5498fd1654d3f26cad0bf036"}
Nov 01 14:16:28.440641 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:28.440483    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl" event={"ID":"1b941fae-3c27-4d68-8205-38e492179ee9","Type":"ContainerStarted","Data":"76d9e3690e99549586ea7897aa4d718f9611f847a9d2fcdd47786aecf036711e"}
Nov 01 14:16:29.280393 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:29.280360    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8mqxj"
Nov 01 14:16:29.280611 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:29.280401    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b6rtr"
Nov 01 14:16:29.280611 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:29.280371    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sc9jq"
Nov 01 14:16:29.280611 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:29.280534    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b6rtr" podUID="ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1"
Nov 01 14:16:29.280611 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:29.280534    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8mqxj" podUID="0d4352db-d2e7-43ba-97ba-9e344abbac8b"
Nov 01 14:16:29.280810 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:29.280609    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sc9jq" podUID="8b1eb53c-8c9a-46c4-b654-2aef42491b14"
Nov 01 14:16:29.449180 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:29.449135    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl" event={"ID":"1b941fae-3c27-4d68-8205-38e492179ee9","Type":"ContainerStarted","Data":"5d72348a66ca56adbfa6d6098816e540d9b9319afacee608271ba5ac4839aa29"}
Nov 01 14:16:30.454107 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:30.454072    2576 generic.go:356] "Generic (PLEG): container finished" podID="bcc6412c-2003-49de-9817-a3713e28afef" containerID="8863917ee3587e3abf9a51b988aec40ad35ef2d6b05bf950f5f530838d62a1df" exitCode=0
Nov 01 14:16:30.454581 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:30.454114    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9vt8t" event={"ID":"bcc6412c-2003-49de-9817-a3713e28afef","Type":"ContainerDied","Data":"8863917ee3587e3abf9a51b988aec40ad35ef2d6b05bf950f5f530838d62a1df"}
Nov 01 14:16:31.279748 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:31.279717    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sc9jq"
Nov 01 14:16:31.279748 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:31.279750    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8mqxj"
Nov 01 14:16:31.279985 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:31.279828    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sc9jq" podUID="8b1eb53c-8c9a-46c4-b654-2aef42491b14"
Nov 01 14:16:31.279985 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:31.279876    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8mqxj" podUID="0d4352db-d2e7-43ba-97ba-9e344abbac8b"
Nov 01 14:16:31.279985 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:31.279938    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b6rtr"
Nov 01 14:16:31.280117 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:31.280015    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b6rtr" podUID="ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1"
Nov 01 14:16:31.459001 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:31.458955    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl" event={"ID":"1b941fae-3c27-4d68-8205-38e492179ee9","Type":"ContainerStarted","Data":"a9b4c578d81e8d9c536a9dca711b8e45273054c279efb5297186d6b7f0288af3"}
Nov 01 14:16:31.461912 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:31.461887    2576 generic.go:356] "Generic (PLEG): container finished" podID="bcc6412c-2003-49de-9817-a3713e28afef" containerID="c34dec7ce4ebed9bd4f05417c7eae6058859d0a0f3bdc5ea5163ead034de55d9" exitCode=0
Nov 01 14:16:31.462032 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:31.461931    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9vt8t" event={"ID":"bcc6412c-2003-49de-9817-a3713e28afef","Type":"ContainerDied","Data":"c34dec7ce4ebed9bd4f05417c7eae6058859d0a0f3bdc5ea5163ead034de55d9"}
Nov 01 14:16:32.306342 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:32.306295    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-qzf26"
Nov 01 14:16:32.306525 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:32.306429    2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 01 14:16:32.307050 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:32.307029    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qzf26"
Nov 01 14:16:32.466994 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:32.466956    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9vt8t" event={"ID":"bcc6412c-2003-49de-9817-a3713e28afef","Type":"ContainerStarted","Data":"0cb0dddf68d361dd230923c8fbd16224ed1bb8d4918a4408f2db23a57cbbbade"}
Nov 01 14:16:32.583651 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:32.583594    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-9vt8t" podStartSLOduration=9.063079855 podStartE2EDuration="22.583575082s" totalImagesPullingTime="13.520495227s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 14:16:10 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:16:32.582613697 +0000 UTC m=+22.742554357" watchObservedRunningTime="2026-11-01 14:16:32.583575082 +0000 UTC m=+22.743515744"
Nov 01 14:16:33.280265 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:33.280064    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b6rtr"
Nov 01 14:16:33.280523 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:33.280108    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sc9jq"
Nov 01 14:16:33.280523 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:33.280344    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b6rtr" podUID="ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1"
Nov 01 14:16:33.280523 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:33.280136    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8mqxj"
Nov 01 14:16:33.280523 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:33.280420    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sc9jq" podUID="8b1eb53c-8c9a-46c4-b654-2aef42491b14"
Nov 01 14:16:33.280523 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:33.280492    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8mqxj" podUID="0d4352db-d2e7-43ba-97ba-9e344abbac8b"
Nov 01 14:16:33.472389 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:33.472348    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl" event={"ID":"1b941fae-3c27-4d68-8205-38e492179ee9","Type":"ContainerStarted","Data":"be7f7399662702e2e014f7c12e44b1c70afdb36368a87ad70366c52c12eb25a0"}
Nov 01 14:16:33.733661 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:33.733613    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl" podStartSLOduration=7.329383021 podStartE2EDuration="22.733599997s" totalImagesPullingTime="15.404216976s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 14:16:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:16:33.733331778 +0000 UTC m=+23.893272441" watchObservedRunningTime="2026-11-01 14:16:33.733599997 +0000 UTC m=+23.893540658"
Nov 01 14:16:34.475310 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:34.475280    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:34.475783 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:34.475355    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:34.475783 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:34.475422    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:34.490592 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:34.490571    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:34.490725 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:34.490698    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:35.277294 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:35.277265    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-sc9jq"]
Nov 01 14:16:35.277449 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:35.277397    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sc9jq"
Nov 01 14:16:35.277503 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:35.277484    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sc9jq" podUID="8b1eb53c-8c9a-46c4-b654-2aef42491b14"
Nov 01 14:16:35.277900 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:35.277881    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8mqxj"]
Nov 01 14:16:35.277981 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:35.277971    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8mqxj"
Nov 01 14:16:35.278053 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:35.278037    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8mqxj" podUID="0d4352db-d2e7-43ba-97ba-9e344abbac8b"
Nov 01 14:16:35.279567 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:35.279551    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-b6rtr"]
Nov 01 14:16:35.279659 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:35.279649    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b6rtr"
Nov 01 14:16:35.279734 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:35.279719    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b6rtr" podUID="ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1"
Nov 01 14:16:37.280534 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:37.280505    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sc9jq"
Nov 01 14:16:37.281192 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:37.280596    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b6rtr"
Nov 01 14:16:37.281192 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:37.280614    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8mqxj"
Nov 01 14:16:37.281192 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:37.280719    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sc9jq" podUID="8b1eb53c-8c9a-46c4-b654-2aef42491b14"
Nov 01 14:16:37.281192 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:37.280834    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b6rtr" podUID="ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1"
Nov 01 14:16:37.281192 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:37.280892    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8mqxj" podUID="0d4352db-d2e7-43ba-97ba-9e344abbac8b"
Nov 01 14:16:39.280227 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:39.280196    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sc9jq"
Nov 01 14:16:39.280618 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:39.280230    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8mqxj"
Nov 01 14:16:39.280618 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:39.280308    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b6rtr"
Nov 01 14:16:39.280618 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:39.280304    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sc9jq" podUID="8b1eb53c-8c9a-46c4-b654-2aef42491b14"
Nov 01 14:16:39.280618 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:39.280410    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8mqxj" podUID="0d4352db-d2e7-43ba-97ba-9e344abbac8b"
Nov 01 14:16:39.280618 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:39.280474    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b6rtr" podUID="ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1"
Nov 01 14:16:41.178565 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.178532    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-37.us-west-2.compute.internal" event="NodeReady"
Nov 01 14:16:41.178943 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.178655    2576 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 01 14:16:41.280278 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.280236    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sc9jq"
Nov 01 14:16:41.280442 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.280319    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b6rtr"
Nov 01 14:16:41.280442 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.280403    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8mqxj"
Nov 01 14:16:41.289878 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.289856    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-zfqjv"]
Nov 01 14:16:41.293979 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.293958    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-zfqjv"
Nov 01 14:16:41.330892 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.330860    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-4lzxm"]
Nov 01 14:16:41.335410 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.335388    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-xmfr4"]
Nov 01 14:16:41.335539 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.335519    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-4lzxm"
Nov 01 14:16:41.339579 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.339556    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-xmfr4"
Nov 01 14:16:41.339922 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.339903    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-96vgd"]
Nov 01 14:16:41.343572 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.343553    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-shpt4"]
Nov 01 14:16:41.343687 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.343662    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-96vgd"
Nov 01 14:16:41.346236 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.346219    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-69c8c9797f-cblkb"]
Nov 01 14:16:41.346328 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.346315    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-shpt4"
Nov 01 14:16:41.349132 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.349113    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-bgmv8"]
Nov 01 14:16:41.349236 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.349221    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69c8c9797f-cblkb"
Nov 01 14:16:41.353063 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.353043    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-87xcd"]
Nov 01 14:16:41.353279 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.353250    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bgmv8"
Nov 01 14:16:41.356015 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.355999    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-7d54k"]
Nov 01 14:16:41.356147 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.356130    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-87xcd"
Nov 01 14:16:41.358854 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.358835    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-cwfck"]
Nov 01 14:16:41.358972 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.358956    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-7d54k"
Nov 01 14:16:41.362021 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.362002    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-cwfck"
Nov 01 14:16:41.381988 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.381962    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7f9cfbc8db-w8cmn"]
Nov 01 14:16:41.384868 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.384849    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-xmwtp"]
Nov 01 14:16:41.385108 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.385094    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f9cfbc8db-w8cmn"
Nov 01 14:16:41.388340 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.388322    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hsdjc"]
Nov 01 14:16:41.388446 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.388428    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-xmwtp"
Nov 01 14:16:41.389627 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.389593    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 01 14:16:41.389761 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.389743    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-x2nj9\""
Nov 01 14:16:41.389822 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.389772    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-ftvdm\""
Nov 01 14:16:41.389822 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.389790    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 01 14:16:41.389822 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.389817    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 01 14:16:41.389962 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.389825    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-mpq57\""
Nov 01 14:16:41.389962 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.389830    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 01 14:16:41.394781 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.394759    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-b9d995dcc-lnk2n"]
Nov 01 14:16:41.396576 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.394917    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hsdjc"
Nov 01 14:16:41.400352 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.400330    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-lhdpr"]
Nov 01 14:16:41.400511 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.400491    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-b9d995dcc-lnk2n"
Nov 01 14:16:41.403601 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.403578    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8fgtw"]
Nov 01 14:16:41.403753 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.403735    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-lhdpr"
Nov 01 14:16:41.406498 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.406483    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-b2lcr"]
Nov 01 14:16:41.406623 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.406609    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8fgtw"
Nov 01 14:16:41.409526 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.409507    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-4lzxm"]
Nov 01 14:16:41.409617 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.409545    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-zfqjv"]
Nov 01 14:16:41.409617 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.409569    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-96vgd"]
Nov 01 14:16:41.409617 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.409593    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-cwfck"]
Nov 01 14:16:41.409617 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.409611    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-bgmv8"]
Nov 01 14:16:41.409811 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.409631    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-7d54k"]
Nov 01 14:16:41.409811 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.409654    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-b2lcr"
Nov 01 14:16:41.431813 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.431747    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-87xcd"]
Nov 01 14:16:41.432632 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.432611    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-69c8c9797f-cblkb"]
Nov 01 14:16:41.433686 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.433667    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f9cfbc8db-w8cmn"]
Nov 01 14:16:41.433757 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.433668    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 01 14:16:41.433757 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.433676    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Nov 01 14:16:41.434281 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.434266    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-b9d995dcc-lnk2n"]
Nov 01 14:16:41.438933 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.438914    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Nov 01 14:16:41.439264 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.439242    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Nov 01 14:16:41.439361 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.439344    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Nov 01 14:16:41.439412 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.439388    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Nov 01 14:16:41.439716 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.439485    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Nov 01 14:16:41.439877 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.439858    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Nov 01 14:16:41.439970 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.439956    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 01 14:16:41.440104 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.440085    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Nov 01 14:16:41.440203 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.440140    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 01 14:16:41.440438 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.440419    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 01 14:16:41.440530 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.440488    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 01 14:16:41.440608 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.440582    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Nov 01 14:16:41.440725 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.440707    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Nov 01 14:16:41.440788 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.440777    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Nov 01 14:16:41.440834 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.440801    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 01 14:16:41.440834 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.440811    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Nov 01 14:16:41.440918 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.440852    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Nov 01 14:16:41.440918 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.440853    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 01 14:16:41.440991 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.440943    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Nov 01 14:16:41.441036 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.440999    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 01 14:16:41.441036 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.441031    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Nov 01 14:16:41.441133 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.441091    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-c5fv9\""
Nov 01 14:16:41.441133 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.441131    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-tp8z2\""
Nov 01 14:16:41.441253 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.441200    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Nov 01 14:16:41.441389 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.441367    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Nov 01 14:16:41.448585 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.448565    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a26b817-3283-4d59-8e8f-3bb4c0fa4987-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-7d54k\" (UID: \"7a26b817-3283-4d59-8e8f-3bb4c0fa4987\") " pod="openshift-insights/insights-operator-7c4986d449-7d54k"
Nov 01 14:16:41.448698 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.448598    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9j226\" (UniqueName: \"kubernetes.io/projected/6a8f947a-0f8f-41aa-b646-4459fa8126a3-kube-api-access-9j226\") pod \"kas-connection-checker-685895c7c9-xmfr4\" (UID: \"6a8f947a-0f8f-41aa-b646-4459fa8126a3\") " pod="kube-system/kas-connection-checker-685895c7c9-xmfr4"
Nov 01 14:16:41.448698 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.448626    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/7a26b817-3283-4d59-8e8f-3bb4c0fa4987-snapshots\") pod \"insights-operator-7c4986d449-7d54k\" (UID: \"7a26b817-3283-4d59-8e8f-3bb4c0fa4987\") " pod="openshift-insights/insights-operator-7c4986d449-7d54k"
Nov 01 14:16:41.448806 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.448762    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e7ea769b-e719-4090-bcd4-05b1b88fcc06-image-registry-private-configuration\") pod \"image-registry-69c8c9797f-cblkb\" (UID: \"e7ea769b-e719-4090-bcd4-05b1b88fcc06\") " pod="openshift-image-registry/image-registry-69c8c9797f-cblkb"
Nov 01 14:16:41.448862 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.448815    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bv8hp\" (UniqueName: \"kubernetes.io/projected/7a26b817-3283-4d59-8e8f-3bb4c0fa4987-kube-api-access-bv8hp\") pod \"insights-operator-7c4986d449-7d54k\" (UID: \"7a26b817-3283-4d59-8e8f-3bb4c0fa4987\") " pod="openshift-insights/insights-operator-7c4986d449-7d54k"
Nov 01 14:16:41.448862 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.448855    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3e7e18b9-78ee-4349-b00c-a0800e8f8341-config\") pod \"service-ca-operator-544d749c8c-87xcd\" (UID: \"3e7e18b9-78ee-4349-b00c-a0800e8f8341\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-87xcd"
Nov 01 14:16:41.448963 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.448874    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e7ea769b-e719-4090-bcd4-05b1b88fcc06-installation-pull-secrets\") pod \"image-registry-69c8c9797f-cblkb\" (UID: \"e7ea769b-e719-4090-bcd4-05b1b88fcc06\") " pod="openshift-image-registry/image-registry-69c8c9797f-cblkb"
Nov 01 14:16:41.448963 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.448919    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/037ca099-5ad7-4874-a984-462c07801e70-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-bgmv8\" (UID: \"037ca099-5ad7-4874-a984-462c07801e70\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bgmv8"
Nov 01 14:16:41.449203 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.449034    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/037ca099-5ad7-4874-a984-462c07801e70-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-bgmv8\" (UID: \"037ca099-5ad7-4874-a984-462c07801e70\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bgmv8"
Nov 01 14:16:41.449203 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.449065    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e7ea769b-e719-4090-bcd4-05b1b88fcc06-trusted-ca\") pod \"image-registry-69c8c9797f-cblkb\" (UID: \"e7ea769b-e719-4090-bcd4-05b1b88fcc06\") " pod="openshift-image-registry/image-registry-69c8c9797f-cblkb"
Nov 01 14:16:41.449203 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.449119    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e7ea769b-e719-4090-bcd4-05b1b88fcc06-bound-sa-token\") pod \"image-registry-69c8c9797f-cblkb\" (UID: \"e7ea769b-e719-4090-bcd4-05b1b88fcc06\") " pod="openshift-image-registry/image-registry-69c8c9797f-cblkb"
Nov 01 14:16:41.449203 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.449161    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7a26b817-3283-4d59-8e8f-3bb4c0fa4987-serving-cert\") pod \"insights-operator-7c4986d449-7d54k\" (UID: \"7a26b817-3283-4d59-8e8f-3bb4c0fa4987\") " pod="openshift-insights/insights-operator-7c4986d449-7d54k"
Nov 01 14:16:41.449364 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.449243    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7a26b817-3283-4d59-8e8f-3bb4c0fa4987-tmp\") pod \"insights-operator-7c4986d449-7d54k\" (UID: \"7a26b817-3283-4d59-8e8f-3bb4c0fa4987\") " pod="openshift-insights/insights-operator-7c4986d449-7d54k"
Nov 01 14:16:41.449364 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.449275    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfj7s\" (UniqueName: \"kubernetes.io/projected/4a724964-bd57-4d56-a042-a0d490115eaa-kube-api-access-kfj7s\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-96vgd\" (UID: \"4a724964-bd57-4d56-a042-a0d490115eaa\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-96vgd"
Nov 01 14:16:41.449364 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.449296    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4a724964-bd57-4d56-a042-a0d490115eaa-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-96vgd\" (UID: \"4a724964-bd57-4d56-a042-a0d490115eaa\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-96vgd"
Nov 01 14:16:41.449364 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.449319    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e7ea769b-e719-4090-bcd4-05b1b88fcc06-registry-certificates\") pod \"image-registry-69c8c9797f-cblkb\" (UID: \"e7ea769b-e719-4090-bcd4-05b1b88fcc06\") " pod="openshift-image-registry/image-registry-69c8c9797f-cblkb"
Nov 01 14:16:41.449364 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.449340    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdxs8\" (UniqueName: \"kubernetes.io/projected/e7ea769b-e719-4090-bcd4-05b1b88fcc06-kube-api-access-rdxs8\") pod \"image-registry-69c8c9797f-cblkb\" (UID: \"e7ea769b-e719-4090-bcd4-05b1b88fcc06\") " pod="openshift-image-registry/image-registry-69c8c9797f-cblkb"
Nov 01 14:16:41.449364 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.449362    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b5a4fefb-cc48-4638-b0f8-590e36bd0118-nginx-conf\") pod \"networking-console-plugin-86c478bb88-cwfck\" (UID: \"b5a4fefb-cc48-4638-b0f8-590e36bd0118\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-cwfck"
Nov 01 14:16:41.449567 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.449386    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b5a4fefb-cc48-4638-b0f8-590e36bd0118-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-cwfck\" (UID: \"b5a4fefb-cc48-4638-b0f8-590e36bd0118\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-cwfck"
Nov 01 14:16:41.449567 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.449414    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e7ea769b-e719-4090-bcd4-05b1b88fcc06-ca-trust-extracted\") pod \"image-registry-69c8c9797f-cblkb\" (UID: \"e7ea769b-e719-4090-bcd4-05b1b88fcc06\") " pod="openshift-image-registry/image-registry-69c8c9797f-cblkb"
Nov 01 14:16:41.449567 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.449455    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stw2n\" (UniqueName: \"kubernetes.io/projected/3e7e18b9-78ee-4349-b00c-a0800e8f8341-kube-api-access-stw2n\") pod \"service-ca-operator-544d749c8c-87xcd\" (UID: \"3e7e18b9-78ee-4349-b00c-a0800e8f8341\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-87xcd"
Nov 01 14:16:41.449567 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.449524    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e7ea769b-e719-4090-bcd4-05b1b88fcc06-registry-tls\") pod \"image-registry-69c8c9797f-cblkb\" (UID: \"e7ea769b-e719-4090-bcd4-05b1b88fcc06\") " pod="openshift-image-registry/image-registry-69c8c9797f-cblkb"
Nov 01 14:16:41.449697 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.449603    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxdcx\" (UniqueName: \"kubernetes.io/projected/e78d2494-62c4-4070-a1eb-8ce40329ff57-kube-api-access-wxdcx\") pod \"volume-data-source-validator-58c9f5f46d-shpt4\" (UID: \"e78d2494-62c4-4070-a1eb-8ce40329ff57\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-shpt4"
Nov 01 14:16:41.449697 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.449644    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6cqj\" (UniqueName: \"kubernetes.io/projected/d8a5ccab-4d5e-4aaf-ad08-766238eca05f-kube-api-access-q6cqj\") pod \"kas-connection-checker-685895c7c9-zfqjv\" (UID: \"d8a5ccab-4d5e-4aaf-ad08-766238eca05f\") " pod="kube-system/kas-connection-checker-685895c7c9-zfqjv"
Nov 01 14:16:41.449697 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.449675    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4a724964-bd57-4d56-a042-a0d490115eaa-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-96vgd\" (UID: \"4a724964-bd57-4d56-a042-a0d490115eaa\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-96vgd"
Nov 01 14:16:41.449794 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.449712    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8thtk\" (UniqueName: \"kubernetes.io/projected/037ca099-5ad7-4874-a984-462c07801e70-kube-api-access-8thtk\") pod \"cluster-monitoring-operator-884576c57-bgmv8\" (UID: \"037ca099-5ad7-4874-a984-462c07801e70\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bgmv8"
Nov 01 14:16:41.449794 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.449756    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jbv4\" (UniqueName: \"kubernetes.io/projected/2460be58-d51b-4876-bf9b-2a0e57901386-kube-api-access-7jbv4\") pod \"kas-connection-checker-685895c7c9-4lzxm\" (UID: \"2460be58-d51b-4876-bf9b-2a0e57901386\") " pod="kube-system/kas-connection-checker-685895c7c9-4lzxm"
Nov 01 14:16:41.449794 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.449775    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3e7e18b9-78ee-4349-b00c-a0800e8f8341-serving-cert\") pod \"service-ca-operator-544d749c8c-87xcd\" (UID: \"3e7e18b9-78ee-4349-b00c-a0800e8f8341\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-87xcd"
Nov 01 14:16:41.449893 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.449799    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a26b817-3283-4d59-8e8f-3bb4c0fa4987-service-ca-bundle\") pod \"insights-operator-7c4986d449-7d54k\" (UID: \"7a26b817-3283-4d59-8e8f-3bb4c0fa4987\") " pod="openshift-insights/insights-operator-7c4986d449-7d54k"
Nov 01 14:16:41.490492 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.490466    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 01 14:16:41.490636 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.490468    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 01 14:16:41.491380 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.491359    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8fgtw"]
Nov 01 14:16:41.492018 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.491994    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-shpt4"]
Nov 01 14:16:41.492533 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.492517    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-xmfr4"]
Nov 01 14:16:41.494048 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.494033    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-xmwtp"]
Nov 01 14:16:41.494658 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.494643    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-lhdpr"]
Nov 01 14:16:41.495344 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.495332    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-b2lcr"]
Nov 01 14:16:41.496063 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.496045    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hsdjc"]
Nov 01 14:16:41.533277 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.533245    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Nov 01 14:16:41.533277 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.533251    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 01 14:16:41.533497 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.533260    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Nov 01 14:16:41.533497 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.533290    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Nov 01 14:16:41.533604 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.533519    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Nov 01 14:16:41.533604 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.533527    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Nov 01 14:16:41.533604 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.533527    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 01 14:16:41.533604 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.533549    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Nov 01 14:16:41.533604 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.533523    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 01 14:16:41.533604 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.533579    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Nov 01 14:16:41.533604 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.533586    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Nov 01 14:16:41.533604 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.533598    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Nov 01 14:16:41.533604 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.533606    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 01 14:16:41.533927 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.533609    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 01 14:16:41.533927 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.533625    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 01 14:16:41.533927 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.533702    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Nov 01 14:16:41.533927 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.533717    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 01 14:16:41.533927 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.533706    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Nov 01 14:16:41.533927 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.533788    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-gxzsc\""
Nov 01 14:16:41.533927 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.533803    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Nov 01 14:16:41.533927 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.533809    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-x2zjc\""
Nov 01 14:16:41.533927 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.533844    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zjzzz\""
Nov 01 14:16:41.534764 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.534748    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Nov 01 14:16:41.550777 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.550748    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b5a4fefb-cc48-4638-b0f8-590e36bd0118-nginx-conf\") pod \"networking-console-plugin-86c478bb88-cwfck\" (UID: \"b5a4fefb-cc48-4638-b0f8-590e36bd0118\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-cwfck"
Nov 01 14:16:41.550903 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.550792    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b5a4fefb-cc48-4638-b0f8-590e36bd0118-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-cwfck\" (UID: \"b5a4fefb-cc48-4638-b0f8-590e36bd0118\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-cwfck"
Nov 01 14:16:41.550903 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.550828    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e7ea769b-e719-4090-bcd4-05b1b88fcc06-ca-trust-extracted\") pod \"image-registry-69c8c9797f-cblkb\" (UID: \"e7ea769b-e719-4090-bcd4-05b1b88fcc06\") " pod="openshift-image-registry/image-registry-69c8c9797f-cblkb"
Nov 01 14:16:41.550903 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.550870    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7970eec1-3765-48a4-b09c-3dec2d29956b-trusted-ca\") pod \"image-registry-7f9cfbc8db-w8cmn\" (UID: \"7970eec1-3765-48a4-b09c-3dec2d29956b\") " pod="openshift-image-registry/image-registry-7f9cfbc8db-w8cmn"
Nov 01 14:16:41.550903 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.550900    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/70bb916a-832b-4bce-9c0a-f01bf4e3abc4-cert\") pod \"ingress-canary-8fgtw\" (UID: \"70bb916a-832b-4bce-9c0a-f01bf4e3abc4\") " pod="openshift-ingress-canary/ingress-canary-8fgtw"
Nov 01 14:16:41.551100 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.550932    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stw2n\" (UniqueName: \"kubernetes.io/projected/3e7e18b9-78ee-4349-b00c-a0800e8f8341-kube-api-access-stw2n\") pod \"service-ca-operator-544d749c8c-87xcd\" (UID: \"3e7e18b9-78ee-4349-b00c-a0800e8f8341\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-87xcd"
Nov 01 14:16:41.551100 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.550980    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e7ea769b-e719-4090-bcd4-05b1b88fcc06-registry-tls\") pod \"image-registry-69c8c9797f-cblkb\" (UID: \"e7ea769b-e719-4090-bcd4-05b1b88fcc06\") " pod="openshift-image-registry/image-registry-69c8c9797f-cblkb"
Nov 01 14:16:41.551100 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.551013    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c27ff402-235a-4d73-a139-cf928a6d5cbe-metrics-certs\") pod \"router-default-b9d995dcc-lnk2n\" (UID: \"c27ff402-235a-4d73-a139-cf928a6d5cbe\") " pod="openshift-ingress/router-default-b9d995dcc-lnk2n"
Nov 01 14:16:41.551100 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.551068    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxdcx\" (UniqueName: \"kubernetes.io/projected/e78d2494-62c4-4070-a1eb-8ce40329ff57-kube-api-access-wxdcx\") pod \"volume-data-source-validator-58c9f5f46d-shpt4\" (UID: \"e78d2494-62c4-4070-a1eb-8ce40329ff57\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-shpt4"
Nov 01 14:16:41.551320 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.551105    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6cqj\" (UniqueName: \"kubernetes.io/projected/d8a5ccab-4d5e-4aaf-ad08-766238eca05f-kube-api-access-q6cqj\") pod \"kas-connection-checker-685895c7c9-zfqjv\" (UID: \"d8a5ccab-4d5e-4aaf-ad08-766238eca05f\") " pod="kube-system/kas-connection-checker-685895c7c9-zfqjv"
Nov 01 14:16:41.551320 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.551137    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0c8cc47b-fb01-4195-a7fc-eeefc706017e-config-volume\") pod \"dns-default-b2lcr\" (UID: \"0c8cc47b-fb01-4195-a7fc-eeefc706017e\") " pod="openshift-dns/dns-default-b2lcr"
Nov 01 14:16:41.551320 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.551190    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4a724964-bd57-4d56-a042-a0d490115eaa-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-96vgd\" (UID: \"4a724964-bd57-4d56-a042-a0d490115eaa\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-96vgd"
Nov 01 14:16:41.551320 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.551222    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c8cc47b-fb01-4195-a7fc-eeefc706017e-metrics-tls\") pod \"dns-default-b2lcr\" (UID: \"0c8cc47b-fb01-4195-a7fc-eeefc706017e\") " pod="openshift-dns/dns-default-b2lcr"
Nov 01 14:16:41.551320 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.551251    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8thtk\" (UniqueName: \"kubernetes.io/projected/037ca099-5ad7-4874-a984-462c07801e70-kube-api-access-8thtk\") pod \"cluster-monitoring-operator-884576c57-bgmv8\" (UID: \"037ca099-5ad7-4874-a984-462c07801e70\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bgmv8"
Nov 01 14:16:41.551320 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.551280    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bb59051d-c2df-4362-a157-00f14f915e79-trusted-ca\") pod \"console-operator-7cd56b8dbc-xmwtp\" (UID: \"bb59051d-c2df-4362-a157-00f14f915e79\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-xmwtp"
Nov 01 14:16:41.551320 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.551312    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7970eec1-3765-48a4-b09c-3dec2d29956b-registry-certificates\") pod \"image-registry-7f9cfbc8db-w8cmn\" (UID: \"7970eec1-3765-48a4-b09c-3dec2d29956b\") " pod="openshift-image-registry/image-registry-7f9cfbc8db-w8cmn"
Nov 01 14:16:41.551592 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.551346    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jbv4\" (UniqueName: \"kubernetes.io/projected/2460be58-d51b-4876-bf9b-2a0e57901386-kube-api-access-7jbv4\") pod \"kas-connection-checker-685895c7c9-4lzxm\" (UID: \"2460be58-d51b-4876-bf9b-2a0e57901386\") " pod="kube-system/kas-connection-checker-685895c7c9-4lzxm"
Nov 01 14:16:41.551592 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.551377    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gnmx\" (UniqueName: \"kubernetes.io/projected/7970eec1-3765-48a4-b09c-3dec2d29956b-kube-api-access-8gnmx\") pod \"image-registry-7f9cfbc8db-w8cmn\" (UID: \"7970eec1-3765-48a4-b09c-3dec2d29956b\") " pod="openshift-image-registry/image-registry-7f9cfbc8db-w8cmn"
Nov 01 14:16:41.551592 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.551402    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3e7e18b9-78ee-4349-b00c-a0800e8f8341-serving-cert\") pod \"service-ca-operator-544d749c8c-87xcd\" (UID: \"3e7e18b9-78ee-4349-b00c-a0800e8f8341\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-87xcd"
Nov 01 14:16:41.551592 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.551433    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a26b817-3283-4d59-8e8f-3bb4c0fa4987-service-ca-bundle\") pod \"insights-operator-7c4986d449-7d54k\" (UID: \"7a26b817-3283-4d59-8e8f-3bb4c0fa4987\") " pod="openshift-insights/insights-operator-7c4986d449-7d54k"
Nov 01 14:16:41.551592 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.551466    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d407313d-3333-4ec4-85bb-cd27cf4d7422-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-hsdjc\" (UID: \"d407313d-3333-4ec4-85bb-cd27cf4d7422\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hsdjc"
Nov 01 14:16:41.551592 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.551495    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkzgz\" (UniqueName: \"kubernetes.io/projected/70bb916a-832b-4bce-9c0a-f01bf4e3abc4-kube-api-access-xkzgz\") pod \"ingress-canary-8fgtw\" (UID: \"70bb916a-832b-4bce-9c0a-f01bf4e3abc4\") " pod="openshift-ingress-canary/ingress-canary-8fgtw"
Nov 01 14:16:41.551592 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.551529    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c27ff402-235a-4d73-a139-cf928a6d5cbe-stats-auth\") pod \"router-default-b9d995dcc-lnk2n\" (UID: \"c27ff402-235a-4d73-a139-cf928a6d5cbe\") " pod="openshift-ingress/router-default-b9d995dcc-lnk2n"
Nov 01 14:16:41.551592 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.551555    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a26b817-3283-4d59-8e8f-3bb4c0fa4987-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-7d54k\" (UID: \"7a26b817-3283-4d59-8e8f-3bb4c0fa4987\") " pod="openshift-insights/insights-operator-7c4986d449-7d54k"
Nov 01 14:16:41.551592 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.551587    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7970eec1-3765-48a4-b09c-3dec2d29956b-bound-sa-token\") pod \"image-registry-7f9cfbc8db-w8cmn\" (UID: \"7970eec1-3765-48a4-b09c-3dec2d29956b\") " pod="openshift-image-registry/image-registry-7f9cfbc8db-w8cmn"
Nov 01 14:16:41.552002 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.551619    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsmvn\" (UniqueName: \"kubernetes.io/projected/d407313d-3333-4ec4-85bb-cd27cf4d7422-kube-api-access-fsmvn\") pod \"cluster-samples-operator-6ddb6c8b94-hsdjc\" (UID: \"d407313d-3333-4ec4-85bb-cd27cf4d7422\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hsdjc"
Nov 01 14:16:41.552002 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.551652    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j226\" (UniqueName: \"kubernetes.io/projected/6a8f947a-0f8f-41aa-b646-4459fa8126a3-kube-api-access-9j226\") pod \"kas-connection-checker-685895c7c9-xmfr4\" (UID: \"6a8f947a-0f8f-41aa-b646-4459fa8126a3\") " pod="kube-system/kas-connection-checker-685895c7c9-xmfr4"
Nov 01 14:16:41.552002 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.551677    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/7a26b817-3283-4d59-8e8f-3bb4c0fa4987-snapshots\") pod \"insights-operator-7c4986d449-7d54k\" (UID: \"7a26b817-3283-4d59-8e8f-3bb4c0fa4987\") " pod="openshift-insights/insights-operator-7c4986d449-7d54k"
Nov 01 14:16:41.552002 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.551709    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjj74\" (UniqueName: \"kubernetes.io/projected/f219e3c2-6da8-4c88-9ec4-b26cf4d10326-kube-api-access-pjj74\") pod \"network-check-source-6f78d59f99-lhdpr\" (UID: \"f219e3c2-6da8-4c88-9ec4-b26cf4d10326\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-lhdpr"
Nov 01 14:16:41.552002 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.551746    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c8cc47b-fb01-4195-a7fc-eeefc706017e-tmp-dir\") pod \"dns-default-b2lcr\" (UID: \"0c8cc47b-fb01-4195-a7fc-eeefc706017e\") " pod="openshift-dns/dns-default-b2lcr"
Nov 01 14:16:41.552002 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.551780    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e7ea769b-e719-4090-bcd4-05b1b88fcc06-image-registry-private-configuration\") pod \"image-registry-69c8c9797f-cblkb\" (UID: \"e7ea769b-e719-4090-bcd4-05b1b88fcc06\") " pod="openshift-image-registry/image-registry-69c8c9797f-cblkb"
Nov 01 14:16:41.552002 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.551820    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv8hp\" (UniqueName: \"kubernetes.io/projected/7a26b817-3283-4d59-8e8f-3bb4c0fa4987-kube-api-access-bv8hp\") pod \"insights-operator-7c4986d449-7d54k\" (UID: \"7a26b817-3283-4d59-8e8f-3bb4c0fa4987\") " pod="openshift-insights/insights-operator-7c4986d449-7d54k"
Nov 01 14:16:41.552002 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.551853    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3e7e18b9-78ee-4349-b00c-a0800e8f8341-config\") pod \"service-ca-operator-544d749c8c-87xcd\" (UID: \"3e7e18b9-78ee-4349-b00c-a0800e8f8341\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-87xcd"
Nov 01 14:16:41.552002 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.551883    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e7ea769b-e719-4090-bcd4-05b1b88fcc06-installation-pull-secrets\") pod \"image-registry-69c8c9797f-cblkb\" (UID: \"e7ea769b-e719-4090-bcd4-05b1b88fcc06\") " pod="openshift-image-registry/image-registry-69c8c9797f-cblkb"
Nov 01 14:16:41.552002 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.551907    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bb59051d-c2df-4362-a157-00f14f915e79-serving-cert\") pod \"console-operator-7cd56b8dbc-xmwtp\" (UID: \"bb59051d-c2df-4362-a157-00f14f915e79\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-xmwtp"
Nov 01 14:16:41.552002 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.551937    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92xdl\" (UniqueName: \"kubernetes.io/projected/bb59051d-c2df-4362-a157-00f14f915e79-kube-api-access-92xdl\") pod \"console-operator-7cd56b8dbc-xmwtp\" (UID: \"bb59051d-c2df-4362-a157-00f14f915e79\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-xmwtp"
Nov 01 14:16:41.552002 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.551966    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgmsd\" (UniqueName: \"kubernetes.io/projected/0c8cc47b-fb01-4195-a7fc-eeefc706017e-kube-api-access-wgmsd\") pod \"dns-default-b2lcr\" (UID: \"0c8cc47b-fb01-4195-a7fc-eeefc706017e\") " pod="openshift-dns/dns-default-b2lcr"
Nov 01 14:16:41.552002 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.551999    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/037ca099-5ad7-4874-a984-462c07801e70-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-bgmv8\" (UID: \"037ca099-5ad7-4874-a984-462c07801e70\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bgmv8"
Nov 01 14:16:41.552612 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.552035    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/037ca099-5ad7-4874-a984-462c07801e70-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-bgmv8\" (UID: \"037ca099-5ad7-4874-a984-462c07801e70\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bgmv8"
Nov 01 14:16:41.552612 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.552069    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e7ea769b-e719-4090-bcd4-05b1b88fcc06-trusted-ca\") pod \"image-registry-69c8c9797f-cblkb\" (UID: \"e7ea769b-e719-4090-bcd4-05b1b88fcc06\") " pod="openshift-image-registry/image-registry-69c8c9797f-cblkb"
Nov 01 14:16:41.552612 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.552100    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e7ea769b-e719-4090-bcd4-05b1b88fcc06-bound-sa-token\") pod \"image-registry-69c8c9797f-cblkb\" (UID: \"e7ea769b-e719-4090-bcd4-05b1b88fcc06\") " pod="openshift-image-registry/image-registry-69c8c9797f-cblkb"
Nov 01 14:16:41.552612 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.552153    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7a26b817-3283-4d59-8e8f-3bb4c0fa4987-serving-cert\") pod \"insights-operator-7c4986d449-7d54k\" (UID: \"7a26b817-3283-4d59-8e8f-3bb4c0fa4987\") " pod="openshift-insights/insights-operator-7c4986d449-7d54k"
Nov 01 14:16:41.552612 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.552208    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bb59051d-c2df-4362-a157-00f14f915e79-config\") pod \"console-operator-7cd56b8dbc-xmwtp\" (UID: \"bb59051d-c2df-4362-a157-00f14f915e79\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-xmwtp"
Nov 01 14:16:41.552612 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.552239    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c27ff402-235a-4d73-a139-cf928a6d5cbe-service-ca-bundle\") pod \"router-default-b9d995dcc-lnk2n\" (UID: \"c27ff402-235a-4d73-a139-cf928a6d5cbe\") " pod="openshift-ingress/router-default-b9d995dcc-lnk2n"
Nov 01 14:16:41.552612 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.552279    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7a26b817-3283-4d59-8e8f-3bb4c0fa4987-tmp\") pod \"insights-operator-7c4986d449-7d54k\" (UID: \"7a26b817-3283-4d59-8e8f-3bb4c0fa4987\") " pod="openshift-insights/insights-operator-7c4986d449-7d54k"
Nov 01 14:16:41.552612 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.552369    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfj7s\" (UniqueName: \"kubernetes.io/projected/4a724964-bd57-4d56-a042-a0d490115eaa-kube-api-access-kfj7s\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-96vgd\" (UID: \"4a724964-bd57-4d56-a042-a0d490115eaa\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-96vgd"
Nov 01 14:16:41.552612 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.552404    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjv76\" (UniqueName: \"kubernetes.io/projected/c27ff402-235a-4d73-a139-cf928a6d5cbe-kube-api-access-rjv76\") pod \"router-default-b9d995dcc-lnk2n\" (UID: \"c27ff402-235a-4d73-a139-cf928a6d5cbe\") " pod="openshift-ingress/router-default-b9d995dcc-lnk2n"
Nov 01 14:16:41.552612 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.552441    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4a724964-bd57-4d56-a042-a0d490115eaa-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-96vgd\" (UID: \"4a724964-bd57-4d56-a042-a0d490115eaa\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-96vgd"
Nov 01 14:16:41.552612 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.552472    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e7ea769b-e719-4090-bcd4-05b1b88fcc06-registry-certificates\") pod \"image-registry-69c8c9797f-cblkb\" (UID: \"e7ea769b-e719-4090-bcd4-05b1b88fcc06\") " pod="openshift-image-registry/image-registry-69c8c9797f-cblkb"
Nov 01 14:16:41.552612 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.552503    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7970eec1-3765-48a4-b09c-3dec2d29956b-ca-trust-extracted\") pod \"image-registry-7f9cfbc8db-w8cmn\" (UID: \"7970eec1-3765-48a4-b09c-3dec2d29956b\") " pod="openshift-image-registry/image-registry-7f9cfbc8db-w8cmn"
Nov 01 14:16:41.552612 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.552534    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/d407313d-3333-4ec4-85bb-cd27cf4d7422-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-hsdjc\" (UID: \"d407313d-3333-4ec4-85bb-cd27cf4d7422\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hsdjc"
Nov 01 14:16:41.552612 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.552571    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdxs8\" (UniqueName: \"kubernetes.io/projected/e7ea769b-e719-4090-bcd4-05b1b88fcc06-kube-api-access-rdxs8\") pod \"image-registry-69c8c9797f-cblkb\" (UID: \"e7ea769b-e719-4090-bcd4-05b1b88fcc06\") " pod="openshift-image-registry/image-registry-69c8c9797f-cblkb"
Nov 01 14:16:41.552612 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.552601    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7970eec1-3765-48a4-b09c-3dec2d29956b-image-registry-private-configuration\") pod \"image-registry-7f9cfbc8db-w8cmn\" (UID: \"7970eec1-3765-48a4-b09c-3dec2d29956b\") " pod="openshift-image-registry/image-registry-7f9cfbc8db-w8cmn"
Nov 01 14:16:41.553327 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.552627    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7970eec1-3765-48a4-b09c-3dec2d29956b-installation-pull-secrets\") pod \"image-registry-7f9cfbc8db-w8cmn\" (UID: \"7970eec1-3765-48a4-b09c-3dec2d29956b\") " pod="openshift-image-registry/image-registry-7f9cfbc8db-w8cmn"
Nov 01 14:16:41.553327 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.552656    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c27ff402-235a-4d73-a139-cf928a6d5cbe-default-certificate\") pod \"router-default-b9d995dcc-lnk2n\" (UID: \"c27ff402-235a-4d73-a139-cf928a6d5cbe\") " pod="openshift-ingress/router-default-b9d995dcc-lnk2n"
Nov 01 14:16:41.553327 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.552691    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7970eec1-3765-48a4-b09c-3dec2d29956b-registry-tls\") pod \"image-registry-7f9cfbc8db-w8cmn\" (UID: \"7970eec1-3765-48a4-b09c-3dec2d29956b\") " pod="openshift-image-registry/image-registry-7f9cfbc8db-w8cmn"
Nov 01 14:16:41.553563 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.553533    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b5a4fefb-cc48-4638-b0f8-590e36bd0118-nginx-conf\") pod \"networking-console-plugin-86c478bb88-cwfck\" (UID: \"b5a4fefb-cc48-4638-b0f8-590e36bd0118\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-cwfck"
Nov 01 14:16:41.553682 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:41.553666    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 01 14:16:41.553760 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:41.553752    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5a4fefb-cc48-4638-b0f8-590e36bd0118-networking-console-plugin-cert podName:b5a4fefb-cc48-4638-b0f8-590e36bd0118 nodeName:}" failed. No retries permitted until 2026-11-01 14:16:42.053727838 +0000 UTC m=+32.213668492 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b5a4fefb-cc48-4638-b0f8-590e36bd0118-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-cwfck" (UID: "b5a4fefb-cc48-4638-b0f8-590e36bd0118") : secret "networking-console-plugin-cert" not found
Nov 01 14:16:41.554468 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.554414    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e7ea769b-e719-4090-bcd4-05b1b88fcc06-ca-trust-extracted\") pod \"image-registry-69c8c9797f-cblkb\" (UID: \"e7ea769b-e719-4090-bcd4-05b1b88fcc06\") " pod="openshift-image-registry/image-registry-69c8c9797f-cblkb"
Nov 01 14:16:41.555188 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:41.555139    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 14:16:41.555188 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:41.555162    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69c8c9797f-cblkb: secret "image-registry-tls" not found
Nov 01 14:16:41.555402 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:41.555243    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7ea769b-e719-4090-bcd4-05b1b88fcc06-registry-tls podName:e7ea769b-e719-4090-bcd4-05b1b88fcc06 nodeName:}" failed. No retries permitted until 2026-11-01 14:16:42.055224881 +0000 UTC m=+32.215165540 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e7ea769b-e719-4090-bcd4-05b1b88fcc06-registry-tls") pod "image-registry-69c8c9797f-cblkb" (UID: "e7ea769b-e719-4090-bcd4-05b1b88fcc06") : secret "image-registry-tls" not found
Nov 01 14:16:41.555793 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.555710    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e7ea769b-e719-4090-bcd4-05b1b88fcc06-registry-certificates\") pod \"image-registry-69c8c9797f-cblkb\" (UID: \"e7ea769b-e719-4090-bcd4-05b1b88fcc06\") " pod="openshift-image-registry/image-registry-69c8c9797f-cblkb"
Nov 01 14:16:41.555925 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.555898    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7a26b817-3283-4d59-8e8f-3bb4c0fa4987-tmp\") pod \"insights-operator-7c4986d449-7d54k\" (UID: \"7a26b817-3283-4d59-8e8f-3bb4c0fa4987\") " pod="openshift-insights/insights-operator-7c4986d449-7d54k"
Nov 01 14:16:41.556561 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.556289    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a26b817-3283-4d59-8e8f-3bb4c0fa4987-service-ca-bundle\") pod \"insights-operator-7c4986d449-7d54k\" (UID: \"7a26b817-3283-4d59-8e8f-3bb4c0fa4987\") " pod="openshift-insights/insights-operator-7c4986d449-7d54k"
Nov 01 14:16:41.556561 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.556420    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4a724964-bd57-4d56-a042-a0d490115eaa-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-96vgd\" (UID: \"4a724964-bd57-4d56-a042-a0d490115eaa\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-96vgd"
Nov 01 14:16:41.556561 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:41.556505    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 01 14:16:41.556766 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.556558    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3e7e18b9-78ee-4349-b00c-a0800e8f8341-config\") pod \"service-ca-operator-544d749c8c-87xcd\" (UID: \"3e7e18b9-78ee-4349-b00c-a0800e8f8341\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-87xcd"
Nov 01 14:16:41.556766 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:41.556574    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/037ca099-5ad7-4874-a984-462c07801e70-cluster-monitoring-operator-tls podName:037ca099-5ad7-4874-a984-462c07801e70 nodeName:}" failed. No retries permitted until 2026-11-01 14:16:42.05655779 +0000 UTC m=+32.216498444 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/037ca099-5ad7-4874-a984-462c07801e70-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-bgmv8" (UID: "037ca099-5ad7-4874-a984-462c07801e70") : secret "cluster-monitoring-operator-tls" not found
Nov 01 14:16:41.557054 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.557031    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/7a26b817-3283-4d59-8e8f-3bb4c0fa4987-snapshots\") pod \"insights-operator-7c4986d449-7d54k\" (UID: \"7a26b817-3283-4d59-8e8f-3bb4c0fa4987\") " pod="openshift-insights/insights-operator-7c4986d449-7d54k"
Nov 01 14:16:41.557192 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.557136    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/037ca099-5ad7-4874-a984-462c07801e70-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-bgmv8\" (UID: \"037ca099-5ad7-4874-a984-462c07801e70\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bgmv8"
Nov 01 14:16:41.557370 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.557354    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a26b817-3283-4d59-8e8f-3bb4c0fa4987-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-7d54k\" (UID: \"7a26b817-3283-4d59-8e8f-3bb4c0fa4987\") " pod="openshift-insights/insights-operator-7c4986d449-7d54k"
Nov 01 14:16:41.557440 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.557353    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e7ea769b-e719-4090-bcd4-05b1b88fcc06-trusted-ca\") pod \"image-registry-69c8c9797f-cblkb\" (UID: \"e7ea769b-e719-4090-bcd4-05b1b88fcc06\") " pod="openshift-image-registry/image-registry-69c8c9797f-cblkb"
Nov 01 14:16:41.559067 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.559041    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4a724964-bd57-4d56-a042-a0d490115eaa-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-96vgd\" (UID: \"4a724964-bd57-4d56-a042-a0d490115eaa\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-96vgd"
Nov 01 14:16:41.559291 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.559274    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3e7e18b9-78ee-4349-b00c-a0800e8f8341-serving-cert\") pod \"service-ca-operator-544d749c8c-87xcd\" (UID: \"3e7e18b9-78ee-4349-b00c-a0800e8f8341\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-87xcd"
Nov 01 14:16:41.559354 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.559311    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e7ea769b-e719-4090-bcd4-05b1b88fcc06-image-registry-private-configuration\") pod \"image-registry-69c8c9797f-cblkb\" (UID: \"e7ea769b-e719-4090-bcd4-05b1b88fcc06\") " pod="openshift-image-registry/image-registry-69c8c9797f-cblkb"
Nov 01 14:16:41.559354 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.559314    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e7ea769b-e719-4090-bcd4-05b1b88fcc06-installation-pull-secrets\") pod \"image-registry-69c8c9797f-cblkb\" (UID: \"e7ea769b-e719-4090-bcd4-05b1b88fcc06\") " pod="openshift-image-registry/image-registry-69c8c9797f-cblkb"
Nov 01 14:16:41.559491 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.559473    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7a26b817-3283-4d59-8e8f-3bb4c0fa4987-serving-cert\") pod \"insights-operator-7c4986d449-7d54k\" (UID: \"7a26b817-3283-4d59-8e8f-3bb4c0fa4987\") " pod="openshift-insights/insights-operator-7c4986d449-7d54k"
Nov 01 14:16:41.616248 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.616158    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e7ea769b-e719-4090-bcd4-05b1b88fcc06-bound-sa-token\") pod \"image-registry-69c8c9797f-cblkb\" (UID: \"e7ea769b-e719-4090-bcd4-05b1b88fcc06\") " pod="openshift-image-registry/image-registry-69c8c9797f-cblkb"
Nov 01 14:16:41.616248 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.616220    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6cqj\" (UniqueName: \"kubernetes.io/projected/d8a5ccab-4d5e-4aaf-ad08-766238eca05f-kube-api-access-q6cqj\") pod \"kas-connection-checker-685895c7c9-zfqjv\" (UID: \"d8a5ccab-4d5e-4aaf-ad08-766238eca05f\") " pod="kube-system/kas-connection-checker-685895c7c9-zfqjv"
Nov 01 14:16:41.616445 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.616320    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxdcx\" (UniqueName: \"kubernetes.io/projected/e78d2494-62c4-4070-a1eb-8ce40329ff57-kube-api-access-wxdcx\") pod \"volume-data-source-validator-58c9f5f46d-shpt4\" (UID: \"e78d2494-62c4-4070-a1eb-8ce40329ff57\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-shpt4"
Nov 01 14:16:41.616513 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.616441    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stw2n\" (UniqueName: \"kubernetes.io/projected/3e7e18b9-78ee-4349-b00c-a0800e8f8341-kube-api-access-stw2n\") pod \"service-ca-operator-544d749c8c-87xcd\" (UID: \"3e7e18b9-78ee-4349-b00c-a0800e8f8341\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-87xcd"
Nov 01 14:16:41.616574 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.616547    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8thtk\" (UniqueName: \"kubernetes.io/projected/037ca099-5ad7-4874-a984-462c07801e70-kube-api-access-8thtk\") pod \"cluster-monitoring-operator-884576c57-bgmv8\" (UID: \"037ca099-5ad7-4874-a984-462c07801e70\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bgmv8"
Nov 01 14:16:41.616922 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.616897    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jbv4\" (UniqueName: \"kubernetes.io/projected/2460be58-d51b-4876-bf9b-2a0e57901386-kube-api-access-7jbv4\") pod \"kas-connection-checker-685895c7c9-4lzxm\" (UID: \"2460be58-d51b-4876-bf9b-2a0e57901386\") " pod="kube-system/kas-connection-checker-685895c7c9-4lzxm"
Nov 01 14:16:41.617008 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.616989    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9j226\" (UniqueName: \"kubernetes.io/projected/6a8f947a-0f8f-41aa-b646-4459fa8126a3-kube-api-access-9j226\") pod \"kas-connection-checker-685895c7c9-xmfr4\" (UID: \"6a8f947a-0f8f-41aa-b646-4459fa8126a3\") " pod="kube-system/kas-connection-checker-685895c7c9-xmfr4"
Nov 01 14:16:41.617092 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.617075    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv8hp\" (UniqueName: \"kubernetes.io/projected/7a26b817-3283-4d59-8e8f-3bb4c0fa4987-kube-api-access-bv8hp\") pod \"insights-operator-7c4986d449-7d54k\" (UID: \"7a26b817-3283-4d59-8e8f-3bb4c0fa4987\") " pod="openshift-insights/insights-operator-7c4986d449-7d54k"
Nov 01 14:16:41.617571 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.617549    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfj7s\" (UniqueName: \"kubernetes.io/projected/4a724964-bd57-4d56-a042-a0d490115eaa-kube-api-access-kfj7s\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-96vgd\" (UID: \"4a724964-bd57-4d56-a042-a0d490115eaa\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-96vgd"
Nov 01 14:16:41.618354 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.618331    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdxs8\" (UniqueName: \"kubernetes.io/projected/e7ea769b-e719-4090-bcd4-05b1b88fcc06-kube-api-access-rdxs8\") pod \"image-registry-69c8c9797f-cblkb\" (UID: \"e7ea769b-e719-4090-bcd4-05b1b88fcc06\") " pod="openshift-image-registry/image-registry-69c8c9797f-cblkb"
Nov 01 14:16:41.648144 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.648109    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-4lzxm"
Nov 01 14:16:41.653568 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.653541    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bb59051d-c2df-4362-a157-00f14f915e79-config\") pod \"console-operator-7cd56b8dbc-xmwtp\" (UID: \"bb59051d-c2df-4362-a157-00f14f915e79\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-xmwtp"
Nov 01 14:16:41.653635 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.653576    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c27ff402-235a-4d73-a139-cf928a6d5cbe-service-ca-bundle\") pod \"router-default-b9d995dcc-lnk2n\" (UID: \"c27ff402-235a-4d73-a139-cf928a6d5cbe\") " pod="openshift-ingress/router-default-b9d995dcc-lnk2n"
Nov 01 14:16:41.653635 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.653601    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjv76\" (UniqueName: \"kubernetes.io/projected/c27ff402-235a-4d73-a139-cf928a6d5cbe-kube-api-access-rjv76\") pod \"router-default-b9d995dcc-lnk2n\" (UID: \"c27ff402-235a-4d73-a139-cf928a6d5cbe\") " pod="openshift-ingress/router-default-b9d995dcc-lnk2n"
Nov 01 14:16:41.653635 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.653620    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7970eec1-3765-48a4-b09c-3dec2d29956b-ca-trust-extracted\") pod \"image-registry-7f9cfbc8db-w8cmn\" (UID: \"7970eec1-3765-48a4-b09c-3dec2d29956b\") " pod="openshift-image-registry/image-registry-7f9cfbc8db-w8cmn"
Nov 01 14:16:41.653768 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.653643    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/d407313d-3333-4ec4-85bb-cd27cf4d7422-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-hsdjc\" (UID: \"d407313d-3333-4ec4-85bb-cd27cf4d7422\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hsdjc"
Nov 01 14:16:41.653768 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.653669    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7970eec1-3765-48a4-b09c-3dec2d29956b-image-registry-private-configuration\") pod \"image-registry-7f9cfbc8db-w8cmn\" (UID: \"7970eec1-3765-48a4-b09c-3dec2d29956b\") " pod="openshift-image-registry/image-registry-7f9cfbc8db-w8cmn"
Nov 01 14:16:41.653768 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.653693    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7970eec1-3765-48a4-b09c-3dec2d29956b-installation-pull-secrets\") pod \"image-registry-7f9cfbc8db-w8cmn\" (UID: \"7970eec1-3765-48a4-b09c-3dec2d29956b\") " pod="openshift-image-registry/image-registry-7f9cfbc8db-w8cmn"
Nov 01 14:16:41.653768 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:41.653708    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c27ff402-235a-4d73-a139-cf928a6d5cbe-service-ca-bundle podName:c27ff402-235a-4d73-a139-cf928a6d5cbe nodeName:}" failed. No retries permitted until 2026-11-01 14:16:42.153693185 +0000 UTC m=+32.313633846 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c27ff402-235a-4d73-a139-cf928a6d5cbe-service-ca-bundle") pod "router-default-b9d995dcc-lnk2n" (UID: "c27ff402-235a-4d73-a139-cf928a6d5cbe") : configmap references non-existent config key: service-ca.crt
Nov 01 14:16:41.653768 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.653741    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c27ff402-235a-4d73-a139-cf928a6d5cbe-default-certificate\") pod \"router-default-b9d995dcc-lnk2n\" (UID: \"c27ff402-235a-4d73-a139-cf928a6d5cbe\") " pod="openshift-ingress/router-default-b9d995dcc-lnk2n"
Nov 01 14:16:41.654014 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.653770    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7970eec1-3765-48a4-b09c-3dec2d29956b-registry-tls\") pod \"image-registry-7f9cfbc8db-w8cmn\" (UID: \"7970eec1-3765-48a4-b09c-3dec2d29956b\") " pod="openshift-image-registry/image-registry-7f9cfbc8db-w8cmn"
Nov 01 14:16:41.654014 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.653815    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7970eec1-3765-48a4-b09c-3dec2d29956b-trusted-ca\") pod \"image-registry-7f9cfbc8db-w8cmn\" (UID: \"7970eec1-3765-48a4-b09c-3dec2d29956b\") " pod="openshift-image-registry/image-registry-7f9cfbc8db-w8cmn"
Nov 01 14:16:41.654014 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.653838    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/70bb916a-832b-4bce-9c0a-f01bf4e3abc4-cert\") pod \"ingress-canary-8fgtw\" (UID: \"70bb916a-832b-4bce-9c0a-f01bf4e3abc4\") " pod="openshift-ingress-canary/ingress-canary-8fgtw"
Nov 01 14:16:41.654014 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.653886    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c27ff402-235a-4d73-a139-cf928a6d5cbe-metrics-certs\") pod \"router-default-b9d995dcc-lnk2n\" (UID: \"c27ff402-235a-4d73-a139-cf928a6d5cbe\") " pod="openshift-ingress/router-default-b9d995dcc-lnk2n"
Nov 01 14:16:41.654014 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.653920    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0c8cc47b-fb01-4195-a7fc-eeefc706017e-config-volume\") pod \"dns-default-b2lcr\" (UID: \"0c8cc47b-fb01-4195-a7fc-eeefc706017e\") " pod="openshift-dns/dns-default-b2lcr"
Nov 01 14:16:41.654014 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.653948    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c8cc47b-fb01-4195-a7fc-eeefc706017e-metrics-tls\") pod \"dns-default-b2lcr\" (UID: \"0c8cc47b-fb01-4195-a7fc-eeefc706017e\") " pod="openshift-dns/dns-default-b2lcr"
Nov 01 14:16:41.654014 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.653974    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bb59051d-c2df-4362-a157-00f14f915e79-trusted-ca\") pod \"console-operator-7cd56b8dbc-xmwtp\" (UID: \"bb59051d-c2df-4362-a157-00f14f915e79\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-xmwtp"
Nov 01 14:16:41.654014 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.654000    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7970eec1-3765-48a4-b09c-3dec2d29956b-registry-certificates\") pod \"image-registry-7f9cfbc8db-w8cmn\" (UID: \"7970eec1-3765-48a4-b09c-3dec2d29956b\") " pod="openshift-image-registry/image-registry-7f9cfbc8db-w8cmn"
Nov 01 14:16:41.654014 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.654002    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7970eec1-3765-48a4-b09c-3dec2d29956b-ca-trust-extracted\") pod \"image-registry-7f9cfbc8db-w8cmn\" (UID: \"7970eec1-3765-48a4-b09c-3dec2d29956b\") " pod="openshift-image-registry/image-registry-7f9cfbc8db-w8cmn"
Nov 01 14:16:41.654455 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.654031    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gnmx\" (UniqueName: \"kubernetes.io/projected/7970eec1-3765-48a4-b09c-3dec2d29956b-kube-api-access-8gnmx\") pod \"image-registry-7f9cfbc8db-w8cmn\" (UID: \"7970eec1-3765-48a4-b09c-3dec2d29956b\") " pod="openshift-image-registry/image-registry-7f9cfbc8db-w8cmn"
Nov 01 14:16:41.654455 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.654063    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d407313d-3333-4ec4-85bb-cd27cf4d7422-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-hsdjc\" (UID: \"d407313d-3333-4ec4-85bb-cd27cf4d7422\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hsdjc"
Nov 01 14:16:41.654455 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.654095    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkzgz\" (UniqueName: \"kubernetes.io/projected/70bb916a-832b-4bce-9c0a-f01bf4e3abc4-kube-api-access-xkzgz\") pod \"ingress-canary-8fgtw\" (UID: \"70bb916a-832b-4bce-9c0a-f01bf4e3abc4\") " pod="openshift-ingress-canary/ingress-canary-8fgtw"
Nov 01 14:16:41.654455 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.654125    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c27ff402-235a-4d73-a139-cf928a6d5cbe-stats-auth\") pod \"router-default-b9d995dcc-lnk2n\" (UID: \"c27ff402-235a-4d73-a139-cf928a6d5cbe\") " pod="openshift-ingress/router-default-b9d995dcc-lnk2n"
Nov 01 14:16:41.654455 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.654151    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7970eec1-3765-48a4-b09c-3dec2d29956b-bound-sa-token\") pod \"image-registry-7f9cfbc8db-w8cmn\" (UID: \"7970eec1-3765-48a4-b09c-3dec2d29956b\") " pod="openshift-image-registry/image-registry-7f9cfbc8db-w8cmn"
Nov 01 14:16:41.654455 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.654201    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsmvn\" (UniqueName: \"kubernetes.io/projected/d407313d-3333-4ec4-85bb-cd27cf4d7422-kube-api-access-fsmvn\") pod \"cluster-samples-operator-6ddb6c8b94-hsdjc\" (UID: \"d407313d-3333-4ec4-85bb-cd27cf4d7422\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hsdjc"
Nov 01 14:16:41.654455 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.654233    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjj74\" (UniqueName: \"kubernetes.io/projected/f219e3c2-6da8-4c88-9ec4-b26cf4d10326-kube-api-access-pjj74\") pod \"network-check-source-6f78d59f99-lhdpr\" (UID: \"f219e3c2-6da8-4c88-9ec4-b26cf4d10326\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-lhdpr"
Nov 01 14:16:41.654455 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.654257    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c8cc47b-fb01-4195-a7fc-eeefc706017e-tmp-dir\") pod \"dns-default-b2lcr\" (UID: \"0c8cc47b-fb01-4195-a7fc-eeefc706017e\") " pod="openshift-dns/dns-default-b2lcr"
Nov 01 14:16:41.654455 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.654302    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bb59051d-c2df-4362-a157-00f14f915e79-serving-cert\") pod \"console-operator-7cd56b8dbc-xmwtp\" (UID: \"bb59051d-c2df-4362-a157-00f14f915e79\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-xmwtp"
Nov 01 14:16:41.654455 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.654331    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92xdl\" (UniqueName: \"kubernetes.io/projected/bb59051d-c2df-4362-a157-00f14f915e79-kube-api-access-92xdl\") pod \"console-operator-7cd56b8dbc-xmwtp\" (UID: \"bb59051d-c2df-4362-a157-00f14f915e79\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-xmwtp"
Nov 01 14:16:41.654455 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.654333    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bb59051d-c2df-4362-a157-00f14f915e79-config\") pod \"console-operator-7cd56b8dbc-xmwtp\" (UID: \"bb59051d-c2df-4362-a157-00f14f915e79\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-xmwtp"
Nov 01 14:16:41.654455 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.654358    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgmsd\" (UniqueName: \"kubernetes.io/projected/0c8cc47b-fb01-4195-a7fc-eeefc706017e-kube-api-access-wgmsd\") pod \"dns-default-b2lcr\" (UID: \"0c8cc47b-fb01-4195-a7fc-eeefc706017e\") " pod="openshift-dns/dns-default-b2lcr"
Nov 01 14:16:41.654455 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.654386    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/d407313d-3333-4ec4-85bb-cd27cf4d7422-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-hsdjc\" (UID: \"d407313d-3333-4ec4-85bb-cd27cf4d7422\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hsdjc"
Nov 01 14:16:41.655078 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:41.654565    2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 01 14:16:41.655078 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:41.654618    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d407313d-3333-4ec4-85bb-cd27cf4d7422-samples-operator-tls podName:d407313d-3333-4ec4-85bb-cd27cf4d7422 nodeName:}" failed. No retries permitted until 2026-11-01 14:16:42.154599015 +0000 UTC m=+32.314539668 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d407313d-3333-4ec4-85bb-cd27cf4d7422-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-hsdjc" (UID: "d407313d-3333-4ec4-85bb-cd27cf4d7422") : secret "samples-operator-tls" not found
Nov 01 14:16:41.655078 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.655049    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c8cc47b-fb01-4195-a7fc-eeefc706017e-tmp-dir\") pod \"dns-default-b2lcr\" (UID: \"0c8cc47b-fb01-4195-a7fc-eeefc706017e\") " pod="openshift-dns/dns-default-b2lcr"
Nov 01 14:16:41.655261 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.655078    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7970eec1-3765-48a4-b09c-3dec2d29956b-registry-certificates\") pod \"image-registry-7f9cfbc8db-w8cmn\" (UID: \"7970eec1-3765-48a4-b09c-3dec2d29956b\") " pod="openshift-image-registry/image-registry-7f9cfbc8db-w8cmn"
Nov 01 14:16:41.655261 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.655158    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bb59051d-c2df-4362-a157-00f14f915e79-trusted-ca\") pod \"console-operator-7cd56b8dbc-xmwtp\" (UID: \"bb59051d-c2df-4362-a157-00f14f915e79\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-xmwtp"
Nov 01 14:16:41.655362 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:41.655333    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 01 14:16:41.655409 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:41.655380    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 14:16:41.655533 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:41.655385    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c27ff402-235a-4d73-a139-cf928a6d5cbe-metrics-certs podName:c27ff402-235a-4d73-a139-cf928a6d5cbe nodeName:}" failed. No retries permitted until 2026-11-01 14:16:42.155368685 +0000 UTC m=+32.315309346 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c27ff402-235a-4d73-a139-cf928a6d5cbe-metrics-certs") pod "router-default-b9d995dcc-lnk2n" (UID: "c27ff402-235a-4d73-a139-cf928a6d5cbe") : secret "router-metrics-certs-default" not found
Nov 01 14:16:41.655533 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:41.655471    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 14:16:41.655533 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:41.655486    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f9cfbc8db-w8cmn: secret "image-registry-tls" not found
Nov 01 14:16:41.655533 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:41.655489    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c8cc47b-fb01-4195-a7fc-eeefc706017e-metrics-tls podName:0c8cc47b-fb01-4195-a7fc-eeefc706017e nodeName:}" failed. No retries permitted until 2026-11-01 14:16:42.155471677 +0000 UTC m=+32.315412323 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0c8cc47b-fb01-4195-a7fc-eeefc706017e-metrics-tls") pod "dns-default-b2lcr" (UID: "0c8cc47b-fb01-4195-a7fc-eeefc706017e") : secret "dns-default-metrics-tls" not found
Nov 01 14:16:41.655533 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:41.655525    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7970eec1-3765-48a4-b09c-3dec2d29956b-registry-tls podName:7970eec1-3765-48a4-b09c-3dec2d29956b nodeName:}" failed. No retries permitted until 2026-11-01 14:16:42.155512166 +0000 UTC m=+32.315452819 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7970eec1-3765-48a4-b09c-3dec2d29956b-registry-tls") pod "image-registry-7f9cfbc8db-w8cmn" (UID: "7970eec1-3765-48a4-b09c-3dec2d29956b") : secret "image-registry-tls" not found
Nov 01 14:16:41.656552 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.655730    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0c8cc47b-fb01-4195-a7fc-eeefc706017e-config-volume\") pod \"dns-default-b2lcr\" (UID: \"0c8cc47b-fb01-4195-a7fc-eeefc706017e\") " pod="openshift-dns/dns-default-b2lcr"
Nov 01 14:16:41.656552 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.656296    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7970eec1-3765-48a4-b09c-3dec2d29956b-installation-pull-secrets\") pod \"image-registry-7f9cfbc8db-w8cmn\" (UID: \"7970eec1-3765-48a4-b09c-3dec2d29956b\") " pod="openshift-image-registry/image-registry-7f9cfbc8db-w8cmn"
Nov 01 14:16:41.656552 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.656300    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7970eec1-3765-48a4-b09c-3dec2d29956b-trusted-ca\") pod \"image-registry-7f9cfbc8db-w8cmn\" (UID: \"7970eec1-3765-48a4-b09c-3dec2d29956b\") " pod="openshift-image-registry/image-registry-7f9cfbc8db-w8cmn"
Nov 01 14:16:41.656552 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.656406    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7970eec1-3765-48a4-b09c-3dec2d29956b-image-registry-private-configuration\") pod \"image-registry-7f9cfbc8db-w8cmn\" (UID: \"7970eec1-3765-48a4-b09c-3dec2d29956b\") " pod="openshift-image-registry/image-registry-7f9cfbc8db-w8cmn"
Nov 01 14:16:41.657355 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.657325    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-xmfr4"
Nov 01 14:16:41.657914 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.657891    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/70bb916a-832b-4bce-9c0a-f01bf4e3abc4-cert\") pod \"ingress-canary-8fgtw\" (UID: \"70bb916a-832b-4bce-9c0a-f01bf4e3abc4\") " pod="openshift-ingress-canary/ingress-canary-8fgtw"
Nov 01 14:16:41.658310 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.658295    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c27ff402-235a-4d73-a139-cf928a6d5cbe-stats-auth\") pod \"router-default-b9d995dcc-lnk2n\" (UID: \"c27ff402-235a-4d73-a139-cf928a6d5cbe\") " pod="openshift-ingress/router-default-b9d995dcc-lnk2n"
Nov 01 14:16:41.659382 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.659364    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bb59051d-c2df-4362-a157-00f14f915e79-serving-cert\") pod \"console-operator-7cd56b8dbc-xmwtp\" (UID: \"bb59051d-c2df-4362-a157-00f14f915e79\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-xmwtp"
Nov 01 14:16:41.659462 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.659445    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c27ff402-235a-4d73-a139-cf928a6d5cbe-default-certificate\") pod \"router-default-b9d995dcc-lnk2n\" (UID: \"c27ff402-235a-4d73-a139-cf928a6d5cbe\") " pod="openshift-ingress/router-default-b9d995dcc-lnk2n"
Nov 01 14:16:41.663011 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.662991    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-96vgd"
Nov 01 14:16:41.669284 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.669261    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-shpt4"
Nov 01 14:16:41.689674 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.689598    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-87xcd"
Nov 01 14:16:41.700204 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.698464    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-7d54k"
Nov 01 14:16:41.718457 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.718047    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92xdl\" (UniqueName: \"kubernetes.io/projected/bb59051d-c2df-4362-a157-00f14f915e79-kube-api-access-92xdl\") pod \"console-operator-7cd56b8dbc-xmwtp\" (UID: \"bb59051d-c2df-4362-a157-00f14f915e79\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-xmwtp"
Nov 01 14:16:41.722191 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.722091    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjj74\" (UniqueName: \"kubernetes.io/projected/f219e3c2-6da8-4c88-9ec4-b26cf4d10326-kube-api-access-pjj74\") pod \"network-check-source-6f78d59f99-lhdpr\" (UID: \"f219e3c2-6da8-4c88-9ec4-b26cf4d10326\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-lhdpr"
Nov 01 14:16:41.724939 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.724684    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7970eec1-3765-48a4-b09c-3dec2d29956b-bound-sa-token\") pod \"image-registry-7f9cfbc8db-w8cmn\" (UID: \"7970eec1-3765-48a4-b09c-3dec2d29956b\") " pod="openshift-image-registry/image-registry-7f9cfbc8db-w8cmn"
Nov 01 14:16:41.725492 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.725466    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-xmwtp"
Nov 01 14:16:41.727193 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.727133    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsmvn\" (UniqueName: \"kubernetes.io/projected/d407313d-3333-4ec4-85bb-cd27cf4d7422-kube-api-access-fsmvn\") pod \"cluster-samples-operator-6ddb6c8b94-hsdjc\" (UID: \"d407313d-3333-4ec4-85bb-cd27cf4d7422\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hsdjc"
Nov 01 14:16:41.727193 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.727162    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgmsd\" (UniqueName: \"kubernetes.io/projected/0c8cc47b-fb01-4195-a7fc-eeefc706017e-kube-api-access-wgmsd\") pod \"dns-default-b2lcr\" (UID: \"0c8cc47b-fb01-4195-a7fc-eeefc706017e\") " pod="openshift-dns/dns-default-b2lcr"
Nov 01 14:16:41.727684 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.727643    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gnmx\" (UniqueName: \"kubernetes.io/projected/7970eec1-3765-48a4-b09c-3dec2d29956b-kube-api-access-8gnmx\") pod \"image-registry-7f9cfbc8db-w8cmn\" (UID: \"7970eec1-3765-48a4-b09c-3dec2d29956b\") " pod="openshift-image-registry/image-registry-7f9cfbc8db-w8cmn"
Nov 01 14:16:41.728613 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.728589    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkzgz\" (UniqueName: \"kubernetes.io/projected/70bb916a-832b-4bce-9c0a-f01bf4e3abc4-kube-api-access-xkzgz\") pod \"ingress-canary-8fgtw\" (UID: \"70bb916a-832b-4bce-9c0a-f01bf4e3abc4\") " pod="openshift-ingress-canary/ingress-canary-8fgtw"
Nov 01 14:16:41.729418 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.729376    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjv76\" (UniqueName: \"kubernetes.io/projected/c27ff402-235a-4d73-a139-cf928a6d5cbe-kube-api-access-rjv76\") pod \"router-default-b9d995dcc-lnk2n\" (UID: \"c27ff402-235a-4d73-a139-cf928a6d5cbe\") " pod="openshift-ingress/router-default-b9d995dcc-lnk2n"
Nov 01 14:16:41.757760 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.756368    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8fgtw"
Nov 01 14:16:41.757760 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.756682    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-lhdpr"
Nov 01 14:16:41.911536 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:41.911512    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-zfqjv"
Nov 01 14:16:42.042455 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:42.042423    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-xmfr4"]
Nov 01 14:16:42.045663 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:42.045639    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a8f947a_0f8f_41aa_b646_4459fa8126a3.slice/crio-6f895928b8cb30ce26ce5335d0bc02d8bfc8edfb5035df303cbdf4e67fe480ac WatchSource:0}: Error finding container 6f895928b8cb30ce26ce5335d0bc02d8bfc8edfb5035df303cbdf4e67fe480ac: Status 404 returned error can't find the container with id 6f895928b8cb30ce26ce5335d0bc02d8bfc8edfb5035df303cbdf4e67fe480ac
Nov 01 14:16:42.048973 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:42.048950    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-4lzxm"]
Nov 01 14:16:42.053684 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:42.053651    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2460be58_d51b_4876_bf9b_2a0e57901386.slice/crio-880c12f6309a8d768f868b8369ee6a928463263c6f30c39a8cb01909d7aa9902 WatchSource:0}: Error finding container 880c12f6309a8d768f868b8369ee6a928463263c6f30c39a8cb01909d7aa9902: Status 404 returned error can't find the container with id 880c12f6309a8d768f868b8369ee6a928463263c6f30c39a8cb01909d7aa9902
Nov 01 14:16:42.061905 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:42.061873    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b5a4fefb-cc48-4638-b0f8-590e36bd0118-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-cwfck\" (UID: \"b5a4fefb-cc48-4638-b0f8-590e36bd0118\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-cwfck"
Nov 01 14:16:42.061982 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:42.061942    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e7ea769b-e719-4090-bcd4-05b1b88fcc06-registry-tls\") pod \"image-registry-69c8c9797f-cblkb\" (UID: \"e7ea769b-e719-4090-bcd4-05b1b88fcc06\") " pod="openshift-image-registry/image-registry-69c8c9797f-cblkb"
Nov 01 14:16:42.062055 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:42.062038    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/037ca099-5ad7-4874-a984-462c07801e70-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-bgmv8\" (UID: \"037ca099-5ad7-4874-a984-462c07801e70\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bgmv8"
Nov 01 14:16:42.062229 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:42.062211    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 01 14:16:42.062294 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:42.062260    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/037ca099-5ad7-4874-a984-462c07801e70-cluster-monitoring-operator-tls podName:037ca099-5ad7-4874-a984-462c07801e70 nodeName:}" failed. No retries permitted until 2026-11-01 14:16:43.062247348 +0000 UTC m=+33.222187987 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/037ca099-5ad7-4874-a984-462c07801e70-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-bgmv8" (UID: "037ca099-5ad7-4874-a984-462c07801e70") : secret "cluster-monitoring-operator-tls" not found
Nov 01 14:16:42.062336 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:42.062300    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 01 14:16:42.062336 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:42.062324    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5a4fefb-cc48-4638-b0f8-590e36bd0118-networking-console-plugin-cert podName:b5a4fefb-cc48-4638-b0f8-590e36bd0118 nodeName:}" failed. No retries permitted until 2026-11-01 14:16:43.062317754 +0000 UTC m=+33.222258394 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b5a4fefb-cc48-4638-b0f8-590e36bd0118-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-cwfck" (UID: "b5a4fefb-cc48-4638-b0f8-590e36bd0118") : secret "networking-console-plugin-cert" not found
Nov 01 14:16:42.062396 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:42.062361    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 14:16:42.062396 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:42.062369    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69c8c9797f-cblkb: secret "image-registry-tls" not found
Nov 01 14:16:42.062396 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:42.062387    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7ea769b-e719-4090-bcd4-05b1b88fcc06-registry-tls podName:e7ea769b-e719-4090-bcd4-05b1b88fcc06 nodeName:}" failed. No retries permitted until 2026-11-01 14:16:43.062381865 +0000 UTC m=+33.222322504 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e7ea769b-e719-4090-bcd4-05b1b88fcc06-registry-tls") pod "image-registry-69c8c9797f-cblkb" (UID: "e7ea769b-e719-4090-bcd4-05b1b88fcc06") : secret "image-registry-tls" not found
Nov 01 14:16:42.064710 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:42.064690    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-shpt4"]
Nov 01 14:16:42.067852 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:42.067814    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-96vgd"]
Nov 01 14:16:42.069025 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:42.068975    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode78d2494_62c4_4070_a1eb_8ce40329ff57.slice/crio-2c371efbc265a2789bc9913d731f39b76b38dfbd89aa2beb1cf181b6b2a82a88 WatchSource:0}: Error finding container 2c371efbc265a2789bc9913d731f39b76b38dfbd89aa2beb1cf181b6b2a82a88: Status 404 returned error can't find the container with id 2c371efbc265a2789bc9913d731f39b76b38dfbd89aa2beb1cf181b6b2a82a88
Nov 01 14:16:42.120492 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:42.120464    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-87xcd"]
Nov 01 14:16:42.124224 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:42.124195    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e7e18b9_78ee_4349_b00c_a0800e8f8341.slice/crio-9226216fff100a82b2a6715ce444110115c3d396b98acc865c89fc819868e4b7 WatchSource:0}: Error finding container 9226216fff100a82b2a6715ce444110115c3d396b98acc865c89fc819868e4b7: Status 404 returned error can't find the container with id 9226216fff100a82b2a6715ce444110115c3d396b98acc865c89fc819868e4b7
Nov 01 14:16:42.129586 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:42.129566    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-7d54k"]
Nov 01 14:16:42.134907 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:42.134881    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a26b817_3283_4d59_8e8f_3bb4c0fa4987.slice/crio-7b2912f5c7df5c0bd7c90ec4fae06b2a295b3fc96e054a6682e822f05bbc3be2 WatchSource:0}: Error finding container 7b2912f5c7df5c0bd7c90ec4fae06b2a295b3fc96e054a6682e822f05bbc3be2: Status 404 returned error can't find the container with id 7b2912f5c7df5c0bd7c90ec4fae06b2a295b3fc96e054a6682e822f05bbc3be2
Nov 01 14:16:42.162525 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:42.162494    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c27ff402-235a-4d73-a139-cf928a6d5cbe-service-ca-bundle\") pod \"router-default-b9d995dcc-lnk2n\" (UID: \"c27ff402-235a-4d73-a139-cf928a6d5cbe\") " pod="openshift-ingress/router-default-b9d995dcc-lnk2n"
Nov 01 14:16:42.162632 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:42.162558    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7970eec1-3765-48a4-b09c-3dec2d29956b-registry-tls\") pod \"image-registry-7f9cfbc8db-w8cmn\" (UID: \"7970eec1-3765-48a4-b09c-3dec2d29956b\") " pod="openshift-image-registry/image-registry-7f9cfbc8db-w8cmn"
Nov 01 14:16:42.162689 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:42.162674    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 14:16:42.162744 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:42.162682    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c27ff402-235a-4d73-a139-cf928a6d5cbe-metrics-certs\") pod \"router-default-b9d995dcc-lnk2n\" (UID: \"c27ff402-235a-4d73-a139-cf928a6d5cbe\") " pod="openshift-ingress/router-default-b9d995dcc-lnk2n"
Nov 01 14:16:42.162744 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:42.162700    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f9cfbc8db-w8cmn: secret "image-registry-tls" not found
Nov 01 14:16:42.162744 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:42.162719    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c8cc47b-fb01-4195-a7fc-eeefc706017e-metrics-tls\") pod \"dns-default-b2lcr\" (UID: \"0c8cc47b-fb01-4195-a7fc-eeefc706017e\") " pod="openshift-dns/dns-default-b2lcr"
Nov 01 14:16:42.162879 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:42.162753    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7970eec1-3765-48a4-b09c-3dec2d29956b-registry-tls podName:7970eec1-3765-48a4-b09c-3dec2d29956b nodeName:}" failed. No retries permitted until 2026-11-01 14:16:43.162737074 +0000 UTC m=+33.322677719 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7970eec1-3765-48a4-b09c-3dec2d29956b-registry-tls") pod "image-registry-7f9cfbc8db-w8cmn" (UID: "7970eec1-3765-48a4-b09c-3dec2d29956b") : secret "image-registry-tls" not found
Nov 01 14:16:42.162879 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:42.162777    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d407313d-3333-4ec4-85bb-cd27cf4d7422-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-hsdjc\" (UID: \"d407313d-3333-4ec4-85bb-cd27cf4d7422\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hsdjc"
Nov 01 14:16:42.162879 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:42.162797    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 01 14:16:42.162879 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:42.162828    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c27ff402-235a-4d73-a139-cf928a6d5cbe-service-ca-bundle podName:c27ff402-235a-4d73-a139-cf928a6d5cbe nodeName:}" failed. No retries permitted until 2026-11-01 14:16:43.162809204 +0000 UTC m=+33.322749866 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c27ff402-235a-4d73-a139-cf928a6d5cbe-service-ca-bundle") pod "router-default-b9d995dcc-lnk2n" (UID: "c27ff402-235a-4d73-a139-cf928a6d5cbe") : configmap references non-existent config key: service-ca.crt
Nov 01 14:16:42.162879 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:42.162861    2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 01 14:16:42.162879 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:42.162867    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 14:16:42.163096 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:42.162892    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c27ff402-235a-4d73-a139-cf928a6d5cbe-metrics-certs podName:c27ff402-235a-4d73-a139-cf928a6d5cbe nodeName:}" failed. No retries permitted until 2026-11-01 14:16:43.16288237 +0000 UTC m=+33.322823012 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c27ff402-235a-4d73-a139-cf928a6d5cbe-metrics-certs") pod "router-default-b9d995dcc-lnk2n" (UID: "c27ff402-235a-4d73-a139-cf928a6d5cbe") : secret "router-metrics-certs-default" not found
Nov 01 14:16:42.163096 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:42.162909    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d407313d-3333-4ec4-85bb-cd27cf4d7422-samples-operator-tls podName:d407313d-3333-4ec4-85bb-cd27cf4d7422 nodeName:}" failed. No retries permitted until 2026-11-01 14:16:43.162900757 +0000 UTC m=+33.322841397 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d407313d-3333-4ec4-85bb-cd27cf4d7422-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-hsdjc" (UID: "d407313d-3333-4ec4-85bb-cd27cf4d7422") : secret "samples-operator-tls" not found
Nov 01 14:16:42.163096 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:42.162929    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c8cc47b-fb01-4195-a7fc-eeefc706017e-metrics-tls podName:0c8cc47b-fb01-4195-a7fc-eeefc706017e nodeName:}" failed. No retries permitted until 2026-11-01 14:16:43.162921076 +0000 UTC m=+33.322861724 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0c8cc47b-fb01-4195-a7fc-eeefc706017e-metrics-tls") pod "dns-default-b2lcr" (UID: "0c8cc47b-fb01-4195-a7fc-eeefc706017e") : secret "dns-default-metrics-tls" not found
Nov 01 14:16:42.163940 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:42.163919    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-xmwtp"]
Nov 01 14:16:42.166573 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:42.166543    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb59051d_c2df_4362_a157_00f14f915e79.slice/crio-e08e52ee478f9a7272ba40ba4b90357351ebf80ac971cebdba7f80f520e407e2 WatchSource:0}: Error finding container e08e52ee478f9a7272ba40ba4b90357351ebf80ac971cebdba7f80f520e407e2: Status 404 returned error can't find the container with id e08e52ee478f9a7272ba40ba4b90357351ebf80ac971cebdba7f80f520e407e2
Nov 01 14:16:42.187728 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:42.187667    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8fgtw"]
Nov 01 14:16:42.190922 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:42.190901    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70bb916a_832b_4bce_9c0a_f01bf4e3abc4.slice/crio-cafdbe7c69ef251c3ee7143456fc8e488e67d6ac1609f2906deea2ffb53664cd WatchSource:0}: Error finding container cafdbe7c69ef251c3ee7143456fc8e488e67d6ac1609f2906deea2ffb53664cd: Status 404 returned error can't find the container with id cafdbe7c69ef251c3ee7143456fc8e488e67d6ac1609f2906deea2ffb53664cd
Nov 01 14:16:42.303371 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:42.303340    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-zfqjv"]
Nov 01 14:16:42.306136 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:42.306117    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-lhdpr"]
Nov 01 14:16:42.306615 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:42.306594    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8a5ccab_4d5e_4aaf_ad08_766238eca05f.slice/crio-ef98d8e5539559612e84235ecac8dad90436bb395a86ef237ac3bf99da120f9a WatchSource:0}: Error finding container ef98d8e5539559612e84235ecac8dad90436bb395a86ef237ac3bf99da120f9a: Status 404 returned error can't find the container with id ef98d8e5539559612e84235ecac8dad90436bb395a86ef237ac3bf99da120f9a
Nov 01 14:16:42.309181 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:42.309140    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf219e3c2_6da8_4c88_9ec4_b26cf4d10326.slice/crio-690b74893e54482894df2b6f978f53c0f1b1fa7c44c3184bfd47a6a6b3758137 WatchSource:0}: Error finding container 690b74893e54482894df2b6f978f53c0f1b1fa7c44c3184bfd47a6a6b3758137: Status 404 returned error can't find the container with id 690b74893e54482894df2b6f978f53c0f1b1fa7c44c3184bfd47a6a6b3758137
Nov 01 14:16:42.491935 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:42.491903    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-lhdpr" event={"ID":"f219e3c2-6da8-4c88-9ec4-b26cf4d10326","Type":"ContainerStarted","Data":"690b74893e54482894df2b6f978f53c0f1b1fa7c44c3184bfd47a6a6b3758137"}
Nov 01 14:16:42.492951 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:42.492926    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8fgtw" event={"ID":"70bb916a-832b-4bce-9c0a-f01bf4e3abc4","Type":"ContainerStarted","Data":"cafdbe7c69ef251c3ee7143456fc8e488e67d6ac1609f2906deea2ffb53664cd"}
Nov 01 14:16:42.494343 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:42.494322    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-xmfr4" event={"ID":"6a8f947a-0f8f-41aa-b646-4459fa8126a3","Type":"ContainerStarted","Data":"bb5eb0ad76d756c66c90c36fdd072827c8a46d4925a228d9e80166e646245eb8"}
Nov 01 14:16:42.494420 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:42.494359    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-xmfr4" event={"ID":"6a8f947a-0f8f-41aa-b646-4459fa8126a3","Type":"ContainerStarted","Data":"6f895928b8cb30ce26ce5335d0bc02d8bfc8edfb5035df303cbdf4e67fe480ac"}
Nov 01 14:16:42.495815 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:42.495788    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-4lzxm" event={"ID":"2460be58-d51b-4876-bf9b-2a0e57901386","Type":"ContainerStarted","Data":"60013d8dd39ea3ab7bdfd045dbce83968a4264359392fd4a15f0b98ad8c065bc"}
Nov 01 14:16:42.495897 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:42.495821    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-4lzxm" event={"ID":"2460be58-d51b-4876-bf9b-2a0e57901386","Type":"ContainerStarted","Data":"880c12f6309a8d768f868b8369ee6a928463263c6f30c39a8cb01909d7aa9902"}
Nov 01 14:16:42.497195 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:42.497146    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-zfqjv" event={"ID":"d8a5ccab-4d5e-4aaf-ad08-766238eca05f","Type":"ContainerStarted","Data":"70beebf6a4619db03c40cc63421172d951686b22e235e08077fc9bbc8f58fbc8"}
Nov 01 14:16:42.497280 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:42.497211    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-zfqjv" event={"ID":"d8a5ccab-4d5e-4aaf-ad08-766238eca05f","Type":"ContainerStarted","Data":"ef98d8e5539559612e84235ecac8dad90436bb395a86ef237ac3bf99da120f9a"}
Nov 01 14:16:42.498260 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:42.498242    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-xmwtp" event={"ID":"bb59051d-c2df-4362-a157-00f14f915e79","Type":"ContainerStarted","Data":"e08e52ee478f9a7272ba40ba4b90357351ebf80ac971cebdba7f80f520e407e2"}
Nov 01 14:16:42.499315 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:42.499292    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-7d54k" event={"ID":"7a26b817-3283-4d59-8e8f-3bb4c0fa4987","Type":"ContainerStarted","Data":"7b2912f5c7df5c0bd7c90ec4fae06b2a295b3fc96e054a6682e822f05bbc3be2"}
Nov 01 14:16:42.500351 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:42.500334    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-shpt4" event={"ID":"e78d2494-62c4-4070-a1eb-8ce40329ff57","Type":"ContainerStarted","Data":"2c371efbc265a2789bc9913d731f39b76b38dfbd89aa2beb1cf181b6b2a82a88"}
Nov 01 14:16:42.501318 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:42.501300    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-96vgd" event={"ID":"4a724964-bd57-4d56-a042-a0d490115eaa","Type":"ContainerStarted","Data":"bac08a467e759b5f9eb601b19c0cafeffee46e2ef10d11078b6c9d8231c3d408"}
Nov 01 14:16:42.502242 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:42.502224    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-87xcd" event={"ID":"3e7e18b9-78ee-4349-b00c-a0800e8f8341","Type":"ContainerStarted","Data":"9226216fff100a82b2a6715ce444110115c3d396b98acc865c89fc819868e4b7"}
Nov 01 14:16:43.072743 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:43.072705    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b5a4fefb-cc48-4638-b0f8-590e36bd0118-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-cwfck\" (UID: \"b5a4fefb-cc48-4638-b0f8-590e36bd0118\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-cwfck"
Nov 01 14:16:43.072915 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:43.072753    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e7ea769b-e719-4090-bcd4-05b1b88fcc06-registry-tls\") pod \"image-registry-69c8c9797f-cblkb\" (UID: \"e7ea769b-e719-4090-bcd4-05b1b88fcc06\") " pod="openshift-image-registry/image-registry-69c8c9797f-cblkb"
Nov 01 14:16:43.072915 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:43.072826    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/037ca099-5ad7-4874-a984-462c07801e70-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-bgmv8\" (UID: \"037ca099-5ad7-4874-a984-462c07801e70\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bgmv8"
Nov 01 14:16:43.072915 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:43.072850    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 01 14:16:43.072915 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:43.072906    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5a4fefb-cc48-4638-b0f8-590e36bd0118-networking-console-plugin-cert podName:b5a4fefb-cc48-4638-b0f8-590e36bd0118 nodeName:}" failed. No retries permitted until 2026-11-01 14:16:45.072890136 +0000 UTC m=+35.232830809 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b5a4fefb-cc48-4638-b0f8-590e36bd0118-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-cwfck" (UID: "b5a4fefb-cc48-4638-b0f8-590e36bd0118") : secret "networking-console-plugin-cert" not found
Nov 01 14:16:43.073065 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:43.072928    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 14:16:43.073065 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:43.072949    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69c8c9797f-cblkb: secret "image-registry-tls" not found
Nov 01 14:16:43.073065 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:43.072987    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 01 14:16:43.073065 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:43.072999    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7ea769b-e719-4090-bcd4-05b1b88fcc06-registry-tls podName:e7ea769b-e719-4090-bcd4-05b1b88fcc06 nodeName:}" failed. No retries permitted until 2026-11-01 14:16:45.072982249 +0000 UTC m=+35.232922901 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e7ea769b-e719-4090-bcd4-05b1b88fcc06-registry-tls") pod "image-registry-69c8c9797f-cblkb" (UID: "e7ea769b-e719-4090-bcd4-05b1b88fcc06") : secret "image-registry-tls" not found
Nov 01 14:16:43.073065 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:43.073025    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/037ca099-5ad7-4874-a984-462c07801e70-cluster-monitoring-operator-tls podName:037ca099-5ad7-4874-a984-462c07801e70 nodeName:}" failed. No retries permitted until 2026-11-01 14:16:45.073014879 +0000 UTC m=+35.232955556 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/037ca099-5ad7-4874-a984-462c07801e70-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-bgmv8" (UID: "037ca099-5ad7-4874-a984-462c07801e70") : secret "cluster-monitoring-operator-tls" not found
Nov 01 14:16:43.174120 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:43.174088    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c27ff402-235a-4d73-a139-cf928a6d5cbe-metrics-certs\") pod \"router-default-b9d995dcc-lnk2n\" (UID: \"c27ff402-235a-4d73-a139-cf928a6d5cbe\") " pod="openshift-ingress/router-default-b9d995dcc-lnk2n"
Nov 01 14:16:43.174283 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:43.174130    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c8cc47b-fb01-4195-a7fc-eeefc706017e-metrics-tls\") pod \"dns-default-b2lcr\" (UID: \"0c8cc47b-fb01-4195-a7fc-eeefc706017e\") " pod="openshift-dns/dns-default-b2lcr"
Nov 01 14:16:43.174283 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:43.174183    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d407313d-3333-4ec4-85bb-cd27cf4d7422-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-hsdjc\" (UID: \"d407313d-3333-4ec4-85bb-cd27cf4d7422\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hsdjc"
Nov 01 14:16:43.174283 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:43.174245    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 01 14:16:43.174400 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:43.174282    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c27ff402-235a-4d73-a139-cf928a6d5cbe-service-ca-bundle\") pod \"router-default-b9d995dcc-lnk2n\" (UID: \"c27ff402-235a-4d73-a139-cf928a6d5cbe\") " pod="openshift-ingress/router-default-b9d995dcc-lnk2n"
Nov 01 14:16:43.174400 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:43.174286    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 14:16:43.174400 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:43.174318    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c27ff402-235a-4d73-a139-cf928a6d5cbe-metrics-certs podName:c27ff402-235a-4d73-a139-cf928a6d5cbe nodeName:}" failed. No retries permitted until 2026-11-01 14:16:45.17429888 +0000 UTC m=+35.334239542 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c27ff402-235a-4d73-a139-cf928a6d5cbe-metrics-certs") pod "router-default-b9d995dcc-lnk2n" (UID: "c27ff402-235a-4d73-a139-cf928a6d5cbe") : secret "router-metrics-certs-default" not found
Nov 01 14:16:43.174400 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:43.174325    2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 01 14:16:43.174400 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:43.174343    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c8cc47b-fb01-4195-a7fc-eeefc706017e-metrics-tls podName:0c8cc47b-fb01-4195-a7fc-eeefc706017e nodeName:}" failed. No retries permitted until 2026-11-01 14:16:45.174332952 +0000 UTC m=+35.334273593 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0c8cc47b-fb01-4195-a7fc-eeefc706017e-metrics-tls") pod "dns-default-b2lcr" (UID: "0c8cc47b-fb01-4195-a7fc-eeefc706017e") : secret "dns-default-metrics-tls" not found
Nov 01 14:16:43.174400 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:43.174363    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d407313d-3333-4ec4-85bb-cd27cf4d7422-samples-operator-tls podName:d407313d-3333-4ec4-85bb-cd27cf4d7422 nodeName:}" failed. No retries permitted until 2026-11-01 14:16:45.174354267 +0000 UTC m=+35.334294918 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d407313d-3333-4ec4-85bb-cd27cf4d7422-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-hsdjc" (UID: "d407313d-3333-4ec4-85bb-cd27cf4d7422") : secret "samples-operator-tls" not found
Nov 01 14:16:43.174400 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:43.174386    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c27ff402-235a-4d73-a139-cf928a6d5cbe-service-ca-bundle podName:c27ff402-235a-4d73-a139-cf928a6d5cbe nodeName:}" failed. No retries permitted until 2026-11-01 14:16:45.174375338 +0000 UTC m=+35.334315977 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c27ff402-235a-4d73-a139-cf928a6d5cbe-service-ca-bundle") pod "router-default-b9d995dcc-lnk2n" (UID: "c27ff402-235a-4d73-a139-cf928a6d5cbe") : configmap references non-existent config key: service-ca.crt
Nov 01 14:16:43.174656 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:43.174421    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7970eec1-3765-48a4-b09c-3dec2d29956b-registry-tls\") pod \"image-registry-7f9cfbc8db-w8cmn\" (UID: \"7970eec1-3765-48a4-b09c-3dec2d29956b\") " pod="openshift-image-registry/image-registry-7f9cfbc8db-w8cmn"
Nov 01 14:16:43.174656 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:43.174510    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 14:16:43.174656 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:43.174527    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f9cfbc8db-w8cmn: secret "image-registry-tls" not found
Nov 01 14:16:43.174656 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:43.174568    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7970eec1-3765-48a4-b09c-3dec2d29956b-registry-tls podName:7970eec1-3765-48a4-b09c-3dec2d29956b nodeName:}" failed. No retries permitted until 2026-11-01 14:16:45.174552423 +0000 UTC m=+35.334493075 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7970eec1-3765-48a4-b09c-3dec2d29956b-registry-tls") pod "image-registry-7f9cfbc8db-w8cmn" (UID: "7970eec1-3765-48a4-b09c-3dec2d29956b") : secret "image-registry-tls" not found
Nov 01 14:16:43.275004 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:43.274969    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1-metrics-certs\") pod \"network-metrics-daemon-b6rtr\" (UID: \"ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1\") " pod="openshift-multus/network-metrics-daemon-b6rtr"
Nov 01 14:16:43.275468 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:43.275150    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 01 14:16:43.275468 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:43.275254    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1-metrics-certs podName:ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1 nodeName:}" failed. No retries permitted until 2026-11-01 14:17:15.275214977 +0000 UTC m=+65.435155638 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1-metrics-certs") pod "network-metrics-daemon-b6rtr" (UID: "ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1") : secret "metrics-daemon-secret" not found
Nov 01 14:16:43.275468 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:43.275292    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8b1eb53c-8c9a-46c4-b654-2aef42491b14-original-pull-secret\") pod \"global-pull-secret-syncer-sc9jq\" (UID: \"8b1eb53c-8c9a-46c4-b654-2aef42491b14\") " pod="kube-system/global-pull-secret-syncer-sc9jq"
Nov 01 14:16:43.277945 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:43.277926    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8b1eb53c-8c9a-46c4-b654-2aef42491b14-original-pull-secret\") pod \"global-pull-secret-syncer-sc9jq\" (UID: \"8b1eb53c-8c9a-46c4-b654-2aef42491b14\") " pod="kube-system/global-pull-secret-syncer-sc9jq"
Nov 01 14:16:43.377094 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:43.376916    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp6rs\" (UniqueName: \"kubernetes.io/projected/0d4352db-d2e7-43ba-97ba-9e344abbac8b-kube-api-access-vp6rs\") pod \"network-check-target-8mqxj\" (UID: \"0d4352db-d2e7-43ba-97ba-9e344abbac8b\") " pod="openshift-network-diagnostics/network-check-target-8mqxj"
Nov 01 14:16:43.379860 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:43.379831    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp6rs\" (UniqueName: \"kubernetes.io/projected/0d4352db-d2e7-43ba-97ba-9e344abbac8b-kube-api-access-vp6rs\") pod \"network-check-target-8mqxj\" (UID: \"0d4352db-d2e7-43ba-97ba-9e344abbac8b\") " pod="openshift-network-diagnostics/network-check-target-8mqxj"
Nov 01 14:16:43.395681 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:43.395656    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sc9jq"
Nov 01 14:16:43.400268 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:43.400241    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8mqxj"
Nov 01 14:16:43.409320 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:43.409260    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-zfqjv" podStartSLOduration=433.409242951 podStartE2EDuration="7m13.409242951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:09:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:16:43.407608034 +0000 UTC m=+33.567548697" watchObservedRunningTime="2026-11-01 14:16:43.409242951 +0000 UTC m=+33.569183616"
Nov 01 14:16:43.523837 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:43.523784    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-xmfr4" podStartSLOduration=433.523772153 podStartE2EDuration="7m13.523772153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:09:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:16:43.522191622 +0000 UTC m=+33.682132283" watchObservedRunningTime="2026-11-01 14:16:43.523772153 +0000 UTC m=+33.683712813"
Nov 01 14:16:43.635314 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:43.635230    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-4lzxm" podStartSLOduration=433.635215613 podStartE2EDuration="7m13.635215613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:09:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:16:43.634424291 +0000 UTC m=+33.794364952" watchObservedRunningTime="2026-11-01 14:16:43.635215613 +0000 UTC m=+33.795156274"
Nov 01 14:16:43.789885 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:43.789850    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-sc9jq"]
Nov 01 14:16:43.794964 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:43.794891    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b1eb53c_8c9a_46c4_b654_2aef42491b14.slice/crio-ba1d2e479174d30eac7876b3f30fff0702c316d073a37353f03d9a09e2110c84 WatchSource:0}: Error finding container ba1d2e479174d30eac7876b3f30fff0702c316d073a37353f03d9a09e2110c84: Status 404 returned error can't find the container with id ba1d2e479174d30eac7876b3f30fff0702c316d073a37353f03d9a09e2110c84
Nov 01 14:16:43.837505 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:43.837478    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8mqxj"]
Nov 01 14:16:44.530842 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:44.530774    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-sc9jq" event={"ID":"8b1eb53c-8c9a-46c4-b654-2aef42491b14","Type":"ContainerStarted","Data":"ba1d2e479174d30eac7876b3f30fff0702c316d073a37353f03d9a09e2110c84"}
Nov 01 14:16:44.532498 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:44.532390    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8mqxj" event={"ID":"0d4352db-d2e7-43ba-97ba-9e344abbac8b","Type":"ContainerStarted","Data":"ae783b4a23f870f431b151c343e9c32bba480f883ab647d3f796595900a6e53e"}
Nov 01 14:16:45.096643 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:45.096550    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/037ca099-5ad7-4874-a984-462c07801e70-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-bgmv8\" (UID: \"037ca099-5ad7-4874-a984-462c07801e70\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bgmv8"
Nov 01 14:16:45.096796 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:45.096688    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b5a4fefb-cc48-4638-b0f8-590e36bd0118-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-cwfck\" (UID: \"b5a4fefb-cc48-4638-b0f8-590e36bd0118\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-cwfck"
Nov 01 14:16:45.096796 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:45.096739    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e7ea769b-e719-4090-bcd4-05b1b88fcc06-registry-tls\") pod \"image-registry-69c8c9797f-cblkb\" (UID: \"e7ea769b-e719-4090-bcd4-05b1b88fcc06\") " pod="openshift-image-registry/image-registry-69c8c9797f-cblkb"
Nov 01 14:16:45.096975 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:45.096953    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 14:16:45.096975 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:45.096975    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69c8c9797f-cblkb: secret "image-registry-tls" not found
Nov 01 14:16:45.097089 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:45.097032    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7ea769b-e719-4090-bcd4-05b1b88fcc06-registry-tls podName:e7ea769b-e719-4090-bcd4-05b1b88fcc06 nodeName:}" failed. No retries permitted until 2026-11-01 14:16:49.097013765 +0000 UTC m=+39.256954418 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e7ea769b-e719-4090-bcd4-05b1b88fcc06-registry-tls") pod "image-registry-69c8c9797f-cblkb" (UID: "e7ea769b-e719-4090-bcd4-05b1b88fcc06") : secret "image-registry-tls" not found
Nov 01 14:16:45.097160 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:45.097144    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 01 14:16:45.097535 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:45.097307    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/037ca099-5ad7-4874-a984-462c07801e70-cluster-monitoring-operator-tls podName:037ca099-5ad7-4874-a984-462c07801e70 nodeName:}" failed. No retries permitted until 2026-11-01 14:16:49.097185562 +0000 UTC m=+39.257126214 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/037ca099-5ad7-4874-a984-462c07801e70-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-bgmv8" (UID: "037ca099-5ad7-4874-a984-462c07801e70") : secret "cluster-monitoring-operator-tls" not found
Nov 01 14:16:45.097535 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:45.097392    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 01 14:16:45.097535 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:45.097431    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5a4fefb-cc48-4638-b0f8-590e36bd0118-networking-console-plugin-cert podName:b5a4fefb-cc48-4638-b0f8-590e36bd0118 nodeName:}" failed. No retries permitted until 2026-11-01 14:16:49.097419636 +0000 UTC m=+39.257360291 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b5a4fefb-cc48-4638-b0f8-590e36bd0118-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-cwfck" (UID: "b5a4fefb-cc48-4638-b0f8-590e36bd0118") : secret "networking-console-plugin-cert" not found
Nov 01 14:16:45.197861 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:45.197819    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c27ff402-235a-4d73-a139-cf928a6d5cbe-service-ca-bundle\") pod \"router-default-b9d995dcc-lnk2n\" (UID: \"c27ff402-235a-4d73-a139-cf928a6d5cbe\") " pod="openshift-ingress/router-default-b9d995dcc-lnk2n"
Nov 01 14:16:45.198032 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:45.197950    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7970eec1-3765-48a4-b09c-3dec2d29956b-registry-tls\") pod \"image-registry-7f9cfbc8db-w8cmn\" (UID: \"7970eec1-3765-48a4-b09c-3dec2d29956b\") " pod="openshift-image-registry/image-registry-7f9cfbc8db-w8cmn"
Nov 01 14:16:45.198093 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:45.198038    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c27ff402-235a-4d73-a139-cf928a6d5cbe-metrics-certs\") pod \"router-default-b9d995dcc-lnk2n\" (UID: \"c27ff402-235a-4d73-a139-cf928a6d5cbe\") " pod="openshift-ingress/router-default-b9d995dcc-lnk2n"
Nov 01 14:16:45.198093 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:45.198076    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c8cc47b-fb01-4195-a7fc-eeefc706017e-metrics-tls\") pod \"dns-default-b2lcr\" (UID: \"0c8cc47b-fb01-4195-a7fc-eeefc706017e\") " pod="openshift-dns/dns-default-b2lcr"
Nov 01 14:16:45.198217 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:45.198112    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d407313d-3333-4ec4-85bb-cd27cf4d7422-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-hsdjc\" (UID: \"d407313d-3333-4ec4-85bb-cd27cf4d7422\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hsdjc"
Nov 01 14:16:45.198359 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:45.198328    2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 01 14:16:45.198512 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:45.198439    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c27ff402-235a-4d73-a139-cf928a6d5cbe-service-ca-bundle podName:c27ff402-235a-4d73-a139-cf928a6d5cbe nodeName:}" failed. No retries permitted until 2026-11-01 14:16:49.198412915 +0000 UTC m=+39.358353569 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c27ff402-235a-4d73-a139-cf928a6d5cbe-service-ca-bundle") pod "router-default-b9d995dcc-lnk2n" (UID: "c27ff402-235a-4d73-a139-cf928a6d5cbe") : configmap references non-existent config key: service-ca.crt
Nov 01 14:16:45.198589 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:45.198514    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 14:16:45.198589 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:45.198526    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f9cfbc8db-w8cmn: secret "image-registry-tls" not found
Nov 01 14:16:45.198589 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:45.198561    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7970eec1-3765-48a4-b09c-3dec2d29956b-registry-tls podName:7970eec1-3765-48a4-b09c-3dec2d29956b nodeName:}" failed. No retries permitted until 2026-11-01 14:16:49.198550234 +0000 UTC m=+39.358490888 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7970eec1-3765-48a4-b09c-3dec2d29956b-registry-tls") pod "image-registry-7f9cfbc8db-w8cmn" (UID: "7970eec1-3765-48a4-b09c-3dec2d29956b") : secret "image-registry-tls" not found
Nov 01 14:16:45.198740 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:45.198620    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 01 14:16:45.198740 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:45.198677    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c27ff402-235a-4d73-a139-cf928a6d5cbe-metrics-certs podName:c27ff402-235a-4d73-a139-cf928a6d5cbe nodeName:}" failed. No retries permitted until 2026-11-01 14:16:49.19866733 +0000 UTC m=+39.358607983 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c27ff402-235a-4d73-a139-cf928a6d5cbe-metrics-certs") pod "router-default-b9d995dcc-lnk2n" (UID: "c27ff402-235a-4d73-a139-cf928a6d5cbe") : secret "router-metrics-certs-default" not found
Nov 01 14:16:45.198740 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:45.198730    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 14:16:45.198886 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:45.198755    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c8cc47b-fb01-4195-a7fc-eeefc706017e-metrics-tls podName:0c8cc47b-fb01-4195-a7fc-eeefc706017e nodeName:}" failed. No retries permitted until 2026-11-01 14:16:49.198746408 +0000 UTC m=+39.358687069 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0c8cc47b-fb01-4195-a7fc-eeefc706017e-metrics-tls") pod "dns-default-b2lcr" (UID: "0c8cc47b-fb01-4195-a7fc-eeefc706017e") : secret "dns-default-metrics-tls" not found
Nov 01 14:16:45.198886 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:45.198841    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d407313d-3333-4ec4-85bb-cd27cf4d7422-samples-operator-tls podName:d407313d-3333-4ec4-85bb-cd27cf4d7422 nodeName:}" failed. No retries permitted until 2026-11-01 14:16:49.198832463 +0000 UTC m=+39.358773115 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d407313d-3333-4ec4-85bb-cd27cf4d7422-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-hsdjc" (UID: "d407313d-3333-4ec4-85bb-cd27cf4d7422") : secret "samples-operator-tls" not found
Nov 01 14:16:45.541405 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:45.541361    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-shpt4" event={"ID":"e78d2494-62c4-4070-a1eb-8ce40329ff57","Type":"ContainerStarted","Data":"188500b6b0aa4dc26606743d3a809efdc82a1e34e09974f38b3f60f9cceaebe7"}
Nov 01 14:16:45.676498 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:45.676423    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-shpt4" podStartSLOduration=403.012847464 podStartE2EDuration="6m45.676404844s" totalImagesPullingTime="2.66355738s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:10:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:16:45.675641521 +0000 UTC m=+35.835582183" watchObservedRunningTime="2026-11-01 14:16:45.676404844 +0000 UTC m=+35.836345507"
Nov 01 14:16:47.552091 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:47.552054    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-xmwtp" event={"ID":"bb59051d-c2df-4362-a157-00f14f915e79","Type":"ContainerStarted","Data":"87d474983f1bb78a53c1a0bac38617cb2c7ef47ce10c2b8cc97003f0f3fa8e29"}
Nov 01 14:16:47.553090 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:47.553068    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-xmwtp"
Nov 01 14:16:47.554101 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:47.554073    2576 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-xmwtp container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.19:8443/readyz\": dial tcp 10.132.0.19:8443: connect: connection refused" start-of-body=
Nov 01 14:16:47.554223 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:47.554126    2576 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-xmwtp" podUID="bb59051d-c2df-4362-a157-00f14f915e79" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.19:8443/readyz\": dial tcp 10.132.0.19:8443: connect: connection refused"
Nov 01 14:16:47.667385 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:47.667280    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7cd56b8dbc-xmwtp" podStartSLOduration=398.498558045 podStartE2EDuration="6m43.667262213s" totalImagesPullingTime="5.168704168s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:10:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:16:47.666091939 +0000 UTC m=+37.826032604" watchObservedRunningTime="2026-11-01 14:16:47.667262213 +0000 UTC m=+37.827202881"
Nov 01 14:16:48.557108 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:48.557074    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xmwtp_bb59051d-c2df-4362-a157-00f14f915e79/console-operator/0.log"
Nov 01 14:16:48.557597 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:48.557129    2576 generic.go:356] "Generic (PLEG): container finished" podID="bb59051d-c2df-4362-a157-00f14f915e79" containerID="87d474983f1bb78a53c1a0bac38617cb2c7ef47ce10c2b8cc97003f0f3fa8e29" exitCode=255
Nov 01 14:16:48.557597 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:48.557219    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-xmwtp" event={"ID":"bb59051d-c2df-4362-a157-00f14f915e79","Type":"ContainerDied","Data":"87d474983f1bb78a53c1a0bac38617cb2c7ef47ce10c2b8cc97003f0f3fa8e29"}
Nov 01 14:16:48.557906 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:48.557856    2576 scope.go:122] "RemoveContainer" containerID="87d474983f1bb78a53c1a0bac38617cb2c7ef47ce10c2b8cc97003f0f3fa8e29"
Nov 01 14:16:48.561061 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:48.561032    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-7d54k" event={"ID":"7a26b817-3283-4d59-8e8f-3bb4c0fa4987","Type":"ContainerStarted","Data":"56c3f8610d5a54df92228250cdf61c4e3bf3c664f5ff6f24ed0c5fc5c65aecd8"}
Nov 01 14:16:48.792946 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:48.790798    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-7d54k" podStartSLOduration=400.127268989 podStartE2EDuration="6m45.790777873s" totalImagesPullingTime="5.663508884s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:10:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:16:48.789986737 +0000 UTC m=+38.949927399" watchObservedRunningTime="2026-11-01 14:16:48.790777873 +0000 UTC m=+38.950718534"
Nov 01 14:16:49.139196 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:49.139144    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b5a4fefb-cc48-4638-b0f8-590e36bd0118-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-cwfck\" (UID: \"b5a4fefb-cc48-4638-b0f8-590e36bd0118\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-cwfck"
Nov 01 14:16:49.139353 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:49.139224    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e7ea769b-e719-4090-bcd4-05b1b88fcc06-registry-tls\") pod \"image-registry-69c8c9797f-cblkb\" (UID: \"e7ea769b-e719-4090-bcd4-05b1b88fcc06\") " pod="openshift-image-registry/image-registry-69c8c9797f-cblkb"
Nov 01 14:16:49.139353 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:49.139340    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/037ca099-5ad7-4874-a984-462c07801e70-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-bgmv8\" (UID: \"037ca099-5ad7-4874-a984-462c07801e70\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bgmv8"
Nov 01 14:16:49.139513 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:49.139494    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 01 14:16:49.139585 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:49.139564    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/037ca099-5ad7-4874-a984-462c07801e70-cluster-monitoring-operator-tls podName:037ca099-5ad7-4874-a984-462c07801e70 nodeName:}" failed. No retries permitted until 2026-11-01 14:16:57.139544538 +0000 UTC m=+47.299485180 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/037ca099-5ad7-4874-a984-462c07801e70-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-bgmv8" (UID: "037ca099-5ad7-4874-a984-462c07801e70") : secret "cluster-monitoring-operator-tls" not found
Nov 01 14:16:49.139653 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:49.139639    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 01 14:16:49.139708 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:49.139672    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5a4fefb-cc48-4638-b0f8-590e36bd0118-networking-console-plugin-cert podName:b5a4fefb-cc48-4638-b0f8-590e36bd0118 nodeName:}" failed. No retries permitted until 2026-11-01 14:16:57.139662669 +0000 UTC m=+47.299603312 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b5a4fefb-cc48-4638-b0f8-590e36bd0118-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-cwfck" (UID: "b5a4fefb-cc48-4638-b0f8-590e36bd0118") : secret "networking-console-plugin-cert" not found
Nov 01 14:16:49.139767 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:49.139731    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 14:16:49.139767 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:49.139744    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69c8c9797f-cblkb: secret "image-registry-tls" not found
Nov 01 14:16:49.139865 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:49.139773    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7ea769b-e719-4090-bcd4-05b1b88fcc06-registry-tls podName:e7ea769b-e719-4090-bcd4-05b1b88fcc06 nodeName:}" failed. No retries permitted until 2026-11-01 14:16:57.139763531 +0000 UTC m=+47.299704172 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e7ea769b-e719-4090-bcd4-05b1b88fcc06-registry-tls") pod "image-registry-69c8c9797f-cblkb" (UID: "e7ea769b-e719-4090-bcd4-05b1b88fcc06") : secret "image-registry-tls" not found
Nov 01 14:16:49.204198 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:49.197939    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-zq4pm"]
Nov 01 14:16:49.221970 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:49.221290    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zq4pm"
Nov 01 14:16:49.240374 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:49.240336    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/058820ef-eef1-41ca-b331-670bab146861-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zq4pm\" (UID: \"058820ef-eef1-41ca-b331-670bab146861\") " pod="openshift-insights/insights-runtime-extractor-zq4pm"
Nov 01 14:16:49.240526 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:49.240401    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c27ff402-235a-4d73-a139-cf928a6d5cbe-service-ca-bundle\") pod \"router-default-b9d995dcc-lnk2n\" (UID: \"c27ff402-235a-4d73-a139-cf928a6d5cbe\") " pod="openshift-ingress/router-default-b9d995dcc-lnk2n"
Nov 01 14:16:49.240526 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:49.240446    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7970eec1-3765-48a4-b09c-3dec2d29956b-registry-tls\") pod \"image-registry-7f9cfbc8db-w8cmn\" (UID: \"7970eec1-3765-48a4-b09c-3dec2d29956b\") " pod="openshift-image-registry/image-registry-7f9cfbc8db-w8cmn"
Nov 01 14:16:49.240526 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:49.240483    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c27ff402-235a-4d73-a139-cf928a6d5cbe-metrics-certs\") pod \"router-default-b9d995dcc-lnk2n\" (UID: \"c27ff402-235a-4d73-a139-cf928a6d5cbe\") " pod="openshift-ingress/router-default-b9d995dcc-lnk2n"
Nov 01 14:16:49.240526 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:49.240507    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c8cc47b-fb01-4195-a7fc-eeefc706017e-metrics-tls\") pod \"dns-default-b2lcr\" (UID: \"0c8cc47b-fb01-4195-a7fc-eeefc706017e\") " pod="openshift-dns/dns-default-b2lcr"
Nov 01 14:16:49.240526 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:49.240523    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/058820ef-eef1-41ca-b331-670bab146861-data-volume\") pod \"insights-runtime-extractor-zq4pm\" (UID: \"058820ef-eef1-41ca-b331-670bab146861\") " pod="openshift-insights/insights-runtime-extractor-zq4pm"
Nov 01 14:16:49.240792 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:49.240554    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d407313d-3333-4ec4-85bb-cd27cf4d7422-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-hsdjc\" (UID: \"d407313d-3333-4ec4-85bb-cd27cf4d7422\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hsdjc"
Nov 01 14:16:49.240792 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:49.240572    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/058820ef-eef1-41ca-b331-670bab146861-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zq4pm\" (UID: \"058820ef-eef1-41ca-b331-670bab146861\") " pod="openshift-insights/insights-runtime-extractor-zq4pm"
Nov 01 14:16:49.240792 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:49.240593    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/058820ef-eef1-41ca-b331-670bab146861-crio-socket\") pod \"insights-runtime-extractor-zq4pm\" (UID: \"058820ef-eef1-41ca-b331-670bab146861\") " pod="openshift-insights/insights-runtime-extractor-zq4pm"
Nov 01 14:16:49.240792 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:49.240616    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx82t\" (UniqueName: \"kubernetes.io/projected/058820ef-eef1-41ca-b331-670bab146861-kube-api-access-gx82t\") pod \"insights-runtime-extractor-zq4pm\" (UID: \"058820ef-eef1-41ca-b331-670bab146861\") " pod="openshift-insights/insights-runtime-extractor-zq4pm"
Nov 01 14:16:49.240792 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:49.240746    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c27ff402-235a-4d73-a139-cf928a6d5cbe-service-ca-bundle podName:c27ff402-235a-4d73-a139-cf928a6d5cbe nodeName:}" failed. No retries permitted until 2026-11-01 14:16:57.240732328 +0000 UTC m=+47.400672966 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c27ff402-235a-4d73-a139-cf928a6d5cbe-service-ca-bundle") pod "router-default-b9d995dcc-lnk2n" (UID: "c27ff402-235a-4d73-a139-cf928a6d5cbe") : configmap references non-existent config key: service-ca.crt
Nov 01 14:16:49.241039 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:49.240802    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 14:16:49.241039 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:49.240810    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f9cfbc8db-w8cmn: secret "image-registry-tls" not found
Nov 01 14:16:49.241039 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:49.240834    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7970eec1-3765-48a4-b09c-3dec2d29956b-registry-tls podName:7970eec1-3765-48a4-b09c-3dec2d29956b nodeName:}" failed. No retries permitted until 2026-11-01 14:16:57.240824408 +0000 UTC m=+47.400765060 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7970eec1-3765-48a4-b09c-3dec2d29956b-registry-tls") pod "image-registry-7f9cfbc8db-w8cmn" (UID: "7970eec1-3765-48a4-b09c-3dec2d29956b") : secret "image-registry-tls" not found
Nov 01 14:16:49.241039 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:49.240883    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 01 14:16:49.241259 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:49.240913    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c27ff402-235a-4d73-a139-cf928a6d5cbe-metrics-certs podName:c27ff402-235a-4d73-a139-cf928a6d5cbe nodeName:}" failed. No retries permitted until 2026-11-01 14:16:57.240903209 +0000 UTC m=+47.400843864 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c27ff402-235a-4d73-a139-cf928a6d5cbe-metrics-certs") pod "router-default-b9d995dcc-lnk2n" (UID: "c27ff402-235a-4d73-a139-cf928a6d5cbe") : secret "router-metrics-certs-default" not found
Nov 01 14:16:49.242072 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:49.242040    2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 01 14:16:49.242216 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:49.242086    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 14:16:49.242216 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:49.242185    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d407313d-3333-4ec4-85bb-cd27cf4d7422-samples-operator-tls podName:d407313d-3333-4ec4-85bb-cd27cf4d7422 nodeName:}" failed. No retries permitted until 2026-11-01 14:16:57.242149797 +0000 UTC m=+47.402090460 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d407313d-3333-4ec4-85bb-cd27cf4d7422-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-hsdjc" (UID: "d407313d-3333-4ec4-85bb-cd27cf4d7422") : secret "samples-operator-tls" not found
Nov 01 14:16:49.242338 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:49.242228    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c8cc47b-fb01-4195-a7fc-eeefc706017e-metrics-tls podName:0c8cc47b-fb01-4195-a7fc-eeefc706017e nodeName:}" failed. No retries permitted until 2026-11-01 14:16:57.242217972 +0000 UTC m=+47.402158615 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0c8cc47b-fb01-4195-a7fc-eeefc706017e-metrics-tls") pod "dns-default-b2lcr" (UID: "0c8cc47b-fb01-4195-a7fc-eeefc706017e") : secret "dns-default-metrics-tls" not found
Nov 01 14:16:49.256081 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:49.256053    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zq4pm"]
Nov 01 14:16:49.274891 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:49.274799    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8xv6q\""
Nov 01 14:16:49.274891 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:49.274842    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 01 14:16:49.274891 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:49.274865    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 01 14:16:49.341768 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:49.341728    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/058820ef-eef1-41ca-b331-670bab146861-data-volume\") pod \"insights-runtime-extractor-zq4pm\" (UID: \"058820ef-eef1-41ca-b331-670bab146861\") " pod="openshift-insights/insights-runtime-extractor-zq4pm"
Nov 01 14:16:49.341946 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:49.341796    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/058820ef-eef1-41ca-b331-670bab146861-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zq4pm\" (UID: \"058820ef-eef1-41ca-b331-670bab146861\") " pod="openshift-insights/insights-runtime-extractor-zq4pm"
Nov 01 14:16:49.341946 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:49.341829    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/058820ef-eef1-41ca-b331-670bab146861-crio-socket\") pod \"insights-runtime-extractor-zq4pm\" (UID: \"058820ef-eef1-41ca-b331-670bab146861\") " pod="openshift-insights/insights-runtime-extractor-zq4pm"
Nov 01 14:16:49.341946 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:49.341862    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx82t\" (UniqueName: \"kubernetes.io/projected/058820ef-eef1-41ca-b331-670bab146861-kube-api-access-gx82t\") pod \"insights-runtime-extractor-zq4pm\" (UID: \"058820ef-eef1-41ca-b331-670bab146861\") " pod="openshift-insights/insights-runtime-extractor-zq4pm"
Nov 01 14:16:49.341946 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:49.341906    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/058820ef-eef1-41ca-b331-670bab146861-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zq4pm\" (UID: \"058820ef-eef1-41ca-b331-670bab146861\") " pod="openshift-insights/insights-runtime-extractor-zq4pm"
Nov 01 14:16:49.342084 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:49.342001    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 01 14:16:49.342084 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:49.342061    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/058820ef-eef1-41ca-b331-670bab146861-kube-rbac-proxy-cm podName:058820ef-eef1-41ca-b331-670bab146861 nodeName:}" failed. No retries permitted until 2026-11-01 14:16:49.842041841 +0000 UTC m=+40.001982486 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/058820ef-eef1-41ca-b331-670bab146861-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zq4pm" (UID: "058820ef-eef1-41ca-b331-670bab146861") : configmap "kube-rbac-proxy" not found
Nov 01 14:16:49.342196 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:49.342078    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/058820ef-eef1-41ca-b331-670bab146861-data-volume\") pod \"insights-runtime-extractor-zq4pm\" (UID: \"058820ef-eef1-41ca-b331-670bab146861\") " pod="openshift-insights/insights-runtime-extractor-zq4pm"
Nov 01 14:16:49.342196 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:49.342150    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 14:16:49.342196 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:49.342180    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/058820ef-eef1-41ca-b331-670bab146861-crio-socket\") pod \"insights-runtime-extractor-zq4pm\" (UID: \"058820ef-eef1-41ca-b331-670bab146861\") " pod="openshift-insights/insights-runtime-extractor-zq4pm"
Nov 01 14:16:49.342328 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:49.342219    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/058820ef-eef1-41ca-b331-670bab146861-insights-runtime-extractor-tls podName:058820ef-eef1-41ca-b331-670bab146861 nodeName:}" failed. No retries permitted until 2026-11-01 14:16:49.842199451 +0000 UTC m=+40.002140105 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/058820ef-eef1-41ca-b331-670bab146861-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zq4pm" (UID: "058820ef-eef1-41ca-b331-670bab146861") : secret "insights-runtime-extractor-tls" not found
Nov 01 14:16:49.399072 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:49.399039    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx82t\" (UniqueName: \"kubernetes.io/projected/058820ef-eef1-41ca-b331-670bab146861-kube-api-access-gx82t\") pod \"insights-runtime-extractor-zq4pm\" (UID: \"058820ef-eef1-41ca-b331-670bab146861\") " pod="openshift-insights/insights-runtime-extractor-zq4pm"
Nov 01 14:16:49.437746 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:49.436572    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-rlgcn-h5sk4"]
Nov 01 14:16:49.458114 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:49.458075    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-rlgcn-h5sk4"
Nov 01 14:16:49.508878 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:49.508736    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-rlgcn-h5sk4"]
Nov 01 14:16:49.510669 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:49.510622    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-55h6z\""
Nov 01 14:16:49.574060 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:49.573286    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8fgtw" event={"ID":"70bb916a-832b-4bce-9c0a-f01bf4e3abc4","Type":"ContainerStarted","Data":"6b17c6583493602c622ceb56cf480cb4b4f133441a8b39e29e61c94aae352153"}
Nov 01 14:16:49.576386 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:49.576361    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xmwtp_bb59051d-c2df-4362-a157-00f14f915e79/console-operator/1.log"
Nov 01 14:16:49.577208 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:49.577185    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xmwtp_bb59051d-c2df-4362-a157-00f14f915e79/console-operator/0.log"
Nov 01 14:16:49.577353 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:49.577225    2576 generic.go:356] "Generic (PLEG): container finished" podID="bb59051d-c2df-4362-a157-00f14f915e79" containerID="cbc84fec5c099225a4e49365bdccb7876e559e9347bbfff1ca76af14561a16c1" exitCode=255
Nov 01 14:16:49.577353 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:49.577299    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-xmwtp" event={"ID":"bb59051d-c2df-4362-a157-00f14f915e79","Type":"ContainerDied","Data":"cbc84fec5c099225a4e49365bdccb7876e559e9347bbfff1ca76af14561a16c1"}
Nov 01 14:16:49.577353 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:49.577343    2576 scope.go:122] "RemoveContainer" containerID="87d474983f1bb78a53c1a0bac38617cb2c7ef47ce10c2b8cc97003f0f3fa8e29"
Nov 01 14:16:49.577744 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:49.577725    2576 scope.go:122] "RemoveContainer" containerID="cbc84fec5c099225a4e49365bdccb7876e559e9347bbfff1ca76af14561a16c1"
Nov 01 14:16:49.577950 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:49.577926    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-xmwtp_openshift-console-operator(bb59051d-c2df-4362-a157-00f14f915e79)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-xmwtp" podUID="bb59051d-c2df-4362-a157-00f14f915e79"
Nov 01 14:16:49.580684 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:49.580602    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-96vgd" event={"ID":"4a724964-bd57-4d56-a042-a0d490115eaa","Type":"ContainerStarted","Data":"d2d1ecbf767e973e102e7a31b0be6ebf1172e08e054d8003f951a2e74e73c6a7"}
Nov 01 14:16:49.583720 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:49.583657    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-87xcd" event={"ID":"3e7e18b9-78ee-4349-b00c-a0800e8f8341","Type":"ContainerStarted","Data":"229a7f9ea816f632d9b1f883cd515cbea5787bc4a656de6e50c9101f2ca3f073"}
Nov 01 14:16:49.643921 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:49.643891    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/384cd50b-370f-474f-b80e-89a544a9bbbd-trusted-ca-bundle\") pod \"periodic-gathering-rlgcn-h5sk4\" (UID: \"384cd50b-370f-474f-b80e-89a544a9bbbd\") " pod="openshift-insights/periodic-gathering-rlgcn-h5sk4"
Nov 01 14:16:49.644255 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:49.644230    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/384cd50b-370f-474f-b80e-89a544a9bbbd-service-ca-bundle\") pod \"periodic-gathering-rlgcn-h5sk4\" (UID: \"384cd50b-370f-474f-b80e-89a544a9bbbd\") " pod="openshift-insights/periodic-gathering-rlgcn-h5sk4"
Nov 01 14:16:49.644515 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:49.644487    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chwvj\" (UniqueName: \"kubernetes.io/projected/384cd50b-370f-474f-b80e-89a544a9bbbd-kube-api-access-chwvj\") pod \"periodic-gathering-rlgcn-h5sk4\" (UID: \"384cd50b-370f-474f-b80e-89a544a9bbbd\") " pod="openshift-insights/periodic-gathering-rlgcn-h5sk4"
Nov 01 14:16:49.645125 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:49.645103    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/384cd50b-370f-474f-b80e-89a544a9bbbd-archives-path\") pod \"periodic-gathering-rlgcn-h5sk4\" (UID: \"384cd50b-370f-474f-b80e-89a544a9bbbd\") " pod="openshift-insights/periodic-gathering-rlgcn-h5sk4"
Nov 01 14:16:49.688750 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:49.688231    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8fgtw" podStartSLOduration=1.874970515 podStartE2EDuration="8.688212067s" totalImagesPullingTime="6.813241552s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:16:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:16:49.686074073 +0000 UTC m=+39.846014735" watchObservedRunningTime="2026-11-01 14:16:49.688212067 +0000 UTC m=+39.848152730"
Nov 01 14:16:49.747023 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:49.746989    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/384cd50b-370f-474f-b80e-89a544a9bbbd-service-ca-bundle\") pod \"periodic-gathering-rlgcn-h5sk4\" (UID: \"384cd50b-370f-474f-b80e-89a544a9bbbd\") " pod="openshift-insights/periodic-gathering-rlgcn-h5sk4"
Nov 01 14:16:49.747217 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:49.747041    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chwvj\" (UniqueName: \"kubernetes.io/projected/384cd50b-370f-474f-b80e-89a544a9bbbd-kube-api-access-chwvj\") pod \"periodic-gathering-rlgcn-h5sk4\" (UID: \"384cd50b-370f-474f-b80e-89a544a9bbbd\") " pod="openshift-insights/periodic-gathering-rlgcn-h5sk4"
Nov 01 14:16:49.747217 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:49.747069    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/384cd50b-370f-474f-b80e-89a544a9bbbd-archives-path\") pod \"periodic-gathering-rlgcn-h5sk4\" (UID: \"384cd50b-370f-474f-b80e-89a544a9bbbd\") " pod="openshift-insights/periodic-gathering-rlgcn-h5sk4"
Nov 01 14:16:49.747329 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:49.747258    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/384cd50b-370f-474f-b80e-89a544a9bbbd-trusted-ca-bundle\") pod \"periodic-gathering-rlgcn-h5sk4\" (UID: \"384cd50b-370f-474f-b80e-89a544a9bbbd\") " pod="openshift-insights/periodic-gathering-rlgcn-h5sk4"
Nov 01 14:16:49.747616 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:49.747593    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/384cd50b-370f-474f-b80e-89a544a9bbbd-service-ca-bundle\") pod \"periodic-gathering-rlgcn-h5sk4\" (UID: \"384cd50b-370f-474f-b80e-89a544a9bbbd\") " pod="openshift-insights/periodic-gathering-rlgcn-h5sk4"
Nov 01 14:16:49.748000 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:49.747977    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/384cd50b-370f-474f-b80e-89a544a9bbbd-trusted-ca-bundle\") pod \"periodic-gathering-rlgcn-h5sk4\" (UID: \"384cd50b-370f-474f-b80e-89a544a9bbbd\") " pod="openshift-insights/periodic-gathering-rlgcn-h5sk4"
Nov 01 14:16:49.749540 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:49.748243    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/384cd50b-370f-474f-b80e-89a544a9bbbd-archives-path\") pod \"periodic-gathering-rlgcn-h5sk4\" (UID: \"384cd50b-370f-474f-b80e-89a544a9bbbd\") " pod="openshift-insights/periodic-gathering-rlgcn-h5sk4"
Nov 01 14:16:49.798785 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:49.798687    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-87xcd" podStartSLOduration=402.329043516 podStartE2EDuration="6m48.798669935s" totalImagesPullingTime="6.469626419s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:10:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:16:49.79731661 +0000 UTC m=+39.957257273" watchObservedRunningTime="2026-11-01 14:16:49.798669935 +0000 UTC m=+39.958610597"
Nov 01 14:16:49.809051 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:49.809019    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chwvj\" (UniqueName: \"kubernetes.io/projected/384cd50b-370f-474f-b80e-89a544a9bbbd-kube-api-access-chwvj\") pod \"periodic-gathering-rlgcn-h5sk4\" (UID: \"384cd50b-370f-474f-b80e-89a544a9bbbd\") " pod="openshift-insights/periodic-gathering-rlgcn-h5sk4"
Nov 01 14:16:49.848750 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:49.848716    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/058820ef-eef1-41ca-b331-670bab146861-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zq4pm\" (UID: \"058820ef-eef1-41ca-b331-670bab146861\") " pod="openshift-insights/insights-runtime-extractor-zq4pm"
Nov 01 14:16:49.848970 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:49.848795    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/058820ef-eef1-41ca-b331-670bab146861-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zq4pm\" (UID: \"058820ef-eef1-41ca-b331-670bab146861\") " pod="openshift-insights/insights-runtime-extractor-zq4pm"
Nov 01 14:16:49.848970 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:49.848880    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 14:16:49.848970 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:49.848946    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 01 14:16:49.848970 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:49.848957    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/058820ef-eef1-41ca-b331-670bab146861-insights-runtime-extractor-tls podName:058820ef-eef1-41ca-b331-670bab146861 nodeName:}" failed. No retries permitted until 2026-11-01 14:16:50.848937157 +0000 UTC m=+41.008877802 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/058820ef-eef1-41ca-b331-670bab146861-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zq4pm" (UID: "058820ef-eef1-41ca-b331-670bab146861") : secret "insights-runtime-extractor-tls" not found
Nov 01 14:16:49.849209 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:49.849037    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/058820ef-eef1-41ca-b331-670bab146861-kube-rbac-proxy-cm podName:058820ef-eef1-41ca-b331-670bab146861 nodeName:}" failed. No retries permitted until 2026-11-01 14:16:50.849019873 +0000 UTC m=+41.008960512 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/058820ef-eef1-41ca-b331-670bab146861-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zq4pm" (UID: "058820ef-eef1-41ca-b331-670bab146861") : configmap "kube-rbac-proxy" not found
Nov 01 14:16:50.022446 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:50.022374    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-96vgd" podStartSLOduration=403.487123674 podStartE2EDuration="6m50.022355111s" totalImagesPullingTime="6.535231437s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:10:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:16:50.021525682 +0000 UTC m=+40.181466343" watchObservedRunningTime="2026-11-01 14:16:50.022355111 +0000 UTC m=+40.182295774"
Nov 01 14:16:50.079237 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:50.078153    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-rlgcn-h5sk4"
Nov 01 14:16:50.478279 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:50.478195    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-rlgcn-h5sk4"]
Nov 01 14:16:50.481137 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:50.481099    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod384cd50b_370f_474f_b80e_89a544a9bbbd.slice/crio-927f16040227de27cef857b98df6409aba54e22122880ff2d3dc48f638daff3d WatchSource:0}: Error finding container 927f16040227de27cef857b98df6409aba54e22122880ff2d3dc48f638daff3d: Status 404 returned error can't find the container with id 927f16040227de27cef857b98df6409aba54e22122880ff2d3dc48f638daff3d
Nov 01 14:16:50.588790 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:50.588744    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-rlgcn-h5sk4" event={"ID":"384cd50b-370f-474f-b80e-89a544a9bbbd","Type":"ContainerStarted","Data":"927f16040227de27cef857b98df6409aba54e22122880ff2d3dc48f638daff3d"}
Nov 01 14:16:50.591646 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:50.591603    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-lhdpr" event={"ID":"f219e3c2-6da8-4c88-9ec4-b26cf4d10326","Type":"ContainerStarted","Data":"21eae426ce05fb0bdcfb1335a6388e38649a03504ee06d807ad5b12de0a9775e"}
Nov 01 14:16:50.594550 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:50.594527    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xmwtp_bb59051d-c2df-4362-a157-00f14f915e79/console-operator/1.log"
Nov 01 14:16:50.595276 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:50.595245    2576 scope.go:122] "RemoveContainer" containerID="cbc84fec5c099225a4e49365bdccb7876e559e9347bbfff1ca76af14561a16c1"
Nov 01 14:16:50.595438 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:50.595399    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-xmwtp_openshift-console-operator(bb59051d-c2df-4362-a157-00f14f915e79)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-xmwtp" podUID="bb59051d-c2df-4362-a157-00f14f915e79"
Nov 01 14:16:50.673685 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:50.673650    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-4mmw7"]
Nov 01 14:16:50.698241 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:50.698214    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-8rp2x"]
Nov 01 14:16:50.698352 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:50.698333    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4mmw7"
Nov 01 14:16:50.726032 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:50.725196    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8rp2x"
Nov 01 14:16:50.728937 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:50.728853    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-lhdpr" podStartSLOduration=398.085539089 podStartE2EDuration="6m45.728836738s" totalImagesPullingTime="7.643297649s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:10:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:16:50.70502312 +0000 UTC m=+40.864963781" watchObservedRunningTime="2026-11-01 14:16:50.728836738 +0000 UTC m=+40.888777459"
Nov 01 14:16:50.730053 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:50.729560    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-4mmw7"]
Nov 01 14:16:50.750863 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:50.750823    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 01 14:16:50.750863 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:50.750844    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 01 14:16:50.751037 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:50.750940    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-jptnz\""
Nov 01 14:16:50.751037 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:50.750974    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-8rp2x"]
Nov 01 14:16:50.759864 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:50.759773    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpdrz\" (UniqueName: \"kubernetes.io/projected/17570aba-70d9-46ea-a0ca-3a465635a17c-kube-api-access-zpdrz\") pod \"migrator-66df5d496b-4mmw7\" (UID: \"17570aba-70d9-46ea-a0ca-3a465635a17c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4mmw7"
Nov 01 14:16:50.759864 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:50.759808    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khv87\" (UniqueName: \"kubernetes.io/projected/fba1e7f9-ca37-46dc-8c66-56f1d9563acd-kube-api-access-khv87\") pod \"migrator-66df5d496b-8rp2x\" (UID: \"fba1e7f9-ca37-46dc-8c66-56f1d9563acd\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8rp2x"
Nov 01 14:16:50.861273 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:50.861231    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/058820ef-eef1-41ca-b331-670bab146861-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zq4pm\" (UID: \"058820ef-eef1-41ca-b331-670bab146861\") " pod="openshift-insights/insights-runtime-extractor-zq4pm"
Nov 01 14:16:50.861453 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:50.861321    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/058820ef-eef1-41ca-b331-670bab146861-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zq4pm\" (UID: \"058820ef-eef1-41ca-b331-670bab146861\") " pod="openshift-insights/insights-runtime-extractor-zq4pm"
Nov 01 14:16:50.861453 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:50.861391    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 14:16:50.861453 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:50.861399    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpdrz\" (UniqueName: \"kubernetes.io/projected/17570aba-70d9-46ea-a0ca-3a465635a17c-kube-api-access-zpdrz\") pod \"migrator-66df5d496b-4mmw7\" (UID: \"17570aba-70d9-46ea-a0ca-3a465635a17c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4mmw7"
Nov 01 14:16:50.861453 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:50.861427    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khv87\" (UniqueName: \"kubernetes.io/projected/fba1e7f9-ca37-46dc-8c66-56f1d9563acd-kube-api-access-khv87\") pod \"migrator-66df5d496b-8rp2x\" (UID: \"fba1e7f9-ca37-46dc-8c66-56f1d9563acd\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8rp2x"
Nov 01 14:16:50.861453 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:50.861401    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 01 14:16:50.861701 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:50.861460    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/058820ef-eef1-41ca-b331-670bab146861-insights-runtime-extractor-tls podName:058820ef-eef1-41ca-b331-670bab146861 nodeName:}" failed. No retries permitted until 2026-11-01 14:16:52.86144065 +0000 UTC m=+43.021381291 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/058820ef-eef1-41ca-b331-670bab146861-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zq4pm" (UID: "058820ef-eef1-41ca-b331-670bab146861") : secret "insights-runtime-extractor-tls" not found
Nov 01 14:16:50.861701 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:50.861484    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/058820ef-eef1-41ca-b331-670bab146861-kube-rbac-proxy-cm podName:058820ef-eef1-41ca-b331-670bab146861 nodeName:}" failed. No retries permitted until 2026-11-01 14:16:52.861470378 +0000 UTC m=+43.021411032 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/058820ef-eef1-41ca-b331-670bab146861-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zq4pm" (UID: "058820ef-eef1-41ca-b331-670bab146861") : configmap "kube-rbac-proxy" not found
Nov 01 14:16:50.920809 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:50.920775    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpdrz\" (UniqueName: \"kubernetes.io/projected/17570aba-70d9-46ea-a0ca-3a465635a17c-kube-api-access-zpdrz\") pod \"migrator-66df5d496b-4mmw7\" (UID: \"17570aba-70d9-46ea-a0ca-3a465635a17c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4mmw7"
Nov 01 14:16:50.920985 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:50.920947    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khv87\" (UniqueName: \"kubernetes.io/projected/fba1e7f9-ca37-46dc-8c66-56f1d9563acd-kube-api-access-khv87\") pod \"migrator-66df5d496b-8rp2x\" (UID: \"fba1e7f9-ca37-46dc-8c66-56f1d9563acd\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8rp2x"
Nov 01 14:16:51.026292 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:51.026260    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4mmw7"
Nov 01 14:16:51.061826 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:51.061798    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8rp2x"
Nov 01 14:16:51.415418 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:51.415392    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-4mmw7"]
Nov 01 14:16:51.525970 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:51.525787    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-8rp2x"]
Nov 01 14:16:51.528734 ip-10-0-134-37 kubenswrapper[2576]: W1101 14:16:51.528692    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfba1e7f9_ca37_46dc_8c66_56f1d9563acd.slice/crio-ccb76443164367970060f9ea26b8693190d38ed29108de716b17d69abc051c04 WatchSource:0}: Error finding container ccb76443164367970060f9ea26b8693190d38ed29108de716b17d69abc051c04: Status 404 returned error can't find the container with id ccb76443164367970060f9ea26b8693190d38ed29108de716b17d69abc051c04
Nov 01 14:16:51.600765 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:51.600725    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-rlgcn-h5sk4" event={"ID":"384cd50b-370f-474f-b80e-89a544a9bbbd","Type":"ContainerStarted","Data":"ede52eea27808ab3afc48dbf2065c4c6f1cc4b6c8c2f925d899e121398b002bb"}
Nov 01 14:16:51.602829 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:51.602798    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8rp2x" event={"ID":"fba1e7f9-ca37-46dc-8c66-56f1d9563acd","Type":"ContainerStarted","Data":"ccb76443164367970060f9ea26b8693190d38ed29108de716b17d69abc051c04"}
Nov 01 14:16:51.605190 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:51.605141    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8mqxj" event={"ID":"0d4352db-d2e7-43ba-97ba-9e344abbac8b","Type":"ContainerStarted","Data":"3fd0647235b7477bf76673ae96f4f1fc473783bad177ee8503f3f8387505eeff"}
Nov 01 14:16:51.605311 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:51.605293    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8mqxj"
Nov 01 14:16:51.607239 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:51.607213    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4mmw7" event={"ID":"17570aba-70d9-46ea-a0ca-3a465635a17c","Type":"ContainerStarted","Data":"0aea4ba3a0f3f845dfc67cf48042c64292a3373bea0a78fe95a6a0a1ce6fdfaf"}
Nov 01 14:16:51.725581 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:51.725526    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-rlgcn-h5sk4" podStartSLOduration=2.725511974 podStartE2EDuration="2.725511974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:16:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:16:51.724060552 +0000 UTC m=+41.884001216" watchObservedRunningTime="2026-11-01 14:16:51.725511974 +0000 UTC m=+41.885452635"
Nov 01 14:16:51.727395 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:51.727359    2576 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-xmwtp"
Nov 01 14:16:51.727841 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:51.727808    2576 scope.go:122] "RemoveContainer" containerID="cbc84fec5c099225a4e49365bdccb7876e559e9347bbfff1ca76af14561a16c1"
Nov 01 14:16:51.728046 ip-10-0-134-37 kubenswrapper[2576]: E1101 14:16:51.728024    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-xmwtp_openshift-console-operator(bb59051d-c2df-4362-a157-00f14f915e79)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-xmwtp" podUID="bb59051d-c2df-4362-a157-00f14f915e79"
Nov 01 14:16:51.751216 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:51.751189    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-l25pw"]
Nov 01 14:16:51.772605 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:51.772570    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-l25pw"
Nov 01 14:16:51.807289 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:51.807251    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-l25pw"]
Nov 01 14:16:51.824931 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:51.824887    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Nov 01 14:16:51.825282 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:51.825250    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Nov 01 14:16:51.825393 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:51.825287    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-4cd6v\""
Nov 01 14:16:51.825393 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:51.825320    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Nov 01 14:16:51.825487 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:51.825450    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Nov 01 14:16:51.837622 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:51.837564    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-8mqxj" podStartSLOduration=35.08026349 podStartE2EDuration="41.837550691s" totalImagesPullingTime="6.757287201s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:16:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:16:51.834830332 +0000 UTC m=+41.994770997" watchObservedRunningTime="2026-11-01 14:16:51.837550691 +0000 UTC m=+41.997491352"
Nov 01 14:16:51.873739 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:51.873699    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bhpq\" (UniqueName: \"kubernetes.io/projected/321ecf30-5108-4320-9a58-6b204001639c-kube-api-access-8bhpq\") pod \"service-ca-5fffc97bdf-l25pw\" (UID: \"321ecf30-5108-4320-9a58-6b204001639c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-l25pw"
Nov 01 14:16:51.873911 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:51.873867    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/321ecf30-5108-4320-9a58-6b204001639c-signing-cabundle\") pod \"service-ca-5fffc97bdf-l25pw\" (UID: \"321ecf30-5108-4320-9a58-6b204001639c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-l25pw"
Nov 01 14:16:51.874052 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:51.874023    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/321ecf30-5108-4320-9a58-6b204001639c-signing-key\") pod \"service-ca-5fffc97bdf-l25pw\" (UID: \"321ecf30-5108-4320-9a58-6b204001639c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-l25pw"
Nov 01 14:16:51.974719 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:51.974642    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bhpq\" (UniqueName: \"kubernetes.io/projected/321ecf30-5108-4320-9a58-6b204001639c-kube-api-access-8bhpq\") pod \"service-ca-5fffc97bdf-l25pw\" (UID: \"321ecf30-5108-4320-9a58-6b204001639c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-l25pw"
Nov 01 14:16:51.974858 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:51.974736    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/321ecf30-5108-4320-9a58-6b204001639c-signing-cabundle\") pod \"service-ca-5fffc97bdf-l25pw\" (UID: \"321ecf30-5108-4320-9a58-6b204001639c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-l25pw"
Nov 01 14:16:51.974975 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:51.974947    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/321ecf30-5108-4320-9a58-6b204001639c-signing-key\") pod \"service-ca-5fffc97bdf-l25pw\" (UID: \"321ecf30-5108-4320-9a58-6b204001639c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-l25pw"
Nov 01 14:16:51.975440 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:51.975414    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/321ecf30-5108-4320-9a58-6b204001639c-signing-cabundle\") pod \"service-ca-5fffc97bdf-l25pw\" (UID: \"321ecf30-5108-4320-9a58-6b204001639c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-l25pw"
Nov 01 14:16:51.977163 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:51.977143    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/321ecf30-5108-4320-9a58-6b204001639c-signing-key\") pod \"service-ca-5fffc97bdf-l25pw\" (UID: \"321ecf30-5108-4320-9a58-6b204001639c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-l25pw"
Nov 01 14:16:52.031010 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:52.030982    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bhpq\" (UniqueName: \"kubernetes.io/projected/321ecf30-5108-4320-9a58-6b204001639c-kube-api-access-8bhpq\") pod \"service-ca-5fffc97bdf-l25pw\" (UID: \"321ecf30-5108-4320-9a58-6b204001639c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-l25pw"
Nov 01 14:16:52.089142 ip-10-0-134-37 kubenswrapper[2576]: I1101 14:16:52.089107    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-l25pw"
Nov 01 14:16:52.589966 ip-10-0-134-37 systemd[1]: Stopping Kubernetes Kubelet...
Nov 01 14:16:52.597792 ip-10-0-134-37 systemd[1]: kubelet.service: Deactivated successfully.
Nov 01 14:16:52.598001 ip-10-0-134-37 systemd[1]: Stopped Kubernetes Kubelet.
Nov 01 14:16:52.598062 ip-10-0-134-37 systemd[1]: kubelet.service: Consumed 3.142s CPU time.
Nov 01 14:16:52.607895 ip-10-0-134-37 systemd[1]: Starting Kubernetes Kubelet...
Nov 01 14:16:52.700189 ip-10-0-134-37 kubenswrapper[4993]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 14:16:52.700189 ip-10-0-134-37 kubenswrapper[4993]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 01 14:16:52.700189 ip-10-0-134-37 kubenswrapper[4993]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 14:16:52.700189 ip-10-0-134-37 kubenswrapper[4993]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 14:16:52.702438 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702351    4993 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 14:16:52.702498 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702444    4993 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 14:16:52.702498 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702448    4993 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 14:16:52.702498 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702453    4993 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 14:16:52.702498 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702455    4993 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 14:16:52.702498 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702459    4993 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 14:16:52.702498 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702461    4993 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 14:16:52.702498 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702464    4993 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 14:16:52.702498 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702468    4993 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 14:16:52.702498 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702471    4993 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 14:16:52.702498 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702474    4993 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 14:16:52.702498 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702477    4993 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 14:16:52.702498 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702480    4993 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 14:16:52.702498 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702482    4993 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 14:16:52.702498 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702485    4993 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 14:16:52.702498 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702488    4993 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 14:16:52.702498 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702490    4993 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 14:16:52.702498 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702493    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 14:16:52.702498 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702495    4993 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 14:16:52.702498 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702498    4993 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 14:16:52.702498 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702504    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 14:16:52.703001 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702507    4993 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 14:16:52.703001 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702510    4993 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 14:16:52.703001 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702513    4993 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 14:16:52.703001 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702516    4993 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 14:16:52.703001 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702518    4993 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 14:16:52.703001 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702521    4993 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 14:16:52.703001 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702524    4993 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 14:16:52.703001 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702526    4993 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 14:16:52.703001 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702529    4993 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 14:16:52.703001 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702531    4993 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 14:16:52.703001 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702534    4993 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 14:16:52.703001 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702536    4993 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 14:16:52.703001 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702539    4993 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 14:16:52.703001 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702541    4993 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 14:16:52.703001 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702544    4993 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 14:16:52.703001 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702546    4993 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 14:16:52.703001 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702549    4993 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 14:16:52.703001 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702551    4993 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 14:16:52.703001 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702560    4993 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 14:16:52.703001 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702562    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 14:16:52.703499 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702565    4993 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 14:16:52.703499 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702567    4993 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 14:16:52.703499 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702570    4993 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 14:16:52.703499 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702572    4993 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 14:16:52.703499 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702582    4993 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 14:16:52.703499 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702585    4993 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 14:16:52.703499 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702588    4993 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 14:16:52.703499 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702590    4993 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 14:16:52.703499 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702593    4993 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 14:16:52.703499 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702595    4993 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 14:16:52.703499 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702598    4993 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 14:16:52.703499 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702600    4993 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 14:16:52.703499 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702603    4993 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 14:16:52.703499 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702609    4993 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 14:16:52.703499 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702611    4993 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 14:16:52.703499 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702614    4993 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 14:16:52.703499 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702616    4993 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 14:16:52.703499 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702619    4993 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 14:16:52.703499 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702621    4993 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 14:16:52.703499 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702624    4993 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 14:16:52.703990 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702627    4993 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 14:16:52.703990 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702630    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 14:16:52.703990 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702633    4993 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 14:16:52.703990 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702635    4993 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 14:16:52.703990 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702638    4993 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 14:16:52.703990 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702640    4993 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 14:16:52.703990 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702643    4993 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 14:16:52.703990 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702646    4993 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 14:16:52.703990 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702648    4993 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 14:16:52.703990 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702651    4993 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 14:16:52.703990 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702653    4993 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 14:16:52.703990 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702666    4993 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 14:16:52.703990 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702669    4993 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 14:16:52.703990 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702672    4993 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 14:16:52.703990 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702674    4993 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 14:16:52.703990 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702677    4993 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 14:16:52.703990 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702680    4993 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 14:16:52.703990 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702683    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 14:16:52.703990 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702685    4993 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 14:16:52.704466 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702689    4993 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 14:16:52.704466 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702694    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 14:16:52.704466 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702698    4993 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 14:16:52.704466 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702702    4993 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 14:16:52.704466 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702705    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 14:16:52.704466 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702708    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 14:16:52.704466 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702713    4993 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 14:16:52.704466 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702716    4993 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 14:16:52.704466 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702718    4993 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 14:16:52.704466 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702721    4993 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 14:16:52.704466 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702723    4993 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 14:16:52.704466 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702726    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 14:16:52.704466 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702729    4993 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 14:16:52.704466 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702731    4993 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 14:16:52.704466 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702734    4993 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 14:16:52.704466 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702736    4993 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 14:16:52.704466 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702739    4993 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 14:16:52.704466 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702741    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 14:16:52.704466 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702744    4993 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 14:16:52.704937 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702746    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 14:16:52.704937 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702749    4993 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 14:16:52.704937 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702751    4993 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 14:16:52.704937 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702754    4993 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 14:16:52.704937 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702757    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 14:16:52.704937 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702760    4993 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 14:16:52.704937 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.702768    4993 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 14:16:52.704937 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703586    4993 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 14:16:52.704937 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703593    4993 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 14:16:52.704937 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703596    4993 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 14:16:52.704937 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703600    4993 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 14:16:52.704937 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703603    4993 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 14:16:52.704937 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703606    4993 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 14:16:52.704937 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703609    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 14:16:52.704937 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703612    4993 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 14:16:52.704937 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703615    4993 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 14:16:52.704937 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703617    4993 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 14:16:52.704937 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703620    4993 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 14:16:52.704937 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703622    4993 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 14:16:52.704937 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703626    4993 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 14:16:52.705437 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703629    4993 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 14:16:52.705437 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703632    4993 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 14:16:52.705437 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703635    4993 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 14:16:52.705437 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703637    4993 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 14:16:52.705437 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703640    4993 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 14:16:52.705437 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703643    4993 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 14:16:52.705437 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703645    4993 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 14:16:52.705437 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703648    4993 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 14:16:52.705437 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703650    4993 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 14:16:52.705437 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703653    4993 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 14:16:52.705437 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703655    4993 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 14:16:52.705437 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703658    4993 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 14:16:52.705437 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703661    4993 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 14:16:52.705437 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703663    4993 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 14:16:52.705437 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703666    4993 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 14:16:52.705437 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703668    4993 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 14:16:52.705437 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703671    4993 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 14:16:52.705437 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703673    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 14:16:52.705437 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703676    4993 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 14:16:52.705993 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703686    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 14:16:52.705993 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703689    4993 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 14:16:52.705993 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703691    4993 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 14:16:52.705993 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703694    4993 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 14:16:52.705993 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703697    4993 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 14:16:52.705993 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703700    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 14:16:52.705993 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703703    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 14:16:52.705993 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703705    4993 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 14:16:52.705993 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703708    4993 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 14:16:52.705993 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703710    4993 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 14:16:52.705993 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703713    4993 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 14:16:52.705993 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703716    4993 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 14:16:52.705993 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703718    4993 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 14:16:52.705993 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703721    4993 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 14:16:52.705993 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703724    4993 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 14:16:52.705993 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703726    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 14:16:52.705993 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703729    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 14:16:52.705993 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703731    4993 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 14:16:52.705993 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703734    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 14:16:52.706772 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703737    4993 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 14:16:52.706772 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703739    4993 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 14:16:52.706772 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703742    4993 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 14:16:52.706772 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703745    4993 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 14:16:52.706772 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703747    4993 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 14:16:52.706772 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703750    4993 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 14:16:52.706772 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703753    4993 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 14:16:52.706772 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703755    4993 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 14:16:52.706772 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703758    4993 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 14:16:52.706772 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703761    4993 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 14:16:52.706772 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703766    4993 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 14:16:52.706772 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703769    4993 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 14:16:52.706772 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703772    4993 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 14:16:52.706772 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703775    4993 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 14:16:52.706772 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703778    4993 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 14:16:52.706772 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703781    4993 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 14:16:52.706772 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703783    4993 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 14:16:52.706772 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703786    4993 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 14:16:52.706772 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703788    4993 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 14:16:52.706772 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703791    4993 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 14:16:52.707298 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703794    4993 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 14:16:52.707298 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703796    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 14:16:52.707298 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703799    4993 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 14:16:52.707298 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703801    4993 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 14:16:52.707298 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703804    4993 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 14:16:52.707298 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703807    4993 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 14:16:52.707298 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703810    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 14:16:52.707298 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703813    4993 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 14:16:52.707298 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703816    4993 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 14:16:52.707298 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703818    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 14:16:52.707298 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703821    4993 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 14:16:52.707298 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703824    4993 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 14:16:52.707298 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703826    4993 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 14:16:52.707298 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703829    4993 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 14:16:52.707298 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703831    4993 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 14:16:52.707298 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703835    4993 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 14:16:52.707298 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703839    4993 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 14:16:52.707298 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703842    4993 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 14:16:52.707298 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703845    4993 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 14:16:52.707755 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703848    4993 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 14:16:52.707755 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703850    4993 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 14:16:52.707755 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703853    4993 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 14:16:52.707755 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703856    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 14:16:52.707755 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703858    4993 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 14:16:52.707755 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703861    4993 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 14:16:52.707755 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703863    4993 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 14:16:52.707755 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703867    4993 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 14:16:52.707755 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703870    4993 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 14:16:52.707755 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703873    4993 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 14:16:52.707755 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703876    4993 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 14:16:52.707755 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703878    4993 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 14:16:52.707755 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703881    4993 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 14:16:52.707755 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703883    4993 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 14:16:52.707755 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703885    4993 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 14:16:52.707755 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.703888    4993 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 14:16:52.707755 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704490    4993 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 14:16:52.707755 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704496    4993 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 14:16:52.707755 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704499    4993 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 14:16:52.708228 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704502    4993 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 14:16:52.708228 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704505    4993 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 14:16:52.708228 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704510    4993 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 14:16:52.708228 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704513    4993 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 14:16:52.708228 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704516    4993 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 14:16:52.708228 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704519    4993 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 14:16:52.708228 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704521    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 14:16:52.708228 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704524    4993 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 14:16:52.708228 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704527    4993 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 14:16:52.708228 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704529    4993 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 14:16:52.708228 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704532    4993 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 14:16:52.708228 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704534    4993 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 14:16:52.708228 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704537    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 14:16:52.708228 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704539    4993 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 14:16:52.708228 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704542    4993 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 14:16:52.708228 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704545    4993 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 14:16:52.708228 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704549    4993 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 14:16:52.708228 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704553    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 14:16:52.708228 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704556    4993 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 14:16:52.708730 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704561    4993 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 14:16:52.708730 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704566    4993 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 14:16:52.708730 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704577    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 14:16:52.708730 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704581    4993 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 14:16:52.708730 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704583    4993 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 14:16:52.708730 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704586    4993 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 14:16:52.708730 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704589    4993 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 14:16:52.708730 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704591    4993 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 14:16:52.708730 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704594    4993 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 14:16:52.708730 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704596    4993 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 14:16:52.708730 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704599    4993 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 14:16:52.708730 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704602    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 14:16:52.708730 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704606    4993 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 14:16:52.708730 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704609    4993 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 14:16:52.708730 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704611    4993 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 14:16:52.708730 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704615    4993 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 14:16:52.708730 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704618    4993 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 14:16:52.708730 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704620    4993 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 14:16:52.708730 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704623    4993 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 14:16:52.708730 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704626    4993 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 14:16:52.709214 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704628    4993 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 14:16:52.709214 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704631    4993 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 14:16:52.709214 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704633    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 14:16:52.709214 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704636    4993 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 14:16:52.709214 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704639    4993 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 14:16:52.709214 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704641    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 14:16:52.709214 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704644    4993 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 14:16:52.709214 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704646    4993 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 14:16:52.709214 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704649    4993 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 14:16:52.709214 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704651    4993 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 14:16:52.709214 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704653    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 14:16:52.709214 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704656    4993 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 14:16:52.709214 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704659    4993 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 14:16:52.709214 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704661    4993 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 14:16:52.709214 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704664    4993 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 14:16:52.709214 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704671    4993 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 14:16:52.709214 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704674    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 14:16:52.709214 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704677    4993 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 14:16:52.709214 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704679    4993 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 14:16:52.709214 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704682    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 14:16:52.709707 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704684    4993 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 14:16:52.709707 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704687    4993 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 14:16:52.709707 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704689    4993 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 14:16:52.709707 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704692    4993 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 14:16:52.709707 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704696    4993 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 14:16:52.709707 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704699    4993 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 14:16:52.709707 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704701    4993 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 14:16:52.709707 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704705    4993 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 14:16:52.709707 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704708    4993 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 14:16:52.709707 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704711    4993 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 14:16:52.709707 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704713    4993 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 14:16:52.709707 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704716    4993 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 14:16:52.709707 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704719    4993 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 14:16:52.709707 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704722    4993 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 14:16:52.709707 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704724    4993 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 14:16:52.709707 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704727    4993 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 14:16:52.709707 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704730    4993 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 14:16:52.709707 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704732    4993 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 14:16:52.709707 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704735    4993 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 14:16:52.709707 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704737    4993 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 14:16:52.710268 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704740    4993 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 14:16:52.710268 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704742    4993 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 14:16:52.710268 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704745    4993 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 14:16:52.710268 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704748    4993 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 14:16:52.710268 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704750    4993 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 14:16:52.710268 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704752    4993 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 14:16:52.710268 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704755    4993 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 14:16:52.710268 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704757    4993 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 14:16:52.710268 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704765    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 14:16:52.710268 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704770    4993 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 14:16:52.710268 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704774    4993 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 14:16:52.710268 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704777    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 14:16:52.710268 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704781    4993 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 14:16:52.710268 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704784    4993 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 14:16:52.710268 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704786    4993 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 14:16:52.710268 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704789    4993 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 14:16:52.710268 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704793    4993 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 14:16:52.710268 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704795    4993 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 14:16:52.710268 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704798    4993 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 14:16:52.710730 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704801    4993 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 14:16:52.710730 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704804    4993 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 14:16:52.710730 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704806    4993 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 14:16:52.710730 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704809    4993 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 14:16:52.710730 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.704811    4993 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 14:16:52.710730 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.704909    4993 flags.go:64] FLAG: --address="0.0.0.0"
Nov 01 14:16:52.710730 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.704916    4993 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 01 14:16:52.710730 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.704925    4993 flags.go:64] FLAG: --anonymous-auth="true"
Nov 01 14:16:52.710730 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.704930    4993 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 01 14:16:52.710730 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.704934    4993 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 01 14:16:52.710730 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.704938    4993 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 01 14:16:52.710730 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.704943    4993 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 01 14:16:52.710730 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.704947    4993 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 01 14:16:52.710730 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.704950    4993 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 01 14:16:52.710730 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.704953    4993 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 01 14:16:52.710730 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.704957    4993 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 01 14:16:52.710730 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.704960    4993 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 01 14:16:52.710730 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.704963    4993 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 01 14:16:52.710730 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.704966    4993 flags.go:64] FLAG: --cgroup-root=""
Nov 01 14:16:52.710730 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.704969    4993 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 01 14:16:52.710730 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.704973    4993 flags.go:64] FLAG: --client-ca-file=""
Nov 01 14:16:52.710730 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.704976    4993 flags.go:64] FLAG: --cloud-provider="external"
Nov 01 14:16:52.710730 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.704985    4993 flags.go:64] FLAG: --cluster-dns="[]"
Nov 01 14:16:52.710730 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.704992    4993 flags.go:64] FLAG: --cluster-domain=""
Nov 01 14:16:52.711324 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.704995    4993 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 01 14:16:52.711324 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.704998    4993 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 01 14:16:52.711324 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705002    4993 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 01 14:16:52.711324 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705005    4993 flags.go:64] FLAG: --container-log-max-files="5"
Nov 01 14:16:52.711324 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705009    4993 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 01 14:16:52.711324 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705013    4993 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 01 14:16:52.711324 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705017    4993 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 01 14:16:52.711324 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705020    4993 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 01 14:16:52.711324 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705024    4993 flags.go:64] FLAG: --contention-profiling="false"
Nov 01 14:16:52.711324 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705027    4993 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 01 14:16:52.711324 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705030    4993 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 01 14:16:52.711324 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705033    4993 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 01 14:16:52.711324 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705036    4993 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 01 14:16:52.711324 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705041    4993 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 01 14:16:52.711324 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705044    4993 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 01 14:16:52.711324 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705046    4993 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 01 14:16:52.711324 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705049    4993 flags.go:64] FLAG: --enable-load-reader="false"
Nov 01 14:16:52.711324 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705052    4993 flags.go:64] FLAG: --enable-server="true"
Nov 01 14:16:52.711324 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705055    4993 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 01 14:16:52.711324 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705062    4993 flags.go:64] FLAG: --event-burst="100"
Nov 01 14:16:52.711324 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705065    4993 flags.go:64] FLAG: --event-qps="50"
Nov 01 14:16:52.711324 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705068    4993 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 01 14:16:52.711324 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705071    4993 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 01 14:16:52.711324 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705074    4993 flags.go:64] FLAG: --eviction-hard=""
Nov 01 14:16:52.711901 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705078    4993 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 01 14:16:52.711901 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705081    4993 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 01 14:16:52.711901 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705084    4993 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 01 14:16:52.711901 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705087    4993 flags.go:64] FLAG: --eviction-soft=""
Nov 01 14:16:52.711901 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705090    4993 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 01 14:16:52.711901 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705093    4993 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 01 14:16:52.711901 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705096    4993 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 01 14:16:52.711901 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705105    4993 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 01 14:16:52.711901 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705108    4993 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 01 14:16:52.711901 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705111    4993 flags.go:64] FLAG: --fail-swap-on="true"
Nov 01 14:16:52.711901 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705114    4993 flags.go:64] FLAG: --feature-gates=""
Nov 01 14:16:52.711901 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705118    4993 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 01 14:16:52.711901 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705121    4993 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 01 14:16:52.711901 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705126    4993 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 01 14:16:52.711901 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705129    4993 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 01 14:16:52.711901 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705132    4993 flags.go:64] FLAG: --healthz-port="10248"
Nov 01 14:16:52.711901 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705135    4993 flags.go:64] FLAG: --help="false"
Nov 01 14:16:52.711901 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705139    4993 flags.go:64] FLAG: --hostname-override="ip-10-0-134-37.us-west-2.compute.internal"
Nov 01 14:16:52.711901 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705142    4993 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 01 14:16:52.711901 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705145    4993 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 01 14:16:52.711901 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705148    4993 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 01 14:16:52.711901 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705151    4993 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 01 14:16:52.711901 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705155    4993 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 01 14:16:52.712467 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705158    4993 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 01 14:16:52.712467 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705161    4993 flags.go:64] FLAG: --image-service-endpoint=""
Nov 01 14:16:52.712467 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705176    4993 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 01 14:16:52.712467 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705180    4993 flags.go:64] FLAG: --kube-api-burst="100"
Nov 01 14:16:52.712467 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705183    4993 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 01 14:16:52.712467 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705186    4993 flags.go:64] FLAG: --kube-api-qps="50"
Nov 01 14:16:52.712467 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705189    4993 flags.go:64] FLAG: --kube-reserved=""
Nov 01 14:16:52.712467 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705192    4993 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 01 14:16:52.712467 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705195    4993 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 01 14:16:52.712467 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705198    4993 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 01 14:16:52.712467 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705201    4993 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 01 14:16:52.712467 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705204    4993 flags.go:64] FLAG: --lock-file=""
Nov 01 14:16:52.712467 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705207    4993 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 01 14:16:52.712467 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705210    4993 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 01 14:16:52.712467 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705213    4993 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 01 14:16:52.712467 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705218    4993 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 01 14:16:52.712467 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705221    4993 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 01 14:16:52.712467 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705230    4993 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 01 14:16:52.712467 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705233    4993 flags.go:64] FLAG: --logging-format="text"
Nov 01 14:16:52.712467 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705236    4993 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 01 14:16:52.712467 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705239    4993 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 01 14:16:52.712467 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705242    4993 flags.go:64] FLAG: --manifest-url=""
Nov 01 14:16:52.712467 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705246    4993 flags.go:64] FLAG: --manifest-url-header=""
Nov 01 14:16:52.712467 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705250    4993 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 01 14:16:52.712467 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705254    4993 flags.go:64] FLAG: --max-open-files="1000000"
Nov 01 14:16:52.713071 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705257    4993 flags.go:64] FLAG: --max-pods="110"
Nov 01 14:16:52.713071 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705260    4993 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 01 14:16:52.713071 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705263    4993 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 01 14:16:52.713071 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705266    4993 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 01 14:16:52.713071 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705269    4993 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 01 14:16:52.713071 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705272    4993 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 01 14:16:52.713071 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705275    4993 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 01 14:16:52.713071 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705278    4993 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 01 14:16:52.713071 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705285    4993 flags.go:64] FLAG: --node-status-max-images="50"
Nov 01 14:16:52.713071 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705288    4993 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 01 14:16:52.713071 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705291    4993 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 01 14:16:52.713071 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705294    4993 flags.go:64] FLAG: --pod-cidr=""
Nov 01 14:16:52.713071 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705297    4993 flags.go:64] FLAG: --pod-manifest-path=""
Nov 01 14:16:52.713071 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705300    4993 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 01 14:16:52.713071 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705303    4993 flags.go:64] FLAG: --pods-per-core="0"
Nov 01 14:16:52.713071 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705306    4993 flags.go:64] FLAG: --port="10250"
Nov 01 14:16:52.713071 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705309    4993 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 01 14:16:52.713071 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705312    4993 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-06bb2c418a73ad5ed"
Nov 01 14:16:52.713071 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705315    4993 flags.go:64] FLAG: --qos-reserved=""
Nov 01 14:16:52.713071 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705318    4993 flags.go:64] FLAG: --read-only-port="10255"
Nov 01 14:16:52.713071 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705321    4993 flags.go:64] FLAG: --register-node="true"
Nov 01 14:16:52.713071 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705324    4993 flags.go:64] FLAG: --register-with-taints=""
Nov 01 14:16:52.713071 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705330    4993 flags.go:64] FLAG: --registry-burst="10"
Nov 01 14:16:52.713071 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705333    4993 flags.go:64] FLAG: --registry-qps="5"
Nov 01 14:16:52.713071 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705336    4993 flags.go:64] FLAG: --reserved-cpus=""
Nov 01 14:16:52.713071 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705345    4993 flags.go:64] FLAG: --reserved-memory=""
Nov 01 14:16:52.713695 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705349    4993 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 01 14:16:52.713695 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705352    4993 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 01 14:16:52.713695 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705355    4993 flags.go:64] FLAG: --rotate-certificates="false"
Nov 01 14:16:52.713695 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705359    4993 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 01 14:16:52.713695 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705362    4993 flags.go:64] FLAG: --runonce="false"
Nov 01 14:16:52.713695 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705364    4993 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 01 14:16:52.713695 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705368    4993 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 01 14:16:52.713695 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705370    4993 flags.go:64] FLAG: --seccomp-default="false"
Nov 01 14:16:52.713695 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705373    4993 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 01 14:16:52.713695 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705376    4993 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 01 14:16:52.713695 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705379    4993 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 01 14:16:52.713695 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705382    4993 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 01 14:16:52.713695 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705388    4993 flags.go:64] FLAG: --storage-driver-password="root"
Nov 01 14:16:52.713695 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705391    4993 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 01 14:16:52.713695 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705394    4993 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 01 14:16:52.713695 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705397    4993 flags.go:64] FLAG: --storage-driver-user="root"
Nov 01 14:16:52.713695 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705400    4993 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 01 14:16:52.713695 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705403    4993 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 01 14:16:52.713695 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705407    4993 flags.go:64] FLAG: --system-cgroups=""
Nov 01 14:16:52.713695 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705410    4993 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 01 14:16:52.713695 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705415    4993 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 01 14:16:52.713695 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705418    4993 flags.go:64] FLAG: --tls-cert-file=""
Nov 01 14:16:52.713695 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705421    4993 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 01 14:16:52.713695 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705428    4993 flags.go:64] FLAG: --tls-min-version=""
Nov 01 14:16:52.713695 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705431    4993 flags.go:64] FLAG: --tls-private-key-file=""
Nov 01 14:16:52.714366 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705434    4993 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 01 14:16:52.714366 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705437    4993 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 01 14:16:52.714366 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705440    4993 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 01 14:16:52.714366 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705443    4993 flags.go:64] FLAG: --v="2"
Nov 01 14:16:52.714366 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705448    4993 flags.go:64] FLAG: --version="false"
Nov 01 14:16:52.714366 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705452    4993 flags.go:64] FLAG: --vmodule=""
Nov 01 14:16:52.714366 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705456    4993 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 01 14:16:52.714366 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.705466    4993 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 01 14:16:52.714366 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.705656    4993 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 14:16:52.714366 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.705660    4993 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 14:16:52.714366 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.705665    4993 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 14:16:52.714366 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.705668    4993 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 14:16:52.714366 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.705670    4993 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 14:16:52.714366 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.705673    4993 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 14:16:52.714366 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.705676    4993 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 14:16:52.714366 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.705678    4993 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 14:16:52.714366 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.705681    4993 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 14:16:52.714366 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.705684    4993 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 14:16:52.714366 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.705687    4993 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 14:16:52.714366 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.705691    4993 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 14:16:52.714366 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.705694    4993 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 14:16:52.714366 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.705696    4993 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 14:16:52.714954 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.705699    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 14:16:52.714954 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.705701    4993 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 14:16:52.714954 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.705704    4993 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 14:16:52.714954 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.705707    4993 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 14:16:52.714954 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.705710    4993 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 14:16:52.714954 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.705712    4993 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 14:16:52.714954 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.705715    4993 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 14:16:52.714954 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.705718    4993 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 14:16:52.714954 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.705720    4993 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 14:16:52.714954 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.705723    4993 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 14:16:52.714954 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.705726    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 14:16:52.714954 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.705729    4993 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 14:16:52.714954 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.705731    4993 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 14:16:52.714954 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.706075    4993 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 14:16:52.714954 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.706101    4993 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 14:16:52.714954 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.706106    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 14:16:52.714954 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.706112    4993 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 14:16:52.714954 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.706117    4993 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 14:16:52.714954 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.706156    4993 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 14:16:52.714954 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.706161    4993 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 14:16:52.715451 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.706181    4993 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 14:16:52.715451 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.706185    4993 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 14:16:52.715451 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.706190    4993 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 14:16:52.715451 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.706194    4993 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 14:16:52.715451 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.706198    4993 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 14:16:52.715451 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.706209    4993 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 14:16:52.715451 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.706214    4993 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 14:16:52.715451 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.706218    4993 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 14:16:52.715451 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.706223    4993 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 14:16:52.715451 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.706230    4993 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 14:16:52.715451 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.706235    4993 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 14:16:52.715451 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.706240    4993 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 14:16:52.715451 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.706244    4993 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 14:16:52.715451 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.706248    4993 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 14:16:52.715451 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.706253    4993 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 14:16:52.715451 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.706257    4993 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 14:16:52.715451 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.706261    4993 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 14:16:52.715451 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.706265    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 14:16:52.715451 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.706275    4993 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 14:16:52.715451 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.706279    4993 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 14:16:52.715972 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.706284    4993 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 14:16:52.715972 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.706288    4993 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 14:16:52.715972 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.706295    4993 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 14:16:52.715972 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.706301    4993 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 14:16:52.715972 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.706305    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 14:16:52.715972 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.706311    4993 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 14:16:52.715972 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.706315    4993 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 14:16:52.715972 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.706320    4993 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 14:16:52.715972 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.706324    4993 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 14:16:52.715972 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.706328    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 14:16:52.715972 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.706333    4993 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 14:16:52.715972 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.706368    4993 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 14:16:52.715972 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.706373    4993 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 14:16:52.715972 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.706377    4993 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 14:16:52.715972 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.706381    4993 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 14:16:52.715972 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.706385    4993 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 14:16:52.715972 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.706390    4993 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 14:16:52.715972 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.706394    4993 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 14:16:52.715972 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.706398    4993 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 14:16:52.716505 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.706403    4993 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 14:16:52.716505 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.706407    4993 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 14:16:52.716505 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.706414    4993 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 14:16:52.716505 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.706419    4993 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 14:16:52.716505 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.706428    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 14:16:52.716505 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.706432    4993 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 14:16:52.716505 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.706437    4993 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 14:16:52.716505 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.706441    4993 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 14:16:52.716505 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.706446    4993 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 14:16:52.716505 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.706450    4993 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 14:16:52.716505 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.706455    4993 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 14:16:52.716505 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.706459    4993 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 14:16:52.716505 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.706463    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 14:16:52.716505 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.706467    4993 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 14:16:52.716505 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.706471    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 14:16:52.716505 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.706476    4993 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 14:16:52.716505 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.706480    4993 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 14:16:52.716505 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.706489    4993 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 14:16:52.716505 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.706493    4993 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 14:16:52.717187 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.706497    4993 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 14:16:52.717187 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.706501    4993 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 14:16:52.717187 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.706505    4993 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 14:16:52.717187 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.706511    4993 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 14:16:52.717187 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.706517    4993 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 14:16:52.717187 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.706523    4993 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 14:16:52.717187 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.706956    4993 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 14:16:52.717187 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.706985    4993 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 14:16:52.717187 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707001    4993 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 14:16:52.717187 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707005    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 14:16:52.717187 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707009    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 14:16:52.717187 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707013    4993 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 14:16:52.717187 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707017    4993 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 14:16:52.717187 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707020    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 14:16:52.717187 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.707211    4993 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 14:16:52.717593 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707279    4993 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 14:16:52.717593 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707285    4993 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 14:16:52.717593 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707288    4993 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 14:16:52.717593 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707291    4993 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 14:16:52.717593 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707294    4993 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 14:16:52.717593 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707297    4993 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 14:16:52.717593 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707300    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 14:16:52.717593 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707303    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 14:16:52.717593 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707306    4993 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 14:16:52.717593 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707309    4993 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 14:16:52.717593 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707312    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 14:16:52.717593 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707315    4993 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 14:16:52.717593 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707317    4993 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 14:16:52.717593 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707320    4993 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 14:16:52.717593 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707325    4993 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 14:16:52.717593 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707328    4993 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 14:16:52.717593 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707331    4993 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 14:16:52.717593 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707334    4993 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 14:16:52.717593 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707337    4993 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 14:16:52.718048 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707339    4993 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 14:16:52.718048 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707343    4993 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 14:16:52.718048 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707345    4993 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 14:16:52.718048 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707348    4993 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 14:16:52.718048 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707351    4993 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 14:16:52.718048 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707353    4993 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 14:16:52.718048 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707356    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 14:16:52.718048 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707358    4993 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 14:16:52.718048 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707361    4993 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 14:16:52.718048 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707363    4993 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 14:16:52.718048 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707366    4993 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 14:16:52.718048 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707369    4993 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 14:16:52.718048 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707372    4993 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 14:16:52.718048 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707375    4993 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 14:16:52.718048 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707378    4993 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 14:16:52.718048 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707380    4993 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 14:16:52.718048 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707383    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 14:16:52.718048 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707386    4993 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 14:16:52.718048 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707388    4993 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 14:16:52.718048 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707391    4993 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 14:16:52.718670 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707393    4993 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 14:16:52.718670 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707396    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 14:16:52.718670 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707399    4993 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 14:16:52.718670 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707402    4993 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 14:16:52.718670 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707404    4993 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 14:16:52.718670 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707407    4993 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 14:16:52.718670 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707410    4993 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 14:16:52.718670 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707413    4993 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 14:16:52.718670 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707415    4993 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 14:16:52.718670 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707418    4993 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 14:16:52.718670 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707421    4993 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 14:16:52.718670 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707423    4993 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 14:16:52.718670 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707426    4993 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 14:16:52.718670 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707431    4993 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 14:16:52.718670 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707434    4993 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 14:16:52.718670 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707437    4993 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 14:16:52.718670 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707440    4993 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 14:16:52.718670 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707443    4993 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 14:16:52.718670 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707446    4993 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 14:16:52.719259 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707449    4993 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 14:16:52.719259 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707451    4993 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 14:16:52.719259 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707454    4993 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 14:16:52.719259 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707457    4993 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 14:16:52.719259 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707459    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 14:16:52.719259 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707462    4993 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 14:16:52.719259 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707465    4993 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 14:16:52.719259 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707467    4993 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 14:16:52.719259 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707470    4993 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 14:16:52.719259 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707474    4993 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 14:16:52.719259 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707479    4993 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 14:16:52.719259 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707482    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 14:16:52.719259 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707485    4993 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 14:16:52.719259 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707488    4993 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 14:16:52.719259 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707491    4993 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 14:16:52.719259 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707493    4993 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 14:16:52.719259 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707496    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 14:16:52.719259 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707498    4993 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 14:16:52.719259 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707501    4993 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 14:16:52.719768 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707504    4993 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 14:16:52.719768 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707507    4993 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 14:16:52.719768 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707509    4993 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 14:16:52.719768 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707512    4993 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 14:16:52.719768 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707515    4993 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 14:16:52.719768 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707517    4993 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 14:16:52.719768 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707520    4993 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 14:16:52.719768 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707524    4993 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 14:16:52.719768 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707527    4993 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 14:16:52.719768 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707529    4993 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 14:16:52.719768 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707532    4993 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 14:16:52.719768 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707535    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 14:16:52.719768 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707538    4993 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 14:16:52.719768 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707541    4993 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 14:16:52.719768 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707543    4993 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 14:16:52.719768 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707547    4993 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 14:16:52.719768 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707550    4993 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 14:16:52.719768 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707553    4993 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 14:16:52.719768 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707555    4993 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 14:16:52.720420 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707558    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 14:16:52.720420 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707560    4993 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 14:16:52.720420 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707563    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 14:16:52.720420 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707565    4993 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 14:16:52.720420 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707568    4993 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 14:16:52.720420 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707571    4993 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 14:16:52.720420 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707573    4993 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 14:16:52.720420 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707592    4993 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 14:16:52.720420 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707596    4993 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 14:16:52.720420 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.707600    4993 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 14:16:52.720420 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.711588    4993 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 01 14:16:52.720420 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.711604    4993 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 01 14:16:52.720420 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.711626    4993 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 01 14:16:52.720420 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.711631    4993 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 01 14:16:52.720420 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711685    4993 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 14:16:52.720420 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711691    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 14:16:52.720420 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711695    4993 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 14:16:52.720420 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711698    4993 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 14:16:52.720420 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711701    4993 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 14:16:52.720420 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711704    4993 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 14:16:52.720896 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711707    4993 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 14:16:52.720896 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711710    4993 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 14:16:52.720896 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711713    4993 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 14:16:52.720896 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711716    4993 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 14:16:52.720896 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711719    4993 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 14:16:52.720896 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711721    4993 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 14:16:52.720896 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711724    4993 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 14:16:52.720896 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711727    4993 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 14:16:52.720896 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711729    4993 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 14:16:52.720896 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711732    4993 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 14:16:52.720896 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711735    4993 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 14:16:52.720896 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711737    4993 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 14:16:52.720896 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711740    4993 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 14:16:52.720896 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711743    4993 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 14:16:52.720896 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711745    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 14:16:52.720896 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711748    4993 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 14:16:52.720896 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711750    4993 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 14:16:52.720896 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711753    4993 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 14:16:52.720896 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711756    4993 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 14:16:52.720896 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711759    4993 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 14:16:52.721551 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711762    4993 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 14:16:52.721551 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711765    4993 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 14:16:52.721551 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711767    4993 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 14:16:52.721551 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711770    4993 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 14:16:52.721551 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711773    4993 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 14:16:52.721551 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711776    4993 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 14:16:52.721551 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711779    4993 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 14:16:52.721551 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711782    4993 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 14:16:52.721551 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711784    4993 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 14:16:52.721551 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711787    4993 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 14:16:52.721551 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711790    4993 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 14:16:52.721551 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711793    4993 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 14:16:52.721551 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711795    4993 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 14:16:52.721551 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711798    4993 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 14:16:52.721551 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711801    4993 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 14:16:52.721551 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711804    4993 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 14:16:52.721551 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711806    4993 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 14:16:52.721551 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711809    4993 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 14:16:52.721551 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711811    4993 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 14:16:52.721551 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711814    4993 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 14:16:52.722370 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711816    4993 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 14:16:52.722370 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711820    4993 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 14:16:52.722370 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711822    4993 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 14:16:52.722370 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711825    4993 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 14:16:52.722370 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711827    4993 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 14:16:52.722370 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711830    4993 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 14:16:52.722370 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711832    4993 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 14:16:52.722370 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711837    4993 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 14:16:52.722370 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711841    4993 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 14:16:52.722370 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711844    4993 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 14:16:52.722370 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711847    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 14:16:52.722370 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711851    4993 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 14:16:52.722370 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711853    4993 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 14:16:52.722370 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711856    4993 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 14:16:52.722370 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711859    4993 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 14:16:52.722370 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711862    4993 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 14:16:52.722370 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711864    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 14:16:52.722370 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711867    4993 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 14:16:52.722370 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711870    4993 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 14:16:52.723045 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711872    4993 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 14:16:52.723045 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711875    4993 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 14:16:52.723045 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711877    4993 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 14:16:52.723045 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711880    4993 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 14:16:52.723045 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711883    4993 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 14:16:52.723045 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711885    4993 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 14:16:52.723045 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711888    4993 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 14:16:52.723045 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711890    4993 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 14:16:52.723045 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711893    4993 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 14:16:52.723045 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711896    4993 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 14:16:52.723045 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711898    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 14:16:52.723045 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711901    4993 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 14:16:52.723045 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711903    4993 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 14:16:52.723045 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711906    4993 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 14:16:52.723045 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711909    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 14:16:52.723045 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711912    4993 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 14:16:52.723045 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711915    4993 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 14:16:52.723045 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711920    4993 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 14:16:52.723045 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711924    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 14:16:52.723045 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711928    4993 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 14:16:52.723543 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711932    4993 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 14:16:52.723543 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711936    4993 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 14:16:52.723543 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711940    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 14:16:52.723543 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711944    4993 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 14:16:52.723543 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711947    4993 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 14:16:52.723543 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711950    4993 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 14:16:52.723543 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711953    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 14:16:52.723543 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711956    4993 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 14:16:52.723543 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711959    4993 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 14:16:52.723543 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711961    4993 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 14:16:52.723543 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711965    4993 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 14:16:52.723543 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711969    4993 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 14:16:52.723543 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711972    4993 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 14:16:52.723543 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711976    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 14:16:52.723543 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711980    4993 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 14:16:52.723543 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711984    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 14:16:52.723543 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711988    4993 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 14:16:52.723543 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711991    4993 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 14:16:52.723543 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.711996    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 14:16:52.724001 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712000    4993 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 14:16:52.724001 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712004    4993 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 14:16:52.724001 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.712040    4993 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 14:16:52.724001 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712198    4993 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 14:16:52.724001 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712203    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 14:16:52.724001 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712206    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 14:16:52.724001 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712209    4993 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 14:16:52.724001 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712212    4993 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 14:16:52.724001 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712214    4993 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 14:16:52.724001 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712217    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 14:16:52.724001 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712220    4993 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 14:16:52.724001 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712223    4993 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 14:16:52.724001 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712225    4993 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 14:16:52.724001 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712228    4993 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 14:16:52.724001 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712230    4993 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 14:16:52.724428 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712233    4993 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 14:16:52.724428 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712236    4993 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 14:16:52.724428 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712238    4993 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 14:16:52.724428 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712241    4993 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 14:16:52.724428 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712244    4993 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 14:16:52.724428 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712248    4993 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 14:16:52.724428 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712250    4993 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 14:16:52.724428 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712253    4993 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 14:16:52.724428 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712255    4993 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 14:16:52.724428 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712258    4993 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 14:16:52.724428 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712261    4993 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 14:16:52.724428 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712263    4993 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 14:16:52.724428 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712267    4993 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 14:16:52.724428 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712272    4993 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 14:16:52.724428 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712275    4993 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 14:16:52.724428 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712277    4993 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 14:16:52.724428 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712280    4993 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 14:16:52.724428 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712283    4993 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 14:16:52.724428 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712285    4993 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 14:16:52.724890 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712288    4993 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 14:16:52.724890 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712290    4993 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 14:16:52.724890 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712293    4993 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 14:16:52.724890 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712296    4993 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 14:16:52.724890 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712298    4993 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 14:16:52.724890 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712301    4993 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 14:16:52.724890 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712303    4993 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 14:16:52.724890 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712306    4993 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 14:16:52.724890 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712308    4993 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 14:16:52.724890 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712311    4993 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 14:16:52.724890 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712314    4993 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 14:16:52.724890 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712317    4993 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 14:16:52.724890 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712319    4993 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 14:16:52.724890 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712322    4993 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 14:16:52.724890 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712324    4993 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 14:16:52.724890 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712327    4993 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 14:16:52.724890 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712329    4993 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 14:16:52.724890 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712332    4993 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 14:16:52.724890 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712335    4993 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 14:16:52.724890 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712337    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 14:16:52.725380 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712340    4993 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 14:16:52.725380 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712342    4993 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 14:16:52.725380 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712345    4993 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 14:16:52.725380 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712347    4993 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 14:16:52.725380 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712350    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 14:16:52.725380 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712354    4993 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 14:16:52.725380 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712357    4993 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 14:16:52.725380 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712360    4993 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 14:16:52.725380 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712363    4993 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 14:16:52.725380 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712366    4993 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 14:16:52.725380 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712369    4993 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 14:16:52.725380 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712371    4993 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 14:16:52.725380 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712374    4993 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 14:16:52.725380 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712377    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 14:16:52.725380 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712379    4993 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 14:16:52.725380 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712382    4993 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 14:16:52.725380 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712385    4993 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 14:16:52.725380 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712387    4993 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 14:16:52.725380 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712390    4993 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 14:16:52.725956 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712392    4993 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 14:16:52.725956 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712395    4993 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 14:16:52.725956 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712397    4993 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 14:16:52.725956 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712400    4993 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 14:16:52.725956 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712402    4993 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 14:16:52.725956 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712405    4993 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 14:16:52.725956 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712407    4993 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 14:16:52.725956 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712410    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 14:16:52.725956 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712412    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 14:16:52.725956 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712415    4993 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 14:16:52.725956 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712417    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 14:16:52.725956 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712420    4993 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 14:16:52.725956 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712423    4993 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 14:16:52.725956 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712426    4993 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 14:16:52.725956 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712429    4993 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 14:16:52.725956 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712431    4993 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 14:16:52.725956 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712434    4993 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 14:16:52.725956 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712436    4993 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 14:16:52.725956 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712439    4993 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 14:16:52.726606 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712444    4993 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 14:16:52.726606 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712447    4993 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 14:16:52.726606 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712450    4993 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 14:16:52.726606 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712452    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 14:16:52.726606 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712455    4993 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 14:16:52.726606 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712458    4993 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 14:16:52.726606 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712460    4993 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 14:16:52.726606 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712463    4993 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 14:16:52.726606 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712466    4993 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 14:16:52.726606 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712468    4993 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 14:16:52.726606 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712471    4993 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 14:16:52.726606 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712474    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 14:16:52.726606 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712477    4993 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 14:16:52.726606 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712479    4993 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 14:16:52.726606 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712482    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 14:16:52.726606 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712485    4993 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 14:16:52.726606 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712487    4993 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 14:16:52.727448 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.712511    4993 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 14:16:52.727448 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712554    4993 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 14:16:52.727448 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712558    4993 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 14:16:52.727448 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712561    4993 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 14:16:52.727448 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712564    4993 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 14:16:52.727448 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712567    4993 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 14:16:52.727448 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712569    4993 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 14:16:52.727448 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712572    4993 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 14:16:52.727448 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712574    4993 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 14:16:52.727448 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712578    4993 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 14:16:52.727448 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712581    4993 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 14:16:52.727448 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712584    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 14:16:52.727448 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712586    4993 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 14:16:52.727448 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712589    4993 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 14:16:52.727448 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712591    4993 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 14:16:52.727448 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712594    4993 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 14:16:52.728024 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712598    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 14:16:52.728024 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712602    4993 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 14:16:52.728024 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712605    4993 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 14:16:52.728024 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712608    4993 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 14:16:52.728024 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712612    4993 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 14:16:52.728024 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712614    4993 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 14:16:52.728024 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712617    4993 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 14:16:52.728024 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712619    4993 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 14:16:52.728024 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712622    4993 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 14:16:52.728024 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712625    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 14:16:52.728024 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712627    4993 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 14:16:52.728024 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712630    4993 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 14:16:52.728024 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712632    4993 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 14:16:52.728024 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712635    4993 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 14:16:52.728024 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712637    4993 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 14:16:52.728024 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712640    4993 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 14:16:52.728024 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712643    4993 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 14:16:52.728024 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712645    4993 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 14:16:52.728024 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712648    4993 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 14:16:52.728708 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712650    4993 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 14:16:52.728708 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712653    4993 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 14:16:52.728708 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712655    4993 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 14:16:52.728708 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712658    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 14:16:52.728708 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712661    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 14:16:52.728708 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712664    4993 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 14:16:52.728708 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712667    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 14:16:52.728708 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712670    4993 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 14:16:52.728708 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712673    4993 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 14:16:52.728708 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712676    4993 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 14:16:52.728708 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712678    4993 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 14:16:52.728708 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712681    4993 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 14:16:52.728708 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712683    4993 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 14:16:52.728708 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712687    4993 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 14:16:52.728708 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712690    4993 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 14:16:52.728708 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712692    4993 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 14:16:52.728708 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712695    4993 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 14:16:52.728708 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712698    4993 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 14:16:52.728708 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712700    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 14:16:52.729363 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712704    4993 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 14:16:52.729363 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712706    4993 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 14:16:52.729363 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712709    4993 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 14:16:52.729363 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712712    4993 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 14:16:52.729363 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712714    4993 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 14:16:52.729363 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712717    4993 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 14:16:52.729363 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712719    4993 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 14:16:52.729363 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712722    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 14:16:52.729363 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712724    4993 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 14:16:52.729363 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712727    4993 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 14:16:52.729363 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712729    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 14:16:52.729363 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712732    4993 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 14:16:52.729363 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712734    4993 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 14:16:52.729363 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712737    4993 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 14:16:52.729363 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712739    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 14:16:52.729363 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712742    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 14:16:52.729363 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712744    4993 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 14:16:52.729363 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712747    4993 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 14:16:52.729363 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712749    4993 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 14:16:52.729919 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712752    4993 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 14:16:52.729919 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712754    4993 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 14:16:52.729919 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712758    4993 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 14:16:52.729919 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712760    4993 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 14:16:52.729919 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712763    4993 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 14:16:52.729919 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712765    4993 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 14:16:52.729919 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712768    4993 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 14:16:52.729919 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712775    4993 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 14:16:52.729919 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712777    4993 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 14:16:52.729919 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712780    4993 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 14:16:52.729919 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712783    4993 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 14:16:52.729919 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712785    4993 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 14:16:52.729919 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712788    4993 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 14:16:52.729919 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712790    4993 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 14:16:52.729919 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712793    4993 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 14:16:52.729919 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712795    4993 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 14:16:52.729919 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712798    4993 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 14:16:52.729919 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712801    4993 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 14:16:52.729919 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712805    4993 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 14:16:52.730403 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712808    4993 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 14:16:52.730403 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712811    4993 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 14:16:52.730403 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712814    4993 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 14:16:52.730403 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712817    4993 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 14:16:52.730403 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712820    4993 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 14:16:52.730403 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712822    4993 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 14:16:52.730403 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712825    4993 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 14:16:52.730403 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712827    4993 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 14:16:52.730403 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712830    4993 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 14:16:52.730403 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712833    4993 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 14:16:52.730403 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712835    4993 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 14:16:52.730403 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712838    4993 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 14:16:52.730403 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712840    4993 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 14:16:52.730403 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712843    4993 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 14:16:52.730403 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:52.712846    4993 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 14:16:52.730403 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.712900    4993 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 01 14:16:52.730403 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.713894    4993 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 01 14:16:52.730403 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.713946    4993 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 01 14:16:52.730403 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.714127    4993 server.go:1014] "Starting client certificate rotation"
Nov 01 14:16:52.730920 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.714273    4993 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 01 14:16:52.730920 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.714314    4993 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 14:11:10 +0000 UTC" deadline="2028-05-07 13:36:34.188832833 +0000 UTC"
Nov 01 14:16:52.730920 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.714341    4993 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13271h19m41.474493935s"
Nov 01 14:16:52.730920 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.714649    4993 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 14:16:52.730920 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.715691    4993 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 14:16:52.730920 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.716768    4993 log.go:25] "Validated CRI v1 runtime API"
Nov 01 14:16:52.730920 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.718359    4993 log.go:25] "Validated CRI v1 image API"
Nov 01 14:16:52.730920 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.719126    4993 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 01 14:16:52.730920 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.721044    4993 fs.go:135] Filesystem UUIDs: map[0a1d85e9-1934-4ffa-8bb0-39d9aa0fedbc:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 b246a694-e8c5-41c0-b060-ed5c8f313806:/dev/nvme0n1p4]
Nov 01 14:16:52.731195 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.721063    4993 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0aea4ba3a0f3f845dfc67cf48042c64292a3373bea0a78fe95a6a0a1ce6fdfaf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0aea4ba3a0f3f845dfc67cf48042c64292a3373bea0a78fe95a6a0a1ce6fdfaf/userdata/shm major:0 minor:352 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0baeba702a835a54788bc5346bb21d6246e4a408ed299e692f0a26a125c169cc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0baeba702a835a54788bc5346bb21d6246e4a408ed299e692f0a26a125c169cc/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/15e947b3c1de139876bcd194085602f71b94f52d655de40ca926f9773c1ffbea/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/15e947b3c1de139876bcd194085602f71b94f52d655de40ca926f9773c1ffbea/userdata/shm major:0 minor:79 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/177bc7f098ae5cba274d95f4b592129d6acb9976a463ba085b6d43898bfabd73/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/177bc7f098ae5cba274d95f4b592129d6acb9976a463ba085b6d43898bfabd73/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2c371efbc265a2789bc9913d731f39b76b38dfbd89aa2beb1cf181b6b2a82a88/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2c371efbc265a2789bc9913d731f39b76b38dfbd89aa2beb1cf181b6b2a82a88/userdata/shm major:0 minor:226 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4730fc3bf0ae3cee7a9949aff05aa788af46b4d0acdf21c3911d6bc1b09a3d6c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4730fc3bf0ae3cee7a9949aff05aa788af46b4d0acdf21c3911d6bc1b09a3d6c/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/690b74893e54482894df2b6f978f53c0f1b1fa7c44c3184bfd47a6a6b3758137/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/690b74893e54482894df2b6f978f53c0f1b1fa7c44c3184bfd47a6a6b3758137/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6f895928b8cb30ce26ce5335d0bc02d8bfc8edfb5035df303cbdf4e67fe480ac/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6f895928b8cb30ce26ce5335d0bc02d8bfc8edfb5035df303cbdf4e67fe480ac/userdata/shm major:0 minor:222 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7b2912f5c7df5c0bd7c90ec4fae06b2a295b3fc96e054a6682e822f05bbc3be2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7b2912f5c7df5c0bd7c90ec4fae06b2a295b3fc96e054a6682e822f05bbc3be2/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/880c12f6309a8d768f868b8369ee6a928463263c6f30c39a8cb01909d7aa9902/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/880c12f6309a8d768f868b8369ee6a928463263c6f30c39a8cb01909d7aa9902/userdata/shm major:0 minor:220 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9226216fff100a82b2a6715ce444110115c3d396b98acc865c89fc819868e4b7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9226216fff100a82b2a6715ce444110115c3d396b98acc865c89fc819868e4b7/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/927f16040227de27cef857b98df6409aba54e22122880ff2d3dc48f638daff3d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/927f16040227de27cef857b98df6409aba54e22122880ff2d3dc48f638daff3d/userdata/shm major:0 minor:330 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a4620bc86b004bf248b53e4ab03c70adcebaba74a374200d2fdd272ee9eb8eb3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a4620bc86b004bf248b53e4ab03c70adcebaba74a374200d2fdd272ee9eb8eb3/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a5dc0434840b345873d399db22c8ae5810e840e4bc25115b32f7425fe76d5423/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a5dc0434840b345873d399db22c8ae5810e840e4bc25115b32f7425fe76d5423/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/adbd46f9567ea58f2a324a7a72d0fa034b66bae6f1f22ba13011c8ecf111444e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/adbd46f9567ea58f2a324a7a72d0fa034b66bae6f1f22ba13011c8ecf111444e/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ae783b4a23f870f431b151c343e9c32bba480f883ab647d3f796595900a6e53e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ae783b4a23f870f431b151c343e9c32bba480f883ab647d3f796595900a6e53e/userdata/shm major:0 minor:290 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b91c92e51399848fcf2fe6b04cc993ae0e73f896451e011d2df11db69684f74d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b91c92e51399848fcf2fe6b04cc993ae0e73f896451e011d2df11db69684f74d/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ba1d2e479174d30eac7876b3f30fff0702c316d073a37353f03d9a09e2110c84/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ba1d2e479174d30eac7876b3f30fff0702c316d073a37353f03d9a09e2110c84/userdata/shm major:0 minor:288 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bac08a467e759b5f9eb601b19c0cafeffee46e2ef10d11078b6c9d8231c3d408/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bac08a467e759b5f9eb601b19c0cafeffee46e2ef10d11078b6c9d8231c3d408/userdata/shm major:0 minor:224 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cafdbe7c69ef251c3ee7143456fc8e488e67d6ac1609f2906deea2ffb53664cd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cafdbe7c69ef251c3ee7143456fc8e488e67d6ac1609f2906deea2ffb53664cd/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ccb76443164367970060f9ea26b8693190d38ed29108de716b17d69abc051c04/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ccb76443164367970060f9ea26b8693190d38ed29108de716b17d69abc051c04/userdata/shm major:0 minor:354 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e08e52ee478f9a7272ba40ba4b90357351ebf80ac971cebdba7f80f520e407e2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e08e52ee478f9a7272ba40ba4b90357351ebf80ac971cebdba7f80f520e407e2/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ea2f2bba3fe50a234f88c90b77277001e023ccadb96e49135160bda809bc704b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ea2f2bba3fe50a234f88c90b77277001e023ccadb96e49135160bda809bc704b/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ef98d8e5539559612e84235ecac8dad90436bb395a86ef237ac3bf99da120f9a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ef98d8e5539559612e84235ecac8dad90436bb395a86ef237ac3bf99da120f9a/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f29e33b678b841c2e1e6bcf172b77aaacda8e2522ad70f90727388289f22053d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f29e33b678b841c2e1e6bcf172b77aaacda8e2522ad70f90727388289f22053d/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ffb9c2812bfd05c93dc5e7fb90c15e9b272744135cb9e37b50a256bfa0a7432b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ffb9c2812bfd05c93dc5e7fb90c15e9b272744135cb9e37b50a256bfa0a7432b/userdata/shm major:0 minor:362 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/00216deb-5dd0-4b4f-8940-c17dc329303c/volumes/kubernetes.io~projected/kube-api-access-cvgpz:{mountpoint:/var/lib/kubelet/pods/00216deb-5dd0-4b4f-8940-c17dc329303c/volumes/kubernetes.io~projected/kube-api-access-cvgpz major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/037ca099-5ad7-4874-a984-462c07801e70/volumes/kubernetes.io~projected/kube-api-access-8thtk:{mountpoint:/var/lib/kubelet/pods/037ca099-5ad7-4874-a984-462c07801e70/volumes/kubernetes.io~projected/kube-api-access-8thtk major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/058820ef-eef1-41ca-b331-670bab146861/volumes/kubernetes.io~projected/kube-api-access-gx82t:{mountpoint:/var/lib/kubelet/pods/058820ef-eef1-41ca-b331-670bab146861/volumes/kubernetes.io~projected/kube-api-access-gx82t major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c8cc47b-fb01-4195-a7fc-eeefc706017e/volumes/kubernetes.io~projected/kube-api-access-wgmsd:{mountpoint:/var/lib/kubelet/pods/0c8cc47b-fb01-4195-a7fc-eeefc706017e/volumes/kubernetes.io~projected/kube-api-access-wgmsd major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d4352db-d2e7-43ba-97ba-9e344abbac8b/volumes/kubernetes.io~projected/kube-api-access-vp6rs:{mountpoint:/var/lib/kubelet/pods/0d4352db-d2e7-43ba-97ba-9e344abbac8b/volumes/kubernetes.io~projected/kube-api-access-vp6rs major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/17570aba-70d9-46ea-a0ca-3a465635a17c/volumes/kubernetes.io~projected/kube-api-access-zpdrz:{mountpoint:/var/lib/kubelet/pods/17570aba-70d9-46ea-a0ca-3a465635a17c/volumes/kubernetes.io~projected/kube-api-access-zpdrz major:0 minor:342 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1b941fae-3c27-4d68-8205-38e492179ee9/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/1b941fae-3c27-4d68-8205-38e492179ee9/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1b941fae-3c27-4d68-8205-38e492179ee9/volumes/kubernetes.io~projected/kube-api-access-p8mfv:{mountpoint:/var/lib/kubelet/pods/1b941fae-3c27-4d68-8205-38e492179ee9/volumes/kubernetes.io~projected/kube-api-access-p8mfv major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1b941fae-3c27-4d68-8205-38e492179ee9/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/1b941fae-3c27-4d68-8205-38e492179ee9/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2460be58-d51b-4876-bf9b-2a0e57901386/volumes/kubernetes.io~projected/kube-api-access-7jbv4:{mountpoint:/var/lib/kubelet/pods/2460be58-d51b-4876-bf9b-2a0e57901386/volumes/kubernetes.io~projected/kube-api-access-7jbv4 major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2e013e38-9840-45c2-b4a0-80df0f319109/volumes/kubernetes.io~projected/kube-api-access-wzjlr:{mountpoint:/var/lib/kubelet/pods/2e013e38-9840-45c2-b4a0-80df0f319109/volumes/kubernetes.io~projected/kube-api-access-wzjlr major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/321ecf30-5108-4320-9a58-6b204001639c/volumes/kubernetes.io~projected/kube-api-access-8bhpq:{mountpoint:/var/lib/kubelet/pods/321ecf30-5108-4320-9a58-6b204001639c/volumes/kubernetes.io~projected/kube-api-access-8bhpq major:0 minor:361 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/321ecf30-5108-4320-9a58-6b204001639c/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/321ecf30-5108-4320-9a58-6b204001639c/volumes/kubernetes.io~secret/signing-key major:0 minor:360 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/384cd50b-370f-474f-b80e-89a544a9bbbd/volumes/kubernetes.io~projected/kube-api-access-chwvj:{mountpoint:/var/lib/kubelet/pods/384cd50b-370f-474f-b80e-89a544a9bbbd/volumes/kubernetes.io~projected/kube-api-access-chwvj major:0 minor:323 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3e7e18b9-78ee-4349-b00c-a0800e8f8341/volumes/kubernetes.io~projected/kube-api-access-stw2n:{mountpoint:/var/lib/kubelet/pods/3e7e18b9-78ee-4349-b00c-a0800e8f8341/volumes/kubernetes.io~projected/kube-api-access-stw2n major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3e7e18b9-78ee-4349-b00c-a0800e8f8341/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/3e7e18b9-78ee-4349-b00c-a0800e8f8341/volumes/kubernetes.io~secret/serving-cert major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4a724964-bd57-4d56-a042-a0d490115eaa/volumes/kubernetes.io~projected/kube-api-access-kfj7s:{mountpoint:/var/lib/kubelet/pods/4a724964-bd57-4d56-a042-a0d490115eaa/volumes/kubernetes.io~projected/kube-api-access-kfj7s major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4a724964-bd57-4d56-a042-a0d490115eaa/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/4a724964-bd57-4d56-a042-a0d490115eaa/volumes/kubernetes.io~secret/serving-cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5e15a59f-6aac-45e7-8812-ae97d0559052/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/5e15a59f-6aac-45e7-8812-ae97d0559052/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5e15a59f-6aac-45e7-8812-ae97d0559052/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/5e15a59f-6aac-45e7-8812-ae97d0559052/volumes/kubernetes.io~empty-dir/tmp major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5e15a59f-6aac-45e7-8812-ae97d0559052/volumes/kubernetes.io~projected/kube-api-access-8zcj8:{mountpoint:/var/lib/kubelet/pods/5e15a59f-6aac-45e7-8812-ae97d0559052/volumes/kubernetes.io~projected/kube-api-access-8zcj8 major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6a8f947a-0f8f-41aa-b646-4459fa8126a3/volumes/kubernetes.io~projected/kube-api-access-9j226:{mountpoint:/var/lib/kubelet/pods/6a8f947a-0f8f-41aa-b646-4459fa8126a3/volumes/kubernetes.io~projected/kube-api-access-9j226 major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/70bb916a-832b-4bce-9c0a-f01bf4e3abc4/volumes/kubernetes.io~projected/kube-api-access-xkzgz:{mountpoint:/var/lib/kubelet/pods/70bb916a-832b-4bce-9c0a-f01bf4e3abc4/volumes/kubernetes.io~projected/kube-api-access-xkzgz major:0 minor:238 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/70bb916a-832b-4bce-9c0a-f01bf4e3abc4/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/70bb916a-832b-4bce-9c0a-f01bf4e3abc4/volumes/kubernetes.io~secret/cert major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7970eec1-3765-48a4-b09c-3dec2d29956b/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/7970eec1-3765-48a4-b09c-3dec2d29956b/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:237 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7970eec1-3765-48a4-b09c-3dec2d29956b/volumes/kubernetes.io~projected/kube-api-access-8gnmx:{mountpoint:/var/lib/kubelet/pods/7970eec1-3765-48a4-b09c-3dec2d29956b/volumes/kubernetes.io~projected/kube-api-access-8gnmx major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7970eec1-3765-48a4-b09c-3dec2d29956b/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/7970eec1-3765-48a4-b09c-3dec2d29956b/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7970eec1-3765-48a4-b09c-3dec2d29956b/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/7970eec1-3765-48a4-b09c-3dec2d29956b/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7a26b817-3283-4d59-8e8f-3bb4c0fa4987/volumes/kubernetes.io~projected/kube-api-access-bv8hp:{mountpoint:/var/lib/kubelet/pods/7a26b817-3283-4d59-8e8f-3bb4c0fa4987/volumes/kubernetes.io~projected/kube-api-access-bv8hp major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7a26b817-3283-4d59-8e8f-3bb4c0fa4987/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/7a26b817-3283-4d59-8e8f-3bb4c0fa4987/volumes/kubernetes.io~secret/serving-cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a6e7f39-7853-4fec-9ea2-30099983e5bd/volumes/kubernetes.io~projected/kube-api-access-hbnlg:{mountpoint:/var/lib/kubelet/pods/8a6e7f39-7853-4fec-9ea2-30099983e5bd/volumes/kubernetes.io~projected/kube-api-access-hbnlg major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8b1eb53c-8c9a-46c4-b654-2aef42491b14/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/8b1eb53c-8c9a-46c4-b654-2aef42491b14/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a12d30a7-8564-43c1-9742-3194fe8b975f/volumes/kubernetes.io~projected/kube-api-access-gnzsm:{mountpoint:/var/lib/kubelet/pods/a12d30a7-8564-43c1-9742-3194fe8b975f/volumes/kubernetes.io~projected/kube-api-access-gnzsm major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b73fabd2-0d2c-4988-bbcf-7564d9331390/volumes/kubernetes.io~projected/kube-api-access-ddmmn:{mountpoint:/var/lib/kubelet/pods/b73fabd2-0d2c-4988-bbcf-7564d9331390/volumes/kubernetes.io~projected/kube-api-access-ddmmn major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bb59051d-c2df-4362-a157-00f14f915e79/volumes/kubernetes.io~projected/kube-api-access-92xdl:{mountpoint:/var/lib/kubelet/pods/bb59051d-c2df-4362-a157-00f14f915e79/volumes/kubernetes.io~projected/kube-api-access-92xdl major:0 minor:234 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bb59051d-c2df-4362-a157-00f14f915e79/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/bb59051d-c2df-4362-a157-00f14f915e79/volumes/kubernetes.io~secret/serving-cert major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bcc6412c-2003-49de-9817-a3713e28afef/volumes/kubernetes.io~projected/kube-api-access-vdsp8:{mountpoint:/var/lib/kubelet/pods/bcc6412c-2003-49de-9817-a3713e28afef/volumes/kubernetes.io~projected/kube-api-access-vdsp8 major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c27ff402-235a-4d73-a139-cf928a6d5cbe/volumes/kubernetes.io~projected/kube-api-access-rjv76:{mountpoint:/var/lib/kubelet/pods/c27ff402-235a-4d73-a139-cf928a6d5cbe/volumes/kubernetes.io~projected/kube-api-access-rjv76 major:0 minor:239 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c27ff402-235a-4d73-a139-cf928a6d5cbe/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/c27ff402-235a-4d73-a139-cf928a6d5cbe/volumes/kubernetes.io~secret/default-certificate major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c27ff402-235a-4d73-a139-cf928a6d5cbe/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/c27ff402-235a-4d73-a139-cf928a6d5cbe/volumes/kubernetes.io~secret/stats-auth major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1/volumes/kubernetes.io~projected/kube-api-access-hpm8k:{mountpoint:/var/lib/kubelet/pods/ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1/volumes/kubernetes.io~projected/kube-api-access-hpm8k major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ccc5b32b-07bc-4b3c-91ef-b184b0a9afe3/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/ccc5b32b-07bc-4b3c-91ef-b184b0a9afe3/volumes/kubernetes.io~secret/agent-certs major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d407313d-3333-4ec4-85bb-cd27cf4d7422/volumes/kubernetes.io~projected/kube-api-access-fsmvn:{mountpoint:/var/lib/kubelet/pods/d407313d-3333-4ec4-85bb-cd27cf4d7422/volumes/kubernetes.io~projected/kube-api-access-fsmvn major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d8a5ccab-4d5e-4aaf-ad08-766238eca05f/volumes/kubernetes.io~projected/kube-api-access-q6cqj:{mountpoint:/var/lib/kubelet/pods/d8a5ccab-4d5e-4aaf-ad08-766238eca05f/volumes/kubernetes.io~projected/kube-api-access-q6cqj major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e78d2494-62c4-4070-a1eb-8ce40329ff57/volumes/kubernetes.io~projected/kube-api-access-wxdcx:{mountpoint:/var/lib/kubelet/pods/e78d2494-62c4-4070-a1eb-8ce40329ff57/volumes/kubernetes.io~projected/kube-api-access-wxdcx major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e7ea769b-e719-4090-bcd4-05b1b88fcc06/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/e7ea769b-e719-4090-bcd4-05b1b88fcc06/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e7ea769b-e719-4090-bcd4-05b1b88fcc06/volumes/kubernetes.io~projected/kube-api-access-rdxs8:{mountpoint:/var/lib/kubelet/pods/e7ea769b-e719-4090-bcd4-05b1b88fcc06/volumes/kubernetes.io~projected/kube-api-access-rdxs8 major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e7ea769b-e719-4090-bcd4-05b1b88fcc06/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/e7ea769b-e719-4090-bcd4-05b1b88fcc06/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e7ea769b-e719-4090-bcd4-05b1b88fcc06/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/e7ea769b-e719-4090-bcd4-05b1b88fcc06/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f219e3c2-6da8-4c88-9ec4-b26cf4d10326/volumes/kubernetes.io~projected/kube-api-access-pjj74:{mountpoint:/var/lib/kubelet/pods/f219e3c2-6da8-4c88-9ec4-b26cf4d10326/volumes/kubernetes.io~projected/kube-api-access-pjj74 major:0 minor:235 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fba1e7f9-ca37-46dc-8c66-56f1d9563acd/volumes/kubernetes.io~projected/kube-api-access-khv87:{mountpoint:/var/lib/kubelet/pods/fba1e7f9-ca37-46dc-8c66-56f1d9563acd/volumes/kubernetes.io~projected/kube-api-access-khv87 major:0 minor:347 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/66c68af6b157c8c759c659db8eaa28fdfef30ac79dfcc71bf2df9921097039a4/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/0f5a69c9419647412dcbd9bc97bdbd077c0e9a114a53e2f05eca32c2bd5bdf0c/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/d67d4f993dc1e9b0aa12f7295deb27d9f6a8f9e6f4b5afd435beabe34bb90a69/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/cd6dc431e119d9f426190c557b6eaa4d0fcd40cc65ad547839b026eec4cd7dbb/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/682f26b01467c1b47cc23d6534b037033fea31ec270818875e6e3af09bb149db/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/17b16798876bc12c3f0b12de420c968db667f947b391de9e0c64b20dd887ff1e/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/eb6aed145c122132181d061e43e00f0b5f362828669447e994ddba680462dc45/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/26242090354e605960fc9bab79599463bc9ba8f7c3a1a7fdd2da8af740f6d65c/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/c558c619ae8e60e03bc257fee97431bd0a2508c230da1de263e3d4db6f7ea351/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/011e3b342bc455909f4ec42178589224f04d21bc8ecb983af3eba5d8da29d2f8/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-136:{mountpoint:/var/lib/containers/storage/overlay/7244a13b1ea8b4b9618582df396b9da1c430097380fb4bd67aba3aec92abeec4/merged major:0 minor:136 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/decbd88fe588789d178aa39b1bf7ccc066044cdbac418504e96f78f17341ed7e/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/006d9c158c0c215b56d05671a20722d889327d5f47bd44ef5166244dc12a19d8/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/64a933f7d6e203a8747ace7c13eb366b9ea4072685f5a402f0385ded1055d7b7/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/dd2eade9d6c0512a1bf56d352dfae640ce6b7a2f8f61e9463f502a5765956b87/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/e9b5bb3de94428b641717ed6a3daed37c9a8004fa646d614b741937a10c7e94d/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/75277e312313890472ef2db9fefc6c6fad7d06d720680b52af21a6a14f43be10/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/b3617e797bf20e1f043b46b63a713c35c29267fb63e898ee9f4ff0d87ce7cab0/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/512bd4d514aaf002d74cdcfc465a5aac2d1f8155d1408d9b414e981c61ae7ca3/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/32eab2da393d8e68d6f05a452c4d467b36268b23a9e9720bed59306a04afe012/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-189:{mountpoint:/var/lib/containers/storage/overlay/4d51146ce06815b128ecc25942f5f11b53b86c863df3d14750f6c1116bec6d6d/merged major:0 minor:189 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/ebd2bedf710e57695e4279626f047f51f83b967f50734f30368415a460b80a1d/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/9ac11032320791379aa41cd7164fc1b1479b9cd0a66276c6e2c621996e847a73/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/beae26b1ccedc63cfc5d4cb03e56453ed1d69c4305d1d191d068fd05057767a1/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/201b3ba6b98b5045ac63ec42bfa21b8ade270605d6e0bc3085d70262161fbd69/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/39af173d36a5ae7d19b82718bb6f3ce1f2a66e940464d28b895c13158ea55d85/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/7626bf3124448ccf1b79a56062cf8166be4349238e81ae3353017c83755d271d/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/952c9bc88ec9e92c1b4fdfdbfbfe5b3d3558ad5554454d771853c3c9236a9505/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/31247a3fd593571feed1f4e971916c256f1694146718a2141696ee1dccf72d32/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/f7cf1ad3e27a9e3533f7a2878405799e9f57af75d562448f1757706026e48db4/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-265:{mountpoint:/var/lib/containers/storage/overlay/b0b76ca1401d62a6f46a333c0e0b7f58e365dff7953629560efb07c29e33716a/merged major:0 minor:265 fsType:overlay blockSize:0} overlay_0-274:{mountpoint:/var/lib/containers/storage/overlay/01a5aebf2b78cf2413417211bc4c3353f2af791da1cd0f5d30b8af516e57a18d/merged major:0 minor:274 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/2ea151651ba208df80b06429479ce297310193f0a760530277555f5eecc7c7db/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-280:{mountpoint:/var/lib/containers/storage/overlay/4a32c95691134a9902b87a3e88ac703dae00b5c7112a53db1babcc391ccc5b64/merged major:0 minor:280 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/4f8f8c907bfd2cee456022858df0e73dfa9d9296be960726baa7bb0787465177/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/415ceb749ea8b139b53a9501036958f9c1d4be73d13f5618fc4438b519f9bfc6/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/b3a5b29a821d13724721ee19329bfc4d5ba52514ccc8677d811fcb5af8eb2201/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/d601c7b42db663f57b80502b30523e66b29ed6f155dffae3f3dd45cfb215b4e6/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/cb36da9bc0da491c7caaeb48bd8536a453630a1968e66e151ae3f049316481b0/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/43cd008b83ed3969330fb48cce0be0d5491c0ef1832794dee4b00549418e8cdc/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-312:{mountpoint:/var/lib/containers/storage/overlay/0e4b8f60b791f7f2d78cd916ec77942a3518df057ed077ce99186532f7a6f452/merged major:0 minor:312 fsType:overlay blockSize:0} overlay_0-324:{mountpoint:/var/lib/containers/storage/overlay/64426b85f6be5388bc2583e74410dbf58e2ccc1b9d0e42ee06a071cb1728d133/merged major:0 minor:324 fsType:overlay blockSize:0} overlay_0-336:{mountpoint:/var/lib/containers/storage/overlay/a4b4842ef4c4921eca1332b234a392520a470dce7d66ce989a9908868959f765/merged major:0 minor:336 fsType:overlay blockSize:0} overlay_0-338:{mountpoint:/var/lib/containers/storage/overlay/36ab131eac7f07dd193b3e4f004be2b611f650e41c838eef2d72e5d32088af81/merged major:0 minor:338 fsType:overlay blockSize:0} overlay_0-340:{mountpoint:/var/lib/containers/storage/overlay/ceef53a7e04232a09da2724401bf30bcc757079eff8b086ee7966a6c2e838c21/merged major:0 minor:340 fsType:overlay blockSize:0} overlay_0-356:{mountpoint:/var/lib/containers/storage/overlay/256b8bb17bf693b41321be52624fa79e9364c7652cefcfc0c4f166e310fb12f0/merged major:0 minor:356 fsType:overlay blockSize:0} overlay_0-358:{mountpoint:/var/lib/containers/storage/overlay/c44d3cc8c5f7572f2d0e325c03b14f92eb671226e84cfa2468834b559e29ff51/merged major:0 minor:358 fsType:overlay blockSize:0} overlay_0-364:{mountpoint:/var/lib/containers/storage/overlay/0b5d753640271f6dcabbed2fdf84697cb90240f6930622033ec88a9c7d4f7c3c/merged major:0 minor:364 fsType:overlay blockSize:0} overlay_0-46:{mountpoint:/var/lib/containers/storage/overlay/903f4474d24e722e74ba333b5deb1929681cf8183e4947fed141e35a699c11e8/merged major:0 minor:46 fsType:overlay blockSize:0} overlay_0-64:{mountpoint:/var/lib/containers/storage/overlay/813bbb87d23ef1e51120c70f6a8b4b22b65c70f5628d8dfac165e3487b5d8a18/merged major:0 minor:64 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/b5b2019a077e099d4feb839559d03bf588884ecee00f69943f8dc070ef2ffab4/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/ea20330d67bdde098e3c07cfe76eb6a80e6a4602de4e604714ab4f52ed3e39a0/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/d74a09cfb1cda5ecaf294da1ba1c708027ce6b2b374c76bd53aa437e4177f190/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/cc1d6f73deb6980b9abaf53e1617219aa8658b819179cbfc27199988e660bb0b/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/16ea7469c8ae9edaf94d4f26c2cca99f68baf04972b2601c01ac38f244658de4/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/7050369da09591da585313b831bd84543588dabbc704465781b914a0f604548a/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/7309b47a2cc55fd8fe6ceeb5236c2d49649ecacfdf4f14bed6ed4366967b2bbb/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/a5e09e4ce2b2ca1ee9e5e306f3a53a3202b7212b95bd9a40bc7b6cf0db4f7b5b/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/9acfd72bfeab2e3043b89c05a2be2d5330d5131c1e981e49adacf11fdd2afd86/merged major:0 minor:96 fsType:overlay blockSize:0}]
Nov 01 14:16:52.731447 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.730084    4993 manager.go:217] Machine: {Timestamp:2026-11-01 14:16:52.729609915 +0000 UTC m=+0.059472643 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100267 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b76fc81e3d383e99c32ec3afca649 SystemUUID:ec2b76fc-81e3-d383-e99c-32ec3afca649 BootID:f3985537-feee-4310-b198-e5c2a3ce5604 Filesystems:[{Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/5e15a59f-6aac-45e7-8812-ae97d0559052/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:49 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f29e33b678b841c2e1e6bcf172b77aaacda8e2522ad70f90727388289f22053d/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c27ff402-235a-4d73-a139-cf928a6d5cbe/volumes/kubernetes.io~projected/kube-api-access-rjv76 DeviceMajor:0 DeviceMinor:239 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0c8cc47b-fb01-4195-a7fc-eeefc706017e/volumes/kubernetes.io~projected/kube-api-access-wgmsd DeviceMajor:0 DeviceMinor:232 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ea2f2bba3fe50a234f88c90b77277001e023ccadb96e49135160bda809bc704b/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e7ea769b-e719-4090-bcd4-05b1b88fcc06/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/70bb916a-832b-4bce-9c0a-f01bf4e3abc4/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:216 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3e7e18b9-78ee-4349-b00c-a0800e8f8341/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e78d2494-62c4-4070-a1eb-8ce40329ff57/volumes/kubernetes.io~projected/kube-api-access-wxdcx DeviceMajor:0 DeviceMinor:209 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ef98d8e5539559612e84235ecac8dad90436bb395a86ef237ac3bf99da120f9a/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b73fabd2-0d2c-4988-bbcf-7564d9331390/volumes/kubernetes.io~projected/kube-api-access-ddmmn DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-64 DeviceMajor:0 DeviceMinor:64 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2460be58-d51b-4876-bf9b-2a0e57901386/volumes/kubernetes.io~projected/kube-api-access-7jbv4 DeviceMajor:0 DeviceMinor:204 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/384cd50b-370f-474f-b80e-89a544a9bbbd/volumes/kubernetes.io~projected/kube-api-access-chwvj DeviceMajor:0 DeviceMinor:323 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7970eec1-3765-48a4-b09c-3dec2d29956b/volumes/kubernetes.io~projected/kube-api-access-8gnmx DeviceMajor:0 DeviceMinor:233 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ae783b4a23f870f431b151c343e9c32bba480f883ab647d3f796595900a6e53e/userdata/shm DeviceMajor:0 DeviceMinor:290 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b91c92e51399848fcf2fe6b04cc993ae0e73f896451e011d2df11db69684f74d/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/adbd46f9567ea58f2a324a7a72d0fa034b66bae6f1f22ba13011c8ecf111444e/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8b1eb53c-8c9a-46c4-b654-2aef42491b14/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:282 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/690b74893e54482894df2b6f978f53c0f1b1fa7c44c3184bfd47a6a6b3758137/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e7ea769b-e719-4090-bcd4-05b1b88fcc06/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:205 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3e7e18b9-78ee-4349-b00c-a0800e8f8341/volumes/kubernetes.io~projected/kube-api-access-stw2n DeviceMajor:0 DeviceMinor:210 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-324 DeviceMajor:0 DeviceMinor:324 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ccc5b32b-07bc-4b3c-91ef-b184b0a9afe3/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:48 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/177bc7f098ae5cba274d95f4b592129d6acb9976a463ba085b6d43898bfabd73/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7a26b817-3283-4d59-8e8f-3bb4c0fa4987/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:203 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7a26b817-3283-4d59-8e8f-3bb4c0fa4987/volumes/kubernetes.io~projected/kube-api-access-bv8hp DeviceMajor:0 DeviceMinor:207 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c27ff402-235a-4d73-a139-cf928a6d5cbe/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:217 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-338 DeviceMajor:0 DeviceMinor:338 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/00216deb-5dd0-4b4f-8940-c17dc329303c/volumes/kubernetes.io~projected/kube-api-access-cvgpz DeviceMajor:0 DeviceMinor:51 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1b941fae-3c27-4d68-8205-38e492179ee9/volumes/kubernetes.io~projected/kube-api-access-p8mfv DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7970eec1-3765-48a4-b09c-3dec2d29956b/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:214 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-312 DeviceMajor:0 DeviceMinor:312 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8a6e7f39-7853-4fec-9ea2-30099983e5bd/volumes/kubernetes.io~projected/kube-api-access-hbnlg DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d407313d-3333-4ec4-85bb-cd27cf4d7422/volumes/kubernetes.io~projected/kube-api-access-fsmvn DeviceMajor:0 DeviceMinor:236 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/70bb916a-832b-4bce-9c0a-f01bf4e3abc4/volumes/kubernetes.io~projected/kube-api-access-xkzgz DeviceMajor:0 DeviceMinor:238 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bcc6412c-2003-49de-9817-a3713e28afef/volumes/kubernetes.io~projected/kube-api-access-vdsp8 DeviceMajor:0 DeviceMinor:52 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6f895928b8cb30ce26ce5335d0bc02d8bfc8edfb5035df303cbdf4e67fe480ac/userdata/shm DeviceMajor:0 DeviceMinor:222 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a5dc0434840b345873d399db22c8ae5810e840e4bc25115b32f7425fe76d5423/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ccb76443164367970060f9ea26b8693190d38ed29108de716b17d69abc051c04/userdata/shm DeviceMajor:0 DeviceMinor:354 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e7ea769b-e719-4090-bcd4-05b1b88fcc06/volumes/kubernetes.io~projected/kube-api-access-rdxs8 DeviceMajor:0 DeviceMinor:213 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7b2912f5c7df5c0bd7c90ec4fae06b2a295b3fc96e054a6682e822f05bbc3be2/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-46 DeviceMajor:0 DeviceMinor:46 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bb59051d-c2df-4362-a157-00f14f915e79/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:219 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-280 DeviceMajor:0 DeviceMinor:280 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fba1e7f9-ca37-46dc-8c66-56f1d9563acd/volumes/kubernetes.io~projected/kube-api-access-khv87 DeviceMajor:0 DeviceMinor:347 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-356 DeviceMajor:0 DeviceMinor:356 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2e013e38-9840-45c2-b4a0-80df0f319109/volumes/kubernetes.io~projected/kube-api-access-wzjlr DeviceMajor:0 DeviceMinor:53 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/880c12f6309a8d768f868b8369ee6a928463263c6f30c39a8cb01909d7aa9902/userdata/shm DeviceMajor:0 DeviceMinor:220 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e08e52ee478f9a7272ba40ba4b90357351ebf80ac971cebdba7f80f520e407e2/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ffb9c2812bfd05c93dc5e7fb90c15e9b272744135cb9e37b50a256bfa0a7432b/userdata/shm DeviceMajor:0 DeviceMinor:362 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/5e15a59f-6aac-45e7-8812-ae97d0559052/volumes/kubernetes.io~projected/kube-api-access-8zcj8 DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a12d30a7-8564-43c1-9742-3194fe8b975f/volumes/kubernetes.io~projected/kube-api-access-gnzsm DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1b941fae-3c27-4d68-8205-38e492179ee9/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/bb59051d-c2df-4362-a157-00f14f915e79/volumes/kubernetes.io~projected/kube-api-access-92xdl DeviceMajor:0 DeviceMinor:234 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ba1d2e479174d30eac7876b3f30fff0702c316d073a37353f03d9a09e2110c84/userdata/shm DeviceMajor:0 DeviceMinor:288 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/058820ef-eef1-41ca-b331-670bab146861/volumes/kubernetes.io~projected/kube-api-access-gx82t DeviceMajor:0 DeviceMinor:314 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/927f16040227de27cef857b98df6409aba54e22122880ff2d3dc48f638daff3d/userdata/shm DeviceMajor:0 DeviceMinor:330 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1b941fae-3c27-4d68-8205-38e492179ee9/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:50 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0baeba702a835a54788bc5346bb21d6246e4a408ed299e692f0a26a125c169cc/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4a724964-bd57-4d56-a042-a0d490115eaa/volumes/kubernetes.io~projected/kube-api-access-kfj7s DeviceMajor:0 DeviceMinor:212 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9226216fff100a82b2a6715ce444110115c3d396b98acc865c89fc819868e4b7/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7970eec1-3765-48a4-b09c-3dec2d29956b/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:215 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-265 DeviceMajor:0 DeviceMinor:265 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5e15a59f-6aac-45e7-8812-ae97d0559052/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:47 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-189 DeviceMajor:0 DeviceMinor:189 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0d4352db-d2e7-43ba-97ba-9e344abbac8b/volumes/kubernetes.io~projected/kube-api-access-vp6rs DeviceMajor:0 DeviceMinor:287 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a4620bc86b004bf248b53e4ab03c70adcebaba74a374200d2fdd272ee9eb8eb3/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-136 DeviceMajor:0 DeviceMinor:136 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cafdbe7c69ef251c3ee7143456fc8e488e67d6ac1609f2906deea2ffb53664cd/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-364 DeviceMajor:0 DeviceMinor:364 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-274 DeviceMajor:0 DeviceMinor:274 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4a724964-bd57-4d56-a042-a0d490115eaa/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d8a5ccab-4d5e-4aaf-ad08-766238eca05f/volumes/kubernetes.io~projected/kube-api-access-q6cqj DeviceMajor:0 DeviceMinor:208 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/15e947b3c1de139876bcd194085602f71b94f52d655de40ca926f9773c1ffbea/userdata/shm DeviceMajor:0 DeviceMinor:79 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c27ff402-235a-4d73-a139-cf928a6d5cbe/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:218 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-340 DeviceMajor:0 DeviceMinor:340 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/037ca099-5ad7-4874-a984-462c07801e70/volumes/kubernetes.io~projected/kube-api-access-8thtk DeviceMajor:0 DeviceMinor:211 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2c371efbc265a2789bc9913d731f39b76b38dfbd89aa2beb1cf181b6b2a82a88/userdata/shm DeviceMajor:0 DeviceMinor:226 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0aea4ba3a0f3f845dfc67cf48042c64292a3373bea0a78fe95a6a0a1ce6fdfaf/userdata/shm DeviceMajor:0 DeviceMinor:352 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1/volumes/kubernetes.io~projected/kube-api-access-hpm8k DeviceMajor:0 DeviceMinor:54 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6a8f947a-0f8f-41aa-b646-4459fa8126a3/volumes/kubernetes.io~projected/kube-api-access-9j226 DeviceMajor:0 DeviceMinor:206 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f219e3c2-6da8-4c88-9ec4-b26cf4d10326/volumes/kubernetes.io~projected/kube-api-access-pjj74 DeviceMajor:0 DeviceMinor:235 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/321ecf30-5108-4320-9a58-6b204001639c/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:360 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7970eec1-3765-48a4-b09c-3dec2d29956b/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:237 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/17570aba-70d9-46ea-a0ca-3a465635a17c/volumes/kubernetes.io~projected/kube-api-access-zpdrz DeviceMajor:0 DeviceMinor:342 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-358 DeviceMajor:0 DeviceMinor:358 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4730fc3bf0ae3cee7a9949aff05aa788af46b4d0acdf21c3911d6bc1b09a3d6c/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-336 DeviceMajor:0 DeviceMinor:336 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/321ecf30-5108-4320-9a58-6b204001639c/volumes/kubernetes.io~projected/kube-api-access-8bhpq DeviceMajor:0 DeviceMinor:361 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e7ea769b-e719-4090-bcd4-05b1b88fcc06/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bac08a467e759b5f9eb601b19c0cafeffee46e2ef10d11078b6c9d8231c3d408/userdata/shm DeviceMajor:0 DeviceMinor:224 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0aea4ba3a0f3f84 MacAddress:86:05:dd:3b:7b:12 Speed:10000 Mtu:8901} {Name:2c371efbc265a27 MacAddress:f6:81:26:f3:7f:39 Speed:10000 Mtu:8901} {Name:690b74893e54482 MacAddress:aa:66:f2:3a:8b:42 Speed:10000 Mtu:8901} {Name:6f895928b8cb30c MacAddress:02:fe:63:bf:d5:d0 Speed:10000 Mtu:8901} {Name:7b2912f5c7df5c0 MacAddress:fe:3b:f8:61:c5:96 Speed:10000 Mtu:8901} {Name:880c12f6309a8d7 MacAddress:ba:5e:fb:d7:07:e7 Speed:10000 Mtu:8901} {Name:9226216fff100a8 MacAddress:0a:29:8e:86:99:b0 Speed:10000 Mtu:8901} {Name:927f16040227de2 MacAddress:fe:37:bb:76:e5:56 Speed:10000 Mtu:8901} {Name:ae783b4a23f870f MacAddress:2e:b9:a4:f4:71:95 Speed:10000 Mtu:8901} {Name:ba1d2e479174d30 MacAddress:1a:3f:89:9b:70:c5 Speed:10000 Mtu:8901} {Name:bac08a467e759b5 MacAddress:d6:06:58:2e:47:27 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:ea:0f:98:35 Speed:0 Mtu:9001} {Name:br-int MacAddress:1a:22:1c:33:34:2f Speed:0 Mtu:8901} {Name:cafdbe7c69ef251 MacAddress:86:ec:58:5d:22:29 Speed:10000 Mtu:8901} {Name:ccb764431643679 MacAddress:ae:a6:19:96:14:4e Speed:10000 Mtu:8901} {Name:e08e52ee478f9a7 MacAddress:9e:ea:56:10:03:a3 Speed:10000 Mtu:8901} {Name:ef98d8e55395596 MacAddress:72:00:4c:c9:6d:c6 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:ea:0f:98:35 Speed:0 Mtu:9001} {Name:ffb9c2812bfd05c MacAddress:c6:c3:62:da:d1:91 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:2a:94:6d:45:25:2c Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:b2:46:ac:74:d9:06 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 01 14:16:52.731447 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.730485    4993 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 01 14:16:52.731447 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.730532    4993 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 01 14:16:52.731447 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.730833    4993 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 01 14:16:52.731447 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.730858    4993 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-37.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 01 14:16:52.731447 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.730987    4993 topology_manager.go:143] "Creating topology manager with none policy"
Nov 01 14:16:52.731447 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.730994    4993 container_manager_linux.go:311] "Creating device plugin manager"
Nov 01 14:16:52.731447 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.731007    4993 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 14:16:52.731447 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.731020    4993 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 14:16:52.731447 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.731038    4993 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 01 14:16:52.731447 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.731219    4993 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 01 14:16:52.731447 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.731330    4993 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 01 14:16:52.731447 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.731365    4993 kubelet.go:493] "Attempting to sync node with API server"
Nov 01 14:16:52.731447 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.731380    4993 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 01 14:16:52.731447 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.731395    4993 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 01 14:16:52.731447 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.731403    4993 kubelet.go:405] "Adding apiserver pod source"
Nov 01 14:16:52.731447 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.731411    4993 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 01 14:16:52.732332 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.732316    4993 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 01 14:16:52.732380 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.732334    4993 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 14:16:52.732815 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.732799    4993 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 01 14:16:52.744288 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.744270    4993 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 01 14:16:52.744288 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.744285    4993 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 14:16:52.744414 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.744383    4993 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 01 14:16:52.744581 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.744569    4993 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 01 14:16:52.744623 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.744599    4993 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 01 14:16:52.744695 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.744685    4993 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 01 14:16:52.744734 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.744700    4993 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 01 14:16:52.744734 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.744706    4993 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 01 14:16:52.744734 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.744712    4993 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 01 14:16:52.744734 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.744717    4993 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 01 14:16:52.744734 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.744723    4993 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 01 14:16:52.744734 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.744728    4993 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 01 14:16:52.744734 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.744733    4993 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 01 14:16:52.744976 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.744740    4993 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 01 14:16:52.744976 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.744746    4993 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 01 14:16:52.744976 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.744754    4993 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 01 14:16:52.744976 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.744763    4993 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 01 14:16:52.744976 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.744786    4993 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 01 14:16:52.744976 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.744793    4993 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 01 14:16:52.748458 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.748443    4993 server.go:1263] "Started kubelet"
Nov 01 14:16:52.748552 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.748503    4993 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 01 14:16:52.749029 ip-10-0-134-37 systemd[1]: Started Kubernetes Kubelet.
Nov 01 14:16:52.752636 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.749124    4993 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 01 14:16:52.752636 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.751149    4993 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 01 14:16:52.752636 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.751205    4993 server.go:317] "Adding debug handlers to kubelet server"
Nov 01 14:16:52.752636 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.751445    4993 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 01 14:16:52.755810 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.755790    4993 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 01 14:16:52.755897 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.755860    4993 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 01 14:16:52.755967 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.755931    4993 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 14:11:11 +0000 UTC" deadline="2028-08-07 04:21:34.113234576 +0000 UTC"
Nov 01 14:16:52.759209 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.755957    4993 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15470h4m41.357280495s"
Nov 01 14:16:52.759292 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.756009    4993 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 01 14:16:52.759292 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.759235    4993 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 01 14:16:52.759292 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.756023    4993 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 01 14:16:52.759292 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:52.756065    4993 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-37.us-west-2.compute.internal\" not found"
Nov 01 14:16:52.760415 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.760397    4993 factory.go:55] Registering systemd factory
Nov 01 14:16:52.760415 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.760417    4993 factory.go:223] Registration of the systemd container factory successfully
Nov 01 14:16:52.760660 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.760644    4993 factory.go:153] Registering CRI-O factory
Nov 01 14:16:52.760726 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.760664    4993 factory.go:223] Registration of the crio container factory successfully
Nov 01 14:16:52.760726 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.760720    4993 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 01 14:16:52.760825 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.760738    4993 factory.go:103] Registering Raw factory
Nov 01 14:16:52.760825 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.760756    4993 manager.go:1199] Started watching for new ooms in manager
Nov 01 14:16:52.761065 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.761053    4993 manager.go:319] Starting recovery of all containers
Nov 01 14:16:52.762535 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.762181    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e7e18b9-78ee-4349-b00c-a0800e8f8341" volumeName="kubernetes.io/projected/3e7e18b9-78ee-4349-b00c-a0800e8f8341-kube-api-access-stw2n" seLinuxMountContext=""
Nov 01 14:16:52.762642 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.762541    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7970eec1-3765-48a4-b09c-3dec2d29956b" volumeName="kubernetes.io/secret/7970eec1-3765-48a4-b09c-3dec2d29956b-image-registry-private-configuration" seLinuxMountContext=""
Nov 01 14:16:52.762642 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.762567    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b73fabd2-0d2c-4988-bbcf-7564d9331390" volumeName="kubernetes.io/configmap/b73fabd2-0d2c-4988-bbcf-7564d9331390-multus-daemon-config" seLinuxMountContext=""
Nov 01 14:16:52.762642 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.762607    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c8cc47b-fb01-4195-a7fc-eeefc706017e" volumeName="kubernetes.io/projected/0c8cc47b-fb01-4195-a7fc-eeefc706017e-kube-api-access-wgmsd" seLinuxMountContext=""
Nov 01 14:16:52.762642 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.762628    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4a724964-bd57-4d56-a042-a0d490115eaa" volumeName="kubernetes.io/configmap/4a724964-bd57-4d56-a042-a0d490115eaa-config" seLinuxMountContext=""
Nov 01 14:16:52.762943 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.762656    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e15a59f-6aac-45e7-8812-ae97d0559052" volumeName="kubernetes.io/empty-dir/5e15a59f-6aac-45e7-8812-ae97d0559052-tmp" seLinuxMountContext=""
Nov 01 14:16:52.762943 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.762674    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70bb916a-832b-4bce-9c0a-f01bf4e3abc4" volumeName="kubernetes.io/secret/70bb916a-832b-4bce-9c0a-f01bf4e3abc4-cert" seLinuxMountContext=""
Nov 01 14:16:52.762943 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.762693    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7970eec1-3765-48a4-b09c-3dec2d29956b" volumeName="kubernetes.io/empty-dir/7970eec1-3765-48a4-b09c-3dec2d29956b-ca-trust-extracted" seLinuxMountContext=""
Nov 01 14:16:52.762943 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.762754    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7970eec1-3765-48a4-b09c-3dec2d29956b" volumeName="kubernetes.io/secret/7970eec1-3765-48a4-b09c-3dec2d29956b-installation-pull-secrets" seLinuxMountContext=""
Nov 01 14:16:52.762943 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.762774    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7a26b817-3283-4d59-8e8f-3bb4c0fa4987" volumeName="kubernetes.io/empty-dir/7a26b817-3283-4d59-8e8f-3bb4c0fa4987-tmp" seLinuxMountContext=""
Nov 01 14:16:52.762943 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.762800    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bb59051d-c2df-4362-a157-00f14f915e79" volumeName="kubernetes.io/configmap/bb59051d-c2df-4362-a157-00f14f915e79-config" seLinuxMountContext=""
Nov 01 14:16:52.762943 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.762820    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="058820ef-eef1-41ca-b331-670bab146861" volumeName="kubernetes.io/empty-dir/058820ef-eef1-41ca-b331-670bab146861-data-volume" seLinuxMountContext=""
Nov 01 14:16:52.762943 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.762851    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d4352db-d2e7-43ba-97ba-9e344abbac8b" volumeName="kubernetes.io/projected/0d4352db-d2e7-43ba-97ba-9e344abbac8b-kube-api-access-vp6rs" seLinuxMountContext=""
Nov 01 14:16:52.762943 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.762873    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="384cd50b-370f-474f-b80e-89a544a9bbbd" volumeName="kubernetes.io/configmap/384cd50b-370f-474f-b80e-89a544a9bbbd-service-ca-bundle" seLinuxMountContext=""
Nov 01 14:16:52.762943 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.762896    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7970eec1-3765-48a4-b09c-3dec2d29956b" volumeName="kubernetes.io/projected/7970eec1-3765-48a4-b09c-3dec2d29956b-kube-api-access-8gnmx" seLinuxMountContext=""
Nov 01 14:16:52.762943 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.762909    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7a26b817-3283-4d59-8e8f-3bb4c0fa4987" volumeName="kubernetes.io/secret/7a26b817-3283-4d59-8e8f-3bb4c0fa4987-serving-cert" seLinuxMountContext=""
Nov 01 14:16:52.762943 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.762922    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b1eb53c-8c9a-46c4-b654-2aef42491b14" volumeName="kubernetes.io/secret/8b1eb53c-8c9a-46c4-b654-2aef42491b14-original-pull-secret" seLinuxMountContext=""
Nov 01 14:16:52.763766 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.763020    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5a4fefb-cc48-4638-b0f8-590e36bd0118" volumeName="kubernetes.io/configmap/b5a4fefb-cc48-4638-b0f8-590e36bd0118-nginx-conf" seLinuxMountContext=""
Nov 01 14:16:52.763766 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.763035    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ccc5b32b-07bc-4b3c-91ef-b184b0a9afe3" volumeName="kubernetes.io/secret/ccc5b32b-07bc-4b3c-91ef-b184b0a9afe3-agent-certs" seLinuxMountContext=""
Nov 01 14:16:52.763766 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.763056    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="321ecf30-5108-4320-9a58-6b204001639c" volumeName="kubernetes.io/projected/321ecf30-5108-4320-9a58-6b204001639c-kube-api-access-8bhpq" seLinuxMountContext=""
Nov 01 14:16:52.763766 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.763071    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6a8f947a-0f8f-41aa-b646-4459fa8126a3" volumeName="kubernetes.io/projected/6a8f947a-0f8f-41aa-b646-4459fa8126a3-kube-api-access-9j226" seLinuxMountContext=""
Nov 01 14:16:52.763766 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.763083    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7970eec1-3765-48a4-b09c-3dec2d29956b" volumeName="kubernetes.io/projected/7970eec1-3765-48a4-b09c-3dec2d29956b-bound-sa-token" seLinuxMountContext=""
Nov 01 14:16:52.763766 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.763101    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c27ff402-235a-4d73-a139-cf928a6d5cbe" volumeName="kubernetes.io/projected/c27ff402-235a-4d73-a139-cf928a6d5cbe-kube-api-access-rjv76" seLinuxMountContext=""
Nov 01 14:16:52.763766 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.763114    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e78d2494-62c4-4070-a1eb-8ce40329ff57" volumeName="kubernetes.io/projected/e78d2494-62c4-4070-a1eb-8ce40329ff57-kube-api-access-wxdcx" seLinuxMountContext=""
Nov 01 14:16:52.763766 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.763131    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f219e3c2-6da8-4c88-9ec4-b26cf4d10326" volumeName="kubernetes.io/projected/f219e3c2-6da8-4c88-9ec4-b26cf4d10326-kube-api-access-pjj74" seLinuxMountContext=""
Nov 01 14:16:52.763766 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.763147    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7970eec1-3765-48a4-b09c-3dec2d29956b" volumeName="kubernetes.io/configmap/7970eec1-3765-48a4-b09c-3dec2d29956b-registry-certificates" seLinuxMountContext=""
Nov 01 14:16:52.763766 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.763161    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a12d30a7-8564-43c1-9742-3194fe8b975f" volumeName="kubernetes.io/projected/a12d30a7-8564-43c1-9742-3194fe8b975f-kube-api-access-gnzsm" seLinuxMountContext=""
Nov 01 14:16:52.763766 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.763202    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c27ff402-235a-4d73-a139-cf928a6d5cbe" volumeName="kubernetes.io/secret/c27ff402-235a-4d73-a139-cf928a6d5cbe-default-certificate" seLinuxMountContext=""
Nov 01 14:16:52.763766 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.763233    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7ea769b-e719-4090-bcd4-05b1b88fcc06" volumeName="kubernetes.io/empty-dir/e7ea769b-e719-4090-bcd4-05b1b88fcc06-ca-trust-extracted" seLinuxMountContext=""
Nov 01 14:16:52.763766 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.763246    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="00216deb-5dd0-4b4f-8940-c17dc329303c" volumeName="kubernetes.io/projected/00216deb-5dd0-4b4f-8940-c17dc329303c-kube-api-access-cvgpz" seLinuxMountContext=""
Nov 01 14:16:52.763766 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.763259    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b941fae-3c27-4d68-8205-38e492179ee9" volumeName="kubernetes.io/configmap/1b941fae-3c27-4d68-8205-38e492179ee9-ovnkube-script-lib" seLinuxMountContext=""
Nov 01 14:16:52.763766 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.763279    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bcc6412c-2003-49de-9817-a3713e28afef" volumeName="kubernetes.io/projected/bcc6412c-2003-49de-9817-a3713e28afef-kube-api-access-vdsp8" seLinuxMountContext=""
Nov 01 14:16:52.763766 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.763291    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7ea769b-e719-4090-bcd4-05b1b88fcc06" volumeName="kubernetes.io/configmap/e7ea769b-e719-4090-bcd4-05b1b88fcc06-trusted-ca" seLinuxMountContext=""
Nov 01 14:16:52.763766 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.763308    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="384cd50b-370f-474f-b80e-89a544a9bbbd" volumeName="kubernetes.io/empty-dir/384cd50b-370f-474f-b80e-89a544a9bbbd-archives-path" seLinuxMountContext=""
Nov 01 14:16:52.763766 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.763322    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e15a59f-6aac-45e7-8812-ae97d0559052" volumeName="kubernetes.io/empty-dir/5e15a59f-6aac-45e7-8812-ae97d0559052-etc-tuned" seLinuxMountContext=""
Nov 01 14:16:52.763766 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.763334    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7a26b817-3283-4d59-8e8f-3bb4c0fa4987" volumeName="kubernetes.io/configmap/7a26b817-3283-4d59-8e8f-3bb4c0fa4987-service-ca-bundle" seLinuxMountContext=""
Nov 01 14:16:52.763766 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.763351    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bb59051d-c2df-4362-a157-00f14f915e79" volumeName="kubernetes.io/configmap/bb59051d-c2df-4362-a157-00f14f915e79-trusted-ca" seLinuxMountContext=""
Nov 01 14:16:52.763766 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.763365    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bcc6412c-2003-49de-9817-a3713e28afef" volumeName="kubernetes.io/configmap/bcc6412c-2003-49de-9817-a3713e28afef-cni-binary-copy" seLinuxMountContext=""
Nov 01 14:16:52.763766 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.763387    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1" volumeName="kubernetes.io/projected/ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1-kube-api-access-hpm8k" seLinuxMountContext=""
Nov 01 14:16:52.763766 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.763400    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7ea769b-e719-4090-bcd4-05b1b88fcc06" volumeName="kubernetes.io/configmap/e7ea769b-e719-4090-bcd4-05b1b88fcc06-registry-certificates" seLinuxMountContext=""
Nov 01 14:16:52.763766 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.763412    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e15a59f-6aac-45e7-8812-ae97d0559052" volumeName="kubernetes.io/projected/5e15a59f-6aac-45e7-8812-ae97d0559052-kube-api-access-8zcj8" seLinuxMountContext=""
Nov 01 14:16:52.763766 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.763432    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bb59051d-c2df-4362-a157-00f14f915e79" volumeName="kubernetes.io/projected/bb59051d-c2df-4362-a157-00f14f915e79-kube-api-access-92xdl" seLinuxMountContext=""
Nov 01 14:16:52.763766 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.763445    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8a5ccab-4d5e-4aaf-ad08-766238eca05f" volumeName="kubernetes.io/projected/d8a5ccab-4d5e-4aaf-ad08-766238eca05f-kube-api-access-q6cqj" seLinuxMountContext=""
Nov 01 14:16:52.763766 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.763464    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fba1e7f9-ca37-46dc-8c66-56f1d9563acd" volumeName="kubernetes.io/projected/fba1e7f9-ca37-46dc-8c66-56f1d9563acd-kube-api-access-khv87" seLinuxMountContext=""
Nov 01 14:16:52.763766 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.763478    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="058820ef-eef1-41ca-b331-670bab146861" volumeName="kubernetes.io/projected/058820ef-eef1-41ca-b331-670bab146861-kube-api-access-gx82t" seLinuxMountContext=""
Nov 01 14:16:52.763766 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.763509    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e7e18b9-78ee-4349-b00c-a0800e8f8341" volumeName="kubernetes.io/secret/3e7e18b9-78ee-4349-b00c-a0800e8f8341-serving-cert" seLinuxMountContext=""
Nov 01 14:16:52.763766 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.763529    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7a26b817-3283-4d59-8e8f-3bb4c0fa4987" volumeName="kubernetes.io/projected/7a26b817-3283-4d59-8e8f-3bb4c0fa4987-kube-api-access-bv8hp" seLinuxMountContext=""
Nov 01 14:16:52.763766 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.763545    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a6e7f39-7853-4fec-9ea2-30099983e5bd" volumeName="kubernetes.io/projected/8a6e7f39-7853-4fec-9ea2-30099983e5bd-kube-api-access-hbnlg" seLinuxMountContext=""
Nov 01 14:16:52.763766 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.763563    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a12d30a7-8564-43c1-9742-3194fe8b975f" volumeName="kubernetes.io/configmap/a12d30a7-8564-43c1-9742-3194fe8b975f-serviceca" seLinuxMountContext=""
Nov 01 14:16:52.763766 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.763586    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c8cc47b-fb01-4195-a7fc-eeefc706017e" volumeName="kubernetes.io/configmap/0c8cc47b-fb01-4195-a7fc-eeefc706017e-config-volume" seLinuxMountContext=""
Nov 01 14:16:52.763766 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.763603    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b941fae-3c27-4d68-8205-38e492179ee9" volumeName="kubernetes.io/projected/1b941fae-3c27-4d68-8205-38e492179ee9-kube-api-access-p8mfv" seLinuxMountContext=""
Nov 01 14:16:52.763766 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.763621    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b941fae-3c27-4d68-8205-38e492179ee9" volumeName="kubernetes.io/secret/1b941fae-3c27-4d68-8205-38e492179ee9-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 01 14:16:52.763766 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.763637    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="384cd50b-370f-474f-b80e-89a544a9bbbd" volumeName="kubernetes.io/configmap/384cd50b-370f-474f-b80e-89a544a9bbbd-trusted-ca-bundle" seLinuxMountContext=""
Nov 01 14:16:52.763766 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.763657    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bb59051d-c2df-4362-a157-00f14f915e79" volumeName="kubernetes.io/secret/bb59051d-c2df-4362-a157-00f14f915e79-serving-cert" seLinuxMountContext=""
Nov 01 14:16:52.763766 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.763672    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ccc5b32b-07bc-4b3c-91ef-b184b0a9afe3" volumeName="kubernetes.io/configmap/ccc5b32b-07bc-4b3c-91ef-b184b0a9afe3-konnectivity-ca" seLinuxMountContext=""
Nov 01 14:16:52.763766 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.763687    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="037ca099-5ad7-4874-a984-462c07801e70" volumeName="kubernetes.io/projected/037ca099-5ad7-4874-a984-462c07801e70-kube-api-access-8thtk" seLinuxMountContext=""
Nov 01 14:16:52.763766 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.763748    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="321ecf30-5108-4320-9a58-6b204001639c" volumeName="kubernetes.io/configmap/321ecf30-5108-4320-9a58-6b204001639c-signing-cabundle" seLinuxMountContext=""
Nov 01 14:16:52.763766 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.763762    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="321ecf30-5108-4320-9a58-6b204001639c" volumeName="kubernetes.io/secret/321ecf30-5108-4320-9a58-6b204001639c-signing-key" seLinuxMountContext=""
Nov 01 14:16:52.763766 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.763778    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7970eec1-3765-48a4-b09c-3dec2d29956b" volumeName="kubernetes.io/configmap/7970eec1-3765-48a4-b09c-3dec2d29956b-trusted-ca" seLinuxMountContext=""
Nov 01 14:16:52.763766 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.763791    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a6e7f39-7853-4fec-9ea2-30099983e5bd" volumeName="kubernetes.io/configmap/8a6e7f39-7853-4fec-9ea2-30099983e5bd-iptables-alerter-script" seLinuxMountContext=""
Nov 01 14:16:52.766032 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.763803    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c8cc47b-fb01-4195-a7fc-eeefc706017e" volumeName="kubernetes.io/empty-dir/0c8cc47b-fb01-4195-a7fc-eeefc706017e-tmp-dir" seLinuxMountContext=""
Nov 01 14:16:52.766032 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.763826    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2460be58-d51b-4876-bf9b-2a0e57901386" volumeName="kubernetes.io/projected/2460be58-d51b-4876-bf9b-2a0e57901386-kube-api-access-7jbv4" seLinuxMountContext=""
Nov 01 14:16:52.766032 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.763840    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="384cd50b-370f-474f-b80e-89a544a9bbbd" volumeName="kubernetes.io/projected/384cd50b-370f-474f-b80e-89a544a9bbbd-kube-api-access-chwvj" seLinuxMountContext=""
Nov 01 14:16:52.766032 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.763859    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d407313d-3333-4ec4-85bb-cd27cf4d7422" volumeName="kubernetes.io/configmap/d407313d-3333-4ec4-85bb-cd27cf4d7422-samples-operator-config" seLinuxMountContext=""
Nov 01 14:16:52.766032 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.763872    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="037ca099-5ad7-4874-a984-462c07801e70" volumeName="kubernetes.io/configmap/037ca099-5ad7-4874-a984-462c07801e70-telemetry-config" seLinuxMountContext=""
Nov 01 14:16:52.766032 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.763884    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4a724964-bd57-4d56-a042-a0d490115eaa" volumeName="kubernetes.io/secret/4a724964-bd57-4d56-a042-a0d490115eaa-serving-cert" seLinuxMountContext=""
Nov 01 14:16:52.766032 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.763901    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70bb916a-832b-4bce-9c0a-f01bf4e3abc4" volumeName="kubernetes.io/projected/70bb916a-832b-4bce-9c0a-f01bf4e3abc4-kube-api-access-xkzgz" seLinuxMountContext=""
Nov 01 14:16:52.766032 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.763915    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7a26b817-3283-4d59-8e8f-3bb4c0fa4987" volumeName="kubernetes.io/empty-dir/7a26b817-3283-4d59-8e8f-3bb4c0fa4987-snapshots" seLinuxMountContext=""
Nov 01 14:16:52.766032 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.763936    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b73fabd2-0d2c-4988-bbcf-7564d9331390" volumeName="kubernetes.io/configmap/b73fabd2-0d2c-4988-bbcf-7564d9331390-cni-binary-copy" seLinuxMountContext=""
Nov 01 14:16:52.766032 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.763973    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bcc6412c-2003-49de-9817-a3713e28afef" volumeName="kubernetes.io/configmap/bcc6412c-2003-49de-9817-a3713e28afef-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 01 14:16:52.766032 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.763987    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c27ff402-235a-4d73-a139-cf928a6d5cbe" volumeName="kubernetes.io/secret/c27ff402-235a-4d73-a139-cf928a6d5cbe-stats-auth" seLinuxMountContext=""
Nov 01 14:16:52.766032 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.764007    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7ea769b-e719-4090-bcd4-05b1b88fcc06" volumeName="kubernetes.io/projected/e7ea769b-e719-4090-bcd4-05b1b88fcc06-kube-api-access-rdxs8" seLinuxMountContext=""
Nov 01 14:16:52.766032 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.764022    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="00216deb-5dd0-4b4f-8940-c17dc329303c" volumeName="kubernetes.io/empty-dir/00216deb-5dd0-4b4f-8940-c17dc329303c-tmp-dir" seLinuxMountContext=""
Nov 01 14:16:52.766032 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.764047    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17570aba-70d9-46ea-a0ca-3a465635a17c" volumeName="kubernetes.io/projected/17570aba-70d9-46ea-a0ca-3a465635a17c-kube-api-access-zpdrz" seLinuxMountContext=""
Nov 01 14:16:52.766032 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.764061    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b73fabd2-0d2c-4988-bbcf-7564d9331390" volumeName="kubernetes.io/projected/b73fabd2-0d2c-4988-bbcf-7564d9331390-kube-api-access-ddmmn" seLinuxMountContext=""
Nov 01 14:16:52.766032 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.764074    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d407313d-3333-4ec4-85bb-cd27cf4d7422" volumeName="kubernetes.io/projected/d407313d-3333-4ec4-85bb-cd27cf4d7422-kube-api-access-fsmvn" seLinuxMountContext=""
Nov 01 14:16:52.766032 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.764091    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7ea769b-e719-4090-bcd4-05b1b88fcc06" volumeName="kubernetes.io/secret/e7ea769b-e719-4090-bcd4-05b1b88fcc06-image-registry-private-configuration" seLinuxMountContext=""
Nov 01 14:16:52.766032 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.764117    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e7e18b9-78ee-4349-b00c-a0800e8f8341" volumeName="kubernetes.io/configmap/3e7e18b9-78ee-4349-b00c-a0800e8f8341-config" seLinuxMountContext=""
Nov 01 14:16:52.766032 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.764133    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b941fae-3c27-4d68-8205-38e492179ee9" volumeName="kubernetes.io/configmap/1b941fae-3c27-4d68-8205-38e492179ee9-env-overrides" seLinuxMountContext=""
Nov 01 14:16:52.766032 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.764148    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7ea769b-e719-4090-bcd4-05b1b88fcc06" volumeName="kubernetes.io/secret/e7ea769b-e719-4090-bcd4-05b1b88fcc06-installation-pull-secrets" seLinuxMountContext=""
Nov 01 14:16:52.766032 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.764160    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b941fae-3c27-4d68-8205-38e492179ee9" volumeName="kubernetes.io/configmap/1b941fae-3c27-4d68-8205-38e492179ee9-ovnkube-config" seLinuxMountContext=""
Nov 01 14:16:52.766032 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.764225    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e013e38-9840-45c2-b4a0-80df0f319109" volumeName="kubernetes.io/projected/2e013e38-9840-45c2-b4a0-80df0f319109-kube-api-access-wzjlr" seLinuxMountContext=""
Nov 01 14:16:52.766032 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.764221    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xmwtp_bb59051d-c2df-4362-a157-00f14f915e79/console-operator/1.log"
Nov 01 14:16:52.766032 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.764241    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4a724964-bd57-4d56-a042-a0d490115eaa" volumeName="kubernetes.io/projected/4a724964-bd57-4d56-a042-a0d490115eaa-kube-api-access-kfj7s" seLinuxMountContext=""
Nov 01 14:16:52.766032 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.764353    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7a26b817-3283-4d59-8e8f-3bb4c0fa4987" volumeName="kubernetes.io/configmap/7a26b817-3283-4d59-8e8f-3bb4c0fa4987-trusted-ca-bundle" seLinuxMountContext=""
Nov 01 14:16:52.766032 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.764367    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bcc6412c-2003-49de-9817-a3713e28afef" volumeName="kubernetes.io/configmap/bcc6412c-2003-49de-9817-a3713e28afef-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 01 14:16:52.766032 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.764386    4993 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7ea769b-e719-4090-bcd4-05b1b88fcc06" volumeName="kubernetes.io/projected/e7ea769b-e719-4090-bcd4-05b1b88fcc06-bound-sa-token" seLinuxMountContext=""
Nov 01 14:16:52.766032 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.764398    4993 reconstruct.go:97] "Volume reconstruction finished"
Nov 01 14:16:52.766032 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.764420    4993 reconciler.go:29] "Reconciler: start to sync state"
Nov 01 14:16:52.770490 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.770469    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xmwtp_bb59051d-c2df-4362-a157-00f14f915e79/console-operator/1.log"
Nov 01 14:16:52.773043 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.773010    4993 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 01 14:16:52.786938 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.786910    4993 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 01 14:16:52.786938 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.786936    4993 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 01 14:16:52.787112 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.787048    4993 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 01 14:16:52.787159 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:52.787114    4993 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 01 14:16:52.817351 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.817317    4993 generic.go:356] "Generic (PLEG): container finished" podID="bcc6412c-2003-49de-9817-a3713e28afef" containerID="c34dec7ce4ebed9bd4f05417c7eae6058859d0a0f3bdc5ea5163ead034de55d9" exitCode=0
Nov 01 14:16:52.817351 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.817346    4993 generic.go:356] "Generic (PLEG): container finished" podID="bcc6412c-2003-49de-9817-a3713e28afef" containerID="8863917ee3587e3abf9a51b988aec40ad35ef2d6b05bf950f5f530838d62a1df" exitCode=0
Nov 01 14:16:52.817351 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.817354    4993 generic.go:356] "Generic (PLEG): container finished" podID="bcc6412c-2003-49de-9817-a3713e28afef" containerID="5126218ef0fab3767c5e0b17a270a071f3da376d0abd638a12cea24cfb10e6e4" exitCode=0
Nov 01 14:16:52.817563 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.817360    4993 generic.go:356] "Generic (PLEG): container finished" podID="bcc6412c-2003-49de-9817-a3713e28afef" containerID="e0dbf291ebc64df129cf689c155ac541c396ab67b535413524ec389de84fe542" exitCode=0
Nov 01 14:16:52.817563 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.817366    4993 generic.go:356] "Generic (PLEG): container finished" podID="bcc6412c-2003-49de-9817-a3713e28afef" containerID="02ca50c2f61f7e649a9fbbe54e49525427915c8948a281e26f6bd17f137475ed" exitCode=0
Nov 01 14:16:52.817563 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.817372    4993 generic.go:356] "Generic (PLEG): container finished" podID="bcc6412c-2003-49de-9817-a3713e28afef" containerID="752cbe43539da3516c58551b06f593c22d6f7d257f4af36aeb85556ed092bb1b" exitCode=0
Nov 01 14:16:52.819259 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.819227    4993 generic.go:356] "Generic (PLEG): container finished" podID="4d681fe8617362d2405aa92b260bf206" containerID="4cfff79920975d1bc452be97e1e84bbbcb57c55587987af90f476708a7dff340" exitCode=0
Nov 01 14:16:52.839083 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.839062    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xmwtp_bb59051d-c2df-4362-a157-00f14f915e79/console-operator/1.log"
Nov 01 14:16:52.839206 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.839093    4993 generic.go:356] "Generic (PLEG): container finished" podID="bb59051d-c2df-4362-a157-00f14f915e79" containerID="cbc84fec5c099225a4e49365bdccb7876e559e9347bbfff1ca76af14561a16c1" exitCode=255
Nov 01 14:16:52.858490 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.858440    4993 manager.go:324] Recovery completed
Nov 01 14:16:52.859319 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:52.859302    4993 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-37.us-west-2.compute.internal\" not found"
Nov 01 14:16:52.877666 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.877636    4993 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 14:16:52.878622 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.878596    4993 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 14:16:52.878622 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.878562    4993 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 14:16:52.879021 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.878993    4993 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-37.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 14:16:52.879066 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.879039    4993 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-37.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 14:16:52.879066 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.879049    4993 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-37.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 14:16:52.881613 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.881596    4993 cpu_manager.go:226] "Starting" policy="none"
Nov 01 14:16:52.881613 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.881612    4993 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 01 14:16:52.881701 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.881630    4993 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 01 14:16:52.881731 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.881603    4993 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 14:16:52.881780 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.881764    4993 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 01 14:16:52.881814 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.881782    4993 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 01 14:16:52.881814 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.881806    4993 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 01 14:16:52.881876 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.881813    4993 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 01 14:16:52.881876 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.881821    4993 policy_none.go:50] "Start"
Nov 01 14:16:52.881876 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.881845    4993 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 01 14:16:52.881876 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.881861    4993 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 01 14:16:52.882034 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.882024    4993 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 01 14:16:52.882064 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.882034    4993 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 01 14:16:52.882064 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.882040    4993 policy_none.go:44] "Start"
Nov 01 14:16:52.885264 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.885246    4993 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-37.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 14:16:52.885341 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.885278    4993 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-37.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 14:16:52.885341 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.885290    4993 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-37.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 14:16:52.886896 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.886869    4993 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 01 14:16:52.887476 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:52.887461    4993 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Nov 01 14:16:52.891794 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.891782    4993 manager.go:341] "Starting Device Plugin manager"
Nov 01 14:16:52.891871 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:52.891806    4993 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 01 14:16:52.891871 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.891815    4993 server.go:87] "Starting device plugin registration server"
Nov 01 14:16:52.892569 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.892553    4993 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 01 14:16:52.892670 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.892568    4993 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 01 14:16:52.892734 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.892691    4993 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 01 14:16:52.892836 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.892786    4993 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 01 14:16:52.892836 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.892822    4993 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 01 14:16:52.892836 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.892828    4993 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 01 14:16:52.892993 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.892893    4993 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T14:16:52.892812848Z","UUID":"94b3fcea-ede5-4a1e-8ae5-b188285c3644","Handler":null,"Name":"","Endpoint":""}
Nov 01 14:16:52.894322 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:52.894301    4993 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 01 14:16:52.894450 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:52.894331    4993 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-37.us-west-2.compute.internal\" not found"
Nov 01 14:16:52.895728 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.895438    4993 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 01 14:16:52.895728 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.895730    4993 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 01 14:16:52.924492 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.924457    4993 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 14:16:52.970971 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.970940    4993 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 14:16:52.971163 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.971094    4993 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 01 14:16:52.992984 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.992946    4993 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 14:16:52.994220 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.994200    4993 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-37.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 14:16:52.994300 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.994236    4993 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-37.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 14:16:52.994300 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.994248    4993 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-37.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 14:16:52.994300 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:52.994296    4993 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-37.us-west-2.compute.internal"
Nov 01 14:16:53.087823 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.087749    4993 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-37.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-37.us-west-2.compute.internal"]
Nov 01 14:16:53.088456 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.088423    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-37.us-west-2.compute.internal" event={"ID":"f4211ca6cf8dc522c889bb47e17f738b","Type":"ContainerStarted","Data":"4803321d47f7ec635ae01afbb5a7446e3efc65ac900182cfe7d948ede6c14287"}
Nov 01 14:16:53.088538 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.088464    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-37.us-west-2.compute.internal" event={"ID":"f4211ca6cf8dc522c889bb47e17f738b","Type":"ContainerStarted","Data":"4730fc3bf0ae3cee7a9949aff05aa788af46b4d0acdf21c3911d6bc1b09a3d6c"}
Nov 01 14:16:53.088538 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.088499    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-37.us-west-2.compute.internal" event={"ID":"4d681fe8617362d2405aa92b260bf206","Type":"ContainerStarted","Data":"c98e7849d6416234bacb0dc2b0cf8d26091985d54a9f636410834d339744de52"}
Nov 01 14:16:53.088538 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.088513    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-37.us-west-2.compute.internal" event={"ID":"4d681fe8617362d2405aa92b260bf206","Type":"ContainerDied","Data":"4cfff79920975d1bc452be97e1e84bbbcb57c55587987af90f476708a7dff340"}
Nov 01 14:16:53.088538 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.088527    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-37.us-west-2.compute.internal" event={"ID":"4d681fe8617362d2405aa92b260bf206","Type":"ContainerStarted","Data":"b91c92e51399848fcf2fe6b04cc993ae0e73f896451e011d2df11db69684f74d"}
Nov 01 14:16:53.088654 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.088564    4993 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-37.us-west-2.compute.internal"
Nov 01 14:16:53.088654 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.088586    4993 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-37.us-west-2.compute.internal"
Nov 01 14:16:53.094889 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.094868    4993 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-134-37.us-west-2.compute.internal"
Nov 01 14:16:53.094992 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.094927    4993 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-37.us-west-2.compute.internal"
Nov 01 14:16:53.146270 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.146199    4993 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 14:16:53.146270 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:53.146249    4993 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-37.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-37.us-west-2.compute.internal"
Nov 01 14:16:53.146782 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.146770    4993 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 14:16:53.146864 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:53.146794    4993 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-37.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-37.us-west-2.compute.internal"
Nov 01 14:16:53.172436 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.172405    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4d681fe8617362d2405aa92b260bf206-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-37.us-west-2.compute.internal\" (UID: \"4d681fe8617362d2405aa92b260bf206\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-37.us-west-2.compute.internal"
Nov 01 14:16:53.172583 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.172443    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d681fe8617362d2405aa92b260bf206-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-37.us-west-2.compute.internal\" (UID: \"4d681fe8617362d2405aa92b260bf206\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-37.us-west-2.compute.internal"
Nov 01 14:16:53.172583 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.172503    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f4211ca6cf8dc522c889bb47e17f738b-config\") pod \"kube-apiserver-proxy-ip-10-0-134-37.us-west-2.compute.internal\" (UID: \"f4211ca6cf8dc522c889bb47e17f738b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-37.us-west-2.compute.internal"
Nov 01 14:16:53.273662 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.273629    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4d681fe8617362d2405aa92b260bf206-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-37.us-west-2.compute.internal\" (UID: \"4d681fe8617362d2405aa92b260bf206\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-37.us-west-2.compute.internal"
Nov 01 14:16:53.273826 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.273673    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d681fe8617362d2405aa92b260bf206-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-37.us-west-2.compute.internal\" (UID: \"4d681fe8617362d2405aa92b260bf206\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-37.us-west-2.compute.internal"
Nov 01 14:16:53.273826 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.273700    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f4211ca6cf8dc522c889bb47e17f738b-config\") pod \"kube-apiserver-proxy-ip-10-0-134-37.us-west-2.compute.internal\" (UID: \"f4211ca6cf8dc522c889bb47e17f738b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-37.us-west-2.compute.internal"
Nov 01 14:16:53.273826 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.273752    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f4211ca6cf8dc522c889bb47e17f738b-config\") pod \"kube-apiserver-proxy-ip-10-0-134-37.us-west-2.compute.internal\" (UID: \"f4211ca6cf8dc522c889bb47e17f738b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-37.us-west-2.compute.internal"
Nov 01 14:16:53.273826 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.273760    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4d681fe8617362d2405aa92b260bf206-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-37.us-west-2.compute.internal\" (UID: \"4d681fe8617362d2405aa92b260bf206\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-37.us-west-2.compute.internal"
Nov 01 14:16:53.273826 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.273760    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d681fe8617362d2405aa92b260bf206-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-37.us-west-2.compute.internal\" (UID: \"4d681fe8617362d2405aa92b260bf206\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-37.us-west-2.compute.internal"
Nov 01 14:16:53.731755 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.731722    4993 apiserver.go:52] "Watching apiserver"
Nov 01 14:16:53.846852 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.846821    4993 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-37.us-west-2.compute.internal"
Nov 01 14:16:53.846990 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.846953    4993 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-37.us-west-2.compute.internal"
Nov 01 14:16:53.874634 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.874604    4993 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 01 14:16:53.875393 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.875368    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-b9d995dcc-lnk2n","openshift-image-registry/image-registry-69c8c9797f-cblkb","openshift-kube-storage-version-migrator/migrator-66df5d496b-4mmw7","kube-system/kas-connection-checker-685895c7c9-zfqjv","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-96vgd","kube-system/konnectivity-agent-qzf26","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hsdjc","openshift-dns/dns-default-b2lcr","openshift-ovn-kubernetes/ovnkube-node-cxjpl","openshift-multus/network-metrics-daemon-b6rtr","openshift-network-diagnostics/network-check-target-8mqxj","openshift-insights/insights-runtime-extractor-zq4pm","openshift-insights/periodic-gathering-rlgcn-h5sk4","openshift-service-ca-operator/service-ca-operator-544d749c8c-87xcd","openshift-ingress-canary/ingress-canary-8fgtw","openshift-service-ca/service-ca-5fffc97bdf-l25pw","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-shpt4","openshift-insights/insights-operator-7c4986d449-7d54k","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-37.us-west-2.compute.internal","openshift-kube-storage-version-migrator/migrator-66df5d496b-8rp2x","openshift-multus/multus-cwzp8","kube-system/kas-connection-checker-685895c7c9-4lzxm","openshift-dns/node-resolver-pwgqr","kube-system/kas-connection-checker-685895c7c9-xmfr4","openshift-cluster-node-tuning-operator/tuned-m5m8p","openshift-network-operator/iptables-alerter-8jxks","kube-system/kube-apiserver-proxy-ip-10-0-134-37.us-west-2.compute.internal","openshift-console-operator/console-operator-7cd56b8dbc-xmwtp","openshift-network-diagnostics/network-check-source-6f78d59f99-lhdpr","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdqx4","openshift-image-registry/image-registry-7f9cfbc8db-w8cmn","openshift-multus/multus-additional-cni-plugins-9vt8t","kube-system/global-pull-secret-syncer-sc9jq","openshift-monitoring/cluster-monitoring-operator-884576c57-bgmv8","openshift-network-console/networking-console-plugin-86c478bb88-cwfck","openshift-image-registry/node-ca-d7z97"]
Nov 01 14:16:53.876774 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.876751    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-b9d995dcc-lnk2n"
Nov 01 14:16:53.877114 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.877092    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hsdjc"
Nov 01 14:16:53.880098 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.880076    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bgmv8"
Nov 01 14:16:53.882980 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.882960    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f9cfbc8db-w8cmn"
Nov 01 14:16:53.883348 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.883329    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-cwfck"
Nov 01 14:16:53.883768 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.883756    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b6rtr"
Nov 01 14:16:53.888695 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.888678    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69c8c9797f-cblkb"
Nov 01 14:16:53.889915 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.889897    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-b2lcr"
Nov 01 14:16:53.892074 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.892056    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zq4pm"
Nov 01 14:16:53.893983 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.893963    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-l25pw"
Nov 01 14:16:53.901037 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.901020    4993 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 14:16:53.901115 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:53.901065    4993 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-37.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-37.us-west-2.compute.internal"
Nov 01 14:16:53.901360 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.901346    4993 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 14:16:53.901400 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:53.901382    4993 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-37.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-37.us-west-2.compute.internal"
Nov 01 14:16:53.924134 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.924084    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Nov 01 14:16:53.924240 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.924225    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Nov 01 14:16:53.925365 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.925342    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Nov 01 14:16:53.925365 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.925355    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 01 14:16:53.925489 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.925369    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 01 14:16:53.925682 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.925662    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Nov 01 14:16:53.925746 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.925708    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Nov 01 14:16:53.925746 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.925712    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Nov 01 14:16:53.925827 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.925774    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Nov 01 14:16:53.926368 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.926349    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Nov 01 14:16:53.926368 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.926366    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Nov 01 14:16:53.926461 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.926371    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Nov 01 14:16:53.926461 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.926407    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Nov 01 14:16:53.926461 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.926406    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Nov 01 14:16:53.926461 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.926427    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Nov 01 14:16:53.926627 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.926567    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Nov 01 14:16:53.927042 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.927029    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Nov 01 14:16:53.927301 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.927288    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Nov 01 14:16:53.927794 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.927782    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 01 14:16:53.928162 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.928151    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Nov 01 14:16:53.928752 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.928735    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 01 14:16:53.928986 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.928972    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Nov 01 14:16:53.929047 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.929022    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Nov 01 14:16:53.929239 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.929224    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Nov 01 14:16:53.929291 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.929276    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Nov 01 14:16:53.929457 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.929445    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Nov 01 14:16:53.929774 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.929763    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 01 14:16:53.929913 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.929900    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Nov 01 14:16:53.929970 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.929959    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 01 14:16:53.930118 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.930105    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Nov 01 14:16:53.930473 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.930456    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 01 14:16:53.930550 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.930538    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Nov 01 14:16:53.930550 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.930546    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Nov 01 14:16:53.930626 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.930548    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Nov 01 14:16:53.930626 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.930558    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-gxzsc\""
Nov 01 14:16:53.963202 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.963160    4993 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 01 14:16:53.966895 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.966877    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-pwgqr_00216deb-5dd0-4b4f-8940-c17dc329303c/dns-node-resolver/0.log"
Nov 01 14:16:53.974751 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.974733    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 01 14:16:53.975190 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.975150    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 01 14:16:53.975268 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.975233    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 01 14:16:53.975348 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.975286    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Nov 01 14:16:53.975348 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.975294    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 01 14:16:53.975348 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.975331    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 01 14:16:53.975476 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.975446    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-v4966\""
Nov 01 14:16:53.975532 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.975485    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-x2nj9\""
Nov 01 14:16:53.975532 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.975496    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 01 14:16:53.975654 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.975545    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 01 14:16:53.975654 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.975558    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-lp9km\""
Nov 01 14:16:53.975654 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.975517    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 01 14:16:53.975654 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.975581    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-cj7jb\""
Nov 01 14:16:53.975654 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.975621    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-mpq57\""
Nov 01 14:16:53.975852 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.975663    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 01 14:16:53.975852 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.975675    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 01 14:16:53.975852 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.975704    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-c5fv9\""
Nov 01 14:16:53.975852 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.975761    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 01 14:16:53.976044 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.975867    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 01 14:16:53.976044 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.975873    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-tp8z2\""
Nov 01 14:16:53.976044 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.975904    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 01 14:16:53.976044 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.975906    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 01 14:16:53.976044 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.975921    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 01 14:16:53.976044 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.975938    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 01 14:16:53.976044 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.975956    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 01 14:16:53.976044 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.975963    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 01 14:16:53.976044 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.975967    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 01 14:16:53.976044 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.975990    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-24rb9\""
Nov 01 14:16:53.976484 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.976064    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-m46fn\""
Nov 01 14:16:53.976484 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.976123    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-ftvdm\""
Nov 01 14:16:53.976484 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.976178    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 01 14:16:53.976484 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.976201    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 01 14:16:53.976484 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.976226    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 01 14:16:53.976484 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.976394    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4fnkb\""
Nov 01 14:16:53.976484 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.976400    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 01 14:16:53.976484 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.976409    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Nov 01 14:16:53.976881 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.976782    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 01 14:16:53.976881 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.976792    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7pw86\""
Nov 01 14:16:53.976881 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.976838    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 01 14:16:53.977037 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.976883    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-9p5x8\""
Nov 01 14:16:53.977037 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.977016    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 01 14:16:53.977037 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.977032    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Nov 01 14:16:53.977222 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.977126    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 01 14:16:53.977222 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.977141    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 01 14:16:53.977222 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.977146    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 01 14:16:53.977222 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.977160    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 01 14:16:53.977222 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.977187    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 01 14:16:53.977222 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.977197    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-smpqh\""
Nov 01 14:16:53.977222 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.977203    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 01 14:16:53.977662 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.977230    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 01 14:16:53.977662 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.977245    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 01 14:16:53.977662 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.977368    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 01 14:16:53.977662 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.977636    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5e15a59f-6aac-45e7-8812-ae97d0559052-etc-kubernetes\") pod \"tuned-m5m8p\" (UID: \"5e15a59f-6aac-45e7-8812-ae97d0559052\") " pod="openshift-cluster-node-tuning-operator/tuned-m5m8p"
Nov 01 14:16:53.977662 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.977658    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7970eec1-3765-48a4-b09c-3dec2d29956b-trusted-ca\") pod \"image-registry-7f9cfbc8db-w8cmn\" (UID: \"7970eec1-3765-48a4-b09c-3dec2d29956b\") " pod="openshift-image-registry/image-registry-7f9cfbc8db-w8cmn"
Nov 01 14:16:53.977901 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.977678    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/058820ef-eef1-41ca-b331-670bab146861-data-volume\") pod \"insights-runtime-extractor-zq4pm\" (UID: \"058820ef-eef1-41ca-b331-670bab146861\") " pod="openshift-insights/insights-runtime-extractor-zq4pm"
Nov 01 14:16:53.977901 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.977695    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/7a26b817-3283-4d59-8e8f-3bb4c0fa4987-snapshots\") pod \"insights-operator-7c4986d449-7d54k\" (UID: \"7a26b817-3283-4d59-8e8f-3bb4c0fa4987\") " pod="openshift-insights/insights-operator-7c4986d449-7d54k"
Nov 01 14:16:53.977901 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.977786    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 01 14:16:53.977901 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.977806    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b73fabd2-0d2c-4988-bbcf-7564d9331390-cni-binary-copy\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:53.977901 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.977841    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b73fabd2-0d2c-4988-bbcf-7564d9331390-host-run-k8s-cni-cncf-io\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:53.977901 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.977846    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/7a26b817-3283-4d59-8e8f-3bb4c0fa4987-snapshots\") pod \"insights-operator-7c4986d449-7d54k\" (UID: \"7a26b817-3283-4d59-8e8f-3bb4c0fa4987\") " pod="openshift-insights/insights-operator-7c4986d449-7d54k"
Nov 01 14:16:53.977901 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.977861    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/058820ef-eef1-41ca-b331-670bab146861-data-volume\") pod \"insights-runtime-extractor-zq4pm\" (UID: \"058820ef-eef1-41ca-b331-670bab146861\") " pod="openshift-insights/insights-runtime-extractor-zq4pm"
Nov 01 14:16:53.977901 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.977868    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvgpz\" (UniqueName: \"kubernetes.io/projected/00216deb-5dd0-4b4f-8940-c17dc329303c-kube-api-access-cvgpz\") pod \"node-resolver-pwgqr\" (UID: \"00216deb-5dd0-4b4f-8940-c17dc329303c\") " pod="openshift-dns/node-resolver-pwgqr"
Nov 01 14:16:53.977901 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.977876    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 01 14:16:53.977901 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.977895    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbnlg\" (UniqueName: \"kubernetes.io/projected/8a6e7f39-7853-4fec-9ea2-30099983e5bd-kube-api-access-hbnlg\") pod \"iptables-alerter-8jxks\" (UID: \"8a6e7f39-7853-4fec-9ea2-30099983e5bd\") " pod="openshift-network-operator/iptables-alerter-8jxks"
Nov 01 14:16:53.977901 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.977907    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 01 14:16:53.978450 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.977921    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b73fabd2-0d2c-4988-bbcf-7564d9331390-cnibin\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:53.978450 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.977955    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Nov 01 14:16:53.978450 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.977980    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 01 14:16:53.978450 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.977978    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b73fabd2-0d2c-4988-bbcf-7564d9331390-host-var-lib-cni-multus\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:53.978450 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.978025    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkzgz\" (UniqueName: \"kubernetes.io/projected/70bb916a-832b-4bce-9c0a-f01bf4e3abc4-kube-api-access-xkzgz\") pod \"ingress-canary-8fgtw\" (UID: \"70bb916a-832b-4bce-9c0a-f01bf4e3abc4\") " pod="openshift-ingress-canary/ingress-canary-8fgtw"
Nov 01 14:16:53.978450 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.978042    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-55h6z\""
Nov 01 14:16:53.978450 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.978112    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-var-lib-openvswitch\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:53.978450 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.978143    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/037ca099-5ad7-4874-a984-462c07801e70-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-bgmv8\" (UID: \"037ca099-5ad7-4874-a984-462c07801e70\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bgmv8"
Nov 01 14:16:53.978450 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.978202    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00216deb-5dd0-4b4f-8940-c17dc329303c-tmp-dir\") pod \"node-resolver-pwgqr\" (UID: \"00216deb-5dd0-4b4f-8940-c17dc329303c\") " pod="openshift-dns/node-resolver-pwgqr"
Nov 01 14:16:53.978450 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.978233    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bhpq\" (UniqueName: \"kubernetes.io/projected/321ecf30-5108-4320-9a58-6b204001639c-kube-api-access-8bhpq\") pod \"service-ca-5fffc97bdf-l25pw\" (UID: \"321ecf30-5108-4320-9a58-6b204001639c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-l25pw"
Nov 01 14:16:53.978450 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.978258    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b73fabd2-0d2c-4988-bbcf-7564d9331390-multus-cni-dir\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:53.978450 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.978287    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c27ff402-235a-4d73-a139-cf928a6d5cbe-default-certificate\") pod \"router-default-b9d995dcc-lnk2n\" (UID: \"c27ff402-235a-4d73-a139-cf928a6d5cbe\") " pod="openshift-ingress/router-default-b9d995dcc-lnk2n"
Nov 01 14:16:53.978450 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.978290    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00216deb-5dd0-4b4f-8940-c17dc329303c-tmp-dir\") pod \"node-resolver-pwgqr\" (UID: \"00216deb-5dd0-4b4f-8940-c17dc329303c\") " pod="openshift-dns/node-resolver-pwgqr"
Nov 01 14:16:53.978450 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.978305    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b73fabd2-0d2c-4988-bbcf-7564d9331390-cni-binary-copy\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:53.978450 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.978326    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bb59051d-c2df-4362-a157-00f14f915e79-config\") pod \"console-operator-7cd56b8dbc-xmwtp\" (UID: \"bb59051d-c2df-4362-a157-00f14f915e79\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-xmwtp"
Nov 01 14:16:53.978450 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.978394    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 01 14:16:53.978450 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.978432    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b73fabd2-0d2c-4988-bbcf-7564d9331390-hostroot\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:53.978450 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.978446    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 01 14:16:53.978450 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.978465    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Nov 01 14:16:53.979519 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.978472    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 01 14:16:53.979519 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.978475    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Nov 01 14:16:53.979519 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.978499    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Nov 01 14:16:53.979519 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.978531    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zjzzz\""
Nov 01 14:16:53.979519 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.978547    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 01 14:16:53.979519 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.978562    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8xv6q\""
Nov 01 14:16:53.979519 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.978564    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c27ff402-235a-4d73-a139-cf928a6d5cbe-default-certificate\") pod \"router-default-b9d995dcc-lnk2n\" (UID: \"c27ff402-235a-4d73-a139-cf928a6d5cbe\") " pod="openshift-ingress/router-default-b9d995dcc-lnk2n"
Nov 01 14:16:53.979519 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.978579    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 01 14:16:53.979519 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.978597    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bb59051d-c2df-4362-a157-00f14f915e79-config\") pod \"console-operator-7cd56b8dbc-xmwtp\" (UID: \"bb59051d-c2df-4362-a157-00f14f915e79\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-xmwtp"
Nov 01 14:16:53.979519 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.978580    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 01 14:16:53.979519 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.978648    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/384cd50b-370f-474f-b80e-89a544a9bbbd-service-ca-bundle\") pod \"periodic-gathering-rlgcn-h5sk4\" (UID: \"384cd50b-370f-474f-b80e-89a544a9bbbd\") " pod="openshift-insights/periodic-gathering-rlgcn-h5sk4"
Nov 01 14:16:53.979519 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.978675    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-4cd6v\""
Nov 01 14:16:53.979519 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.978704    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-etc-openvswitch\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:53.979519 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.978724    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c8cc47b-fb01-4195-a7fc-eeefc706017e-metrics-tls\") pod \"dns-default-b2lcr\" (UID: \"0c8cc47b-fb01-4195-a7fc-eeefc706017e\") " pod="openshift-dns/dns-default-b2lcr"
Nov 01 14:16:53.979519 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.978739    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c27ff402-235a-4d73-a139-cf928a6d5cbe-service-ca-bundle\") pod \"router-default-b9d995dcc-lnk2n\" (UID: \"c27ff402-235a-4d73-a139-cf928a6d5cbe\") " pod="openshift-ingress/router-default-b9d995dcc-lnk2n"
Nov 01 14:16:53.979519 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.978764    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ccc5b32b-07bc-4b3c-91ef-b184b0a9afe3-konnectivity-ca\") pod \"konnectivity-agent-qzf26\" (UID: \"ccc5b32b-07bc-4b3c-91ef-b184b0a9afe3\") " pod="kube-system/konnectivity-agent-qzf26"
Nov 01 14:16:53.979519 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.978739    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-x2zjc\""
Nov 01 14:16:53.979519 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.978792    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-jptnz\""
Nov 01 14:16:53.979519 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.978808    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/384cd50b-370f-474f-b80e-89a544a9bbbd-service-ca-bundle\") pod \"periodic-gathering-rlgcn-h5sk4\" (UID: \"384cd50b-370f-474f-b80e-89a544a9bbbd\") " pod="openshift-insights/periodic-gathering-rlgcn-h5sk4"
Nov 01 14:16:53.979519 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.978876    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-log-socket\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:53.979519 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.978906    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsmvn\" (UniqueName: \"kubernetes.io/projected/d407313d-3333-4ec4-85bb-cd27cf4d7422-kube-api-access-fsmvn\") pod \"cluster-samples-operator-6ddb6c8b94-hsdjc\" (UID: \"d407313d-3333-4ec4-85bb-cd27cf4d7422\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hsdjc"
Nov 01 14:16:53.979519 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.978947    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/321ecf30-5108-4320-9a58-6b204001639c-signing-key\") pod \"service-ca-5fffc97bdf-l25pw\" (UID: \"321ecf30-5108-4320-9a58-6b204001639c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-l25pw"
Nov 01 14:16:53.979519 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.978978    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8a6e7f39-7853-4fec-9ea2-30099983e5bd-host-slash\") pod \"iptables-alerter-8jxks\" (UID: \"8a6e7f39-7853-4fec-9ea2-30099983e5bd\") " pod="openshift-network-operator/iptables-alerter-8jxks"
Nov 01 14:16:53.979519 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.979008    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ccc5b32b-07bc-4b3c-91ef-b184b0a9afe3-konnectivity-ca\") pod \"konnectivity-agent-qzf26\" (UID: \"ccc5b32b-07bc-4b3c-91ef-b184b0a9afe3\") " pod="kube-system/konnectivity-agent-qzf26"
Nov 01 14:16:53.979519 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.979007    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx82t\" (UniqueName: \"kubernetes.io/projected/058820ef-eef1-41ca-b331-670bab146861-kube-api-access-gx82t\") pod \"insights-runtime-extractor-zq4pm\" (UID: \"058820ef-eef1-41ca-b331-670bab146861\") " pod="openshift-insights/insights-runtime-extractor-zq4pm"
Nov 01 14:16:53.979519 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.979144    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjv76\" (UniqueName: \"kubernetes.io/projected/c27ff402-235a-4d73-a139-cf928a6d5cbe-kube-api-access-rjv76\") pod \"router-default-b9d995dcc-lnk2n\" (UID: \"c27ff402-235a-4d73-a139-cf928a6d5cbe\") " pod="openshift-ingress/router-default-b9d995dcc-lnk2n"
Nov 01 14:16:53.979519 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.979160    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/321ecf30-5108-4320-9a58-6b204001639c-signing-key\") pod \"service-ca-5fffc97bdf-l25pw\" (UID: \"321ecf30-5108-4320-9a58-6b204001639c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-l25pw"
Nov 01 14:16:53.979519 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.979209    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3e7e18b9-78ee-4349-b00c-a0800e8f8341-serving-cert\") pod \"service-ca-operator-544d749c8c-87xcd\" (UID: \"3e7e18b9-78ee-4349-b00c-a0800e8f8341\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-87xcd"
Nov 01 14:16:53.979519 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.979232    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5e15a59f-6aac-45e7-8812-ae97d0559052-etc-systemd\") pod \"tuned-m5m8p\" (UID: \"5e15a59f-6aac-45e7-8812-ae97d0559052\") " pod="openshift-cluster-node-tuning-operator/tuned-m5m8p"
Nov 01 14:16:53.979519 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.979255    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92xdl\" (UniqueName: \"kubernetes.io/projected/bb59051d-c2df-4362-a157-00f14f915e79-kube-api-access-92xdl\") pod \"console-operator-7cd56b8dbc-xmwtp\" (UID: \"bb59051d-c2df-4362-a157-00f14f915e79\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-xmwtp"
Nov 01 14:16:53.979519 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.979295    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e7ea769b-e719-4090-bcd4-05b1b88fcc06-registry-tls\") pod \"image-registry-69c8c9797f-cblkb\" (UID: \"e7ea769b-e719-4090-bcd4-05b1b88fcc06\") " pod="openshift-image-registry/image-registry-69c8c9797f-cblkb"
Nov 01 14:16:53.979519 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.979376    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0c8cc47b-fb01-4195-a7fc-eeefc706017e-config-volume\") pod \"dns-default-b2lcr\" (UID: \"0c8cc47b-fb01-4195-a7fc-eeefc706017e\") " pod="openshift-dns/dns-default-b2lcr"
Nov 01 14:16:53.979519 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.979384    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3e7e18b9-78ee-4349-b00c-a0800e8f8341-serving-cert\") pod \"service-ca-operator-544d749c8c-87xcd\" (UID: \"3e7e18b9-78ee-4349-b00c-a0800e8f8341\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-87xcd"
Nov 01 14:16:53.979519 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.979436    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7970eec1-3765-48a4-b09c-3dec2d29956b-registry-tls\") pod \"image-registry-7f9cfbc8db-w8cmn\" (UID: \"7970eec1-3765-48a4-b09c-3dec2d29956b\") " pod="openshift-image-registry/image-registry-7f9cfbc8db-w8cmn"
Nov 01 14:16:53.979519 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.979473    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7970eec1-3765-48a4-b09c-3dec2d29956b-registry-certificates\") pod \"image-registry-7f9cfbc8db-w8cmn\" (UID: \"7970eec1-3765-48a4-b09c-3dec2d29956b\") " pod="openshift-image-registry/image-registry-7f9cfbc8db-w8cmn"
Nov 01 14:16:53.979519 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.979500    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/384cd50b-370f-474f-b80e-89a544a9bbbd-archives-path\") pod \"periodic-gathering-rlgcn-h5sk4\" (UID: \"384cd50b-370f-474f-b80e-89a544a9bbbd\") " pod="openshift-insights/periodic-gathering-rlgcn-h5sk4"
Nov 01 14:16:53.979519 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.979541    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bcc6412c-2003-49de-9817-a3713e28afef-cni-binary-copy\") pod \"multus-additional-cni-plugins-9vt8t\" (UID: \"bcc6412c-2003-49de-9817-a3713e28afef\") " pod="openshift-multus/multus-additional-cni-plugins-9vt8t"
Nov 01 14:16:53.980714 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.979562    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bcc6412c-2003-49de-9817-a3713e28afef-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9vt8t\" (UID: \"bcc6412c-2003-49de-9817-a3713e28afef\") " pod="openshift-multus/multus-additional-cni-plugins-9vt8t"
Nov 01 14:16:53.980714 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.979579    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1b941fae-3c27-4d68-8205-38e492179ee9-env-overrides\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:53.980714 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.979543    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0c8cc47b-fb01-4195-a7fc-eeefc706017e-config-volume\") pod \"dns-default-b2lcr\" (UID: \"0c8cc47b-fb01-4195-a7fc-eeefc706017e\") " pod="openshift-dns/dns-default-b2lcr"
Nov 01 14:16:53.980714 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.979602    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b5a4fefb-cc48-4638-b0f8-590e36bd0118-nginx-conf\") pod \"networking-console-plugin-86c478bb88-cwfck\" (UID: \"b5a4fefb-cc48-4638-b0f8-590e36bd0118\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-cwfck"
Nov 01 14:16:53.980714 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.979628    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bcc6412c-2003-49de-9817-a3713e28afef-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9vt8t\" (UID: \"bcc6412c-2003-49de-9817-a3713e28afef\") " pod="openshift-multus/multus-additional-cni-plugins-9vt8t"
Nov 01 14:16:53.980714 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.979655    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-host-slash\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:53.980714 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.979687    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:53.980714 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.979711    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnzsm\" (UniqueName: \"kubernetes.io/projected/a12d30a7-8564-43c1-9742-3194fe8b975f-kube-api-access-gnzsm\") pod \"node-ca-d7z97\" (UID: \"a12d30a7-8564-43c1-9742-3194fe8b975f\") " pod="openshift-image-registry/node-ca-d7z97"
Nov 01 14:16:53.980714 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.979719    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1b941fae-3c27-4d68-8205-38e492179ee9-env-overrides\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:53.980714 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.979730    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5e15a59f-6aac-45e7-8812-ae97d0559052-etc-sysctl-d\") pod \"tuned-m5m8p\" (UID: \"5e15a59f-6aac-45e7-8812-ae97d0559052\") " pod="openshift-cluster-node-tuning-operator/tuned-m5m8p"
Nov 01 14:16:53.980714 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.979733    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/384cd50b-370f-474f-b80e-89a544a9bbbd-archives-path\") pod \"periodic-gathering-rlgcn-h5sk4\" (UID: \"384cd50b-370f-474f-b80e-89a544a9bbbd\") " pod="openshift-insights/periodic-gathering-rlgcn-h5sk4"
Nov 01 14:16:53.980714 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.979738    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7970eec1-3765-48a4-b09c-3dec2d29956b-registry-certificates\") pod \"image-registry-7f9cfbc8db-w8cmn\" (UID: \"7970eec1-3765-48a4-b09c-3dec2d29956b\") " pod="openshift-image-registry/image-registry-7f9cfbc8db-w8cmn"
Nov 01 14:16:53.980714 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.979783    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxdcx\" (UniqueName: \"kubernetes.io/projected/e78d2494-62c4-4070-a1eb-8ce40329ff57-kube-api-access-wxdcx\") pod \"volume-data-source-validator-58c9f5f46d-shpt4\" (UID: \"e78d2494-62c4-4070-a1eb-8ce40329ff57\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-shpt4"
Nov 01 14:16:53.980714 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.979815    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/058820ef-eef1-41ca-b331-670bab146861-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zq4pm\" (UID: \"058820ef-eef1-41ca-b331-670bab146861\") " pod="openshift-insights/insights-runtime-extractor-zq4pm"
Nov 01 14:16:53.980714 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.979829    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bcc6412c-2003-49de-9817-a3713e28afef-cni-binary-copy\") pod \"multus-additional-cni-plugins-9vt8t\" (UID: \"bcc6412c-2003-49de-9817-a3713e28afef\") " pod="openshift-multus/multus-additional-cni-plugins-9vt8t"
Nov 01 14:16:53.980714 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.979838    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bcc6412c-2003-49de-9817-a3713e28afef-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9vt8t\" (UID: \"bcc6412c-2003-49de-9817-a3713e28afef\") " pod="openshift-multus/multus-additional-cni-plugins-9vt8t"
Nov 01 14:16:53.980714 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.979869    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a26b817-3283-4d59-8e8f-3bb4c0fa4987-service-ca-bundle\") pod \"insights-operator-7c4986d449-7d54k\" (UID: \"7a26b817-3283-4d59-8e8f-3bb4c0fa4987\") " pod="openshift-insights/insights-operator-7c4986d449-7d54k"
Nov 01 14:16:53.980714 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.979888    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bcc6412c-2003-49de-9817-a3713e28afef-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9vt8t\" (UID: \"bcc6412c-2003-49de-9817-a3713e28afef\") " pod="openshift-multus/multus-additional-cni-plugins-9vt8t"
Nov 01 14:16:53.980714 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.979942    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b5a4fefb-cc48-4638-b0f8-590e36bd0118-nginx-conf\") pod \"networking-console-plugin-86c478bb88-cwfck\" (UID: \"b5a4fefb-cc48-4638-b0f8-590e36bd0118\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-cwfck"
Nov 01 14:16:53.980714 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.979970    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddmmn\" (UniqueName: \"kubernetes.io/projected/b73fabd2-0d2c-4988-bbcf-7564d9331390-kube-api-access-ddmmn\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:53.980714 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.980050    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a26b817-3283-4d59-8e8f-3bb4c0fa4987-service-ca-bundle\") pod \"insights-operator-7c4986d449-7d54k\" (UID: \"7a26b817-3283-4d59-8e8f-3bb4c0fa4987\") " pod="openshift-insights/insights-operator-7c4986d449-7d54k"
Nov 01 14:16:53.980714 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.980055    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a12d30a7-8564-43c1-9742-3194fe8b975f-serviceca\") pod \"node-ca-d7z97\" (UID: \"a12d30a7-8564-43c1-9742-3194fe8b975f\") " pod="openshift-image-registry/node-ca-d7z97"
Nov 01 14:16:53.980714 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.980126    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpm8k\" (UniqueName: \"kubernetes.io/projected/ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1-kube-api-access-hpm8k\") pod \"network-metrics-daemon-b6rtr\" (UID: \"ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1\") " pod="openshift-multus/network-metrics-daemon-b6rtr"
Nov 01 14:16:53.980714 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.980211    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bb59051d-c2df-4362-a157-00f14f915e79-trusted-ca\") pod \"console-operator-7cd56b8dbc-xmwtp\" (UID: \"bb59051d-c2df-4362-a157-00f14f915e79\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-xmwtp"
Nov 01 14:16:53.980714 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.980293    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4a724964-bd57-4d56-a042-a0d490115eaa-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-96vgd\" (UID: \"4a724964-bd57-4d56-a042-a0d490115eaa\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-96vgd"
Nov 01 14:16:53.980714 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.980290    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a12d30a7-8564-43c1-9742-3194fe8b975f-serviceca\") pod \"node-ca-d7z97\" (UID: \"a12d30a7-8564-43c1-9742-3194fe8b975f\") " pod="openshift-image-registry/node-ca-d7z97"
Nov 01 14:16:53.980714 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.980390    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a12d30a7-8564-43c1-9742-3194fe8b975f-host\") pod \"node-ca-d7z97\" (UID: \"a12d30a7-8564-43c1-9742-3194fe8b975f\") " pod="openshift-image-registry/node-ca-d7z97"
Nov 01 14:16:53.980714 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.980482    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2e013e38-9840-45c2-b4a0-80df0f319109-device-dir\") pod \"aws-ebs-csi-driver-node-vdqx4\" (UID: \"2e013e38-9840-45c2-b4a0-80df0f319109\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdqx4"
Nov 01 14:16:53.980714 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.980500    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b73fabd2-0d2c-4988-bbcf-7564d9331390-etc-kubernetes\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:53.980714 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.980540    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4a724964-bd57-4d56-a042-a0d490115eaa-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-96vgd\" (UID: \"4a724964-bd57-4d56-a042-a0d490115eaa\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-96vgd"
Nov 01 14:16:53.980714 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.980547    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bb59051d-c2df-4362-a157-00f14f915e79-trusted-ca\") pod \"console-operator-7cd56b8dbc-xmwtp\" (UID: \"bb59051d-c2df-4362-a157-00f14f915e79\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-xmwtp"
Nov 01 14:16:53.980714 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.980545    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stw2n\" (UniqueName: \"kubernetes.io/projected/3e7e18b9-78ee-4349-b00c-a0800e8f8341-kube-api-access-stw2n\") pod \"service-ca-operator-544d749c8c-87xcd\" (UID: \"3e7e18b9-78ee-4349-b00c-a0800e8f8341\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-87xcd"
Nov 01 14:16:53.980714 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.980591    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/d407313d-3333-4ec4-85bb-cd27cf4d7422-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-hsdjc\" (UID: \"d407313d-3333-4ec4-85bb-cd27cf4d7422\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hsdjc"
Nov 01 14:16:53.980714 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.980690    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5e15a59f-6aac-45e7-8812-ae97d0559052-host\") pod \"tuned-m5m8p\" (UID: \"5e15a59f-6aac-45e7-8812-ae97d0559052\") " pod="openshift-cluster-node-tuning-operator/tuned-m5m8p"
Nov 01 14:16:53.980714 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.980720    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/70bb916a-832b-4bce-9c0a-f01bf4e3abc4-cert\") pod \"ingress-canary-8fgtw\" (UID: \"70bb916a-832b-4bce-9c0a-f01bf4e3abc4\") " pod="openshift-ingress-canary/ingress-canary-8fgtw"
Nov 01 14:16:53.982021 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.980745    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ccc5b32b-07bc-4b3c-91ef-b184b0a9afe3-agent-certs\") pod \"konnectivity-agent-qzf26\" (UID: \"ccc5b32b-07bc-4b3c-91ef-b184b0a9afe3\") " pod="kube-system/konnectivity-agent-qzf26"
Nov 01 14:16:53.982021 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.980754    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/d407313d-3333-4ec4-85bb-cd27cf4d7422-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-hsdjc\" (UID: \"d407313d-3333-4ec4-85bb-cd27cf4d7422\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hsdjc"
Nov 01 14:16:53.982021 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.980770    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b5a4fefb-cc48-4638-b0f8-590e36bd0118-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-cwfck\" (UID: \"b5a4fefb-cc48-4638-b0f8-590e36bd0118\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-cwfck"
Nov 01 14:16:53.982021 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.980854    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgmsd\" (UniqueName: \"kubernetes.io/projected/0c8cc47b-fb01-4195-a7fc-eeefc706017e-kube-api-access-wgmsd\") pod \"dns-default-b2lcr\" (UID: \"0c8cc47b-fb01-4195-a7fc-eeefc706017e\") " pod="openshift-dns/dns-default-b2lcr"
Nov 01 14:16:53.982021 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.980934    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjj74\" (UniqueName: \"kubernetes.io/projected/f219e3c2-6da8-4c88-9ec4-b26cf4d10326-kube-api-access-pjj74\") pod \"network-check-source-6f78d59f99-lhdpr\" (UID: \"f219e3c2-6da8-4c88-9ec4-b26cf4d10326\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-lhdpr"
Nov 01 14:16:53.982021 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.980940    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/70bb916a-832b-4bce-9c0a-f01bf4e3abc4-cert\") pod \"ingress-canary-8fgtw\" (UID: \"70bb916a-832b-4bce-9c0a-f01bf4e3abc4\") " pod="openshift-ingress-canary/ingress-canary-8fgtw"
Nov 01 14:16:53.982021 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.980953    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-host-kubelet\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:53.982021 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.980991    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5e15a59f-6aac-45e7-8812-ae97d0559052-run\") pod \"tuned-m5m8p\" (UID: \"5e15a59f-6aac-45e7-8812-ae97d0559052\") " pod="openshift-cluster-node-tuning-operator/tuned-m5m8p"
Nov 01 14:16:53.982021 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.981010    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5e15a59f-6aac-45e7-8812-ae97d0559052-sys\") pod \"tuned-m5m8p\" (UID: \"5e15a59f-6aac-45e7-8812-ae97d0559052\") " pod="openshift-cluster-node-tuning-operator/tuned-m5m8p"
Nov 01 14:16:53.982021 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.981041    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b73fabd2-0d2c-4988-bbcf-7564d9331390-system-cni-dir\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:53.982021 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.981052    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ccc5b32b-07bc-4b3c-91ef-b184b0a9afe3-agent-certs\") pod \"konnectivity-agent-qzf26\" (UID: \"ccc5b32b-07bc-4b3c-91ef-b184b0a9afe3\") " pod="kube-system/konnectivity-agent-qzf26"
Nov 01 14:16:53.982021 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.981120    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bcc6412c-2003-49de-9817-a3713e28afef-cnibin\") pod \"multus-additional-cni-plugins-9vt8t\" (UID: \"bcc6412c-2003-49de-9817-a3713e28afef\") " pod="openshift-multus/multus-additional-cni-plugins-9vt8t"
Nov 01 14:16:53.982021 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.981154    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-host-cni-bin\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:53.982021 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.981215    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e7ea769b-e719-4090-bcd4-05b1b88fcc06-trusted-ca\") pod \"image-registry-69c8c9797f-cblkb\" (UID: \"e7ea769b-e719-4090-bcd4-05b1b88fcc06\") " pod="openshift-image-registry/image-registry-69c8c9797f-cblkb"
Nov 01 14:16:53.982021 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.981283    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khv87\" (UniqueName: \"kubernetes.io/projected/fba1e7f9-ca37-46dc-8c66-56f1d9563acd-kube-api-access-khv87\") pod \"migrator-66df5d496b-8rp2x\" (UID: \"fba1e7f9-ca37-46dc-8c66-56f1d9563acd\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8rp2x"
Nov 01 14:16:53.982021 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.981346    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/00216deb-5dd0-4b4f-8940-c17dc329303c-hosts-file\") pod \"node-resolver-pwgqr\" (UID: \"00216deb-5dd0-4b4f-8940-c17dc329303c\") " pod="openshift-dns/node-resolver-pwgqr"
Nov 01 14:16:53.982021 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.981375    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7970eec1-3765-48a4-b09c-3dec2d29956b-ca-trust-extracted\") pod \"image-registry-7f9cfbc8db-w8cmn\" (UID: \"7970eec1-3765-48a4-b09c-3dec2d29956b\") " pod="openshift-image-registry/image-registry-7f9cfbc8db-w8cmn"
Nov 01 14:16:53.982021 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.981394    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4a724964-bd57-4d56-a042-a0d490115eaa-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-96vgd\" (UID: \"4a724964-bd57-4d56-a042-a0d490115eaa\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-96vgd"
Nov 01 14:16:53.982021 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.981434    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e7ea769b-e719-4090-bcd4-05b1b88fcc06-installation-pull-secrets\") pod \"image-registry-69c8c9797f-cblkb\" (UID: \"e7ea769b-e719-4090-bcd4-05b1b88fcc06\") " pod="openshift-image-registry/image-registry-69c8c9797f-cblkb"
Nov 01 14:16:53.982021 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.981454    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j226\" (UniqueName: \"kubernetes.io/projected/6a8f947a-0f8f-41aa-b646-4459fa8126a3-kube-api-access-9j226\") pod \"kas-connection-checker-685895c7c9-xmfr4\" (UID: \"6a8f947a-0f8f-41aa-b646-4459fa8126a3\") " pod="kube-system/kas-connection-checker-685895c7c9-xmfr4"
Nov 01 14:16:53.982021 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.981470    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8b1eb53c-8c9a-46c4-b654-2aef42491b14-dbus\") pod \"global-pull-secret-syncer-sc9jq\" (UID: \"8b1eb53c-8c9a-46c4-b654-2aef42491b14\") " pod="kube-system/global-pull-secret-syncer-sc9jq"
Nov 01 14:16:53.982021 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.981490    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-run-openvswitch\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:53.982021 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.981493    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7970eec1-3765-48a4-b09c-3dec2d29956b-ca-trust-extracted\") pod \"image-registry-7f9cfbc8db-w8cmn\" (UID: \"7970eec1-3765-48a4-b09c-3dec2d29956b\") " pod="openshift-image-registry/image-registry-7f9cfbc8db-w8cmn"
Nov 01 14:16:53.982021 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.981532    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1-metrics-certs\") pod \"network-metrics-daemon-b6rtr\" (UID: \"ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1\") " pod="openshift-multus/network-metrics-daemon-b6rtr"
Nov 01 14:16:53.982021 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.981559    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zcj8\" (UniqueName: \"kubernetes.io/projected/5e15a59f-6aac-45e7-8812-ae97d0559052-kube-api-access-8zcj8\") pod \"tuned-m5m8p\" (UID: \"5e15a59f-6aac-45e7-8812-ae97d0559052\") " pod="openshift-cluster-node-tuning-operator/tuned-m5m8p"
Nov 01 14:16:53.982021 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.981563    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e7ea769b-e719-4090-bcd4-05b1b88fcc06-trusted-ca\") pod \"image-registry-69c8c9797f-cblkb\" (UID: \"e7ea769b-e719-4090-bcd4-05b1b88fcc06\") " pod="openshift-image-registry/image-registry-69c8c9797f-cblkb"
Nov 01 14:16:53.982021 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.981569    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4a724964-bd57-4d56-a042-a0d490115eaa-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-96vgd\" (UID: \"4a724964-bd57-4d56-a042-a0d490115eaa\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-96vgd"
Nov 01 14:16:53.982021 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.981590    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/037ca099-5ad7-4874-a984-462c07801e70-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-bgmv8\" (UID: \"037ca099-5ad7-4874-a984-462c07801e70\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bgmv8"
Nov 01 14:16:53.982021 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.981620    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/058820ef-eef1-41ca-b331-670bab146861-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zq4pm\" (UID: \"058820ef-eef1-41ca-b331-670bab146861\") " pod="openshift-insights/insights-runtime-extractor-zq4pm"
Nov 01 14:16:53.982021 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.981637    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e7ea769b-e719-4090-bcd4-05b1b88fcc06-installation-pull-secrets\") pod \"image-registry-69c8c9797f-cblkb\" (UID: \"e7ea769b-e719-4090-bcd4-05b1b88fcc06\") " pod="openshift-image-registry/image-registry-69c8c9797f-cblkb"
Nov 01 14:16:53.982021 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.981662    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv8hp\" (UniqueName: \"kubernetes.io/projected/7a26b817-3283-4d59-8e8f-3bb4c0fa4987-kube-api-access-bv8hp\") pod \"insights-operator-7c4986d449-7d54k\" (UID: \"7a26b817-3283-4d59-8e8f-3bb4c0fa4987\") " pod="openshift-insights/insights-operator-7c4986d449-7d54k"
Nov 01 14:16:53.982021 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.981765    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzjlr\" (UniqueName: \"kubernetes.io/projected/2e013e38-9840-45c2-b4a0-80df0f319109-kube-api-access-wzjlr\") pod \"aws-ebs-csi-driver-node-vdqx4\" (UID: \"2e013e38-9840-45c2-b4a0-80df0f319109\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdqx4"
Nov 01 14:16:53.982021 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.981849    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/037ca099-5ad7-4874-a984-462c07801e70-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-bgmv8\" (UID: \"037ca099-5ad7-4874-a984-462c07801e70\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bgmv8"
Nov 01 14:16:53.982021 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.981855    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b73fabd2-0d2c-4988-bbcf-7564d9331390-multus-socket-dir-parent\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:53.982021 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.981950    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8b1eb53c-8c9a-46c4-b654-2aef42491b14-original-pull-secret\") pod \"global-pull-secret-syncer-sc9jq\" (UID: \"8b1eb53c-8c9a-46c4-b654-2aef42491b14\") " pod="kube-system/global-pull-secret-syncer-sc9jq"
Nov 01 14:16:53.982021 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.981974    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5e15a59f-6aac-45e7-8812-ae97d0559052-etc-sysconfig\") pod \"tuned-m5m8p\" (UID: \"5e15a59f-6aac-45e7-8812-ae97d0559052\") " pod="openshift-cluster-node-tuning-operator/tuned-m5m8p"
Nov 01 14:16:53.982021 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.982008    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b73fabd2-0d2c-4988-bbcf-7564d9331390-host-var-lib-cni-bin\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:53.983254 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.982096    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b73fabd2-0d2c-4988-bbcf-7564d9331390-host-run-multus-certs\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:53.983254 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.982125    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-host-cni-netd\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:53.983254 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.982152    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdxs8\" (UniqueName: \"kubernetes.io/projected/e7ea769b-e719-4090-bcd4-05b1b88fcc06-kube-api-access-rdxs8\") pod \"image-registry-69c8c9797f-cblkb\" (UID: \"e7ea769b-e719-4090-bcd4-05b1b88fcc06\") " pod="openshift-image-registry/image-registry-69c8c9797f-cblkb"
Nov 01 14:16:53.983254 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.982185    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8b1eb53c-8c9a-46c4-b654-2aef42491b14-original-pull-secret\") pod \"global-pull-secret-syncer-sc9jq\" (UID: \"8b1eb53c-8c9a-46c4-b654-2aef42491b14\") " pod="kube-system/global-pull-secret-syncer-sc9jq"
Nov 01 14:16:53.983254 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.982214    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c8cc47b-fb01-4195-a7fc-eeefc706017e-tmp-dir\") pod \"dns-default-b2lcr\" (UID: \"0c8cc47b-fb01-4195-a7fc-eeefc706017e\") " pod="openshift-dns/dns-default-b2lcr"
Nov 01 14:16:53.983254 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.982248    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jbv4\" (UniqueName: \"kubernetes.io/projected/2460be58-d51b-4876-bf9b-2a0e57901386-kube-api-access-7jbv4\") pod \"kas-connection-checker-685895c7c9-4lzxm\" (UID: \"2460be58-d51b-4876-bf9b-2a0e57901386\") " pod="kube-system/kas-connection-checker-685895c7c9-4lzxm"
Nov 01 14:16:53.983254 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.982297    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c8cc47b-fb01-4195-a7fc-eeefc706017e-tmp-dir\") pod \"dns-default-b2lcr\" (UID: \"0c8cc47b-fb01-4195-a7fc-eeefc706017e\") " pod="openshift-dns/dns-default-b2lcr"
Nov 01 14:16:53.983254 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.982333    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7970eec1-3765-48a4-b09c-3dec2d29956b-bound-sa-token\") pod \"image-registry-7f9cfbc8db-w8cmn\" (UID: \"7970eec1-3765-48a4-b09c-3dec2d29956b\") " pod="openshift-image-registry/image-registry-7f9cfbc8db-w8cmn"
Nov 01 14:16:53.983254 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.982468    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b73fabd2-0d2c-4988-bbcf-7564d9331390-host-var-lib-kubelet\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:53.983254 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.982498    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chwvj\" (UniqueName: \"kubernetes.io/projected/384cd50b-370f-474f-b80e-89a544a9bbbd-kube-api-access-chwvj\") pod \"periodic-gathering-rlgcn-h5sk4\" (UID: \"384cd50b-370f-474f-b80e-89a544a9bbbd\") " pod="openshift-insights/periodic-gathering-rlgcn-h5sk4"
Nov 01 14:16:53.983254 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.982523    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-host-run-netns\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:53.983254 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.982551    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8thtk\" (UniqueName: \"kubernetes.io/projected/037ca099-5ad7-4874-a984-462c07801e70-kube-api-access-8thtk\") pod \"cluster-monitoring-operator-884576c57-bgmv8\" (UID: \"037ca099-5ad7-4874-a984-462c07801e70\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bgmv8"
Nov 01 14:16:53.983254 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.982578    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfj7s\" (UniqueName: \"kubernetes.io/projected/4a724964-bd57-4d56-a042-a0d490115eaa-kube-api-access-kfj7s\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-96vgd\" (UID: \"4a724964-bd57-4d56-a042-a0d490115eaa\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-96vgd"
Nov 01 14:16:53.983254 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.982613    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bcc6412c-2003-49de-9817-a3713e28afef-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9vt8t\" (UID: \"bcc6412c-2003-49de-9817-a3713e28afef\") " pod="openshift-multus/multus-additional-cni-plugins-9vt8t"
Nov 01 14:16:53.983254 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.982640    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c27ff402-235a-4d73-a139-cf928a6d5cbe-stats-auth\") pod \"router-default-b9d995dcc-lnk2n\" (UID: \"c27ff402-235a-4d73-a139-cf928a6d5cbe\") " pod="openshift-ingress/router-default-b9d995dcc-lnk2n"
Nov 01 14:16:53.983254 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.982664    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3e7e18b9-78ee-4349-b00c-a0800e8f8341-config\") pod \"service-ca-operator-544d749c8c-87xcd\" (UID: \"3e7e18b9-78ee-4349-b00c-a0800e8f8341\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-87xcd"
Nov 01 14:16:53.983254 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.982683    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8b1eb53c-8c9a-46c4-b654-2aef42491b14-kubelet-config\") pod \"global-pull-secret-syncer-sc9jq\" (UID: \"8b1eb53c-8c9a-46c4-b654-2aef42491b14\") " pod="kube-system/global-pull-secret-syncer-sc9jq"
Nov 01 14:16:53.983254 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.982705    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e7ea769b-e719-4090-bcd4-05b1b88fcc06-ca-trust-extracted\") pod \"image-registry-69c8c9797f-cblkb\" (UID: \"e7ea769b-e719-4090-bcd4-05b1b88fcc06\") " pod="openshift-image-registry/image-registry-69c8c9797f-cblkb"
Nov 01 14:16:53.983254 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.982720    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7a26b817-3283-4d59-8e8f-3bb4c0fa4987-serving-cert\") pod \"insights-operator-7c4986d449-7d54k\" (UID: \"7a26b817-3283-4d59-8e8f-3bb4c0fa4987\") " pod="openshift-insights/insights-operator-7c4986d449-7d54k"
Nov 01 14:16:53.983254 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.982736    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bcc6412c-2003-49de-9817-a3713e28afef-system-cni-dir\") pod \"multus-additional-cni-plugins-9vt8t\" (UID: \"bcc6412c-2003-49de-9817-a3713e28afef\") " pod="openshift-multus/multus-additional-cni-plugins-9vt8t"
Nov 01 14:16:53.983254 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.982757    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1b941fae-3c27-4d68-8205-38e492179ee9-ovnkube-config\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:53.983254 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.982777    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8mfv\" (UniqueName: \"kubernetes.io/projected/1b941fae-3c27-4d68-8205-38e492179ee9-kube-api-access-p8mfv\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:53.983254 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.982793    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5e15a59f-6aac-45e7-8812-ae97d0559052-etc-sysctl-conf\") pod \"tuned-m5m8p\" (UID: \"5e15a59f-6aac-45e7-8812-ae97d0559052\") " pod="openshift-cluster-node-tuning-operator/tuned-m5m8p"
Nov 01 14:16:53.983254 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.982812    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e7ea769b-e719-4090-bcd4-05b1b88fcc06-ca-trust-extracted\") pod \"image-registry-69c8c9797f-cblkb\" (UID: \"e7ea769b-e719-4090-bcd4-05b1b88fcc06\") " pod="openshift-image-registry/image-registry-69c8c9797f-cblkb"
Nov 01 14:16:53.983254 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.982885    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e15a59f-6aac-45e7-8812-ae97d0559052-var-lib-kubelet\") pod \"tuned-m5m8p\" (UID: \"5e15a59f-6aac-45e7-8812-ae97d0559052\") " pod="openshift-cluster-node-tuning-operator/tuned-m5m8p"
Nov 01 14:16:53.983254 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.982974    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bb59051d-c2df-4362-a157-00f14f915e79-serving-cert\") pod \"console-operator-7cd56b8dbc-xmwtp\" (UID: \"bb59051d-c2df-4362-a157-00f14f915e79\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-xmwtp"
Nov 01 14:16:53.983254 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.982977    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c27ff402-235a-4d73-a139-cf928a6d5cbe-stats-auth\") pod \"router-default-b9d995dcc-lnk2n\" (UID: \"c27ff402-235a-4d73-a139-cf928a6d5cbe\") " pod="openshift-ingress/router-default-b9d995dcc-lnk2n"
Nov 01 14:16:53.983254 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.982996    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3e7e18b9-78ee-4349-b00c-a0800e8f8341-config\") pod \"service-ca-operator-544d749c8c-87xcd\" (UID: \"3e7e18b9-78ee-4349-b00c-a0800e8f8341\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-87xcd"
Nov 01 14:16:53.983254 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.982999    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7a26b817-3283-4d59-8e8f-3bb4c0fa4987-serving-cert\") pod \"insights-operator-7c4986d449-7d54k\" (UID: \"7a26b817-3283-4d59-8e8f-3bb4c0fa4987\") " pod="openshift-insights/insights-operator-7c4986d449-7d54k"
Nov 01 14:16:53.983254 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.983029    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7970eec1-3765-48a4-b09c-3dec2d29956b-installation-pull-secrets\") pod \"image-registry-7f9cfbc8db-w8cmn\" (UID: \"7970eec1-3765-48a4-b09c-3dec2d29956b\") " pod="openshift-image-registry/image-registry-7f9cfbc8db-w8cmn"
Nov 01 14:16:53.983254 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.983093    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bb59051d-c2df-4362-a157-00f14f915e79-serving-cert\") pod \"console-operator-7cd56b8dbc-xmwtp\" (UID: \"bb59051d-c2df-4362-a157-00f14f915e79\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-xmwtp"
Nov 01 14:16:53.983254 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.983118    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1b941fae-3c27-4d68-8205-38e492179ee9-ovnkube-config\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:53.983254 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.983122    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b73fabd2-0d2c-4988-bbcf-7564d9331390-host-run-netns\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:53.983254 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.983222    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b73fabd2-0d2c-4988-bbcf-7564d9331390-multus-conf-dir\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:53.983254 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.983258    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7970eec1-3765-48a4-b09c-3dec2d29956b-installation-pull-secrets\") pod \"image-registry-7f9cfbc8db-w8cmn\" (UID: \"7970eec1-3765-48a4-b09c-3dec2d29956b\") " pod="openshift-image-registry/image-registry-7f9cfbc8db-w8cmn"
Nov 01 14:16:53.983254 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.983259    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdsp8\" (UniqueName: \"kubernetes.io/projected/bcc6412c-2003-49de-9817-a3713e28afef-kube-api-access-vdsp8\") pod \"multus-additional-cni-plugins-9vt8t\" (UID: \"bcc6412c-2003-49de-9817-a3713e28afef\") " pod="openshift-multus/multus-additional-cni-plugins-9vt8t"
Nov 01 14:16:53.984597 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.983309    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-systemd-units\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:53.984597 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.983371    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-run-ovn\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:53.984597 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.983392    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1b941fae-3c27-4d68-8205-38e492179ee9-ovn-node-metrics-cert\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:53.984597 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.983416    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5e15a59f-6aac-45e7-8812-ae97d0559052-etc-modprobe-d\") pod \"tuned-m5m8p\" (UID: \"5e15a59f-6aac-45e7-8812-ae97d0559052\") " pod="openshift-cluster-node-tuning-operator/tuned-m5m8p"
Nov 01 14:16:53.984597 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.983432    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8a6e7f39-7853-4fec-9ea2-30099983e5bd-iptables-alerter-script\") pod \"iptables-alerter-8jxks\" (UID: \"8a6e7f39-7853-4fec-9ea2-30099983e5bd\") " pod="openshift-network-operator/iptables-alerter-8jxks"
Nov 01 14:16:53.984597 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.983483    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/058820ef-eef1-41ca-b331-670bab146861-crio-socket\") pod \"insights-runtime-extractor-zq4pm\" (UID: \"058820ef-eef1-41ca-b331-670bab146861\") " pod="openshift-insights/insights-runtime-extractor-zq4pm"
Nov 01 14:16:53.984597 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.983525    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7a26b817-3283-4d59-8e8f-3bb4c0fa4987-tmp\") pod \"insights-operator-7c4986d449-7d54k\" (UID: \"7a26b817-3283-4d59-8e8f-3bb4c0fa4987\") " pod="openshift-insights/insights-operator-7c4986d449-7d54k"
Nov 01 14:16:53.984597 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.983555    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1b941fae-3c27-4d68-8205-38e492179ee9-ovn-node-metrics-cert\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:53.984597 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.983560    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2e013e38-9840-45c2-b4a0-80df0f319109-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vdqx4\" (UID: \"2e013e38-9840-45c2-b4a0-80df0f319109\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdqx4"
Nov 01 14:16:53.984597 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.983604    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2e013e38-9840-45c2-b4a0-80df0f319109-socket-dir\") pod \"aws-ebs-csi-driver-node-vdqx4\" (UID: \"2e013e38-9840-45c2-b4a0-80df0f319109\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdqx4"
Nov 01 14:16:53.984597 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.983625    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8a6e7f39-7853-4fec-9ea2-30099983e5bd-iptables-alerter-script\") pod \"iptables-alerter-8jxks\" (UID: \"8a6e7f39-7853-4fec-9ea2-30099983e5bd\") " pod="openshift-network-operator/iptables-alerter-8jxks"
Nov 01 14:16:53.984597 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.983664    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7a26b817-3283-4d59-8e8f-3bb4c0fa4987-tmp\") pod \"insights-operator-7c4986d449-7d54k\" (UID: \"7a26b817-3283-4d59-8e8f-3bb4c0fa4987\") " pod="openshift-insights/insights-operator-7c4986d449-7d54k"
Nov 01 14:16:53.984597 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.983687    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-run-systemd\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:53.984597 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.983705    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-node-log\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:53.984597 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.983734    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5e15a59f-6aac-45e7-8812-ae97d0559052-lib-modules\") pod \"tuned-m5m8p\" (UID: \"5e15a59f-6aac-45e7-8812-ae97d0559052\") " pod="openshift-cluster-node-tuning-operator/tuned-m5m8p"
Nov 01 14:16:53.984597 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.983768    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e7ea769b-e719-4090-bcd4-05b1b88fcc06-image-registry-private-configuration\") pod \"image-registry-69c8c9797f-cblkb\" (UID: \"e7ea769b-e719-4090-bcd4-05b1b88fcc06\") " pod="openshift-image-registry/image-registry-69c8c9797f-cblkb"
Nov 01 14:16:53.984597 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.983801    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e7ea769b-e719-4090-bcd4-05b1b88fcc06-bound-sa-token\") pod \"image-registry-69c8c9797f-cblkb\" (UID: \"e7ea769b-e719-4090-bcd4-05b1b88fcc06\") " pod="openshift-image-registry/image-registry-69c8c9797f-cblkb"
Nov 01 14:16:53.984597 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.983841    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gnmx\" (UniqueName: \"kubernetes.io/projected/7970eec1-3765-48a4-b09c-3dec2d29956b-kube-api-access-8gnmx\") pod \"image-registry-7f9cfbc8db-w8cmn\" (UID: \"7970eec1-3765-48a4-b09c-3dec2d29956b\") " pod="openshift-image-registry/image-registry-7f9cfbc8db-w8cmn"
Nov 01 14:16:53.984597 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.983858    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2e013e38-9840-45c2-b4a0-80df0f319109-sys-fs\") pod \"aws-ebs-csi-driver-node-vdqx4\" (UID: \"2e013e38-9840-45c2-b4a0-80df0f319109\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdqx4"
Nov 01 14:16:53.984597 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.983886    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6cqj\" (UniqueName: \"kubernetes.io/projected/d8a5ccab-4d5e-4aaf-ad08-766238eca05f-kube-api-access-q6cqj\") pod \"kas-connection-checker-685895c7c9-zfqjv\" (UID: \"d8a5ccab-4d5e-4aaf-ad08-766238eca05f\") " pod="kube-system/kas-connection-checker-685895c7c9-zfqjv"
Nov 01 14:16:53.984597 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.983968    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e7ea769b-e719-4090-bcd4-05b1b88fcc06-image-registry-private-configuration\") pod \"image-registry-69c8c9797f-cblkb\" (UID: \"e7ea769b-e719-4090-bcd4-05b1b88fcc06\") " pod="openshift-image-registry/image-registry-69c8c9797f-cblkb"
Nov 01 14:16:53.984597 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.984017    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c27ff402-235a-4d73-a139-cf928a6d5cbe-metrics-certs\") pod \"router-default-b9d995dcc-lnk2n\" (UID: \"c27ff402-235a-4d73-a139-cf928a6d5cbe\") " pod="openshift-ingress/router-default-b9d995dcc-lnk2n"
Nov 01 14:16:53.984597 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.984047    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d407313d-3333-4ec4-85bb-cd27cf4d7422-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-hsdjc\" (UID: \"d407313d-3333-4ec4-85bb-cd27cf4d7422\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hsdjc"
Nov 01 14:16:53.984597 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.984089    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e7ea769b-e719-4090-bcd4-05b1b88fcc06-registry-certificates\") pod \"image-registry-69c8c9797f-cblkb\" (UID: \"e7ea769b-e719-4090-bcd4-05b1b88fcc06\") " pod="openshift-image-registry/image-registry-69c8c9797f-cblkb"
Nov 01 14:16:53.984597 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.984116    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b73fabd2-0d2c-4988-bbcf-7564d9331390-multus-daemon-config\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:53.984597 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.984133    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp6rs\" (UniqueName: \"kubernetes.io/projected/0d4352db-d2e7-43ba-97ba-9e344abbac8b-kube-api-access-vp6rs\") pod \"network-check-target-8mqxj\" (UID: \"0d4352db-d2e7-43ba-97ba-9e344abbac8b\") " pod="openshift-network-diagnostics/network-check-target-8mqxj"
Nov 01 14:16:53.984597 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.984150    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-host-run-ovn-kubernetes\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:53.984597 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.984187    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1b941fae-3c27-4d68-8205-38e492179ee9-ovnkube-script-lib\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:53.984597 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.984212    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5e15a59f-6aac-45e7-8812-ae97d0559052-etc-tuned\") pod \"tuned-m5m8p\" (UID: \"5e15a59f-6aac-45e7-8812-ae97d0559052\") " pod="openshift-cluster-node-tuning-operator/tuned-m5m8p"
Nov 01 14:16:53.984597 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.984239    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e15a59f-6aac-45e7-8812-ae97d0559052-tmp\") pod \"tuned-m5m8p\" (UID: \"5e15a59f-6aac-45e7-8812-ae97d0559052\") " pod="openshift-cluster-node-tuning-operator/tuned-m5m8p"
Nov 01 14:16:53.984597 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.984236    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e7ea769b-e719-4090-bcd4-05b1b88fcc06-registry-certificates\") pod \"image-registry-69c8c9797f-cblkb\" (UID: \"e7ea769b-e719-4090-bcd4-05b1b88fcc06\") " pod="openshift-image-registry/image-registry-69c8c9797f-cblkb"
Nov 01 14:16:53.984597 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.984265    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/321ecf30-5108-4320-9a58-6b204001639c-signing-cabundle\") pod \"service-ca-5fffc97bdf-l25pw\" (UID: \"321ecf30-5108-4320-9a58-6b204001639c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-l25pw"
Nov 01 14:16:53.984597 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.984292    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7970eec1-3765-48a4-b09c-3dec2d29956b-image-registry-private-configuration\") pod \"image-registry-7f9cfbc8db-w8cmn\" (UID: \"7970eec1-3765-48a4-b09c-3dec2d29956b\") " pod="openshift-image-registry/image-registry-7f9cfbc8db-w8cmn"
Nov 01 14:16:53.984597 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.984319    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a26b817-3283-4d59-8e8f-3bb4c0fa4987-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-7d54k\" (UID: \"7a26b817-3283-4d59-8e8f-3bb4c0fa4987\") " pod="openshift-insights/insights-operator-7c4986d449-7d54k"
Nov 01 14:16:53.984597 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.984348    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2e013e38-9840-45c2-b4a0-80df0f319109-registration-dir\") pod \"aws-ebs-csi-driver-node-vdqx4\" (UID: \"2e013e38-9840-45c2-b4a0-80df0f319109\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdqx4"
Nov 01 14:16:53.984597 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.984355    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b73fabd2-0d2c-4988-bbcf-7564d9331390-multus-daemon-config\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:53.984597 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.984377    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e15a59f-6aac-45e7-8812-ae97d0559052-tmp\") pod \"tuned-m5m8p\" (UID: \"5e15a59f-6aac-45e7-8812-ae97d0559052\") " pod="openshift-cluster-node-tuning-operator/tuned-m5m8p"
Nov 01 14:16:53.984597 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.984376    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2e013e38-9840-45c2-b4a0-80df0f319109-etc-selinux\") pod \"aws-ebs-csi-driver-node-vdqx4\" (UID: \"2e013e38-9840-45c2-b4a0-80df0f319109\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdqx4"
Nov 01 14:16:53.984597 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.984321    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5e15a59f-6aac-45e7-8812-ae97d0559052-etc-tuned\") pod \"tuned-m5m8p\" (UID: \"5e15a59f-6aac-45e7-8812-ae97d0559052\") " pod="openshift-cluster-node-tuning-operator/tuned-m5m8p"
Nov 01 14:16:53.984597 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.984410    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1b941fae-3c27-4d68-8205-38e492179ee9-ovnkube-script-lib\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:53.984597 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.984420    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpdrz\" (UniqueName: \"kubernetes.io/projected/17570aba-70d9-46ea-a0ca-3a465635a17c-kube-api-access-zpdrz\") pod \"migrator-66df5d496b-4mmw7\" (UID: \"17570aba-70d9-46ea-a0ca-3a465635a17c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4mmw7"
Nov 01 14:16:53.984597 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.984442    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/321ecf30-5108-4320-9a58-6b204001639c-signing-cabundle\") pod \"service-ca-5fffc97bdf-l25pw\" (UID: \"321ecf30-5108-4320-9a58-6b204001639c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-l25pw"
Nov 01 14:16:53.984597 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.984456    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/384cd50b-370f-474f-b80e-89a544a9bbbd-trusted-ca-bundle\") pod \"periodic-gathering-rlgcn-h5sk4\" (UID: \"384cd50b-370f-474f-b80e-89a544a9bbbd\") " pod="openshift-insights/periodic-gathering-rlgcn-h5sk4"
Nov 01 14:16:53.984597 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.984573    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7970eec1-3765-48a4-b09c-3dec2d29956b-image-registry-private-configuration\") pod \"image-registry-7f9cfbc8db-w8cmn\" (UID: \"7970eec1-3765-48a4-b09c-3dec2d29956b\") " pod="openshift-image-registry/image-registry-7f9cfbc8db-w8cmn"
Nov 01 14:16:53.985811 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.984638    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a26b817-3283-4d59-8e8f-3bb4c0fa4987-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-7d54k\" (UID: \"7a26b817-3283-4d59-8e8f-3bb4c0fa4987\") " pod="openshift-insights/insights-operator-7c4986d449-7d54k"
Nov 01 14:16:53.985811 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.985184    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/384cd50b-370f-474f-b80e-89a544a9bbbd-trusted-ca-bundle\") pod \"periodic-gathering-rlgcn-h5sk4\" (UID: \"384cd50b-370f-474f-b80e-89a544a9bbbd\") " pod="openshift-insights/periodic-gathering-rlgcn-h5sk4"
Nov 01 14:16:53.988573 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:53.988557    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7970eec1-3765-48a4-b09c-3dec2d29956b-trusted-ca\") pod \"image-registry-7f9cfbc8db-w8cmn\" (UID: \"7970eec1-3765-48a4-b09c-3dec2d29956b\") " pod="openshift-image-registry/image-registry-7f9cfbc8db-w8cmn"
Nov 01 14:16:54.030278 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.030243    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbnlg\" (UniqueName: \"kubernetes.io/projected/8a6e7f39-7853-4fec-9ea2-30099983e5bd-kube-api-access-hbnlg\") pod \"iptables-alerter-8jxks\" (UID: \"8a6e7f39-7853-4fec-9ea2-30099983e5bd\") " pod="openshift-network-operator/iptables-alerter-8jxks"
Nov 01 14:16:54.030465 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.030426    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkzgz\" (UniqueName: \"kubernetes.io/projected/70bb916a-832b-4bce-9c0a-f01bf4e3abc4-kube-api-access-xkzgz\") pod \"ingress-canary-8fgtw\" (UID: \"70bb916a-832b-4bce-9c0a-f01bf4e3abc4\") " pod="openshift-ingress-canary/ingress-canary-8fgtw"
Nov 01 14:16:54.030962 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.030945    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjv76\" (UniqueName: \"kubernetes.io/projected/c27ff402-235a-4d73-a139-cf928a6d5cbe-kube-api-access-rjv76\") pod \"router-default-b9d995dcc-lnk2n\" (UID: \"c27ff402-235a-4d73-a139-cf928a6d5cbe\") " pod="openshift-ingress/router-default-b9d995dcc-lnk2n"
Nov 01 14:16:54.031095 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.031073    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvgpz\" (UniqueName: \"kubernetes.io/projected/00216deb-5dd0-4b4f-8940-c17dc329303c-kube-api-access-cvgpz\") pod \"node-resolver-pwgqr\" (UID: \"00216deb-5dd0-4b4f-8940-c17dc329303c\") " pod="openshift-dns/node-resolver-pwgqr"
Nov 01 14:16:54.031162 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.031148    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bhpq\" (UniqueName: \"kubernetes.io/projected/321ecf30-5108-4320-9a58-6b204001639c-kube-api-access-8bhpq\") pod \"service-ca-5fffc97bdf-l25pw\" (UID: \"321ecf30-5108-4320-9a58-6b204001639c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-l25pw"
Nov 01 14:16:54.031260 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.031243    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx82t\" (UniqueName: \"kubernetes.io/projected/058820ef-eef1-41ca-b331-670bab146861-kube-api-access-gx82t\") pod \"insights-runtime-extractor-zq4pm\" (UID: \"058820ef-eef1-41ca-b331-670bab146861\") " pod="openshift-insights/insights-runtime-extractor-zq4pm"
Nov 01 14:16:54.032513 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.032497    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpm8k\" (UniqueName: \"kubernetes.io/projected/ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1-kube-api-access-hpm8k\") pod \"network-metrics-daemon-b6rtr\" (UID: \"ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1\") " pod="openshift-multus/network-metrics-daemon-b6rtr"
Nov 01 14:16:54.032864 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.032844    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzjlr\" (UniqueName: \"kubernetes.io/projected/2e013e38-9840-45c2-b4a0-80df0f319109-kube-api-access-wzjlr\") pod \"aws-ebs-csi-driver-node-vdqx4\" (UID: \"2e013e38-9840-45c2-b4a0-80df0f319109\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdqx4"
Nov 01 14:16:54.032940 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.032924    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stw2n\" (UniqueName: \"kubernetes.io/projected/3e7e18b9-78ee-4349-b00c-a0800e8f8341-kube-api-access-stw2n\") pod \"service-ca-operator-544d749c8c-87xcd\" (UID: \"3e7e18b9-78ee-4349-b00c-a0800e8f8341\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-87xcd"
Nov 01 14:16:54.033365 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.033340    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9j226\" (UniqueName: \"kubernetes.io/projected/6a8f947a-0f8f-41aa-b646-4459fa8126a3-kube-api-access-9j226\") pod \"kas-connection-checker-685895c7c9-xmfr4\" (UID: \"6a8f947a-0f8f-41aa-b646-4459fa8126a3\") " pod="kube-system/kas-connection-checker-685895c7c9-xmfr4"
Nov 01 14:16:54.033695 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.033671    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92xdl\" (UniqueName: \"kubernetes.io/projected/bb59051d-c2df-4362-a157-00f14f915e79-kube-api-access-92xdl\") pod \"console-operator-7cd56b8dbc-xmwtp\" (UID: \"bb59051d-c2df-4362-a157-00f14f915e79\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-xmwtp"
Nov 01 14:16:54.033782 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.033753    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxdcx\" (UniqueName: \"kubernetes.io/projected/e78d2494-62c4-4070-a1eb-8ce40329ff57-kube-api-access-wxdcx\") pod \"volume-data-source-validator-58c9f5f46d-shpt4\" (UID: \"e78d2494-62c4-4070-a1eb-8ce40329ff57\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-shpt4"
Nov 01 14:16:54.033826 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.033785    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnzsm\" (UniqueName: \"kubernetes.io/projected/a12d30a7-8564-43c1-9742-3194fe8b975f-kube-api-access-gnzsm\") pod \"node-ca-d7z97\" (UID: \"a12d30a7-8564-43c1-9742-3194fe8b975f\") " pod="openshift-image-registry/node-ca-d7z97"
Nov 01 14:16:54.033876 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.033856    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddmmn\" (UniqueName: \"kubernetes.io/projected/b73fabd2-0d2c-4988-bbcf-7564d9331390-kube-api-access-ddmmn\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:54.034055 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.034040    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgmsd\" (UniqueName: \"kubernetes.io/projected/0c8cc47b-fb01-4195-a7fc-eeefc706017e-kube-api-access-wgmsd\") pod \"dns-default-b2lcr\" (UID: \"0c8cc47b-fb01-4195-a7fc-eeefc706017e\") " pod="openshift-dns/dns-default-b2lcr"
Nov 01 14:16:54.034090 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.034043    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsmvn\" (UniqueName: \"kubernetes.io/projected/d407313d-3333-4ec4-85bb-cd27cf4d7422-kube-api-access-fsmvn\") pod \"cluster-samples-operator-6ddb6c8b94-hsdjc\" (UID: \"d407313d-3333-4ec4-85bb-cd27cf4d7422\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hsdjc"
Nov 01 14:16:54.034800 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.034783    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjj74\" (UniqueName: \"kubernetes.io/projected/f219e3c2-6da8-4c88-9ec4-b26cf4d10326-kube-api-access-pjj74\") pod \"network-check-source-6f78d59f99-lhdpr\" (UID: \"f219e3c2-6da8-4c88-9ec4-b26cf4d10326\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-lhdpr"
Nov 01 14:16:54.035387 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.035370    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfj7s\" (UniqueName: \"kubernetes.io/projected/4a724964-bd57-4d56-a042-a0d490115eaa-kube-api-access-kfj7s\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-96vgd\" (UID: \"4a724964-bd57-4d56-a042-a0d490115eaa\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-96vgd"
Nov 01 14:16:54.035760 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.035741    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8thtk\" (UniqueName: \"kubernetes.io/projected/037ca099-5ad7-4874-a984-462c07801e70-kube-api-access-8thtk\") pod \"cluster-monitoring-operator-884576c57-bgmv8\" (UID: \"037ca099-5ad7-4874-a984-462c07801e70\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bgmv8"
Nov 01 14:16:54.035930 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.035909    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdsp8\" (UniqueName: \"kubernetes.io/projected/bcc6412c-2003-49de-9817-a3713e28afef-kube-api-access-vdsp8\") pod \"multus-additional-cni-plugins-9vt8t\" (UID: \"bcc6412c-2003-49de-9817-a3713e28afef\") " pod="openshift-multus/multus-additional-cni-plugins-9vt8t"
Nov 01 14:16:54.036048 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.036027    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7970eec1-3765-48a4-b09c-3dec2d29956b-bound-sa-token\") pod \"image-registry-7f9cfbc8db-w8cmn\" (UID: \"7970eec1-3765-48a4-b09c-3dec2d29956b\") " pod="openshift-image-registry/image-registry-7f9cfbc8db-w8cmn"
Nov 01 14:16:54.036121 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.036055    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khv87\" (UniqueName: \"kubernetes.io/projected/fba1e7f9-ca37-46dc-8c66-56f1d9563acd-kube-api-access-khv87\") pod \"migrator-66df5d496b-8rp2x\" (UID: \"fba1e7f9-ca37-46dc-8c66-56f1d9563acd\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8rp2x"
Nov 01 14:16:54.036220 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.036203    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdxs8\" (UniqueName: \"kubernetes.io/projected/e7ea769b-e719-4090-bcd4-05b1b88fcc06-kube-api-access-rdxs8\") pod \"image-registry-69c8c9797f-cblkb\" (UID: \"e7ea769b-e719-4090-bcd4-05b1b88fcc06\") " pod="openshift-image-registry/image-registry-69c8c9797f-cblkb"
Nov 01 14:16:54.036297 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.036280    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chwvj\" (UniqueName: \"kubernetes.io/projected/384cd50b-370f-474f-b80e-89a544a9bbbd-kube-api-access-chwvj\") pod \"periodic-gathering-rlgcn-h5sk4\" (UID: \"384cd50b-370f-474f-b80e-89a544a9bbbd\") " pod="openshift-insights/periodic-gathering-rlgcn-h5sk4"
Nov 01 14:16:54.036461 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.036437    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zcj8\" (UniqueName: \"kubernetes.io/projected/5e15a59f-6aac-45e7-8812-ae97d0559052-kube-api-access-8zcj8\") pod \"tuned-m5m8p\" (UID: \"5e15a59f-6aac-45e7-8812-ae97d0559052\") " pod="openshift-cluster-node-tuning-operator/tuned-m5m8p"
Nov 01 14:16:54.036516 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.036462    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jbv4\" (UniqueName: \"kubernetes.io/projected/2460be58-d51b-4876-bf9b-2a0e57901386-kube-api-access-7jbv4\") pod \"kas-connection-checker-685895c7c9-4lzxm\" (UID: \"2460be58-d51b-4876-bf9b-2a0e57901386\") " pod="kube-system/kas-connection-checker-685895c7c9-4lzxm"
Nov 01 14:16:54.036883 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.036864    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gnmx\" (UniqueName: \"kubernetes.io/projected/7970eec1-3765-48a4-b09c-3dec2d29956b-kube-api-access-8gnmx\") pod \"image-registry-7f9cfbc8db-w8cmn\" (UID: \"7970eec1-3765-48a4-b09c-3dec2d29956b\") " pod="openshift-image-registry/image-registry-7f9cfbc8db-w8cmn"
Nov 01 14:16:54.040379 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.037266    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e7ea769b-e719-4090-bcd4-05b1b88fcc06-bound-sa-token\") pod \"image-registry-69c8c9797f-cblkb\" (UID: \"e7ea769b-e719-4090-bcd4-05b1b88fcc06\") " pod="openshift-image-registry/image-registry-69c8c9797f-cblkb"
Nov 01 14:16:54.040379 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.037274    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6cqj\" (UniqueName: \"kubernetes.io/projected/d8a5ccab-4d5e-4aaf-ad08-766238eca05f-kube-api-access-q6cqj\") pod \"kas-connection-checker-685895c7c9-zfqjv\" (UID: \"d8a5ccab-4d5e-4aaf-ad08-766238eca05f\") " pod="kube-system/kas-connection-checker-685895c7c9-zfqjv"
Nov 01 14:16:54.040379 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.037713    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv8hp\" (UniqueName: \"kubernetes.io/projected/7a26b817-3283-4d59-8e8f-3bb4c0fa4987-kube-api-access-bv8hp\") pod \"insights-operator-7c4986d449-7d54k\" (UID: \"7a26b817-3283-4d59-8e8f-3bb4c0fa4987\") " pod="openshift-insights/insights-operator-7c4986d449-7d54k"
Nov 01 14:16:54.040379 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.037734    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8mfv\" (UniqueName: \"kubernetes.io/projected/1b941fae-3c27-4d68-8205-38e492179ee9-kube-api-access-p8mfv\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:54.066800 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.066771    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpdrz\" (UniqueName: \"kubernetes.io/projected/17570aba-70d9-46ea-a0ca-3a465635a17c-kube-api-access-zpdrz\") pod \"migrator-66df5d496b-4mmw7\" (UID: \"17570aba-70d9-46ea-a0ca-3a465635a17c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4mmw7"
Nov 01 14:16:54.066916 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.066887    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp6rs\" (UniqueName: \"kubernetes.io/projected/0d4352db-d2e7-43ba-97ba-9e344abbac8b-kube-api-access-vp6rs\") pod \"network-check-target-8mqxj\" (UID: \"0d4352db-d2e7-43ba-97ba-9e344abbac8b\") " pod="openshift-network-diagnostics/network-check-target-8mqxj"
Nov 01 14:16:54.085159 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.085130    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8b1eb53c-8c9a-46c4-b654-2aef42491b14-dbus\") pod \"global-pull-secret-syncer-sc9jq\" (UID: \"8b1eb53c-8c9a-46c4-b654-2aef42491b14\") " pod="kube-system/global-pull-secret-syncer-sc9jq"
Nov 01 14:16:54.085316 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.085163    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-run-openvswitch\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:54.085316 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.085202    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1-metrics-certs\") pod \"network-metrics-daemon-b6rtr\" (UID: \"ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1\") " pod="openshift-multus/network-metrics-daemon-b6rtr"
Nov 01 14:16:54.085316 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.085218    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/058820ef-eef1-41ca-b331-670bab146861-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zq4pm\" (UID: \"058820ef-eef1-41ca-b331-670bab146861\") " pod="openshift-insights/insights-runtime-extractor-zq4pm"
Nov 01 14:16:54.085316 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.085236    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b73fabd2-0d2c-4988-bbcf-7564d9331390-multus-socket-dir-parent\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:54.085316 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.085253    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5e15a59f-6aac-45e7-8812-ae97d0559052-etc-sysconfig\") pod \"tuned-m5m8p\" (UID: \"5e15a59f-6aac-45e7-8812-ae97d0559052\") " pod="openshift-cluster-node-tuning-operator/tuned-m5m8p"
Nov 01 14:16:54.085316 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.085258    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-run-openvswitch\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:54.085316 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.085276    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b73fabd2-0d2c-4988-bbcf-7564d9331390-host-var-lib-cni-bin\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:54.085316 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.085276    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8b1eb53c-8c9a-46c4-b654-2aef42491b14-dbus\") pod \"global-pull-secret-syncer-sc9jq\" (UID: \"8b1eb53c-8c9a-46c4-b654-2aef42491b14\") " pod="kube-system/global-pull-secret-syncer-sc9jq"
Nov 01 14:16:54.085316 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.085317    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b73fabd2-0d2c-4988-bbcf-7564d9331390-host-var-lib-cni-bin\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:54.085772 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.085322    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b73fabd2-0d2c-4988-bbcf-7564d9331390-multus-socket-dir-parent\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:54.085772 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:54.085354    4993 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 01 14:16:54.085772 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.085383    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b73fabd2-0d2c-4988-bbcf-7564d9331390-host-run-multus-certs\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:54.085772 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.085412    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5e15a59f-6aac-45e7-8812-ae97d0559052-etc-sysconfig\") pod \"tuned-m5m8p\" (UID: \"5e15a59f-6aac-45e7-8812-ae97d0559052\") " pod="openshift-cluster-node-tuning-operator/tuned-m5m8p"
Nov 01 14:16:54.085772 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.085418    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-host-cni-netd\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:54.085772 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.085449    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-host-cni-netd\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:54.085772 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:54.085457    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1-metrics-certs podName:ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1 nodeName:}" failed. No retries permitted until 2026-11-01 14:16:54.58543691 +0000 UTC m=+1.915299644 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1-metrics-certs") pod "network-metrics-daemon-b6rtr" (UID: "ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1") : secret "metrics-daemon-secret" not found
Nov 01 14:16:54.085772 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.085477    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b73fabd2-0d2c-4988-bbcf-7564d9331390-host-run-multus-certs\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:54.085772 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.085499    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b73fabd2-0d2c-4988-bbcf-7564d9331390-host-var-lib-kubelet\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:54.085772 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.085529    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-host-run-netns\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:54.085772 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.085554    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bcc6412c-2003-49de-9817-a3713e28afef-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9vt8t\" (UID: \"bcc6412c-2003-49de-9817-a3713e28afef\") " pod="openshift-multus/multus-additional-cni-plugins-9vt8t"
Nov 01 14:16:54.085772 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.085561    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b73fabd2-0d2c-4988-bbcf-7564d9331390-host-var-lib-kubelet\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:54.085772 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.085581    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8b1eb53c-8c9a-46c4-b654-2aef42491b14-kubelet-config\") pod \"global-pull-secret-syncer-sc9jq\" (UID: \"8b1eb53c-8c9a-46c4-b654-2aef42491b14\") " pod="kube-system/global-pull-secret-syncer-sc9jq"
Nov 01 14:16:54.085772 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.085600    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-host-run-netns\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:54.085772 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.085608    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bcc6412c-2003-49de-9817-a3713e28afef-system-cni-dir\") pod \"multus-additional-cni-plugins-9vt8t\" (UID: \"bcc6412c-2003-49de-9817-a3713e28afef\") " pod="openshift-multus/multus-additional-cni-plugins-9vt8t"
Nov 01 14:16:54.085772 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.085639    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bcc6412c-2003-49de-9817-a3713e28afef-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9vt8t\" (UID: \"bcc6412c-2003-49de-9817-a3713e28afef\") " pod="openshift-multus/multus-additional-cni-plugins-9vt8t"
Nov 01 14:16:54.085772 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.085649    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5e15a59f-6aac-45e7-8812-ae97d0559052-etc-sysctl-conf\") pod \"tuned-m5m8p\" (UID: \"5e15a59f-6aac-45e7-8812-ae97d0559052\") " pod="openshift-cluster-node-tuning-operator/tuned-m5m8p"
Nov 01 14:16:54.085772 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.085644    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bcc6412c-2003-49de-9817-a3713e28afef-system-cni-dir\") pod \"multus-additional-cni-plugins-9vt8t\" (UID: \"bcc6412c-2003-49de-9817-a3713e28afef\") " pod="openshift-multus/multus-additional-cni-plugins-9vt8t"
Nov 01 14:16:54.085772 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.085679    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e15a59f-6aac-45e7-8812-ae97d0559052-var-lib-kubelet\") pod \"tuned-m5m8p\" (UID: \"5e15a59f-6aac-45e7-8812-ae97d0559052\") " pod="openshift-cluster-node-tuning-operator/tuned-m5m8p"
Nov 01 14:16:54.085772 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.085707    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b73fabd2-0d2c-4988-bbcf-7564d9331390-host-run-netns\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:54.085772 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.085680    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8b1eb53c-8c9a-46c4-b654-2aef42491b14-kubelet-config\") pod \"global-pull-secret-syncer-sc9jq\" (UID: \"8b1eb53c-8c9a-46c4-b654-2aef42491b14\") " pod="kube-system/global-pull-secret-syncer-sc9jq"
Nov 01 14:16:54.085772 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.085716    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e15a59f-6aac-45e7-8812-ae97d0559052-var-lib-kubelet\") pod \"tuned-m5m8p\" (UID: \"5e15a59f-6aac-45e7-8812-ae97d0559052\") " pod="openshift-cluster-node-tuning-operator/tuned-m5m8p"
Nov 01 14:16:54.085772 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.085731    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b73fabd2-0d2c-4988-bbcf-7564d9331390-multus-conf-dir\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:54.085772 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.085751    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b73fabd2-0d2c-4988-bbcf-7564d9331390-host-run-netns\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:54.085772 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.085755    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-systemd-units\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:54.085772 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.085780    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-run-ovn\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:54.085772 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.085786    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-systemd-units\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:54.086902 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.085808    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b73fabd2-0d2c-4988-bbcf-7564d9331390-multus-conf-dir\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:54.086902 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.085817    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5e15a59f-6aac-45e7-8812-ae97d0559052-etc-modprobe-d\") pod \"tuned-m5m8p\" (UID: \"5e15a59f-6aac-45e7-8812-ae97d0559052\") " pod="openshift-cluster-node-tuning-operator/tuned-m5m8p"
Nov 01 14:16:54.086902 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.085823    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-run-ovn\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:54.086902 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.085833    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/058820ef-eef1-41ca-b331-670bab146861-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zq4pm\" (UID: \"058820ef-eef1-41ca-b331-670bab146861\") " pod="openshift-insights/insights-runtime-extractor-zq4pm"
Nov 01 14:16:54.086902 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.085836    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/058820ef-eef1-41ca-b331-670bab146861-crio-socket\") pod \"insights-runtime-extractor-zq4pm\" (UID: \"058820ef-eef1-41ca-b331-670bab146861\") " pod="openshift-insights/insights-runtime-extractor-zq4pm"
Nov 01 14:16:54.086902 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.085874    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2e013e38-9840-45c2-b4a0-80df0f319109-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vdqx4\" (UID: \"2e013e38-9840-45c2-b4a0-80df0f319109\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdqx4"
Nov 01 14:16:54.086902 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.085876    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/058820ef-eef1-41ca-b331-670bab146861-crio-socket\") pod \"insights-runtime-extractor-zq4pm\" (UID: \"058820ef-eef1-41ca-b331-670bab146861\") " pod="openshift-insights/insights-runtime-extractor-zq4pm"
Nov 01 14:16:54.086902 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.085779    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5e15a59f-6aac-45e7-8812-ae97d0559052-etc-sysctl-conf\") pod \"tuned-m5m8p\" (UID: \"5e15a59f-6aac-45e7-8812-ae97d0559052\") " pod="openshift-cluster-node-tuning-operator/tuned-m5m8p"
Nov 01 14:16:54.086902 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.085889    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5e15a59f-6aac-45e7-8812-ae97d0559052-etc-modprobe-d\") pod \"tuned-m5m8p\" (UID: \"5e15a59f-6aac-45e7-8812-ae97d0559052\") " pod="openshift-cluster-node-tuning-operator/tuned-m5m8p"
Nov 01 14:16:54.086902 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.085890    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2e013e38-9840-45c2-b4a0-80df0f319109-socket-dir\") pod \"aws-ebs-csi-driver-node-vdqx4\" (UID: \"2e013e38-9840-45c2-b4a0-80df0f319109\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdqx4"
Nov 01 14:16:54.086902 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.085915    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-run-systemd\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:54.086902 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.085939    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-node-log\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:54.086902 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.085946    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2e013e38-9840-45c2-b4a0-80df0f319109-socket-dir\") pod \"aws-ebs-csi-driver-node-vdqx4\" (UID: \"2e013e38-9840-45c2-b4a0-80df0f319109\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdqx4"
Nov 01 14:16:54.086902 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.085954    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5e15a59f-6aac-45e7-8812-ae97d0559052-lib-modules\") pod \"tuned-m5m8p\" (UID: \"5e15a59f-6aac-45e7-8812-ae97d0559052\") " pod="openshift-cluster-node-tuning-operator/tuned-m5m8p"
Nov 01 14:16:54.086902 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.085952    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2e013e38-9840-45c2-b4a0-80df0f319109-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vdqx4\" (UID: \"2e013e38-9840-45c2-b4a0-80df0f319109\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdqx4"
Nov 01 14:16:54.086902 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.085963    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-run-systemd\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:54.086902 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.085982    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-node-log\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:54.086902 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.085982    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2e013e38-9840-45c2-b4a0-80df0f319109-sys-fs\") pod \"aws-ebs-csi-driver-node-vdqx4\" (UID: \"2e013e38-9840-45c2-b4a0-80df0f319109\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdqx4"
Nov 01 14:16:54.086902 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.086014    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c27ff402-235a-4d73-a139-cf928a6d5cbe-metrics-certs\") pod \"router-default-b9d995dcc-lnk2n\" (UID: \"c27ff402-235a-4d73-a139-cf928a6d5cbe\") " pod="openshift-ingress/router-default-b9d995dcc-lnk2n"
Nov 01 14:16:54.086902 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.086024    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2e013e38-9840-45c2-b4a0-80df0f319109-sys-fs\") pod \"aws-ebs-csi-driver-node-vdqx4\" (UID: \"2e013e38-9840-45c2-b4a0-80df0f319109\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdqx4"
Nov 01 14:16:54.086902 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.086029    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5e15a59f-6aac-45e7-8812-ae97d0559052-lib-modules\") pod \"tuned-m5m8p\" (UID: \"5e15a59f-6aac-45e7-8812-ae97d0559052\") " pod="openshift-cluster-node-tuning-operator/tuned-m5m8p"
Nov 01 14:16:54.086902 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.086039    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d407313d-3333-4ec4-85bb-cd27cf4d7422-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-hsdjc\" (UID: \"d407313d-3333-4ec4-85bb-cd27cf4d7422\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hsdjc"
Nov 01 14:16:54.086902 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:54.086102    4993 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 01 14:16:54.086902 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:54.086121    4993 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 01 14:16:54.086902 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:54.086144    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d407313d-3333-4ec4-85bb-cd27cf4d7422-samples-operator-tls podName:d407313d-3333-4ec4-85bb-cd27cf4d7422 nodeName:}" failed. No retries permitted until 2026-11-01 14:16:54.586133068 +0000 UTC m=+1.915995786 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d407313d-3333-4ec4-85bb-cd27cf4d7422-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-hsdjc" (UID: "d407313d-3333-4ec4-85bb-cd27cf4d7422") : secret "samples-operator-tls" not found
Nov 01 14:16:54.086902 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.086177    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-host-run-ovn-kubernetes\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:54.086902 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.086208    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-host-run-ovn-kubernetes\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:54.086902 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:54.086219    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c27ff402-235a-4d73-a139-cf928a6d5cbe-metrics-certs podName:c27ff402-235a-4d73-a139-cf928a6d5cbe nodeName:}" failed. No retries permitted until 2026-11-01 14:16:54.586200705 +0000 UTC m=+1.916063424 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c27ff402-235a-4d73-a139-cf928a6d5cbe-metrics-certs") pod "router-default-b9d995dcc-lnk2n" (UID: "c27ff402-235a-4d73-a139-cf928a6d5cbe") : secret "router-metrics-certs-default" not found
Nov 01 14:16:54.086902 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.086234    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2e013e38-9840-45c2-b4a0-80df0f319109-registration-dir\") pod \"aws-ebs-csi-driver-node-vdqx4\" (UID: \"2e013e38-9840-45c2-b4a0-80df0f319109\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdqx4"
Nov 01 14:16:54.086902 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.086249    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2e013e38-9840-45c2-b4a0-80df0f319109-etc-selinux\") pod \"aws-ebs-csi-driver-node-vdqx4\" (UID: \"2e013e38-9840-45c2-b4a0-80df0f319109\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdqx4"
Nov 01 14:16:54.086902 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.086270    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5e15a59f-6aac-45e7-8812-ae97d0559052-etc-kubernetes\") pod \"tuned-m5m8p\" (UID: \"5e15a59f-6aac-45e7-8812-ae97d0559052\") " pod="openshift-cluster-node-tuning-operator/tuned-m5m8p"
Nov 01 14:16:54.086902 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.086291    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2e013e38-9840-45c2-b4a0-80df0f319109-registration-dir\") pod \"aws-ebs-csi-driver-node-vdqx4\" (UID: \"2e013e38-9840-45c2-b4a0-80df0f319109\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdqx4"
Nov 01 14:16:54.086902 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.086299    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b73fabd2-0d2c-4988-bbcf-7564d9331390-host-run-k8s-cni-cncf-io\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:54.086902 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.086310    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2e013e38-9840-45c2-b4a0-80df0f319109-etc-selinux\") pod \"aws-ebs-csi-driver-node-vdqx4\" (UID: \"2e013e38-9840-45c2-b4a0-80df0f319109\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdqx4"
Nov 01 14:16:54.086902 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.086323    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5e15a59f-6aac-45e7-8812-ae97d0559052-etc-kubernetes\") pod \"tuned-m5m8p\" (UID: \"5e15a59f-6aac-45e7-8812-ae97d0559052\") " pod="openshift-cluster-node-tuning-operator/tuned-m5m8p"
Nov 01 14:16:54.086902 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.086324    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b73fabd2-0d2c-4988-bbcf-7564d9331390-cnibin\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:54.086902 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.086344    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b73fabd2-0d2c-4988-bbcf-7564d9331390-host-var-lib-cni-multus\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:54.086902 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.086350    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b73fabd2-0d2c-4988-bbcf-7564d9331390-host-run-k8s-cni-cncf-io\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:54.086902 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.086364    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-var-lib-openvswitch\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:54.086902 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.086382    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/037ca099-5ad7-4874-a984-462c07801e70-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-bgmv8\" (UID: \"037ca099-5ad7-4874-a984-462c07801e70\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bgmv8"
Nov 01 14:16:54.086902 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.086387    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-var-lib-openvswitch\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:54.086902 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.086364    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b73fabd2-0d2c-4988-bbcf-7564d9331390-cnibin\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:54.086902 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.086399    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b73fabd2-0d2c-4988-bbcf-7564d9331390-multus-cni-dir\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:54.086902 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.086393    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b73fabd2-0d2c-4988-bbcf-7564d9331390-host-var-lib-cni-multus\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:54.086902 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.086440    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b73fabd2-0d2c-4988-bbcf-7564d9331390-multus-cni-dir\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:54.086902 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:54.086454    4993 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 01 14:16:54.086902 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.086484    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b73fabd2-0d2c-4988-bbcf-7564d9331390-hostroot\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:54.086902 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:54.086496    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/037ca099-5ad7-4874-a984-462c07801e70-cluster-monitoring-operator-tls podName:037ca099-5ad7-4874-a984-462c07801e70 nodeName:}" failed. No retries permitted until 2026-11-01 14:16:54.586482272 +0000 UTC m=+1.916344990 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/037ca099-5ad7-4874-a984-462c07801e70-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-bgmv8" (UID: "037ca099-5ad7-4874-a984-462c07801e70") : secret "cluster-monitoring-operator-tls" not found
Nov 01 14:16:54.086902 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.086509    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b73fabd2-0d2c-4988-bbcf-7564d9331390-hostroot\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:54.086902 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.086525    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-etc-openvswitch\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:54.086902 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.086550    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c8cc47b-fb01-4195-a7fc-eeefc706017e-metrics-tls\") pod \"dns-default-b2lcr\" (UID: \"0c8cc47b-fb01-4195-a7fc-eeefc706017e\") " pod="openshift-dns/dns-default-b2lcr"
Nov 01 14:16:54.086902 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.086564    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-etc-openvswitch\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:54.086902 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.086575    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c27ff402-235a-4d73-a139-cf928a6d5cbe-service-ca-bundle\") pod \"router-default-b9d995dcc-lnk2n\" (UID: \"c27ff402-235a-4d73-a139-cf928a6d5cbe\") " pod="openshift-ingress/router-default-b9d995dcc-lnk2n"
Nov 01 14:16:54.086902 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.086608    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-log-socket\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:54.086902 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:54.086618    4993 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 14:16:54.086902 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.086639    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8a6e7f39-7853-4fec-9ea2-30099983e5bd-host-slash\") pod \"iptables-alerter-8jxks\" (UID: \"8a6e7f39-7853-4fec-9ea2-30099983e5bd\") " pod="openshift-network-operator/iptables-alerter-8jxks"
Nov 01 14:16:54.086902 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:54.086656    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c8cc47b-fb01-4195-a7fc-eeefc706017e-metrics-tls podName:0c8cc47b-fb01-4195-a7fc-eeefc706017e nodeName:}" failed. No retries permitted until 2026-11-01 14:16:54.586645578 +0000 UTC m=+1.916508328 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0c8cc47b-fb01-4195-a7fc-eeefc706017e-metrics-tls") pod "dns-default-b2lcr" (UID: "0c8cc47b-fb01-4195-a7fc-eeefc706017e") : secret "dns-default-metrics-tls" not found
Nov 01 14:16:54.086902 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.086662    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8a6e7f39-7853-4fec-9ea2-30099983e5bd-host-slash\") pod \"iptables-alerter-8jxks\" (UID: \"8a6e7f39-7853-4fec-9ea2-30099983e5bd\") " pod="openshift-network-operator/iptables-alerter-8jxks"
Nov 01 14:16:54.086902 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.086674    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5e15a59f-6aac-45e7-8812-ae97d0559052-etc-systemd\") pod \"tuned-m5m8p\" (UID: \"5e15a59f-6aac-45e7-8812-ae97d0559052\") " pod="openshift-cluster-node-tuning-operator/tuned-m5m8p"
Nov 01 14:16:54.086902 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.086682    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-log-socket\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:54.086902 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:54.086689    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c27ff402-235a-4d73-a139-cf928a6d5cbe-service-ca-bundle podName:c27ff402-235a-4d73-a139-cf928a6d5cbe nodeName:}" failed. No retries permitted until 2026-11-01 14:16:54.586677229 +0000 UTC m=+1.916539990 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c27ff402-235a-4d73-a139-cf928a6d5cbe-service-ca-bundle") pod "router-default-b9d995dcc-lnk2n" (UID: "c27ff402-235a-4d73-a139-cf928a6d5cbe") : configmap references non-existent config key: service-ca.crt
Nov 01 14:16:54.086902 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.086707    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5e15a59f-6aac-45e7-8812-ae97d0559052-etc-systemd\") pod \"tuned-m5m8p\" (UID: \"5e15a59f-6aac-45e7-8812-ae97d0559052\") " pod="openshift-cluster-node-tuning-operator/tuned-m5m8p"
Nov 01 14:16:54.086902 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.086717    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e7ea769b-e719-4090-bcd4-05b1b88fcc06-registry-tls\") pod \"image-registry-69c8c9797f-cblkb\" (UID: \"e7ea769b-e719-4090-bcd4-05b1b88fcc06\") " pod="openshift-image-registry/image-registry-69c8c9797f-cblkb"
Nov 01 14:16:54.086902 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.086735    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7970eec1-3765-48a4-b09c-3dec2d29956b-registry-tls\") pod \"image-registry-7f9cfbc8db-w8cmn\" (UID: \"7970eec1-3765-48a4-b09c-3dec2d29956b\") " pod="openshift-image-registry/image-registry-7f9cfbc8db-w8cmn"
Nov 01 14:16:54.086902 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.086755    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-host-slash\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:54.086902 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.086770    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:54.086902 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.086788    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5e15a59f-6aac-45e7-8812-ae97d0559052-etc-sysctl-d\") pod \"tuned-m5m8p\" (UID: \"5e15a59f-6aac-45e7-8812-ae97d0559052\") " pod="openshift-cluster-node-tuning-operator/tuned-m5m8p"
Nov 01 14:16:54.086902 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.086806    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/058820ef-eef1-41ca-b331-670bab146861-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zq4pm\" (UID: \"058820ef-eef1-41ca-b331-670bab146861\") " pod="openshift-insights/insights-runtime-extractor-zq4pm"
Nov 01 14:16:54.086902 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:54.086825    4993 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 14:16:54.086902 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.086827    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-host-slash\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:54.086902 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:54.086838    4993 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69c8c9797f-cblkb: secret "image-registry-tls" not found
Nov 01 14:16:54.086902 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.086841    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:54.086902 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:54.086850    4993 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 14:16:54.086902 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.086858    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a12d30a7-8564-43c1-9742-3194fe8b975f-host\") pod \"node-ca-d7z97\" (UID: \"a12d30a7-8564-43c1-9742-3194fe8b975f\") " pod="openshift-image-registry/node-ca-d7z97"
Nov 01 14:16:54.086902 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:54.086862    4993 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f9cfbc8db-w8cmn: secret "image-registry-tls" not found
Nov 01 14:16:54.086902 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:54.086880    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7ea769b-e719-4090-bcd4-05b1b88fcc06-registry-tls podName:e7ea769b-e719-4090-bcd4-05b1b88fcc06 nodeName:}" failed. No retries permitted until 2026-11-01 14:16:54.586866187 +0000 UTC m=+1.916728955 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e7ea769b-e719-4090-bcd4-05b1b88fcc06-registry-tls") pod "image-registry-69c8c9797f-cblkb" (UID: "e7ea769b-e719-4090-bcd4-05b1b88fcc06") : secret "image-registry-tls" not found
Nov 01 14:16:54.086902 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:54.086883    4993 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 14:16:54.086902 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.086890    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5e15a59f-6aac-45e7-8812-ae97d0559052-etc-sysctl-d\") pod \"tuned-m5m8p\" (UID: \"5e15a59f-6aac-45e7-8812-ae97d0559052\") " pod="openshift-cluster-node-tuning-operator/tuned-m5m8p"
Nov 01 14:16:54.086902 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.086899    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a12d30a7-8564-43c1-9742-3194fe8b975f-host\") pod \"node-ca-d7z97\" (UID: \"a12d30a7-8564-43c1-9742-3194fe8b975f\") " pod="openshift-image-registry/node-ca-d7z97"
Nov 01 14:16:54.086902 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:54.086916    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7970eec1-3765-48a4-b09c-3dec2d29956b-registry-tls podName:7970eec1-3765-48a4-b09c-3dec2d29956b nodeName:}" failed. No retries permitted until 2026-11-01 14:16:54.586907745 +0000 UTC m=+1.916770463 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7970eec1-3765-48a4-b09c-3dec2d29956b-registry-tls") pod "image-registry-7f9cfbc8db-w8cmn" (UID: "7970eec1-3765-48a4-b09c-3dec2d29956b") : secret "image-registry-tls" not found
Nov 01 14:16:54.086902 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.086935    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2e013e38-9840-45c2-b4a0-80df0f319109-device-dir\") pod \"aws-ebs-csi-driver-node-vdqx4\" (UID: \"2e013e38-9840-45c2-b4a0-80df0f319109\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdqx4"
Nov 01 14:16:54.086902 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.086959    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b73fabd2-0d2c-4988-bbcf-7564d9331390-etc-kubernetes\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:54.089218 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.086982    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5e15a59f-6aac-45e7-8812-ae97d0559052-host\") pod \"tuned-m5m8p\" (UID: \"5e15a59f-6aac-45e7-8812-ae97d0559052\") " pod="openshift-cluster-node-tuning-operator/tuned-m5m8p"
Nov 01 14:16:54.089218 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.086992    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2e013e38-9840-45c2-b4a0-80df0f319109-device-dir\") pod \"aws-ebs-csi-driver-node-vdqx4\" (UID: \"2e013e38-9840-45c2-b4a0-80df0f319109\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdqx4"
Nov 01 14:16:54.089218 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.087009    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b73fabd2-0d2c-4988-bbcf-7564d9331390-etc-kubernetes\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:54.089218 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:54.087018    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/058820ef-eef1-41ca-b331-670bab146861-insights-runtime-extractor-tls podName:058820ef-eef1-41ca-b331-670bab146861 nodeName:}" failed. No retries permitted until 2026-11-01 14:16:54.587007427 +0000 UTC m=+1.916870145 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/058820ef-eef1-41ca-b331-670bab146861-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zq4pm" (UID: "058820ef-eef1-41ca-b331-670bab146861") : secret "insights-runtime-extractor-tls" not found
Nov 01 14:16:54.089218 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.087026    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5e15a59f-6aac-45e7-8812-ae97d0559052-host\") pod \"tuned-m5m8p\" (UID: \"5e15a59f-6aac-45e7-8812-ae97d0559052\") " pod="openshift-cluster-node-tuning-operator/tuned-m5m8p"
Nov 01 14:16:54.089218 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.087035    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b5a4fefb-cc48-4638-b0f8-590e36bd0118-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-cwfck\" (UID: \"b5a4fefb-cc48-4638-b0f8-590e36bd0118\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-cwfck"
Nov 01 14:16:54.089218 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.087065    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-host-kubelet\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:54.089218 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.087091    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5e15a59f-6aac-45e7-8812-ae97d0559052-run\") pod \"tuned-m5m8p\" (UID: \"5e15a59f-6aac-45e7-8812-ae97d0559052\") " pod="openshift-cluster-node-tuning-operator/tuned-m5m8p"
Nov 01 14:16:54.089218 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:54.087103    4993 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 01 14:16:54.089218 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.087107    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-host-kubelet\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:54.089218 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.087113    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5e15a59f-6aac-45e7-8812-ae97d0559052-sys\") pod \"tuned-m5m8p\" (UID: \"5e15a59f-6aac-45e7-8812-ae97d0559052\") " pod="openshift-cluster-node-tuning-operator/tuned-m5m8p"
Nov 01 14:16:54.089218 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:54.087129    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5a4fefb-cc48-4638-b0f8-590e36bd0118-networking-console-plugin-cert podName:b5a4fefb-cc48-4638-b0f8-590e36bd0118 nodeName:}" failed. No retries permitted until 2026-11-01 14:16:54.587120495 +0000 UTC m=+1.916983247 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b5a4fefb-cc48-4638-b0f8-590e36bd0118-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-cwfck" (UID: "b5a4fefb-cc48-4638-b0f8-590e36bd0118") : secret "networking-console-plugin-cert" not found
Nov 01 14:16:54.089218 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.087179    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5e15a59f-6aac-45e7-8812-ae97d0559052-run\") pod \"tuned-m5m8p\" (UID: \"5e15a59f-6aac-45e7-8812-ae97d0559052\") " pod="openshift-cluster-node-tuning-operator/tuned-m5m8p"
Nov 01 14:16:54.089218 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.087187    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b73fabd2-0d2c-4988-bbcf-7564d9331390-system-cni-dir\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:54.089218 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.087212    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bcc6412c-2003-49de-9817-a3713e28afef-cnibin\") pod \"multus-additional-cni-plugins-9vt8t\" (UID: \"bcc6412c-2003-49de-9817-a3713e28afef\") " pod="openshift-multus/multus-additional-cni-plugins-9vt8t"
Nov 01 14:16:54.089218 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.087163    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5e15a59f-6aac-45e7-8812-ae97d0559052-sys\") pod \"tuned-m5m8p\" (UID: \"5e15a59f-6aac-45e7-8812-ae97d0559052\") " pod="openshift-cluster-node-tuning-operator/tuned-m5m8p"
Nov 01 14:16:54.089218 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.087232    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b73fabd2-0d2c-4988-bbcf-7564d9331390-system-cni-dir\") pod \"multus-cwzp8\" (UID: \"b73fabd2-0d2c-4988-bbcf-7564d9331390\") " pod="openshift-multus/multus-cwzp8"
Nov 01 14:16:54.089218 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.087240    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bcc6412c-2003-49de-9817-a3713e28afef-cnibin\") pod \"multus-additional-cni-plugins-9vt8t\" (UID: \"bcc6412c-2003-49de-9817-a3713e28afef\") " pod="openshift-multus/multus-additional-cni-plugins-9vt8t"
Nov 01 14:16:54.089218 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.087262    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-host-cni-bin\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:54.089218 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.087279    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/00216deb-5dd0-4b4f-8940-c17dc329303c-hosts-file\") pod \"node-resolver-pwgqr\" (UID: \"00216deb-5dd0-4b4f-8940-c17dc329303c\") " pod="openshift-dns/node-resolver-pwgqr"
Nov 01 14:16:54.089218 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.087311    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1b941fae-3c27-4d68-8205-38e492179ee9-host-cni-bin\") pod \"ovnkube-node-cxjpl\" (UID: \"1b941fae-3c27-4d68-8205-38e492179ee9\") " pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:54.089218 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.087338    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/00216deb-5dd0-4b4f-8940-c17dc329303c-hosts-file\") pod \"node-resolver-pwgqr\" (UID: \"00216deb-5dd0-4b4f-8940-c17dc329303c\") " pod="openshift-dns/node-resolver-pwgqr"
Nov 01 14:16:54.182581 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.182551    4993 scope.go:122] "RemoveContainer" containerID="cbc84fec5c099225a4e49365bdccb7876e559e9347bbfff1ca76af14561a16c1"
Nov 01 14:16:54.194303 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.194269    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-l25pw"
Nov 01 14:16:54.195156 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.195138    4993 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 14:16:54.591776 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.591694    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c27ff402-235a-4d73-a139-cf928a6d5cbe-metrics-certs\") pod \"router-default-b9d995dcc-lnk2n\" (UID: \"c27ff402-235a-4d73-a139-cf928a6d5cbe\") " pod="openshift-ingress/router-default-b9d995dcc-lnk2n"
Nov 01 14:16:54.591776 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.591733    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d407313d-3333-4ec4-85bb-cd27cf4d7422-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-hsdjc\" (UID: \"d407313d-3333-4ec4-85bb-cd27cf4d7422\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hsdjc"
Nov 01 14:16:54.592103 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.591777    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/037ca099-5ad7-4874-a984-462c07801e70-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-bgmv8\" (UID: \"037ca099-5ad7-4874-a984-462c07801e70\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bgmv8"
Nov 01 14:16:54.592103 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.591808    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c8cc47b-fb01-4195-a7fc-eeefc706017e-metrics-tls\") pod \"dns-default-b2lcr\" (UID: \"0c8cc47b-fb01-4195-a7fc-eeefc706017e\") " pod="openshift-dns/dns-default-b2lcr"
Nov 01 14:16:54.592103 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.591834    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c27ff402-235a-4d73-a139-cf928a6d5cbe-service-ca-bundle\") pod \"router-default-b9d995dcc-lnk2n\" (UID: \"c27ff402-235a-4d73-a139-cf928a6d5cbe\") " pod="openshift-ingress/router-default-b9d995dcc-lnk2n"
Nov 01 14:16:54.592103 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:54.591839    4993 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 01 14:16:54.592103 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:54.591893    4993 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 01 14:16:54.592103 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:54.591905    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c27ff402-235a-4d73-a139-cf928a6d5cbe-metrics-certs podName:c27ff402-235a-4d73-a139-cf928a6d5cbe nodeName:}" failed. No retries permitted until 2026-11-01 14:16:55.591884907 +0000 UTC m=+2.921747640 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c27ff402-235a-4d73-a139-cf928a6d5cbe-metrics-certs") pod "router-default-b9d995dcc-lnk2n" (UID: "c27ff402-235a-4d73-a139-cf928a6d5cbe") : secret "router-metrics-certs-default" not found
Nov 01 14:16:54.592103 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.591892    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e7ea769b-e719-4090-bcd4-05b1b88fcc06-registry-tls\") pod \"image-registry-69c8c9797f-cblkb\" (UID: \"e7ea769b-e719-4090-bcd4-05b1b88fcc06\") " pod="openshift-image-registry/image-registry-69c8c9797f-cblkb"
Nov 01 14:16:54.592103 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:54.591936    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c27ff402-235a-4d73-a139-cf928a6d5cbe-service-ca-bundle podName:c27ff402-235a-4d73-a139-cf928a6d5cbe nodeName:}" failed. No retries permitted until 2026-11-01 14:16:55.591920319 +0000 UTC m=+2.921783054 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c27ff402-235a-4d73-a139-cf928a6d5cbe-service-ca-bundle") pod "router-default-b9d995dcc-lnk2n" (UID: "c27ff402-235a-4d73-a139-cf928a6d5cbe") : configmap references non-existent config key: service-ca.crt
Nov 01 14:16:54.592103 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:54.591938    4993 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 01 14:16:54.592103 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:54.591957    4993 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 14:16:54.592103 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:54.591965    4993 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 14:16:54.592103 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:54.592001    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/037ca099-5ad7-4874-a984-462c07801e70-cluster-monitoring-operator-tls podName:037ca099-5ad7-4874-a984-462c07801e70 nodeName:}" failed. No retries permitted until 2026-11-01 14:16:55.591955122 +0000 UTC m=+2.921817844 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/037ca099-5ad7-4874-a984-462c07801e70-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-bgmv8" (UID: "037ca099-5ad7-4874-a984-462c07801e70") : secret "cluster-monitoring-operator-tls" not found
Nov 01 14:16:54.592103 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:54.591971    4993 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69c8c9797f-cblkb: secret "image-registry-tls" not found
Nov 01 14:16:54.592103 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:54.592043    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c8cc47b-fb01-4195-a7fc-eeefc706017e-metrics-tls podName:0c8cc47b-fb01-4195-a7fc-eeefc706017e nodeName:}" failed. No retries permitted until 2026-11-01 14:16:55.592030522 +0000 UTC m=+2.921893244 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0c8cc47b-fb01-4195-a7fc-eeefc706017e-metrics-tls") pod "dns-default-b2lcr" (UID: "0c8cc47b-fb01-4195-a7fc-eeefc706017e") : secret "dns-default-metrics-tls" not found
Nov 01 14:16:54.592103 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:54.592059    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d407313d-3333-4ec4-85bb-cd27cf4d7422-samples-operator-tls podName:d407313d-3333-4ec4-85bb-cd27cf4d7422 nodeName:}" failed. No retries permitted until 2026-11-01 14:16:55.592050163 +0000 UTC m=+2.921912886 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d407313d-3333-4ec4-85bb-cd27cf4d7422-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-hsdjc" (UID: "d407313d-3333-4ec4-85bb-cd27cf4d7422") : secret "samples-operator-tls" not found
Nov 01 14:16:54.592103 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.592086    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7970eec1-3765-48a4-b09c-3dec2d29956b-registry-tls\") pod \"image-registry-7f9cfbc8db-w8cmn\" (UID: \"7970eec1-3765-48a4-b09c-3dec2d29956b\") " pod="openshift-image-registry/image-registry-7f9cfbc8db-w8cmn"
Nov 01 14:16:54.592675 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.592127    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/058820ef-eef1-41ca-b331-670bab146861-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zq4pm\" (UID: \"058820ef-eef1-41ca-b331-670bab146861\") " pod="openshift-insights/insights-runtime-extractor-zq4pm"
Nov 01 14:16:54.592675 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:54.592135    4993 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 14:16:54.592675 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:54.592144    4993 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f9cfbc8db-w8cmn: secret "image-registry-tls" not found
Nov 01 14:16:54.592675 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.592158    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b5a4fefb-cc48-4638-b0f8-590e36bd0118-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-cwfck\" (UID: \"b5a4fefb-cc48-4638-b0f8-590e36bd0118\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-cwfck"
Nov 01 14:16:54.592675 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:54.592197    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7970eec1-3765-48a4-b09c-3dec2d29956b-registry-tls podName:7970eec1-3765-48a4-b09c-3dec2d29956b nodeName:}" failed. No retries permitted until 2026-11-01 14:16:55.592162424 +0000 UTC m=+2.922025156 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7970eec1-3765-48a4-b09c-3dec2d29956b-registry-tls") pod "image-registry-7f9cfbc8db-w8cmn" (UID: "7970eec1-3765-48a4-b09c-3dec2d29956b") : secret "image-registry-tls" not found
Nov 01 14:16:54.592675 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:54.592218    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7ea769b-e719-4090-bcd4-05b1b88fcc06-registry-tls podName:e7ea769b-e719-4090-bcd4-05b1b88fcc06 nodeName:}" failed. No retries permitted until 2026-11-01 14:16:55.59221137 +0000 UTC m=+2.922074085 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e7ea769b-e719-4090-bcd4-05b1b88fcc06-registry-tls") pod "image-registry-69c8c9797f-cblkb" (UID: "e7ea769b-e719-4090-bcd4-05b1b88fcc06") : secret "image-registry-tls" not found
Nov 01 14:16:54.592675 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:54.592239    4993 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 14:16:54.592675 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.592250    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1-metrics-certs\") pod \"network-metrics-daemon-b6rtr\" (UID: \"ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1\") " pod="openshift-multus/network-metrics-daemon-b6rtr"
Nov 01 14:16:54.592675 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:54.592277    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/058820ef-eef1-41ca-b331-670bab146861-insights-runtime-extractor-tls podName:058820ef-eef1-41ca-b331-670bab146861 nodeName:}" failed. No retries permitted until 2026-11-01 14:16:55.592269091 +0000 UTC m=+2.922131809 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/058820ef-eef1-41ca-b331-670bab146861-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zq4pm" (UID: "058820ef-eef1-41ca-b331-670bab146861") : secret "insights-runtime-extractor-tls" not found
Nov 01 14:16:54.592675 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:54.592241    4993 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 01 14:16:54.592675 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:54.592307    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5a4fefb-cc48-4638-b0f8-590e36bd0118-networking-console-plugin-cert podName:b5a4fefb-cc48-4638-b0f8-590e36bd0118 nodeName:}" failed. No retries permitted until 2026-11-01 14:16:55.592299218 +0000 UTC m=+2.922161932 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b5a4fefb-cc48-4638-b0f8-590e36bd0118-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-cwfck" (UID: "b5a4fefb-cc48-4638-b0f8-590e36bd0118") : secret "networking-console-plugin-cert" not found
Nov 01 14:16:54.592675 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:54.592322    4993 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 01 14:16:54.592675 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:54.592366    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1-metrics-certs podName:ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1 nodeName:}" failed. No retries permitted until 2026-11-01 14:16:55.592343276 +0000 UTC m=+2.922206027 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1-metrics-certs") pod "network-metrics-daemon-b6rtr" (UID: "ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1") : secret "metrics-daemon-secret" not found
Nov 01 14:16:54.850888 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.850816    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xmwtp_bb59051d-c2df-4362-a157-00f14f915e79/console-operator/2.log"
Nov 01 14:16:54.851295 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.851248    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xmwtp_bb59051d-c2df-4362-a157-00f14f915e79/console-operator/1.log"
Nov 01 14:16:54.851295 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.851276    4993 generic.go:356] "Generic (PLEG): container finished" podID="bb59051d-c2df-4362-a157-00f14f915e79" containerID="f3f93526708f60e3b9e8d3460f617d1feeda38b9c13af2a68462d2abb187b712" exitCode=255
Nov 01 14:16:54.851374 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.851355    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-xmwtp" event={"ID":"bb59051d-c2df-4362-a157-00f14f915e79","Type":"ContainerDied","Data":"f3f93526708f60e3b9e8d3460f617d1feeda38b9c13af2a68462d2abb187b712"}
Nov 01 14:16:54.851477 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.851413    4993 scope.go:122] "RemoveContainer" containerID="cbc84fec5c099225a4e49365bdccb7876e559e9347bbfff1ca76af14561a16c1"
Nov 01 14:16:54.851731 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.851711    4993 scope.go:122] "RemoveContainer" containerID="f3f93526708f60e3b9e8d3460f617d1feeda38b9c13af2a68462d2abb187b712"
Nov 01 14:16:54.851959 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:54.851937    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-xmwtp_openshift-console-operator(bb59051d-c2df-4362-a157-00f14f915e79)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-xmwtp" podUID="bb59051d-c2df-4362-a157-00f14f915e79"
Nov 01 14:16:54.853133 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.853101    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-l25pw" event={"ID":"321ecf30-5108-4320-9a58-6b204001639c","Type":"ContainerStarted","Data":"822f70d632801d54810cf8509a55257d0b27fb8def707c0ca021c68c883b7285"}
Nov 01 14:16:54.853233 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.853156    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-l25pw" event={"ID":"321ecf30-5108-4320-9a58-6b204001639c","Type":"ContainerStarted","Data":"ffb9c2812bfd05c93dc5e7fb90c15e9b272744135cb9e37b50a256bfa0a7432b"}
Nov 01 14:16:54.965742 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:54.965718    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-d7z97_a12d30a7-8564-43c1-9742-3194fe8b975f/node-ca/0.log"
Nov 01 14:16:55.126204 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:55.126126    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:55.566333 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:55.566292    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8fgtw_70bb916a-832b-4bce-9c0a-f01bf4e3abc4/serve-healthcheck-canary/0.log"
Nov 01 14:16:55.603097 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:55.603059    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c27ff402-235a-4d73-a139-cf928a6d5cbe-metrics-certs\") pod \"router-default-b9d995dcc-lnk2n\" (UID: \"c27ff402-235a-4d73-a139-cf928a6d5cbe\") " pod="openshift-ingress/router-default-b9d995dcc-lnk2n"
Nov 01 14:16:55.603288 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:55.603102    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d407313d-3333-4ec4-85bb-cd27cf4d7422-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-hsdjc\" (UID: \"d407313d-3333-4ec4-85bb-cd27cf4d7422\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hsdjc"
Nov 01 14:16:55.603288 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:55.603145    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/037ca099-5ad7-4874-a984-462c07801e70-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-bgmv8\" (UID: \"037ca099-5ad7-4874-a984-462c07801e70\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bgmv8"
Nov 01 14:16:55.603288 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:55.603181    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c8cc47b-fb01-4195-a7fc-eeefc706017e-metrics-tls\") pod \"dns-default-b2lcr\" (UID: \"0c8cc47b-fb01-4195-a7fc-eeefc706017e\") " pod="openshift-dns/dns-default-b2lcr"
Nov 01 14:16:55.603288 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:55.603207    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c27ff402-235a-4d73-a139-cf928a6d5cbe-service-ca-bundle\") pod \"router-default-b9d995dcc-lnk2n\" (UID: \"c27ff402-235a-4d73-a139-cf928a6d5cbe\") " pod="openshift-ingress/router-default-b9d995dcc-lnk2n"
Nov 01 14:16:55.603288 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:55.603221    4993 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 01 14:16:55.603288 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:55.603239    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e7ea769b-e719-4090-bcd4-05b1b88fcc06-registry-tls\") pod \"image-registry-69c8c9797f-cblkb\" (UID: \"e7ea769b-e719-4090-bcd4-05b1b88fcc06\") " pod="openshift-image-registry/image-registry-69c8c9797f-cblkb"
Nov 01 14:16:55.603288 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:55.603265    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7970eec1-3765-48a4-b09c-3dec2d29956b-registry-tls\") pod \"image-registry-7f9cfbc8db-w8cmn\" (UID: \"7970eec1-3765-48a4-b09c-3dec2d29956b\") " pod="openshift-image-registry/image-registry-7f9cfbc8db-w8cmn"
Nov 01 14:16:55.603288 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:55.603271    4993 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 01 14:16:55.603288 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:55.603282    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c27ff402-235a-4d73-a139-cf928a6d5cbe-metrics-certs podName:c27ff402-235a-4d73-a139-cf928a6d5cbe nodeName:}" failed. No retries permitted until 2026-11-01 14:16:57.603263279 +0000 UTC m=+4.933125994 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c27ff402-235a-4d73-a139-cf928a6d5cbe-metrics-certs") pod "router-default-b9d995dcc-lnk2n" (UID: "c27ff402-235a-4d73-a139-cf928a6d5cbe") : secret "router-metrics-certs-default" not found
Nov 01 14:16:55.603683 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:55.603291    4993 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 14:16:55.603683 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:55.603326    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d407313d-3333-4ec4-85bb-cd27cf4d7422-samples-operator-tls podName:d407313d-3333-4ec4-85bb-cd27cf4d7422 nodeName:}" failed. No retries permitted until 2026-11-01 14:16:57.603310882 +0000 UTC m=+4.933173611 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d407313d-3333-4ec4-85bb-cd27cf4d7422-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-hsdjc" (UID: "d407313d-3333-4ec4-85bb-cd27cf4d7422") : secret "samples-operator-tls" not found
Nov 01 14:16:55.603683 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:55.603329    4993 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 14:16:55.603683 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:55.603341    4993 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f9cfbc8db-w8cmn: secret "image-registry-tls" not found
Nov 01 14:16:55.603683 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:55.603348    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/058820ef-eef1-41ca-b331-670bab146861-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zq4pm\" (UID: \"058820ef-eef1-41ca-b331-670bab146861\") " pod="openshift-insights/insights-runtime-extractor-zq4pm"
Nov 01 14:16:55.603683 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:55.603376    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7970eec1-3765-48a4-b09c-3dec2d29956b-registry-tls podName:7970eec1-3765-48a4-b09c-3dec2d29956b nodeName:}" failed. No retries permitted until 2026-11-01 14:16:57.603365221 +0000 UTC m=+4.933227953 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7970eec1-3765-48a4-b09c-3dec2d29956b-registry-tls") pod "image-registry-7f9cfbc8db-w8cmn" (UID: "7970eec1-3765-48a4-b09c-3dec2d29956b") : secret "image-registry-tls" not found
Nov 01 14:16:55.603683 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:55.603387    4993 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 14:16:55.603683 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:55.603400    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c8cc47b-fb01-4195-a7fc-eeefc706017e-metrics-tls podName:0c8cc47b-fb01-4195-a7fc-eeefc706017e nodeName:}" failed. No retries permitted until 2026-11-01 14:16:57.603390881 +0000 UTC m=+4.933253611 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0c8cc47b-fb01-4195-a7fc-eeefc706017e-metrics-tls") pod "dns-default-b2lcr" (UID: "0c8cc47b-fb01-4195-a7fc-eeefc706017e") : secret "dns-default-metrics-tls" not found
Nov 01 14:16:55.603683 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:55.603408    4993 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69c8c9797f-cblkb: secret "image-registry-tls" not found
Nov 01 14:16:55.603683 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:55.603414    4993 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 14:16:55.603683 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:55.603414    4993 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 01 14:16:55.603683 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:55.603434    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c27ff402-235a-4d73-a139-cf928a6d5cbe-service-ca-bundle podName:c27ff402-235a-4d73-a139-cf928a6d5cbe nodeName:}" failed. No retries permitted until 2026-11-01 14:16:57.603418782 +0000 UTC m=+4.933281516 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c27ff402-235a-4d73-a139-cf928a6d5cbe-service-ca-bundle") pod "router-default-b9d995dcc-lnk2n" (UID: "c27ff402-235a-4d73-a139-cf928a6d5cbe") : configmap references non-existent config key: service-ca.crt
Nov 01 14:16:55.603683 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:55.603461    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b5a4fefb-cc48-4638-b0f8-590e36bd0118-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-cwfck\" (UID: \"b5a4fefb-cc48-4638-b0f8-590e36bd0118\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-cwfck"
Nov 01 14:16:55.603683 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:55.603477    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/037ca099-5ad7-4874-a984-462c07801e70-cluster-monitoring-operator-tls podName:037ca099-5ad7-4874-a984-462c07801e70 nodeName:}" failed. No retries permitted until 2026-11-01 14:16:57.603462745 +0000 UTC m=+4.933325476 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/037ca099-5ad7-4874-a984-462c07801e70-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-bgmv8" (UID: "037ca099-5ad7-4874-a984-462c07801e70") : secret "cluster-monitoring-operator-tls" not found
Nov 01 14:16:55.603683 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:55.603500    4993 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 01 14:16:55.603683 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:55.603503    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7ea769b-e719-4090-bcd4-05b1b88fcc06-registry-tls podName:e7ea769b-e719-4090-bcd4-05b1b88fcc06 nodeName:}" failed. No retries permitted until 2026-11-01 14:16:57.603495082 +0000 UTC m=+4.933357798 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e7ea769b-e719-4090-bcd4-05b1b88fcc06-registry-tls") pod "image-registry-69c8c9797f-cblkb" (UID: "e7ea769b-e719-4090-bcd4-05b1b88fcc06") : secret "image-registry-tls" not found
Nov 01 14:16:55.603683 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:55.603578    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5a4fefb-cc48-4638-b0f8-590e36bd0118-networking-console-plugin-cert podName:b5a4fefb-cc48-4638-b0f8-590e36bd0118 nodeName:}" failed. No retries permitted until 2026-11-01 14:16:57.603568461 +0000 UTC m=+4.933431176 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b5a4fefb-cc48-4638-b0f8-590e36bd0118-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-cwfck" (UID: "b5a4fefb-cc48-4638-b0f8-590e36bd0118") : secret "networking-console-plugin-cert" not found
Nov 01 14:16:55.603683 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:55.603591    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/058820ef-eef1-41ca-b331-670bab146861-insights-runtime-extractor-tls podName:058820ef-eef1-41ca-b331-670bab146861 nodeName:}" failed. No retries permitted until 2026-11-01 14:16:57.60358459 +0000 UTC m=+4.933447304 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/058820ef-eef1-41ca-b331-670bab146861-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zq4pm" (UID: "058820ef-eef1-41ca-b331-670bab146861") : secret "insights-runtime-extractor-tls" not found
Nov 01 14:16:55.603683 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:55.603648    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1-metrics-certs\") pod \"network-metrics-daemon-b6rtr\" (UID: \"ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1\") " pod="openshift-multus/network-metrics-daemon-b6rtr"
Nov 01 14:16:55.604418 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:55.603732    4993 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 01 14:16:55.604418 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:55.603768    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1-metrics-certs podName:ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1 nodeName:}" failed. No retries permitted until 2026-11-01 14:16:57.603756852 +0000 UTC m=+4.933619601 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1-metrics-certs") pod "network-metrics-daemon-b6rtr" (UID: "ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1") : secret "metrics-daemon-secret" not found
Nov 01 14:16:55.862350 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:55.862269    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xmwtp_bb59051d-c2df-4362-a157-00f14f915e79/console-operator/2.log"
Nov 01 14:16:56.168639 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:56.168566    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-96vgd_4a724964-bd57-4d56-a042-a0d490115eaa/kube-storage-version-migrator-operator/0.log"
Nov 01 14:16:56.867800 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:56.867736    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8rp2x" event={"ID":"fba1e7f9-ca37-46dc-8c66-56f1d9563acd","Type":"ContainerStarted","Data":"1038804c8b9a64e929e0e0016ae0d0a1eb1e1b7b07239f566ba6c133dd838bf6"}
Nov 01 14:16:56.872919 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:56.872886    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4mmw7" event={"ID":"17570aba-70d9-46ea-a0ca-3a465635a17c","Type":"ContainerStarted","Data":"478130e4be5c90bb87e8e4a34de21bf0485f9722e86665ec5fae22654a24ce70"}
Nov 01 14:16:57.069605 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:57.069571    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:16:57.069846 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:57.069829    4993 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 01 14:16:57.623483 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:57.623442    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1-metrics-certs\") pod \"network-metrics-daemon-b6rtr\" (UID: \"ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1\") " pod="openshift-multus/network-metrics-daemon-b6rtr"
Nov 01 14:16:57.623682 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:57.623517    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c27ff402-235a-4d73-a139-cf928a6d5cbe-metrics-certs\") pod \"router-default-b9d995dcc-lnk2n\" (UID: \"c27ff402-235a-4d73-a139-cf928a6d5cbe\") " pod="openshift-ingress/router-default-b9d995dcc-lnk2n"
Nov 01 14:16:57.623682 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:57.623543    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d407313d-3333-4ec4-85bb-cd27cf4d7422-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-hsdjc\" (UID: \"d407313d-3333-4ec4-85bb-cd27cf4d7422\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hsdjc"
Nov 01 14:16:57.623682 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:57.623583    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/037ca099-5ad7-4874-a984-462c07801e70-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-bgmv8\" (UID: \"037ca099-5ad7-4874-a984-462c07801e70\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bgmv8"
Nov 01 14:16:57.623682 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:57.623614    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c8cc47b-fb01-4195-a7fc-eeefc706017e-metrics-tls\") pod \"dns-default-b2lcr\" (UID: \"0c8cc47b-fb01-4195-a7fc-eeefc706017e\") " pod="openshift-dns/dns-default-b2lcr"
Nov 01 14:16:57.623682 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:57.623638    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c27ff402-235a-4d73-a139-cf928a6d5cbe-service-ca-bundle\") pod \"router-default-b9d995dcc-lnk2n\" (UID: \"c27ff402-235a-4d73-a139-cf928a6d5cbe\") " pod="openshift-ingress/router-default-b9d995dcc-lnk2n"
Nov 01 14:16:57.623682 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:57.623668    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e7ea769b-e719-4090-bcd4-05b1b88fcc06-registry-tls\") pod \"image-registry-69c8c9797f-cblkb\" (UID: \"e7ea769b-e719-4090-bcd4-05b1b88fcc06\") " pod="openshift-image-registry/image-registry-69c8c9797f-cblkb"
Nov 01 14:16:57.623953 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:57.623615    4993 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 01 14:16:57.623953 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:57.623692    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7970eec1-3765-48a4-b09c-3dec2d29956b-registry-tls\") pod \"image-registry-7f9cfbc8db-w8cmn\" (UID: \"7970eec1-3765-48a4-b09c-3dec2d29956b\") " pod="openshift-image-registry/image-registry-7f9cfbc8db-w8cmn"
Nov 01 14:16:57.623953 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:57.623741    4993 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 14:16:57.623953 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:57.623756    4993 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69c8c9797f-cblkb: secret "image-registry-tls" not found
Nov 01 14:16:57.623953 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:57.623765    4993 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 14:16:57.623953 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:57.623638    4993 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 01 14:16:57.623953 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:57.623684    4993 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 01 14:16:57.623953 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:57.623689    4993 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 14:16:57.623953 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:57.623771    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1-metrics-certs podName:ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1 nodeName:}" failed. No retries permitted until 2026-11-01 14:17:01.623749639 +0000 UTC m=+8.953612368 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1-metrics-certs") pod "network-metrics-daemon-b6rtr" (UID: "ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1") : secret "metrics-daemon-secret" not found
Nov 01 14:16:57.623953 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:57.623780    4993 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f9cfbc8db-w8cmn: secret "image-registry-tls" not found
Nov 01 14:16:57.623953 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:57.623885    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c27ff402-235a-4d73-a139-cf928a6d5cbe-service-ca-bundle podName:c27ff402-235a-4d73-a139-cf928a6d5cbe nodeName:}" failed. No retries permitted until 2026-11-01 14:17:01.623866501 +0000 UTC m=+8.953729224 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c27ff402-235a-4d73-a139-cf928a6d5cbe-service-ca-bundle") pod "router-default-b9d995dcc-lnk2n" (UID: "c27ff402-235a-4d73-a139-cf928a6d5cbe") : configmap references non-existent config key: service-ca.crt
Nov 01 14:16:57.623953 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:57.623905    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7ea769b-e719-4090-bcd4-05b1b88fcc06-registry-tls podName:e7ea769b-e719-4090-bcd4-05b1b88fcc06 nodeName:}" failed. No retries permitted until 2026-11-01 14:17:01.623894122 +0000 UTC m=+8.953756840 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e7ea769b-e719-4090-bcd4-05b1b88fcc06-registry-tls") pod "image-registry-69c8c9797f-cblkb" (UID: "e7ea769b-e719-4090-bcd4-05b1b88fcc06") : secret "image-registry-tls" not found
Nov 01 14:16:57.623953 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:57.623926    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/058820ef-eef1-41ca-b331-670bab146861-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zq4pm\" (UID: \"058820ef-eef1-41ca-b331-670bab146861\") " pod="openshift-insights/insights-runtime-extractor-zq4pm"
Nov 01 14:16:57.624472 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:57.623967    4993 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 01 14:16:57.624472 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:57.624018    4993 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 14:16:57.624472 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:57.624040    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c27ff402-235a-4d73-a139-cf928a6d5cbe-metrics-certs podName:c27ff402-235a-4d73-a139-cf928a6d5cbe nodeName:}" failed. No retries permitted until 2026-11-01 14:17:01.624018042 +0000 UTC m=+8.953880757 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c27ff402-235a-4d73-a139-cf928a6d5cbe-metrics-certs") pod "router-default-b9d995dcc-lnk2n" (UID: "c27ff402-235a-4d73-a139-cf928a6d5cbe") : secret "router-metrics-certs-default" not found
Nov 01 14:16:57.624472 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:57.624057    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d407313d-3333-4ec4-85bb-cd27cf4d7422-samples-operator-tls podName:d407313d-3333-4ec4-85bb-cd27cf4d7422 nodeName:}" failed. No retries permitted until 2026-11-01 14:17:01.624050104 +0000 UTC m=+8.953912819 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d407313d-3333-4ec4-85bb-cd27cf4d7422-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-hsdjc" (UID: "d407313d-3333-4ec4-85bb-cd27cf4d7422") : secret "samples-operator-tls" not found
Nov 01 14:16:57.624472 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:57.624068    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c8cc47b-fb01-4195-a7fc-eeefc706017e-metrics-tls podName:0c8cc47b-fb01-4195-a7fc-eeefc706017e nodeName:}" failed. No retries permitted until 2026-11-01 14:17:01.624061746 +0000 UTC m=+8.953924461 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0c8cc47b-fb01-4195-a7fc-eeefc706017e-metrics-tls") pod "dns-default-b2lcr" (UID: "0c8cc47b-fb01-4195-a7fc-eeefc706017e") : secret "dns-default-metrics-tls" not found
Nov 01 14:16:57.624472 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:57.624081    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7970eec1-3765-48a4-b09c-3dec2d29956b-registry-tls podName:7970eec1-3765-48a4-b09c-3dec2d29956b nodeName:}" failed. No retries permitted until 2026-11-01 14:17:01.624074619 +0000 UTC m=+8.953937337 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7970eec1-3765-48a4-b09c-3dec2d29956b-registry-tls") pod "image-registry-7f9cfbc8db-w8cmn" (UID: "7970eec1-3765-48a4-b09c-3dec2d29956b") : secret "image-registry-tls" not found
Nov 01 14:16:57.624472 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:57.624095    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/058820ef-eef1-41ca-b331-670bab146861-insights-runtime-extractor-tls podName:058820ef-eef1-41ca-b331-670bab146861 nodeName:}" failed. No retries permitted until 2026-11-01 14:17:01.624088367 +0000 UTC m=+8.953951082 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/058820ef-eef1-41ca-b331-670bab146861-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zq4pm" (UID: "058820ef-eef1-41ca-b331-670bab146861") : secret "insights-runtime-extractor-tls" not found
Nov 01 14:16:57.624472 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:57.623971    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b5a4fefb-cc48-4638-b0f8-590e36bd0118-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-cwfck\" (UID: \"b5a4fefb-cc48-4638-b0f8-590e36bd0118\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-cwfck"
Nov 01 14:16:57.626202 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:57.626159    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/037ca099-5ad7-4874-a984-462c07801e70-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-bgmv8\" (UID: \"037ca099-5ad7-4874-a984-462c07801e70\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bgmv8"
Nov 01 14:16:57.626334 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:57.626316    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b5a4fefb-cc48-4638-b0f8-590e36bd0118-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-cwfck\" (UID: \"b5a4fefb-cc48-4638-b0f8-590e36bd0118\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-cwfck"
Nov 01 14:16:57.781098 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:57.781067    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bgmv8"
Nov 01 14:16:57.783833 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:57.783815    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-cwfck"
Nov 01 14:16:58.098063 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:58.098026    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-xmwtp"
Nov 01 14:16:58.098561 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:58.098383    4993 scope.go:122] "RemoveContainer" containerID="f3f93526708f60e3b9e8d3460f617d1feeda38b9c13af2a68462d2abb187b712"
Nov 01 14:16:58.098623 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:16:58.098569    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-xmwtp_openshift-console-operator(bb59051d-c2df-4362-a157-00f14f915e79)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-xmwtp" podUID="bb59051d-c2df-4362-a157-00f14f915e79"
Nov 01 14:16:58.147918 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:58.147880    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-cwfck"]
Nov 01 14:16:58.154316 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:58.154289    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5a4fefb_cc48_4638_b0f8_590e36bd0118.slice/crio-ed0c260abe302c6428844e785dacb0e269d323dd95feb9ca4d5246c20edcc6ca WatchSource:0}: Error finding container ed0c260abe302c6428844e785dacb0e269d323dd95feb9ca4d5246c20edcc6ca: Status 404 returned error can't find the container with id ed0c260abe302c6428844e785dacb0e269d323dd95feb9ca4d5246c20edcc6ca
Nov 01 14:16:58.197627 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:58.197599    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-bgmv8"]
Nov 01 14:16:58.204412 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:16:58.204384    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod037ca099_5ad7_4874_a984_462c07801e70.slice/crio-87939b69802a96f90e535cac3d64c957b9ec6041ef5b1cda5b4f738543e23ac6 WatchSource:0}: Error finding container 87939b69802a96f90e535cac3d64c957b9ec6041ef5b1cda5b4f738543e23ac6: Status 404 returned error can't find the container with id 87939b69802a96f90e535cac3d64c957b9ec6041ef5b1cda5b4f738543e23ac6
Nov 01 14:16:58.880579 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:58.880541    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-cwfck" event={"ID":"b5a4fefb-cc48-4638-b0f8-590e36bd0118","Type":"ContainerStarted","Data":"ed0c260abe302c6428844e785dacb0e269d323dd95feb9ca4d5246c20edcc6ca"}
Nov 01 14:16:58.881617 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:16:58.881596    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bgmv8" event={"ID":"037ca099-5ad7-4874-a984-462c07801e70","Type":"ContainerStarted","Data":"87939b69802a96f90e535cac3d64c957b9ec6041ef5b1cda5b4f738543e23ac6"}
Nov 01 14:17:00.674450 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:00.674414    4993 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-xmwtp"
Nov 01 14:17:00.674869 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:00.674850    4993 scope.go:122] "RemoveContainer" containerID="f3f93526708f60e3b9e8d3460f617d1feeda38b9c13af2a68462d2abb187b712"
Nov 01 14:17:00.675095 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:17:00.675067    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-xmwtp_openshift-console-operator(bb59051d-c2df-4362-a157-00f14f915e79)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-xmwtp" podUID="bb59051d-c2df-4362-a157-00f14f915e79"
Nov 01 14:17:00.891311 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:00.891213    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-cwfck" event={"ID":"b5a4fefb-cc48-4638-b0f8-590e36bd0118","Type":"ContainerStarted","Data":"68119d516e4e48f51c89e6327a4315c274cc728a9a71d6728074206b08ea2038"}
Nov 01 14:17:00.892639 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:00.892610    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bgmv8" event={"ID":"037ca099-5ad7-4874-a984-462c07801e70","Type":"ContainerStarted","Data":"d7c9fe9bfe1e24eaea8d1756ff590c51d0e707d793eec7cb54b461d48d024fdb"}
Nov 01 14:17:01.668484 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:01.668443    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c8cc47b-fb01-4195-a7fc-eeefc706017e-metrics-tls\") pod \"dns-default-b2lcr\" (UID: \"0c8cc47b-fb01-4195-a7fc-eeefc706017e\") " pod="openshift-dns/dns-default-b2lcr"
Nov 01 14:17:01.668484 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:01.668483    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c27ff402-235a-4d73-a139-cf928a6d5cbe-service-ca-bundle\") pod \"router-default-b9d995dcc-lnk2n\" (UID: \"c27ff402-235a-4d73-a139-cf928a6d5cbe\") " pod="openshift-ingress/router-default-b9d995dcc-lnk2n"
Nov 01 14:17:01.668719 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:01.668508    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e7ea769b-e719-4090-bcd4-05b1b88fcc06-registry-tls\") pod \"image-registry-69c8c9797f-cblkb\" (UID: \"e7ea769b-e719-4090-bcd4-05b1b88fcc06\") " pod="openshift-image-registry/image-registry-69c8c9797f-cblkb"
Nov 01 14:17:01.668719 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:01.668529    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7970eec1-3765-48a4-b09c-3dec2d29956b-registry-tls\") pod \"image-registry-7f9cfbc8db-w8cmn\" (UID: \"7970eec1-3765-48a4-b09c-3dec2d29956b\") " pod="openshift-image-registry/image-registry-7f9cfbc8db-w8cmn"
Nov 01 14:17:01.668719 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:01.668562    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/058820ef-eef1-41ca-b331-670bab146861-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zq4pm\" (UID: \"058820ef-eef1-41ca-b331-670bab146861\") " pod="openshift-insights/insights-runtime-extractor-zq4pm"
Nov 01 14:17:01.668719 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:17:01.668598    4993 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 14:17:01.668719 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:17:01.668615    4993 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 14:17:01.668719 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:17:01.668632    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c27ff402-235a-4d73-a139-cf928a6d5cbe-service-ca-bundle podName:c27ff402-235a-4d73-a139-cf928a6d5cbe nodeName:}" failed. No retries permitted until 2026-11-01 14:17:09.668609266 +0000 UTC m=+16.998471982 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c27ff402-235a-4d73-a139-cf928a6d5cbe-service-ca-bundle") pod "router-default-b9d995dcc-lnk2n" (UID: "c27ff402-235a-4d73-a139-cf928a6d5cbe") : configmap references non-existent config key: service-ca.crt
Nov 01 14:17:01.668719 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:17:01.668661    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c8cc47b-fb01-4195-a7fc-eeefc706017e-metrics-tls podName:0c8cc47b-fb01-4195-a7fc-eeefc706017e nodeName:}" failed. No retries permitted until 2026-11-01 14:17:09.668651361 +0000 UTC m=+16.998514078 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0c8cc47b-fb01-4195-a7fc-eeefc706017e-metrics-tls") pod "dns-default-b2lcr" (UID: "0c8cc47b-fb01-4195-a7fc-eeefc706017e") : secret "dns-default-metrics-tls" not found
Nov 01 14:17:01.668719 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:01.668608    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1-metrics-certs\") pod \"network-metrics-daemon-b6rtr\" (UID: \"ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1\") " pod="openshift-multus/network-metrics-daemon-b6rtr"
Nov 01 14:17:01.668719 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:17:01.668633    4993 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69c8c9797f-cblkb: secret "image-registry-tls" not found
Nov 01 14:17:01.668719 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:17:01.668694    4993 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 14:17:01.668719 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:17:01.668706    4993 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 14:17:01.669149 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:01.668742    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c27ff402-235a-4d73-a139-cf928a6d5cbe-metrics-certs\") pod \"router-default-b9d995dcc-lnk2n\" (UID: \"c27ff402-235a-4d73-a139-cf928a6d5cbe\") " pod="openshift-ingress/router-default-b9d995dcc-lnk2n"
Nov 01 14:17:01.669149 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:17:01.668708    4993 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f9cfbc8db-w8cmn: secret "image-registry-tls" not found
Nov 01 14:17:01.669149 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:17:01.668770    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/058820ef-eef1-41ca-b331-670bab146861-insights-runtime-extractor-tls podName:058820ef-eef1-41ca-b331-670bab146861 nodeName:}" failed. No retries permitted until 2026-11-01 14:17:09.668753475 +0000 UTC m=+16.998616204 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/058820ef-eef1-41ca-b331-670bab146861-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zq4pm" (UID: "058820ef-eef1-41ca-b331-670bab146861") : secret "insights-runtime-extractor-tls" not found
Nov 01 14:17:01.669149 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:17:01.668796    4993 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 01 14:17:01.669149 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:17:01.668809    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7970eec1-3765-48a4-b09c-3dec2d29956b-registry-tls podName:7970eec1-3765-48a4-b09c-3dec2d29956b nodeName:}" failed. No retries permitted until 2026-11-01 14:17:09.668789329 +0000 UTC m=+16.998652058 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7970eec1-3765-48a4-b09c-3dec2d29956b-registry-tls") pod "image-registry-7f9cfbc8db-w8cmn" (UID: "7970eec1-3765-48a4-b09c-3dec2d29956b") : secret "image-registry-tls" not found
Nov 01 14:17:01.669149 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:01.668794    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d407313d-3333-4ec4-85bb-cd27cf4d7422-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-hsdjc\" (UID: \"d407313d-3333-4ec4-85bb-cd27cf4d7422\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hsdjc"
Nov 01 14:17:01.669149 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:17:01.668830    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7ea769b-e719-4090-bcd4-05b1b88fcc06-registry-tls podName:e7ea769b-e719-4090-bcd4-05b1b88fcc06 nodeName:}" failed. No retries permitted until 2026-11-01 14:17:09.66881964 +0000 UTC m=+16.998682372 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e7ea769b-e719-4090-bcd4-05b1b88fcc06-registry-tls") pod "image-registry-69c8c9797f-cblkb" (UID: "e7ea769b-e719-4090-bcd4-05b1b88fcc06") : secret "image-registry-tls" not found
Nov 01 14:17:01.669149 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:17:01.668851    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c27ff402-235a-4d73-a139-cf928a6d5cbe-metrics-certs podName:c27ff402-235a-4d73-a139-cf928a6d5cbe nodeName:}" failed. No retries permitted until 2026-11-01 14:17:09.668842668 +0000 UTC m=+16.998705395 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c27ff402-235a-4d73-a139-cf928a6d5cbe-metrics-certs") pod "router-default-b9d995dcc-lnk2n" (UID: "c27ff402-235a-4d73-a139-cf928a6d5cbe") : secret "router-metrics-certs-default" not found
Nov 01 14:17:01.670917 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:01.670900    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1-metrics-certs\") pod \"network-metrics-daemon-b6rtr\" (UID: \"ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1\") " pod="openshift-multus/network-metrics-daemon-b6rtr"
Nov 01 14:17:01.671132 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:01.671111    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d407313d-3333-4ec4-85bb-cd27cf4d7422-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-hsdjc\" (UID: \"d407313d-3333-4ec4-85bb-cd27cf4d7422\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hsdjc"
Nov 01 14:17:01.677894 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:01.677873    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hsdjc"
Nov 01 14:17:01.684892 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:01.684872    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b6rtr"
Nov 01 14:17:02.042246 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:02.042209    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hsdjc"]
Nov 01 14:17:02.091577 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:02.091549    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-b6rtr"]
Nov 01 14:17:02.102208 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:17:02.102163    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca8eaa6f_ce2b_4cf9_9ea7_3e6358ede5a1.slice/crio-aa489e6b716151f1b0937f732194b1b948f823fb16cf8cdc642ddbade321d01f WatchSource:0}: Error finding container aa489e6b716151f1b0937f732194b1b948f823fb16cf8cdc642ddbade321d01f: Status 404 returned error can't find the container with id aa489e6b716151f1b0937f732194b1b948f823fb16cf8cdc642ddbade321d01f
Nov 01 14:17:02.899864 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:02.899641    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-b6rtr" event={"ID":"ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1","Type":"ContainerStarted","Data":"aa489e6b716151f1b0937f732194b1b948f823fb16cf8cdc642ddbade321d01f"}
Nov 01 14:17:02.901060 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:02.901014    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hsdjc" event={"ID":"d407313d-3333-4ec4-85bb-cd27cf4d7422","Type":"ContainerStarted","Data":"3fc2d574502eb3860a803298f86848baf198611c5fc1a3a6a93e592df822ec39"}
Nov 01 14:17:03.432058 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:03.432014    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cxjpl"
Nov 01 14:17:03.907248 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:03.907213    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-b6rtr" event={"ID":"ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1","Type":"ContainerStarted","Data":"07e1d172ff903a352e1bdbda4c0a952f890551cad8dcd524f3dd8c795274d496"}
Nov 01 14:17:04.173488 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:04.173405    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8mqxj"
Nov 01 14:17:04.913057 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:04.913021    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-b6rtr" event={"ID":"ca8eaa6f-ce2b-4cf9-9ea7-3e6358ede5a1","Type":"ContainerStarted","Data":"5a051574ca7268406c5a0a5d8c55174b115ee93de2b8d596b9c63c734f768cbd"}
Nov 01 14:17:04.914851 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:04.914825    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hsdjc" event={"ID":"d407313d-3333-4ec4-85bb-cd27cf4d7422","Type":"ContainerStarted","Data":"2ad829d47f35a9786bee7154fea458218b07eba87940e72cbd5ffddcdfb3eaa5"}
Nov 01 14:17:04.914971 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:04.914870    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hsdjc" event={"ID":"d407313d-3333-4ec4-85bb-cd27cf4d7422","Type":"ContainerStarted","Data":"d7315bc77ab43eff2afe3041db6da0e52a818c41166e40b2b33b214ac09dbedc"}
Nov 01 14:17:05.788183 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:05.788148    4993 scope.go:122] "RemoveContainer" containerID="f3f93526708f60e3b9e8d3460f617d1feeda38b9c13af2a68462d2abb187b712"
Nov 01 14:17:05.920317 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:05.920294    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xmwtp_bb59051d-c2df-4362-a157-00f14f915e79/console-operator/3.log"
Nov 01 14:17:05.920710 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:05.920694    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xmwtp_bb59051d-c2df-4362-a157-00f14f915e79/console-operator/2.log"
Nov 01 14:17:05.920752 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:05.920727    4993 generic.go:356] "Generic (PLEG): container finished" podID="bb59051d-c2df-4362-a157-00f14f915e79" containerID="57098e775cd5e4f335940539931a0b59dc6621303b8f0714e11d150111008077" exitCode=255
Nov 01 14:17:05.920830 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:05.920808    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-xmwtp" event={"ID":"bb59051d-c2df-4362-a157-00f14f915e79","Type":"ContainerDied","Data":"57098e775cd5e4f335940539931a0b59dc6621303b8f0714e11d150111008077"}
Nov 01 14:17:05.920886 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:05.920877    4993 scope.go:122] "RemoveContainer" containerID="f3f93526708f60e3b9e8d3460f617d1feeda38b9c13af2a68462d2abb187b712"
Nov 01 14:17:05.921432 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:05.921416    4993 scope.go:122] "RemoveContainer" containerID="57098e775cd5e4f335940539931a0b59dc6621303b8f0714e11d150111008077"
Nov 01 14:17:05.921580 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:17:05.921561    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-xmwtp_openshift-console-operator(bb59051d-c2df-4362-a157-00f14f915e79)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-xmwtp" podUID="bb59051d-c2df-4362-a157-00f14f915e79"
Nov 01 14:17:06.927033 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:06.927006    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xmwtp_bb59051d-c2df-4362-a157-00f14f915e79/console-operator/3.log"
Nov 01 14:17:08.099048 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:08.099017    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-xmwtp"
Nov 01 14:17:08.099477 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:08.099391    4993 scope.go:122] "RemoveContainer" containerID="57098e775cd5e4f335940539931a0b59dc6621303b8f0714e11d150111008077"
Nov 01 14:17:08.099610 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:17:08.099589    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-xmwtp_openshift-console-operator(bb59051d-c2df-4362-a157-00f14f915e79)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-xmwtp" podUID="bb59051d-c2df-4362-a157-00f14f915e79"
Nov 01 14:17:09.746903 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:09.746863    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c27ff402-235a-4d73-a139-cf928a6d5cbe-metrics-certs\") pod \"router-default-b9d995dcc-lnk2n\" (UID: \"c27ff402-235a-4d73-a139-cf928a6d5cbe\") " pod="openshift-ingress/router-default-b9d995dcc-lnk2n"
Nov 01 14:17:09.747327 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:09.746916    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c8cc47b-fb01-4195-a7fc-eeefc706017e-metrics-tls\") pod \"dns-default-b2lcr\" (UID: \"0c8cc47b-fb01-4195-a7fc-eeefc706017e\") " pod="openshift-dns/dns-default-b2lcr"
Nov 01 14:17:09.747327 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:09.747090    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c27ff402-235a-4d73-a139-cf928a6d5cbe-service-ca-bundle\") pod \"router-default-b9d995dcc-lnk2n\" (UID: \"c27ff402-235a-4d73-a139-cf928a6d5cbe\") " pod="openshift-ingress/router-default-b9d995dcc-lnk2n"
Nov 01 14:17:09.747327 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:09.747133    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e7ea769b-e719-4090-bcd4-05b1b88fcc06-registry-tls\") pod \"image-registry-69c8c9797f-cblkb\" (UID: \"e7ea769b-e719-4090-bcd4-05b1b88fcc06\") " pod="openshift-image-registry/image-registry-69c8c9797f-cblkb"
Nov 01 14:17:09.747327 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:09.747155    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7970eec1-3765-48a4-b09c-3dec2d29956b-registry-tls\") pod \"image-registry-7f9cfbc8db-w8cmn\" (UID: \"7970eec1-3765-48a4-b09c-3dec2d29956b\") " pod="openshift-image-registry/image-registry-7f9cfbc8db-w8cmn"
Nov 01 14:17:09.747327 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:09.747199    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/058820ef-eef1-41ca-b331-670bab146861-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zq4pm\" (UID: \"058820ef-eef1-41ca-b331-670bab146861\") " pod="openshift-insights/insights-runtime-extractor-zq4pm"
Nov 01 14:17:09.747854 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:09.747822    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c27ff402-235a-4d73-a139-cf928a6d5cbe-service-ca-bundle\") pod \"router-default-b9d995dcc-lnk2n\" (UID: \"c27ff402-235a-4d73-a139-cf928a6d5cbe\") " pod="openshift-ingress/router-default-b9d995dcc-lnk2n"
Nov 01 14:17:09.749539 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:09.749514    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c27ff402-235a-4d73-a139-cf928a6d5cbe-metrics-certs\") pod \"router-default-b9d995dcc-lnk2n\" (UID: \"c27ff402-235a-4d73-a139-cf928a6d5cbe\") " pod="openshift-ingress/router-default-b9d995dcc-lnk2n"
Nov 01 14:17:09.749630 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:09.749541    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/058820ef-eef1-41ca-b331-670bab146861-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zq4pm\" (UID: \"058820ef-eef1-41ca-b331-670bab146861\") " pod="openshift-insights/insights-runtime-extractor-zq4pm"
Nov 01 14:17:09.749698 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:09.749675    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7970eec1-3765-48a4-b09c-3dec2d29956b-registry-tls\") pod \"image-registry-7f9cfbc8db-w8cmn\" (UID: \"7970eec1-3765-48a4-b09c-3dec2d29956b\") " pod="openshift-image-registry/image-registry-7f9cfbc8db-w8cmn"
Nov 01 14:17:09.749843 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:09.749829    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c8cc47b-fb01-4195-a7fc-eeefc706017e-metrics-tls\") pod \"dns-default-b2lcr\" (UID: \"0c8cc47b-fb01-4195-a7fc-eeefc706017e\") " pod="openshift-dns/dns-default-b2lcr"
Nov 01 14:17:09.749878 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:09.749838    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e7ea769b-e719-4090-bcd4-05b1b88fcc06-registry-tls\") pod \"image-registry-69c8c9797f-cblkb\" (UID: \"e7ea769b-e719-4090-bcd4-05b1b88fcc06\") " pod="openshift-image-registry/image-registry-69c8c9797f-cblkb"
Nov 01 14:17:09.777501 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:09.777474    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-b9d995dcc-lnk2n"
Nov 01 14:17:09.783607 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:09.783580    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f9cfbc8db-w8cmn"
Nov 01 14:17:09.789922 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:09.789902    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69c8c9797f-cblkb"
Nov 01 14:17:09.791018 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:09.790989    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-b2lcr"
Nov 01 14:17:09.793028 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:09.793007    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zq4pm"
Nov 01 14:17:10.279710 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:10.279655    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f9cfbc8db-w8cmn"]
Nov 01 14:17:10.286550 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:17:10.286521    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7970eec1_3765_48a4_b09c_3dec2d29956b.slice/crio-8c8289fc7cb3d8a2069b991b62d9d3de818d03311b663c2ec27be51f96d1ebb3 WatchSource:0}: Error finding container 8c8289fc7cb3d8a2069b991b62d9d3de818d03311b663c2ec27be51f96d1ebb3: Status 404 returned error can't find the container with id 8c8289fc7cb3d8a2069b991b62d9d3de818d03311b663c2ec27be51f96d1ebb3
Nov 01 14:17:10.289468 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:17:10.289448    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7ea769b_e719_4090_bcd4_05b1b88fcc06.slice/crio-1388671482ab29fe55f826b17a76268439e016a801764849d2b09b0918caac94 WatchSource:0}: Error finding container 1388671482ab29fe55f826b17a76268439e016a801764849d2b09b0918caac94: Status 404 returned error can't find the container with id 1388671482ab29fe55f826b17a76268439e016a801764849d2b09b0918caac94
Nov 01 14:17:10.290298 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:17:10.290220    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc27ff402_235a_4d73_a139_cf928a6d5cbe.slice/crio-e1a917c680e8f769c70e1096e6330d0e8bb9283244f7020a668221921072642e WatchSource:0}: Error finding container e1a917c680e8f769c70e1096e6330d0e8bb9283244f7020a668221921072642e: Status 404 returned error can't find the container with id e1a917c680e8f769c70e1096e6330d0e8bb9283244f7020a668221921072642e
Nov 01 14:17:10.326383 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:10.326300    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-b9d995dcc-lnk2n"]
Nov 01 14:17:10.328814 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:10.327446    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-69c8c9797f-cblkb"]
Nov 01 14:17:10.338673 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:17:10.338640    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod058820ef_eef1_41ca_b331_670bab146861.slice/crio-991eb39accedb3a42624c30cd525807a727ccd19f5ea935faf2f67b9b9178d30 WatchSource:0}: Error finding container 991eb39accedb3a42624c30cd525807a727ccd19f5ea935faf2f67b9b9178d30: Status 404 returned error can't find the container with id 991eb39accedb3a42624c30cd525807a727ccd19f5ea935faf2f67b9b9178d30
Nov 01 14:17:10.339461 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:17:10.339435    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c8cc47b_fb01_4195_a7fc_eeefc706017e.slice/crio-0d79c7ab03895c7ff5a2398c26478f8a9d003e605afffcb68c7f2622bc1349f7 WatchSource:0}: Error finding container 0d79c7ab03895c7ff5a2398c26478f8a9d003e605afffcb68c7f2622bc1349f7: Status 404 returned error can't find the container with id 0d79c7ab03895c7ff5a2398c26478f8a9d003e605afffcb68c7f2622bc1349f7
Nov 01 14:17:10.372662 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:10.372637    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-b2lcr"]
Nov 01 14:17:10.372951 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:10.372923    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zq4pm"]
Nov 01 14:17:10.674589 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:10.674514    4993 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-xmwtp"
Nov 01 14:17:10.674903 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:10.674889    4993 scope.go:122] "RemoveContainer" containerID="57098e775cd5e4f335940539931a0b59dc6621303b8f0714e11d150111008077"
Nov 01 14:17:10.675086 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:17:10.675068    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-xmwtp_openshift-console-operator(bb59051d-c2df-4362-a157-00f14f915e79)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-xmwtp" podUID="bb59051d-c2df-4362-a157-00f14f915e79"
Nov 01 14:17:10.944886 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:10.944788    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f9cfbc8db-w8cmn" event={"ID":"7970eec1-3765-48a4-b09c-3dec2d29956b","Type":"ContainerStarted","Data":"7e6fcac2b5fe62852f4f7b56fd67b1718e5ff2161a64d21f47f6a6d32ec271db"}
Nov 01 14:17:10.944886 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:10.944847    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7f9cfbc8db-w8cmn"
Nov 01 14:17:10.944886 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:10.944868    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f9cfbc8db-w8cmn" event={"ID":"7970eec1-3765-48a4-b09c-3dec2d29956b","Type":"ContainerStarted","Data":"8c8289fc7cb3d8a2069b991b62d9d3de818d03311b663c2ec27be51f96d1ebb3"}
Nov 01 14:17:10.945996 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:10.945972    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-b2lcr" event={"ID":"0c8cc47b-fb01-4195-a7fc-eeefc706017e","Type":"ContainerStarted","Data":"0d79c7ab03895c7ff5a2398c26478f8a9d003e605afffcb68c7f2622bc1349f7"}
Nov 01 14:17:10.947387 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:10.947356    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zq4pm" event={"ID":"058820ef-eef1-41ca-b331-670bab146861","Type":"ContainerStarted","Data":"68ad33c34bec4f99059d5eba2a9f6ea7fda0d694f8acd1db1dd476488ae82e7d"}
Nov 01 14:17:10.947480 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:10.947403    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zq4pm" event={"ID":"058820ef-eef1-41ca-b331-670bab146861","Type":"ContainerStarted","Data":"991eb39accedb3a42624c30cd525807a727ccd19f5ea935faf2f67b9b9178d30"}
Nov 01 14:17:10.948831 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:10.948804    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69c8c9797f-cblkb" event={"ID":"e7ea769b-e719-4090-bcd4-05b1b88fcc06","Type":"ContainerStarted","Data":"96d5c759e6b4246de1e00096ad2e1fbaf128c249517c335f0c0d7190d52c8826"}
Nov 01 14:17:10.948915 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:10.948847    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-69c8c9797f-cblkb"
Nov 01 14:17:10.948915 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:10.948865    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69c8c9797f-cblkb" event={"ID":"e7ea769b-e719-4090-bcd4-05b1b88fcc06","Type":"ContainerStarted","Data":"1388671482ab29fe55f826b17a76268439e016a801764849d2b09b0918caac94"}
Nov 01 14:17:10.950588 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:10.950566    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-b9d995dcc-lnk2n" event={"ID":"c27ff402-235a-4d73-a139-cf928a6d5cbe","Type":"ContainerStarted","Data":"872c6aca599ca1b963361b77f4a99525c99e6c7f207c3ee2ca1954d149a56b73"}
Nov 01 14:17:10.950690 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:10.950597    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-b9d995dcc-lnk2n" event={"ID":"c27ff402-235a-4d73-a139-cf928a6d5cbe","Type":"ContainerStarted","Data":"e1a917c680e8f769c70e1096e6330d0e8bb9283244f7020a668221921072642e"}
Nov 01 14:17:11.778401 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:11.778371    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-b9d995dcc-lnk2n"
Nov 01 14:17:11.781179 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:11.781140    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-b9d995dcc-lnk2n"
Nov 01 14:17:11.958825 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:11.958710    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zq4pm" event={"ID":"058820ef-eef1-41ca-b331-670bab146861","Type":"ContainerStarted","Data":"63220740b92d8ccf1b662042e35c24d3443632efa66a52d2c01a31a8a9475958"}
Nov 01 14:17:11.959330 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:11.959306    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-b9d995dcc-lnk2n"
Nov 01 14:17:11.960467 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:11.960442    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-b9d995dcc-lnk2n"
Nov 01 14:17:12.971071 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:12.970987    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-b2lcr" event={"ID":"0c8cc47b-fb01-4195-a7fc-eeefc706017e","Type":"ContainerStarted","Data":"6a32156aa911ce5bdca9d8e467d3106411b270d5e6b03601c6e763235ae7a730"}
Nov 01 14:17:12.971071 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:12.971058    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-b2lcr" event={"ID":"0c8cc47b-fb01-4195-a7fc-eeefc706017e","Type":"ContainerStarted","Data":"25236396ba9a734d8b5e4918ddf44d0974d4a75c727238c66abcf3cd8cabc0be"}
Nov 01 14:17:13.977244 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:13.977200    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zq4pm" event={"ID":"058820ef-eef1-41ca-b331-670bab146861","Type":"ContainerStarted","Data":"0746e3d9c04bd1957125436c3ab971f15ac7faffc7a208fafcb21e00944508aa"}
Nov 01 14:17:13.977650 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:13.977311    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-b2lcr"
Nov 01 14:17:14.865491 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:14.865457    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-nrd87"]
Nov 01 14:17:14.868848 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:14.868822    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-nrd87"
Nov 01 14:17:14.911514 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:14.911485    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7f9cfbc8db-w8cmn"]
Nov 01 14:17:14.928726 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:14.928692    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Nov 01 14:17:14.928883 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:14.928772    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-lnqj9\""
Nov 01 14:17:14.960976 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:14.960940    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-79897b5974-8764k"]
Nov 01 14:17:14.964224 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:14.964203    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79897b5974-8764k"
Nov 01 14:17:15.005108 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:15.005078    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/46a13905-2baf-4486-b3d7-ba080d5657cb-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-nrd87\" (UID: \"46a13905-2baf-4486-b3d7-ba080d5657cb\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-nrd87"
Nov 01 14:17:15.074452 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:15.074413    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-79897b5974-8764k"]
Nov 01 14:17:15.091969 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:15.091937    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-nrd87"]
Nov 01 14:17:15.105634 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:15.105599    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/46a13905-2baf-4486-b3d7-ba080d5657cb-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-nrd87\" (UID: \"46a13905-2baf-4486-b3d7-ba080d5657cb\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-nrd87"
Nov 01 14:17:15.105763 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:15.105646    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f35607c3-a795-45fd-a983-9fc0ff3dfa6f-ca-trust-extracted\") pod \"image-registry-79897b5974-8764k\" (UID: \"f35607c3-a795-45fd-a983-9fc0ff3dfa6f\") " pod="openshift-image-registry/image-registry-79897b5974-8764k"
Nov 01 14:17:15.105763 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:15.105677    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f35607c3-a795-45fd-a983-9fc0ff3dfa6f-registry-certificates\") pod \"image-registry-79897b5974-8764k\" (UID: \"f35607c3-a795-45fd-a983-9fc0ff3dfa6f\") " pod="openshift-image-registry/image-registry-79897b5974-8764k"
Nov 01 14:17:15.105763 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:15.105716    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f35607c3-a795-45fd-a983-9fc0ff3dfa6f-registry-tls\") pod \"image-registry-79897b5974-8764k\" (UID: \"f35607c3-a795-45fd-a983-9fc0ff3dfa6f\") " pod="openshift-image-registry/image-registry-79897b5974-8764k"
Nov 01 14:17:15.105931 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:15.105865    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f35607c3-a795-45fd-a983-9fc0ff3dfa6f-bound-sa-token\") pod \"image-registry-79897b5974-8764k\" (UID: \"f35607c3-a795-45fd-a983-9fc0ff3dfa6f\") " pod="openshift-image-registry/image-registry-79897b5974-8764k"
Nov 01 14:17:15.105988 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:15.105931    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f35607c3-a795-45fd-a983-9fc0ff3dfa6f-installation-pull-secrets\") pod \"image-registry-79897b5974-8764k\" (UID: \"f35607c3-a795-45fd-a983-9fc0ff3dfa6f\") " pod="openshift-image-registry/image-registry-79897b5974-8764k"
Nov 01 14:17:15.105988 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:15.105973    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dpjz\" (UniqueName: \"kubernetes.io/projected/f35607c3-a795-45fd-a983-9fc0ff3dfa6f-kube-api-access-8dpjz\") pod \"image-registry-79897b5974-8764k\" (UID: \"f35607c3-a795-45fd-a983-9fc0ff3dfa6f\") " pod="openshift-image-registry/image-registry-79897b5974-8764k"
Nov 01 14:17:15.106132 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:15.106113    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f35607c3-a795-45fd-a983-9fc0ff3dfa6f-trusted-ca\") pod \"image-registry-79897b5974-8764k\" (UID: \"f35607c3-a795-45fd-a983-9fc0ff3dfa6f\") " pod="openshift-image-registry/image-registry-79897b5974-8764k"
Nov 01 14:17:15.106213 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:15.106158    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f35607c3-a795-45fd-a983-9fc0ff3dfa6f-image-registry-private-configuration\") pod \"image-registry-79897b5974-8764k\" (UID: \"f35607c3-a795-45fd-a983-9fc0ff3dfa6f\") " pod="openshift-image-registry/image-registry-79897b5974-8764k"
Nov 01 14:17:15.108196 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:15.108159    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/46a13905-2baf-4486-b3d7-ba080d5657cb-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-nrd87\" (UID: \"46a13905-2baf-4486-b3d7-ba080d5657cb\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-nrd87"
Nov 01 14:17:15.182645 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:15.182572    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-nrd87"
Nov 01 14:17:15.206764 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:15.206733    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f35607c3-a795-45fd-a983-9fc0ff3dfa6f-trusted-ca\") pod \"image-registry-79897b5974-8764k\" (UID: \"f35607c3-a795-45fd-a983-9fc0ff3dfa6f\") " pod="openshift-image-registry/image-registry-79897b5974-8764k"
Nov 01 14:17:15.206898 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:15.206767    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f35607c3-a795-45fd-a983-9fc0ff3dfa6f-image-registry-private-configuration\") pod \"image-registry-79897b5974-8764k\" (UID: \"f35607c3-a795-45fd-a983-9fc0ff3dfa6f\") " pod="openshift-image-registry/image-registry-79897b5974-8764k"
Nov 01 14:17:15.206898 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:15.206793    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f35607c3-a795-45fd-a983-9fc0ff3dfa6f-ca-trust-extracted\") pod \"image-registry-79897b5974-8764k\" (UID: \"f35607c3-a795-45fd-a983-9fc0ff3dfa6f\") " pod="openshift-image-registry/image-registry-79897b5974-8764k"
Nov 01 14:17:15.206898 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:15.206808    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f35607c3-a795-45fd-a983-9fc0ff3dfa6f-registry-certificates\") pod \"image-registry-79897b5974-8764k\" (UID: \"f35607c3-a795-45fd-a983-9fc0ff3dfa6f\") " pod="openshift-image-registry/image-registry-79897b5974-8764k"
Nov 01 14:17:15.207107 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:15.207077    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f35607c3-a795-45fd-a983-9fc0ff3dfa6f-registry-tls\") pod \"image-registry-79897b5974-8764k\" (UID: \"f35607c3-a795-45fd-a983-9fc0ff3dfa6f\") " pod="openshift-image-registry/image-registry-79897b5974-8764k"
Nov 01 14:17:15.207342 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:15.207154    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f35607c3-a795-45fd-a983-9fc0ff3dfa6f-bound-sa-token\") pod \"image-registry-79897b5974-8764k\" (UID: \"f35607c3-a795-45fd-a983-9fc0ff3dfa6f\") " pod="openshift-image-registry/image-registry-79897b5974-8764k"
Nov 01 14:17:15.207342 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:15.207204    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f35607c3-a795-45fd-a983-9fc0ff3dfa6f-installation-pull-secrets\") pod \"image-registry-79897b5974-8764k\" (UID: \"f35607c3-a795-45fd-a983-9fc0ff3dfa6f\") " pod="openshift-image-registry/image-registry-79897b5974-8764k"
Nov 01 14:17:15.207342 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:15.207220    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f35607c3-a795-45fd-a983-9fc0ff3dfa6f-ca-trust-extracted\") pod \"image-registry-79897b5974-8764k\" (UID: \"f35607c3-a795-45fd-a983-9fc0ff3dfa6f\") " pod="openshift-image-registry/image-registry-79897b5974-8764k"
Nov 01 14:17:15.207342 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:15.207248    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dpjz\" (UniqueName: \"kubernetes.io/projected/f35607c3-a795-45fd-a983-9fc0ff3dfa6f-kube-api-access-8dpjz\") pod \"image-registry-79897b5974-8764k\" (UID: \"f35607c3-a795-45fd-a983-9fc0ff3dfa6f\") " pod="openshift-image-registry/image-registry-79897b5974-8764k"
Nov 01 14:17:15.207744 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:15.207710    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f35607c3-a795-45fd-a983-9fc0ff3dfa6f-registry-certificates\") pod \"image-registry-79897b5974-8764k\" (UID: \"f35607c3-a795-45fd-a983-9fc0ff3dfa6f\") " pod="openshift-image-registry/image-registry-79897b5974-8764k"
Nov 01 14:17:15.208020 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:15.207999    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f35607c3-a795-45fd-a983-9fc0ff3dfa6f-trusted-ca\") pod \"image-registry-79897b5974-8764k\" (UID: \"f35607c3-a795-45fd-a983-9fc0ff3dfa6f\") " pod="openshift-image-registry/image-registry-79897b5974-8764k"
Nov 01 14:17:15.209720 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:15.209696    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f35607c3-a795-45fd-a983-9fc0ff3dfa6f-image-registry-private-configuration\") pod \"image-registry-79897b5974-8764k\" (UID: \"f35607c3-a795-45fd-a983-9fc0ff3dfa6f\") " pod="openshift-image-registry/image-registry-79897b5974-8764k"
Nov 01 14:17:15.209925 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:15.209900    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f35607c3-a795-45fd-a983-9fc0ff3dfa6f-installation-pull-secrets\") pod \"image-registry-79897b5974-8764k\" (UID: \"f35607c3-a795-45fd-a983-9fc0ff3dfa6f\") " pod="openshift-image-registry/image-registry-79897b5974-8764k"
Nov 01 14:17:15.209981 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:15.209932    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f35607c3-a795-45fd-a983-9fc0ff3dfa6f-registry-tls\") pod \"image-registry-79897b5974-8764k\" (UID: \"f35607c3-a795-45fd-a983-9fc0ff3dfa6f\") " pod="openshift-image-registry/image-registry-79897b5974-8764k"
Nov 01 14:17:15.259598 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:15.259572    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f35607c3-a795-45fd-a983-9fc0ff3dfa6f-bound-sa-token\") pod \"image-registry-79897b5974-8764k\" (UID: \"f35607c3-a795-45fd-a983-9fc0ff3dfa6f\") " pod="openshift-image-registry/image-registry-79897b5974-8764k"
Nov 01 14:17:15.259735 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:15.259646    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dpjz\" (UniqueName: \"kubernetes.io/projected/f35607c3-a795-45fd-a983-9fc0ff3dfa6f-kube-api-access-8dpjz\") pod \"image-registry-79897b5974-8764k\" (UID: \"f35607c3-a795-45fd-a983-9fc0ff3dfa6f\") " pod="openshift-image-registry/image-registry-79897b5974-8764k"
Nov 01 14:17:15.275531 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:15.275508    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79897b5974-8764k"
Nov 01 14:17:15.553739 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:15.553704    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-nrd87"]
Nov 01 14:17:15.561020 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:17:15.560986    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46a13905_2baf_4486_b3d7_ba080d5657cb.slice/crio-817d3cdfe4d4ac0a9817a880e088469c6bb75d2f8ca4d308dc827a1f3461a36d WatchSource:0}: Error finding container 817d3cdfe4d4ac0a9817a880e088469c6bb75d2f8ca4d308dc827a1f3461a36d: Status 404 returned error can't find the container with id 817d3cdfe4d4ac0a9817a880e088469c6bb75d2f8ca4d308dc827a1f3461a36d
Nov 01 14:17:15.639733 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:15.639702    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-79897b5974-8764k"]
Nov 01 14:17:15.646320 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:17:15.646291    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf35607c3_a795_45fd_a983_9fc0ff3dfa6f.slice/crio-3ab30e5297b0b7919edc57ee9d7f3939a8ad2e6ae180a40141ebe5db648da771 WatchSource:0}: Error finding container 3ab30e5297b0b7919edc57ee9d7f3939a8ad2e6ae180a40141ebe5db648da771: Status 404 returned error can't find the container with id 3ab30e5297b0b7919edc57ee9d7f3939a8ad2e6ae180a40141ebe5db648da771
Nov 01 14:17:15.986989 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:15.986952    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-nrd87" event={"ID":"46a13905-2baf-4486-b3d7-ba080d5657cb","Type":"ContainerStarted","Data":"817d3cdfe4d4ac0a9817a880e088469c6bb75d2f8ca4d308dc827a1f3461a36d"}
Nov 01 14:17:15.988674 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:15.988648    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79897b5974-8764k" event={"ID":"f35607c3-a795-45fd-a983-9fc0ff3dfa6f","Type":"ContainerStarted","Data":"cb773fb3725dff2feaaf400630c9ddeef0dcbeb7f57382547a44068c930c0ca4"}
Nov 01 14:17:15.988833 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:15.988686    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79897b5974-8764k" event={"ID":"f35607c3-a795-45fd-a983-9fc0ff3dfa6f","Type":"ContainerStarted","Data":"3ab30e5297b0b7919edc57ee9d7f3939a8ad2e6ae180a40141ebe5db648da771"}
Nov 01 14:17:15.988833 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:15.988789    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-79897b5974-8764k"
Nov 01 14:17:16.097799 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:16.097630    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-79897b5974-8764k" podStartSLOduration=2.097618082 podStartE2EDuration="2.097618082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:17:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:17:16.096887653 +0000 UTC m=+23.426750402" watchObservedRunningTime="2026-11-01 14:17:16.097618082 +0000 UTC m=+23.427480813"
Nov 01 14:17:17.998520 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:17.998481    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-nrd87" event={"ID":"46a13905-2baf-4486-b3d7-ba080d5657cb","Type":"ContainerStarted","Data":"1be4b79ee01fca3363171658f32fe7acb90f389e9d8ba030403a3390b295da71"}
Nov 01 14:17:17.999006 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:17.998674    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-nrd87"
Nov 01 14:17:18.003464 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:18.003441    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-nrd87"
Nov 01 14:17:18.104456 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:18.104407    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-nrd87" podStartSLOduration=2.5350153349999998 podStartE2EDuration="4.10439343s" totalImagesPullingTime="1.569378095s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:17:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:17:18.103405829 +0000 UTC m=+25.433268566" watchObservedRunningTime="2026-11-01 14:17:18.10439343 +0000 UTC m=+25.434256167"
Nov 01 14:17:18.984432 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:18.984402    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qzf26"
Nov 01 14:17:23.983321 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:23.983284    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-b2lcr"
Nov 01 14:17:24.176267 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:24.176232    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-8mqxj"
Nov 01 14:17:25.594959 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:25.594927    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-bb9q4"]
Nov 01 14:17:25.627480 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:25.627455    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bb9q4"
Nov 01 14:17:25.694220 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:25.694161    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/503865ae-b17e-4535-b61e-29df5033497f-sys\") pod \"node-exporter-bb9q4\" (UID: \"503865ae-b17e-4535-b61e-29df5033497f\") " pod="openshift-monitoring/node-exporter-bb9q4"
Nov 01 14:17:25.694220 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:25.694221    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/503865ae-b17e-4535-b61e-29df5033497f-node-exporter-tls\") pod \"node-exporter-bb9q4\" (UID: \"503865ae-b17e-4535-b61e-29df5033497f\") " pod="openshift-monitoring/node-exporter-bb9q4"
Nov 01 14:17:25.694429 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:25.694281    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/503865ae-b17e-4535-b61e-29df5033497f-root\") pod \"node-exporter-bb9q4\" (UID: \"503865ae-b17e-4535-b61e-29df5033497f\") " pod="openshift-monitoring/node-exporter-bb9q4"
Nov 01 14:17:25.694429 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:25.694332    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/503865ae-b17e-4535-b61e-29df5033497f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bb9q4\" (UID: \"503865ae-b17e-4535-b61e-29df5033497f\") " pod="openshift-monitoring/node-exporter-bb9q4"
Nov 01 14:17:25.694429 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:25.694366    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/503865ae-b17e-4535-b61e-29df5033497f-node-exporter-textfile\") pod \"node-exporter-bb9q4\" (UID: \"503865ae-b17e-4535-b61e-29df5033497f\") " pod="openshift-monitoring/node-exporter-bb9q4"
Nov 01 14:17:25.694429 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:25.694383    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/503865ae-b17e-4535-b61e-29df5033497f-metrics-client-ca\") pod \"node-exporter-bb9q4\" (UID: \"503865ae-b17e-4535-b61e-29df5033497f\") " pod="openshift-monitoring/node-exporter-bb9q4"
Nov 01 14:17:25.694429 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:25.694410    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhzlt\" (UniqueName: \"kubernetes.io/projected/503865ae-b17e-4535-b61e-29df5033497f-kube-api-access-mhzlt\") pod \"node-exporter-bb9q4\" (UID: \"503865ae-b17e-4535-b61e-29df5033497f\") " pod="openshift-monitoring/node-exporter-bb9q4"
Nov 01 14:17:25.694577 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:25.694523    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/503865ae-b17e-4535-b61e-29df5033497f-node-exporter-accelerators-collector-config\") pod \"node-exporter-bb9q4\" (UID: \"503865ae-b17e-4535-b61e-29df5033497f\") " pod="openshift-monitoring/node-exporter-bb9q4"
Nov 01 14:17:25.694577 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:25.694553    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/503865ae-b17e-4535-b61e-29df5033497f-node-exporter-wtmp\") pod \"node-exporter-bb9q4\" (UID: \"503865ae-b17e-4535-b61e-29df5033497f\") " pod="openshift-monitoring/node-exporter-bb9q4"
Nov 01 14:17:25.733849 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:25.733813    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 01 14:17:25.734029 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:25.734015    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 01 14:17:25.734092 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:25.734074    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5wqpw\""
Nov 01 14:17:25.780101 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:25.780071    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 01 14:17:25.780295 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:25.780274    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 01 14:17:25.795649 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:25.795623    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/503865ae-b17e-4535-b61e-29df5033497f-node-exporter-accelerators-collector-config\") pod \"node-exporter-bb9q4\" (UID: \"503865ae-b17e-4535-b61e-29df5033497f\") " pod="openshift-monitoring/node-exporter-bb9q4"
Nov 01 14:17:25.795756 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:25.795652    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/503865ae-b17e-4535-b61e-29df5033497f-node-exporter-wtmp\") pod \"node-exporter-bb9q4\" (UID: \"503865ae-b17e-4535-b61e-29df5033497f\") " pod="openshift-monitoring/node-exporter-bb9q4"
Nov 01 14:17:25.795756 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:25.795678    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/503865ae-b17e-4535-b61e-29df5033497f-sys\") pod \"node-exporter-bb9q4\" (UID: \"503865ae-b17e-4535-b61e-29df5033497f\") " pod="openshift-monitoring/node-exporter-bb9q4"
Nov 01 14:17:25.795756 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:25.795730    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/503865ae-b17e-4535-b61e-29df5033497f-sys\") pod \"node-exporter-bb9q4\" (UID: \"503865ae-b17e-4535-b61e-29df5033497f\") " pod="openshift-monitoring/node-exporter-bb9q4"
Nov 01 14:17:25.795854 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:25.795799    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/503865ae-b17e-4535-b61e-29df5033497f-node-exporter-tls\") pod \"node-exporter-bb9q4\" (UID: \"503865ae-b17e-4535-b61e-29df5033497f\") " pod="openshift-monitoring/node-exporter-bb9q4"
Nov 01 14:17:25.795854 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:25.795812    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/503865ae-b17e-4535-b61e-29df5033497f-node-exporter-wtmp\") pod \"node-exporter-bb9q4\" (UID: \"503865ae-b17e-4535-b61e-29df5033497f\") " pod="openshift-monitoring/node-exporter-bb9q4"
Nov 01 14:17:25.795854 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:25.795824    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/503865ae-b17e-4535-b61e-29df5033497f-root\") pod \"node-exporter-bb9q4\" (UID: \"503865ae-b17e-4535-b61e-29df5033497f\") " pod="openshift-monitoring/node-exporter-bb9q4"
Nov 01 14:17:25.795854 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:25.795851    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/503865ae-b17e-4535-b61e-29df5033497f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bb9q4\" (UID: \"503865ae-b17e-4535-b61e-29df5033497f\") " pod="openshift-monitoring/node-exporter-bb9q4"
Nov 01 14:17:25.796015 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:25.795878    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/503865ae-b17e-4535-b61e-29df5033497f-node-exporter-textfile\") pod \"node-exporter-bb9q4\" (UID: \"503865ae-b17e-4535-b61e-29df5033497f\") " pod="openshift-monitoring/node-exporter-bb9q4"
Nov 01 14:17:25.796015 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:25.795895    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/503865ae-b17e-4535-b61e-29df5033497f-metrics-client-ca\") pod \"node-exporter-bb9q4\" (UID: \"503865ae-b17e-4535-b61e-29df5033497f\") " pod="openshift-monitoring/node-exporter-bb9q4"
Nov 01 14:17:25.796015 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:25.795975    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/503865ae-b17e-4535-b61e-29df5033497f-root\") pod \"node-exporter-bb9q4\" (UID: \"503865ae-b17e-4535-b61e-29df5033497f\") " pod="openshift-monitoring/node-exporter-bb9q4"
Nov 01 14:17:25.796196 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:25.796030    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhzlt\" (UniqueName: \"kubernetes.io/projected/503865ae-b17e-4535-b61e-29df5033497f-kube-api-access-mhzlt\") pod \"node-exporter-bb9q4\" (UID: \"503865ae-b17e-4535-b61e-29df5033497f\") " pod="openshift-monitoring/node-exporter-bb9q4"
Nov 01 14:17:25.796269 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:25.796250    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/503865ae-b17e-4535-b61e-29df5033497f-node-exporter-accelerators-collector-config\") pod \"node-exporter-bb9q4\" (UID: \"503865ae-b17e-4535-b61e-29df5033497f\") " pod="openshift-monitoring/node-exporter-bb9q4"
Nov 01 14:17:25.796334 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:25.796263    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/503865ae-b17e-4535-b61e-29df5033497f-node-exporter-textfile\") pod \"node-exporter-bb9q4\" (UID: \"503865ae-b17e-4535-b61e-29df5033497f\") " pod="openshift-monitoring/node-exporter-bb9q4"
Nov 01 14:17:25.796466 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:25.796448    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/503865ae-b17e-4535-b61e-29df5033497f-metrics-client-ca\") pod \"node-exporter-bb9q4\" (UID: \"503865ae-b17e-4535-b61e-29df5033497f\") " pod="openshift-monitoring/node-exporter-bb9q4"
Nov 01 14:17:25.798238 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:25.798217    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/503865ae-b17e-4535-b61e-29df5033497f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bb9q4\" (UID: \"503865ae-b17e-4535-b61e-29df5033497f\") " pod="openshift-monitoring/node-exporter-bb9q4"
Nov 01 14:17:25.798393 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:25.798377    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/503865ae-b17e-4535-b61e-29df5033497f-node-exporter-tls\") pod \"node-exporter-bb9q4\" (UID: \"503865ae-b17e-4535-b61e-29df5033497f\") " pod="openshift-monitoring/node-exporter-bb9q4"
Nov 01 14:17:25.849441 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:25.849384    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhzlt\" (UniqueName: \"kubernetes.io/projected/503865ae-b17e-4535-b61e-29df5033497f-kube-api-access-mhzlt\") pod \"node-exporter-bb9q4\" (UID: \"503865ae-b17e-4535-b61e-29df5033497f\") " pod="openshift-monitoring/node-exporter-bb9q4"
Nov 01 14:17:25.982841 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:25.982798    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bb9q4"
Nov 01 14:17:26.029819 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:17:26.029786    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod503865ae_b17e_4535_b61e_29df5033497f.slice/crio-90cdeb92486b8a1e9a0c5c145d40359acf1776e49d621382bd247bad4a7a3e0d WatchSource:0}: Error finding container 90cdeb92486b8a1e9a0c5c145d40359acf1776e49d621382bd247bad4a7a3e0d: Status 404 returned error can't find the container with id 90cdeb92486b8a1e9a0c5c145d40359acf1776e49d621382bd247bad4a7a3e0d
Nov 01 14:17:27.035616 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:27.035575    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bb9q4" event={"ID":"503865ae-b17e-4535-b61e-29df5033497f","Type":"ContainerStarted","Data":"90cdeb92486b8a1e9a0c5c145d40359acf1776e49d621382bd247bad4a7a3e0d"}
Nov 01 14:17:27.500095 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:17:27.500052    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:17:29.045726 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:29.045686    4993 generic.go:356] "Generic (PLEG): container finished" podID="503865ae-b17e-4535-b61e-29df5033497f" containerID="fdd9c3b16d9e44b4e209dcec2724643ad1166ddb11a9c67b25b8f76749d9e4a3" exitCode=0
Nov 01 14:17:29.046131 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:29.045780    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bb9q4" event={"ID":"503865ae-b17e-4535-b61e-29df5033497f","Type":"ContainerDied","Data":"fdd9c3b16d9e44b4e209dcec2724643ad1166ddb11a9c67b25b8f76749d9e4a3"}
Nov 01 14:17:29.793901 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:29.793860    4993 patch_prober.go:29] interesting pod/image-registry-69c8c9797f-cblkb container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Nov 01 14:17:29.794040 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:29.793918    4993 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-69c8c9797f-cblkb" podUID="e7ea769b-e719-4090-bcd4-05b1b88fcc06" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 14:17:30.052073 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:30.051986    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bb9q4" event={"ID":"503865ae-b17e-4535-b61e-29df5033497f","Type":"ContainerStarted","Data":"58a816f8001cdb9a485feaf536ba6b093fab7a54a713ea4ab02bed49da9f06e2"}
Nov 01 14:17:30.052073 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:30.052036    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bb9q4" event={"ID":"503865ae-b17e-4535-b61e-29df5033497f","Type":"ContainerStarted","Data":"eecc3c8c3a7f6b2f9c00edfcdd63b9aef13bbe365ec0752627ec86cd9dfd468d"}
Nov 01 14:17:30.159935 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:30.159881    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-bb9q4" podStartSLOduration=3.041894079 podStartE2EDuration="5.159867099s" totalImagesPullingTime="2.043050957s" totalInitContainerRuntime="74.922063ms" isStatefulPod=true podCreationTimestamp="2026-11-01 14:17:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:17:30.159616623 +0000 UTC m=+37.489479359" watchObservedRunningTime="2026-11-01 14:17:30.159867099 +0000 UTC m=+37.489729835"
Nov 01 14:17:31.788089 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:31.788056    4993 scope.go:122] "RemoveContainer" containerID="57098e775cd5e4f335940539931a0b59dc6621303b8f0714e11d150111008077"
Nov 01 14:17:31.963256 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:31.963226    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-69c8c9797f-cblkb"
Nov 01 14:17:32.061597 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:32.061530    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xmwtp_bb59051d-c2df-4362-a157-00f14f915e79/console-operator/3.log"
Nov 01 14:17:32.061740 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:32.061613    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-xmwtp" event={"ID":"bb59051d-c2df-4362-a157-00f14f915e79","Type":"ContainerStarted","Data":"90cd755f8b49f0bc4bfeb517b6744fb7e9f11c32523c5cbec48d1985459491ad"}
Nov 01 14:17:32.061897 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:32.061877    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-xmwtp"
Nov 01 14:17:32.063044 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:32.063015    4993 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-xmwtp container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.19:8443/readyz\": dial tcp 10.132.0.19:8443: connect: connection refused" start-of-body=
Nov 01 14:17:32.063153 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:32.063067    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-xmwtp" podUID="bb59051d-c2df-4362-a157-00f14f915e79" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.19:8443/readyz\": dial tcp 10.132.0.19:8443: connect: connection refused"
Nov 01 14:17:33.069149 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:33.069077    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-xmwtp"
Nov 01 14:17:34.917255 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:34.917221    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7f9cfbc8db-w8cmn"
Nov 01 14:17:36.997735 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:36.997705    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-79897b5974-8764k"
Nov 01 14:17:37.200474 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:37.200430    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-69c8c9797f-cblkb"]
Nov 01 14:17:39.931311 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:39.931268    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7f9cfbc8db-w8cmn" podUID="7970eec1-3765-48a4-b09c-3dec2d29956b" containerName="registry" containerID="cri-o://7e6fcac2b5fe62852f4f7b56fd67b1718e5ff2161a64d21f47f6a6d32ec271db" gracePeriod=30
Nov 01 14:17:40.095217 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:40.095160    4993 generic.go:356] "Generic (PLEG): container finished" podID="7970eec1-3765-48a4-b09c-3dec2d29956b" containerID="7e6fcac2b5fe62852f4f7b56fd67b1718e5ff2161a64d21f47f6a6d32ec271db" exitCode=0
Nov 01 14:17:40.095217 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:40.095210    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f9cfbc8db-w8cmn" event={"ID":"7970eec1-3765-48a4-b09c-3dec2d29956b","Type":"ContainerDied","Data":"7e6fcac2b5fe62852f4f7b56fd67b1718e5ff2161a64d21f47f6a6d32ec271db"}
Nov 01 14:17:40.179367 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:40.179344    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f9cfbc8db-w8cmn"
Nov 01 14:17:40.333884 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:40.333850    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7970eec1-3765-48a4-b09c-3dec2d29956b-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7970eec1-3765-48a4-b09c-3dec2d29956b-bound-sa-token\") pod \"7970eec1-3765-48a4-b09c-3dec2d29956b\" (UID: \"7970eec1-3765-48a4-b09c-3dec2d29956b\") "
Nov 01 14:17:40.334070 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:40.333906    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7970eec1-3765-48a4-b09c-3dec2d29956b-registry-tls\" (UniqueName: \"kubernetes.io/projected/7970eec1-3765-48a4-b09c-3dec2d29956b-registry-tls\") pod \"7970eec1-3765-48a4-b09c-3dec2d29956b\" (UID: \"7970eec1-3765-48a4-b09c-3dec2d29956b\") "
Nov 01 14:17:40.334070 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:40.333959    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7970eec1-3765-48a4-b09c-3dec2d29956b-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7970eec1-3765-48a4-b09c-3dec2d29956b-registry-certificates\") pod \"7970eec1-3765-48a4-b09c-3dec2d29956b\" (UID: \"7970eec1-3765-48a4-b09c-3dec2d29956b\") "
Nov 01 14:17:40.334070 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:40.333991    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7970eec1-3765-48a4-b09c-3dec2d29956b-kube-api-access-8gnmx\" (UniqueName: \"kubernetes.io/projected/7970eec1-3765-48a4-b09c-3dec2d29956b-kube-api-access-8gnmx\") pod \"7970eec1-3765-48a4-b09c-3dec2d29956b\" (UID: \"7970eec1-3765-48a4-b09c-3dec2d29956b\") "
Nov 01 14:17:40.334070 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:40.334034    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7970eec1-3765-48a4-b09c-3dec2d29956b-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7970eec1-3765-48a4-b09c-3dec2d29956b-installation-pull-secrets\") pod \"7970eec1-3765-48a4-b09c-3dec2d29956b\" (UID: \"7970eec1-3765-48a4-b09c-3dec2d29956b\") "
Nov 01 14:17:40.334070 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:40.334068    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7970eec1-3765-48a4-b09c-3dec2d29956b-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7970eec1-3765-48a4-b09c-3dec2d29956b-ca-trust-extracted\") pod \"7970eec1-3765-48a4-b09c-3dec2d29956b\" (UID: \"7970eec1-3765-48a4-b09c-3dec2d29956b\") "
Nov 01 14:17:40.334377 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:40.334099    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7970eec1-3765-48a4-b09c-3dec2d29956b-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7970eec1-3765-48a4-b09c-3dec2d29956b-trusted-ca\") pod \"7970eec1-3765-48a4-b09c-3dec2d29956b\" (UID: \"7970eec1-3765-48a4-b09c-3dec2d29956b\") "
Nov 01 14:17:40.334377 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:40.334155    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7970eec1-3765-48a4-b09c-3dec2d29956b-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7970eec1-3765-48a4-b09c-3dec2d29956b-image-registry-private-configuration\") pod \"7970eec1-3765-48a4-b09c-3dec2d29956b\" (UID: \"7970eec1-3765-48a4-b09c-3dec2d29956b\") "
Nov 01 14:17:40.334861 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:40.334790    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7970eec1-3765-48a4-b09c-3dec2d29956b-registry-certificates" pod "7970eec1-3765-48a4-b09c-3dec2d29956b" (UID: "7970eec1-3765-48a4-b09c-3dec2d29956b"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:17:40.334861 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:40.334803    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7970eec1-3765-48a4-b09c-3dec2d29956b-trusted-ca" pod "7970eec1-3765-48a4-b09c-3dec2d29956b" (UID: "7970eec1-3765-48a4-b09c-3dec2d29956b"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:17:40.336940 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:40.336911    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7970eec1-3765-48a4-b09c-3dec2d29956b-registry-tls" pod "7970eec1-3765-48a4-b09c-3dec2d29956b" (UID: "7970eec1-3765-48a4-b09c-3dec2d29956b"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:17:40.338258 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:40.338221    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7970eec1-3765-48a4-b09c-3dec2d29956b-image-registry-private-configuration" pod "7970eec1-3765-48a4-b09c-3dec2d29956b" (UID: "7970eec1-3765-48a4-b09c-3dec2d29956b"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:17:40.338367 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:40.338340    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7970eec1-3765-48a4-b09c-3dec2d29956b-bound-sa-token" pod "7970eec1-3765-48a4-b09c-3dec2d29956b" (UID: "7970eec1-3765-48a4-b09c-3dec2d29956b"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:17:40.338478 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:40.338449    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7970eec1-3765-48a4-b09c-3dec2d29956b-kube-api-access-8gnmx" pod "7970eec1-3765-48a4-b09c-3dec2d29956b" (UID: "7970eec1-3765-48a4-b09c-3dec2d29956b"). InnerVolumeSpecName "kube-api-access-8gnmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:17:40.338599 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:40.338578    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7970eec1-3765-48a4-b09c-3dec2d29956b-installation-pull-secrets" pod "7970eec1-3765-48a4-b09c-3dec2d29956b" (UID: "7970eec1-3765-48a4-b09c-3dec2d29956b"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:17:40.342108 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:40.342078    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7970eec1-3765-48a4-b09c-3dec2d29956b-ca-trust-extracted" pod "7970eec1-3765-48a4-b09c-3dec2d29956b" (UID: "7970eec1-3765-48a4-b09c-3dec2d29956b"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:17:40.435232 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:40.435193    4993 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7970eec1-3765-48a4-b09c-3dec2d29956b-bound-sa-token\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:17:40.435232 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:40.435224    4993 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7970eec1-3765-48a4-b09c-3dec2d29956b-registry-tls\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:17:40.435232 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:40.435235    4993 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7970eec1-3765-48a4-b09c-3dec2d29956b-registry-certificates\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:17:40.435527 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:40.435246    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gnmx\" (UniqueName: \"kubernetes.io/projected/7970eec1-3765-48a4-b09c-3dec2d29956b-kube-api-access-8gnmx\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:17:40.435527 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:40.435255    4993 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7970eec1-3765-48a4-b09c-3dec2d29956b-installation-pull-secrets\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:17:40.435527 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:40.435266    4993 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7970eec1-3765-48a4-b09c-3dec2d29956b-ca-trust-extracted\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:17:40.435527 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:40.435274    4993 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7970eec1-3765-48a4-b09c-3dec2d29956b-trusted-ca\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:17:40.435527 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:40.435284    4993 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7970eec1-3765-48a4-b09c-3dec2d29956b-image-registry-private-configuration\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:17:41.100135 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:41.100056    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f9cfbc8db-w8cmn" event={"ID":"7970eec1-3765-48a4-b09c-3dec2d29956b","Type":"ContainerDied","Data":"8c8289fc7cb3d8a2069b991b62d9d3de818d03311b663c2ec27be51f96d1ebb3"}
Nov 01 14:17:41.100135 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:41.100117    4993 scope.go:122] "RemoveContainer" containerID="7e6fcac2b5fe62852f4f7b56fd67b1718e5ff2161a64d21f47f6a6d32ec271db"
Nov 01 14:17:41.100135 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:41.100116    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f9cfbc8db-w8cmn"
Nov 01 14:17:41.298872 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:41.298842    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7f9cfbc8db-w8cmn"]
Nov 01 14:17:41.345129 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:41.345093    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7f9cfbc8db-w8cmn"]
Nov 01 14:17:42.792668 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:17:42.792631    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7970eec1-3765-48a4-b09c-3dec2d29956b" path="/var/lib/kubelet/pods/7970eec1-3765-48a4-b09c-3dec2d29956b/volumes"
Nov 01 14:17:57.507458 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:17:57.507388    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:18:02.221910 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:18:02.221851    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-69c8c9797f-cblkb" podUID="e7ea769b-e719-4090-bcd4-05b1b88fcc06" containerName="registry" containerID="cri-o://96d5c759e6b4246de1e00096ad2e1fbaf128c249517c335f0c0d7190d52c8826" gracePeriod=30
Nov 01 14:18:02.481625 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:18:02.481570    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69c8c9797f-cblkb"
Nov 01 14:18:02.522303 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:18:02.522270    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7ea769b-e719-4090-bcd4-05b1b88fcc06-registry-tls\" (UniqueName: \"kubernetes.io/projected/e7ea769b-e719-4090-bcd4-05b1b88fcc06-registry-tls\") pod \"e7ea769b-e719-4090-bcd4-05b1b88fcc06\" (UID: \"e7ea769b-e719-4090-bcd4-05b1b88fcc06\") "
Nov 01 14:18:02.522431 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:18:02.522316    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7ea769b-e719-4090-bcd4-05b1b88fcc06-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e7ea769b-e719-4090-bcd4-05b1b88fcc06-image-registry-private-configuration\") pod \"e7ea769b-e719-4090-bcd4-05b1b88fcc06\" (UID: \"e7ea769b-e719-4090-bcd4-05b1b88fcc06\") "
Nov 01 14:18:02.522431 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:18:02.522345    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7ea769b-e719-4090-bcd4-05b1b88fcc06-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e7ea769b-e719-4090-bcd4-05b1b88fcc06-bound-sa-token\") pod \"e7ea769b-e719-4090-bcd4-05b1b88fcc06\" (UID: \"e7ea769b-e719-4090-bcd4-05b1b88fcc06\") "
Nov 01 14:18:02.522431 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:18:02.522384    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7ea769b-e719-4090-bcd4-05b1b88fcc06-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e7ea769b-e719-4090-bcd4-05b1b88fcc06-installation-pull-secrets\") pod \"e7ea769b-e719-4090-bcd4-05b1b88fcc06\" (UID: \"e7ea769b-e719-4090-bcd4-05b1b88fcc06\") "
Nov 01 14:18:02.522431 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:18:02.522416    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7ea769b-e719-4090-bcd4-05b1b88fcc06-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e7ea769b-e719-4090-bcd4-05b1b88fcc06-ca-trust-extracted\") pod \"e7ea769b-e719-4090-bcd4-05b1b88fcc06\" (UID: \"e7ea769b-e719-4090-bcd4-05b1b88fcc06\") "
Nov 01 14:18:02.522642 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:18:02.522468    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7ea769b-e719-4090-bcd4-05b1b88fcc06-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e7ea769b-e719-4090-bcd4-05b1b88fcc06-trusted-ca\") pod \"e7ea769b-e719-4090-bcd4-05b1b88fcc06\" (UID: \"e7ea769b-e719-4090-bcd4-05b1b88fcc06\") "
Nov 01 14:18:02.523650 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:18:02.523109    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7ea769b-e719-4090-bcd4-05b1b88fcc06-trusted-ca" pod "e7ea769b-e719-4090-bcd4-05b1b88fcc06" (UID: "e7ea769b-e719-4090-bcd4-05b1b88fcc06"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:18:02.524784 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:18:02.524755    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7ea769b-e719-4090-bcd4-05b1b88fcc06-image-registry-private-configuration" pod "e7ea769b-e719-4090-bcd4-05b1b88fcc06" (UID: "e7ea769b-e719-4090-bcd4-05b1b88fcc06"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:18:02.525047 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:18:02.525016    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7ea769b-e719-4090-bcd4-05b1b88fcc06-installation-pull-secrets" pod "e7ea769b-e719-4090-bcd4-05b1b88fcc06" (UID: "e7ea769b-e719-4090-bcd4-05b1b88fcc06"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:18:02.525156 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:18:02.525064    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7ea769b-e719-4090-bcd4-05b1b88fcc06-bound-sa-token" pod "e7ea769b-e719-4090-bcd4-05b1b88fcc06" (UID: "e7ea769b-e719-4090-bcd4-05b1b88fcc06"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:18:02.525156 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:18:02.525117    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7ea769b-e719-4090-bcd4-05b1b88fcc06-registry-tls" pod "e7ea769b-e719-4090-bcd4-05b1b88fcc06" (UID: "e7ea769b-e719-4090-bcd4-05b1b88fcc06"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:18:02.529525 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:18:02.529437    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7ea769b-e719-4090-bcd4-05b1b88fcc06-ca-trust-extracted" pod "e7ea769b-e719-4090-bcd4-05b1b88fcc06" (UID: "e7ea769b-e719-4090-bcd4-05b1b88fcc06"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:18:02.623002 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:18:02.622962    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7ea769b-e719-4090-bcd4-05b1b88fcc06-kube-api-access-rdxs8\" (UniqueName: \"kubernetes.io/projected/e7ea769b-e719-4090-bcd4-05b1b88fcc06-kube-api-access-rdxs8\") pod \"e7ea769b-e719-4090-bcd4-05b1b88fcc06\" (UID: \"e7ea769b-e719-4090-bcd4-05b1b88fcc06\") "
Nov 01 14:18:02.623002 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:18:02.623009    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7ea769b-e719-4090-bcd4-05b1b88fcc06-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e7ea769b-e719-4090-bcd4-05b1b88fcc06-registry-certificates\") pod \"e7ea769b-e719-4090-bcd4-05b1b88fcc06\" (UID: \"e7ea769b-e719-4090-bcd4-05b1b88fcc06\") "
Nov 01 14:18:02.623241 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:18:02.623131    4993 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e7ea769b-e719-4090-bcd4-05b1b88fcc06-registry-tls\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:18:02.623241 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:18:02.623145    4993 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e7ea769b-e719-4090-bcd4-05b1b88fcc06-image-registry-private-configuration\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:18:02.623241 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:18:02.623155    4993 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e7ea769b-e719-4090-bcd4-05b1b88fcc06-bound-sa-token\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:18:02.623241 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:18:02.623184    4993 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e7ea769b-e719-4090-bcd4-05b1b88fcc06-installation-pull-secrets\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:18:02.623241 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:18:02.623198    4993 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e7ea769b-e719-4090-bcd4-05b1b88fcc06-ca-trust-extracted\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:18:02.623241 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:18:02.623210    4993 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e7ea769b-e719-4090-bcd4-05b1b88fcc06-trusted-ca\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:18:02.623457 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:18:02.623429    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7ea769b-e719-4090-bcd4-05b1b88fcc06-registry-certificates" pod "e7ea769b-e719-4090-bcd4-05b1b88fcc06" (UID: "e7ea769b-e719-4090-bcd4-05b1b88fcc06"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:18:02.625224 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:18:02.625206    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7ea769b-e719-4090-bcd4-05b1b88fcc06-kube-api-access-rdxs8" pod "e7ea769b-e719-4090-bcd4-05b1b88fcc06" (UID: "e7ea769b-e719-4090-bcd4-05b1b88fcc06"). InnerVolumeSpecName "kube-api-access-rdxs8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:18:02.723934 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:18:02.723896    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdxs8\" (UniqueName: \"kubernetes.io/projected/e7ea769b-e719-4090-bcd4-05b1b88fcc06-kube-api-access-rdxs8\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:18:02.723934 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:18:02.723928    4993 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e7ea769b-e719-4090-bcd4-05b1b88fcc06-registry-certificates\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:18:03.189974 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:18:03.189896    4993 generic.go:356] "Generic (PLEG): container finished" podID="e7ea769b-e719-4090-bcd4-05b1b88fcc06" containerID="96d5c759e6b4246de1e00096ad2e1fbaf128c249517c335f0c0d7190d52c8826" exitCode=0
Nov 01 14:18:03.189974 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:18:03.189962    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69c8c9797f-cblkb"
Nov 01 14:18:03.189974 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:18:03.189962    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69c8c9797f-cblkb" event={"ID":"e7ea769b-e719-4090-bcd4-05b1b88fcc06","Type":"ContainerDied","Data":"96d5c759e6b4246de1e00096ad2e1fbaf128c249517c335f0c0d7190d52c8826"}
Nov 01 14:18:03.190408 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:18:03.190014    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69c8c9797f-cblkb" event={"ID":"e7ea769b-e719-4090-bcd4-05b1b88fcc06","Type":"ContainerDied","Data":"1388671482ab29fe55f826b17a76268439e016a801764849d2b09b0918caac94"}
Nov 01 14:18:03.190408 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:18:03.190041    4993 scope.go:122] "RemoveContainer" containerID="96d5c759e6b4246de1e00096ad2e1fbaf128c249517c335f0c0d7190d52c8826"
Nov 01 14:18:03.198163 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:18:03.198140    4993 scope.go:122] "RemoveContainer" containerID="96d5c759e6b4246de1e00096ad2e1fbaf128c249517c335f0c0d7190d52c8826"
Nov 01 14:18:03.198426 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:18:03.198405    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96d5c759e6b4246de1e00096ad2e1fbaf128c249517c335f0c0d7190d52c8826\": container with ID starting with 96d5c759e6b4246de1e00096ad2e1fbaf128c249517c335f0c0d7190d52c8826 not found: ID does not exist" containerID="96d5c759e6b4246de1e00096ad2e1fbaf128c249517c335f0c0d7190d52c8826"
Nov 01 14:18:03.198507 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:18:03.198438    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96d5c759e6b4246de1e00096ad2e1fbaf128c249517c335f0c0d7190d52c8826"} err="failed to get container status \"96d5c759e6b4246de1e00096ad2e1fbaf128c249517c335f0c0d7190d52c8826\": rpc error: code = NotFound desc = could not find container \"96d5c759e6b4246de1e00096ad2e1fbaf128c249517c335f0c0d7190d52c8826\": container with ID starting with 96d5c759e6b4246de1e00096ad2e1fbaf128c249517c335f0c0d7190d52c8826 not found: ID does not exist"
Nov 01 14:18:03.387609 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:18:03.387576    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-69c8c9797f-cblkb"]
Nov 01 14:18:03.433893 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:18:03.433864    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-69c8c9797f-cblkb"]
Nov 01 14:18:04.196230 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:18:04.196200    4993 generic.go:356] "Generic (PLEG): container finished" podID="7a26b817-3283-4d59-8e8f-3bb4c0fa4987" containerID="56c3f8610d5a54df92228250cdf61c4e3bf3c664f5ff6f24ed0c5fc5c65aecd8" exitCode=0
Nov 01 14:18:04.196410 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:18:04.196280    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-7d54k" event={"ID":"7a26b817-3283-4d59-8e8f-3bb4c0fa4987","Type":"ContainerDied","Data":"56c3f8610d5a54df92228250cdf61c4e3bf3c664f5ff6f24ed0c5fc5c65aecd8"}
Nov 01 14:18:04.196658 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:18:04.196639    4993 scope.go:122] "RemoveContainer" containerID="56c3f8610d5a54df92228250cdf61c4e3bf3c664f5ff6f24ed0c5fc5c65aecd8"
Nov 01 14:18:04.793579 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:18:04.793548    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7ea769b-e719-4090-bcd4-05b1b88fcc06" path="/var/lib/kubelet/pods/e7ea769b-e719-4090-bcd4-05b1b88fcc06/volumes"
Nov 01 14:18:05.103707 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:18:05.103628    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-rlgcn-h5sk4"]
Nov 01 14:18:05.103872 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:18:05.103827    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-rlgcn-h5sk4" podUID="384cd50b-370f-474f-b80e-89a544a9bbbd" containerName="insights-gathering" containerID="cri-o://ede52eea27808ab3afc48dbf2065c4c6f1cc4b6c8c2f925d899e121398b002bb" gracePeriod=30
Nov 01 14:18:05.208108 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:18:05.208068    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-7d54k" event={"ID":"7a26b817-3283-4d59-8e8f-3bb4c0fa4987","Type":"ContainerStarted","Data":"0d74eecc8ffa32ebe821cc4ac4d75b4d12bc0fc4f087f7bd45d24c3348f6b3fa"}
Nov 01 14:18:05.355969 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:18:05.355916    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-rlgcn-h5sk4"
Nov 01 14:18:05.441063 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:18:05.441033    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/384cd50b-370f-474f-b80e-89a544a9bbbd-kube-api-access-chwvj\" (UniqueName: \"kubernetes.io/projected/384cd50b-370f-474f-b80e-89a544a9bbbd-kube-api-access-chwvj\") pod \"384cd50b-370f-474f-b80e-89a544a9bbbd\" (UID: \"384cd50b-370f-474f-b80e-89a544a9bbbd\") "
Nov 01 14:18:05.441247 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:18:05.441075    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/384cd50b-370f-474f-b80e-89a544a9bbbd-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/384cd50b-370f-474f-b80e-89a544a9bbbd-trusted-ca-bundle\") pod \"384cd50b-370f-474f-b80e-89a544a9bbbd\" (UID: \"384cd50b-370f-474f-b80e-89a544a9bbbd\") "
Nov 01 14:18:05.441247 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:18:05.441098    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/384cd50b-370f-474f-b80e-89a544a9bbbd-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/384cd50b-370f-474f-b80e-89a544a9bbbd-archives-path\") pod \"384cd50b-370f-474f-b80e-89a544a9bbbd\" (UID: \"384cd50b-370f-474f-b80e-89a544a9bbbd\") "
Nov 01 14:18:05.441247 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:18:05.441119    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/384cd50b-370f-474f-b80e-89a544a9bbbd-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/384cd50b-370f-474f-b80e-89a544a9bbbd-service-ca-bundle\") pod \"384cd50b-370f-474f-b80e-89a544a9bbbd\" (UID: \"384cd50b-370f-474f-b80e-89a544a9bbbd\") "
Nov 01 14:18:05.441484 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:18:05.441457    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/384cd50b-370f-474f-b80e-89a544a9bbbd-archives-path" pod "384cd50b-370f-474f-b80e-89a544a9bbbd" (UID: "384cd50b-370f-474f-b80e-89a544a9bbbd"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:18:05.441527 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:18:05.441492    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/384cd50b-370f-474f-b80e-89a544a9bbbd-trusted-ca-bundle" pod "384cd50b-370f-474f-b80e-89a544a9bbbd" (UID: "384cd50b-370f-474f-b80e-89a544a9bbbd"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:18:05.441570 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:18:05.441535    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/384cd50b-370f-474f-b80e-89a544a9bbbd-service-ca-bundle" pod "384cd50b-370f-474f-b80e-89a544a9bbbd" (UID: "384cd50b-370f-474f-b80e-89a544a9bbbd"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:18:05.443438 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:18:05.443410    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/384cd50b-370f-474f-b80e-89a544a9bbbd-kube-api-access-chwvj" pod "384cd50b-370f-474f-b80e-89a544a9bbbd" (UID: "384cd50b-370f-474f-b80e-89a544a9bbbd"). InnerVolumeSpecName "kube-api-access-chwvj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:18:05.542352 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:18:05.542316    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chwvj\" (UniqueName: \"kubernetes.io/projected/384cd50b-370f-474f-b80e-89a544a9bbbd-kube-api-access-chwvj\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:18:05.542352 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:18:05.542352    4993 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/384cd50b-370f-474f-b80e-89a544a9bbbd-trusted-ca-bundle\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:18:05.542555 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:18:05.542363    4993 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/384cd50b-370f-474f-b80e-89a544a9bbbd-archives-path\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:18:05.542555 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:18:05.542373    4993 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/384cd50b-370f-474f-b80e-89a544a9bbbd-service-ca-bundle\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:18:06.213059 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:18:06.213023    4993 generic.go:356] "Generic (PLEG): container finished" podID="384cd50b-370f-474f-b80e-89a544a9bbbd" containerID="ede52eea27808ab3afc48dbf2065c4c6f1cc4b6c8c2f925d899e121398b002bb" exitCode=2
Nov 01 14:18:06.213500 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:18:06.213072    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-rlgcn-h5sk4"
Nov 01 14:18:06.213500 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:18:06.213104    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-rlgcn-h5sk4" event={"ID":"384cd50b-370f-474f-b80e-89a544a9bbbd","Type":"ContainerDied","Data":"ede52eea27808ab3afc48dbf2065c4c6f1cc4b6c8c2f925d899e121398b002bb"}
Nov 01 14:18:06.213500 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:18:06.213157    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-rlgcn-h5sk4" event={"ID":"384cd50b-370f-474f-b80e-89a544a9bbbd","Type":"ContainerDied","Data":"927f16040227de27cef857b98df6409aba54e22122880ff2d3dc48f638daff3d"}
Nov 01 14:18:06.213500 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:18:06.213188    4993 scope.go:122] "RemoveContainer" containerID="ede52eea27808ab3afc48dbf2065c4c6f1cc4b6c8c2f925d899e121398b002bb"
Nov 01 14:18:06.222220 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:18:06.222054    4993 scope.go:122] "RemoveContainer" containerID="ede52eea27808ab3afc48dbf2065c4c6f1cc4b6c8c2f925d899e121398b002bb"
Nov 01 14:18:06.222339 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:18:06.222319    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ede52eea27808ab3afc48dbf2065c4c6f1cc4b6c8c2f925d899e121398b002bb\": container with ID starting with ede52eea27808ab3afc48dbf2065c4c6f1cc4b6c8c2f925d899e121398b002bb not found: ID does not exist" containerID="ede52eea27808ab3afc48dbf2065c4c6f1cc4b6c8c2f925d899e121398b002bb"
Nov 01 14:18:06.222427 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:18:06.222343    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ede52eea27808ab3afc48dbf2065c4c6f1cc4b6c8c2f925d899e121398b002bb"} err="failed to get container status \"ede52eea27808ab3afc48dbf2065c4c6f1cc4b6c8c2f925d899e121398b002bb\": rpc error: code = NotFound desc = could not find container \"ede52eea27808ab3afc48dbf2065c4c6f1cc4b6c8c2f925d899e121398b002bb\": container with ID starting with ede52eea27808ab3afc48dbf2065c4c6f1cc4b6c8c2f925d899e121398b002bb not found: ID does not exist"
Nov 01 14:18:06.362328 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:18:06.362267    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-rlgcn-h5sk4"]
Nov 01 14:18:06.408597 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:18:06.408564    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-rlgcn-h5sk4"]
Nov 01 14:18:06.793015 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:18:06.792971    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="384cd50b-370f-474f-b80e-89a544a9bbbd" path="/var/lib/kubelet/pods/384cd50b-370f-474f-b80e-89a544a9bbbd/volumes"
Nov 01 14:18:10.231452 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:18:10.231418    4993 generic.go:356] "Generic (PLEG): container finished" podID="4a724964-bd57-4d56-a042-a0d490115eaa" containerID="d2d1ecbf767e973e102e7a31b0be6ebf1172e08e054d8003f951a2e74e73c6a7" exitCode=0
Nov 01 14:18:10.231940 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:18:10.231497    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-96vgd" event={"ID":"4a724964-bd57-4d56-a042-a0d490115eaa","Type":"ContainerDied","Data":"d2d1ecbf767e973e102e7a31b0be6ebf1172e08e054d8003f951a2e74e73c6a7"}
Nov 01 14:18:10.231940 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:18:10.231897    4993 scope.go:122] "RemoveContainer" containerID="d2d1ecbf767e973e102e7a31b0be6ebf1172e08e054d8003f951a2e74e73c6a7"
Nov 01 14:18:11.237192 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:18:11.237136    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-96vgd" event={"ID":"4a724964-bd57-4d56-a042-a0d490115eaa","Type":"ContainerStarted","Data":"b6d6745faf7403617a1a2747a435d1a9ab41f0bfdeeba6ce835bd40743f09afe"}
Nov 01 14:18:25.290616 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:18:25.290579    4993 generic.go:356] "Generic (PLEG): container finished" podID="3e7e18b9-78ee-4349-b00c-a0800e8f8341" containerID="229a7f9ea816f632d9b1f883cd515cbea5787bc4a656de6e50c9101f2ca3f073" exitCode=0
Nov 01 14:18:25.291075 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:18:25.290627    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-87xcd" event={"ID":"3e7e18b9-78ee-4349-b00c-a0800e8f8341","Type":"ContainerDied","Data":"229a7f9ea816f632d9b1f883cd515cbea5787bc4a656de6e50c9101f2ca3f073"}
Nov 01 14:18:25.291075 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:18:25.290933    4993 scope.go:122] "RemoveContainer" containerID="229a7f9ea816f632d9b1f883cd515cbea5787bc4a656de6e50c9101f2ca3f073"
Nov 01 14:18:26.295881 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:18:26.295842    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-87xcd" event={"ID":"3e7e18b9-78ee-4349-b00c-a0800e8f8341","Type":"ContainerStarted","Data":"2b743b10b1b7ccec9ef6e797884f195d996950f2cf84a786ed621e68e491f43b"}
Nov 01 14:18:27.499525 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:18:27.499488    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:18:57.498881 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:18:57.498842    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:19:27.499277 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:19:27.499190    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:19:57.499253 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:19:57.499210    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:20:27.497904 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:20:27.497869    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:20:57.498524 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:20:57.498478    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:21:26.551397 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:26.551362    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71znzqm"]
Nov 01 14:21:26.553775 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:26.551811    4993 cpu_manager.go:414] "containerMap: removing container" podUID="7970eec1-3765-48a4-b09c-3dec2d29956b" containerName="registry"
Nov 01 14:21:26.553775 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:26.551829    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7970eec1-3765-48a4-b09c-3dec2d29956b" containerName="registry"
Nov 01 14:21:26.553775 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:26.551859    4993 cpu_manager.go:414] "containerMap: removing container" podUID="e7ea769b-e719-4090-bcd4-05b1b88fcc06" containerName="registry"
Nov 01 14:21:26.553775 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:26.551866    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7ea769b-e719-4090-bcd4-05b1b88fcc06" containerName="registry"
Nov 01 14:21:26.553775 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:26.551879    4993 cpu_manager.go:414] "containerMap: removing container" podUID="384cd50b-370f-474f-b80e-89a544a9bbbd" containerName="insights-gathering"
Nov 01 14:21:26.553775 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:26.551886    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="384cd50b-370f-474f-b80e-89a544a9bbbd" containerName="insights-gathering"
Nov 01 14:21:26.553775 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:26.551986    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="7970eec1-3765-48a4-b09c-3dec2d29956b" containerName="registry"
Nov 01 14:21:26.553775 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:26.552000    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7ea769b-e719-4090-bcd4-05b1b88fcc06" containerName="registry"
Nov 01 14:21:26.553775 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:26.552008    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="384cd50b-370f-474f-b80e-89a544a9bbbd" containerName="insights-gathering"
Nov 01 14:21:26.554882 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:26.554866    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71znzqm"
Nov 01 14:21:26.603406 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:26.603381    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Nov 01 14:21:26.649547 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:26.649524    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Nov 01 14:21:26.649670 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:26.649602    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-dzln8\""
Nov 01 14:21:26.650644 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:26.650624    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71znzqm"]
Nov 01 14:21:26.711664 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:26.711639    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/97690e5c-e9ca-4ccd-86fa-eb05cea6592a-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71znzqm\" (UID: \"97690e5c-e9ca-4ccd-86fa-eb05cea6592a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71znzqm"
Nov 01 14:21:26.711790 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:26.711668    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk8bg\" (UniqueName: \"kubernetes.io/projected/97690e5c-e9ca-4ccd-86fa-eb05cea6592a-kube-api-access-hk8bg\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71znzqm\" (UID: \"97690e5c-e9ca-4ccd-86fa-eb05cea6592a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71znzqm"
Nov 01 14:21:26.711790 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:26.711692    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/97690e5c-e9ca-4ccd-86fa-eb05cea6592a-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71znzqm\" (UID: \"97690e5c-e9ca-4ccd-86fa-eb05cea6592a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71znzqm"
Nov 01 14:21:26.812224 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:26.812126    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/97690e5c-e9ca-4ccd-86fa-eb05cea6592a-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71znzqm\" (UID: \"97690e5c-e9ca-4ccd-86fa-eb05cea6592a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71znzqm"
Nov 01 14:21:26.812224 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:26.812160    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk8bg\" (UniqueName: \"kubernetes.io/projected/97690e5c-e9ca-4ccd-86fa-eb05cea6592a-kube-api-access-hk8bg\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71znzqm\" (UID: \"97690e5c-e9ca-4ccd-86fa-eb05cea6592a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71znzqm"
Nov 01 14:21:26.812224 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:26.812219    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/97690e5c-e9ca-4ccd-86fa-eb05cea6592a-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71znzqm\" (UID: \"97690e5c-e9ca-4ccd-86fa-eb05cea6592a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71znzqm"
Nov 01 14:21:26.812551 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:26.812532    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/97690e5c-e9ca-4ccd-86fa-eb05cea6592a-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71znzqm\" (UID: \"97690e5c-e9ca-4ccd-86fa-eb05cea6592a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71znzqm"
Nov 01 14:21:26.812614 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:26.812597    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/97690e5c-e9ca-4ccd-86fa-eb05cea6592a-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71znzqm\" (UID: \"97690e5c-e9ca-4ccd-86fa-eb05cea6592a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71znzqm"
Nov 01 14:21:26.865151 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:26.865120    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk8bg\" (UniqueName: \"kubernetes.io/projected/97690e5c-e9ca-4ccd-86fa-eb05cea6592a-kube-api-access-hk8bg\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71znzqm\" (UID: \"97690e5c-e9ca-4ccd-86fa-eb05cea6592a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71znzqm"
Nov 01 14:21:26.869949 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:26.869920    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71znzqm"
Nov 01 14:21:27.234485 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:27.234454    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71znzqm"]
Nov 01 14:21:27.241707 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:21:27.241679    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97690e5c_e9ca_4ccd_86fa_eb05cea6592a.slice/crio-81a44116377cd4a440a20684811c6250affb0e12f86aaa569ff49ca68072a3b2 WatchSource:0}: Error finding container 81a44116377cd4a440a20684811c6250affb0e12f86aaa569ff49ca68072a3b2: Status 404 returned error can't find the container with id 81a44116377cd4a440a20684811c6250affb0e12f86aaa569ff49ca68072a3b2
Nov 01 14:21:27.489066 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:27.488986    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4pmtn"]
Nov 01 14:21:27.493355 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:27.493334    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4pmtn"
Nov 01 14:21:27.499159 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:21:27.499132    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:21:27.541936 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:27.541901    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 01 14:21:27.542071 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:27.541944    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-58vzm\""
Nov 01 14:21:27.563458 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:27.563430    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4pmtn"]
Nov 01 14:21:27.620064 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:27.620028    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/306b0405-fb73-435a-ae73-42a90cd2d2f9-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4pmtn\" (UID: \"306b0405-fb73-435a-ae73-42a90cd2d2f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4pmtn"
Nov 01 14:21:27.620064 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:27.620074    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/306b0405-fb73-435a-ae73-42a90cd2d2f9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4pmtn\" (UID: \"306b0405-fb73-435a-ae73-42a90cd2d2f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4pmtn"
Nov 01 14:21:27.620292 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:27.620162    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/306b0405-fb73-435a-ae73-42a90cd2d2f9-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4pmtn\" (UID: \"306b0405-fb73-435a-ae73-42a90cd2d2f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4pmtn"
Nov 01 14:21:27.620292 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:27.620210    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/306b0405-fb73-435a-ae73-42a90cd2d2f9-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4pmtn\" (UID: \"306b0405-fb73-435a-ae73-42a90cd2d2f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4pmtn"
Nov 01 14:21:27.620374 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:27.620284    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/306b0405-fb73-435a-ae73-42a90cd2d2f9-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4pmtn\" (UID: \"306b0405-fb73-435a-ae73-42a90cd2d2f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4pmtn"
Nov 01 14:21:27.620374 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:27.620321    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/306b0405-fb73-435a-ae73-42a90cd2d2f9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4pmtn\" (UID: \"306b0405-fb73-435a-ae73-42a90cd2d2f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4pmtn"
Nov 01 14:21:27.620444 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:27.620413    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/306b0405-fb73-435a-ae73-42a90cd2d2f9-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4pmtn\" (UID: \"306b0405-fb73-435a-ae73-42a90cd2d2f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4pmtn"
Nov 01 14:21:27.620444 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:27.620434    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2kn8\" (UniqueName: \"kubernetes.io/projected/306b0405-fb73-435a-ae73-42a90cd2d2f9-kube-api-access-h2kn8\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4pmtn\" (UID: \"306b0405-fb73-435a-ae73-42a90cd2d2f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4pmtn"
Nov 01 14:21:27.620510 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:27.620448    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/306b0405-fb73-435a-ae73-42a90cd2d2f9-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4pmtn\" (UID: \"306b0405-fb73-435a-ae73-42a90cd2d2f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4pmtn"
Nov 01 14:21:27.721539 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:27.721506    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2kn8\" (UniqueName: \"kubernetes.io/projected/306b0405-fb73-435a-ae73-42a90cd2d2f9-kube-api-access-h2kn8\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4pmtn\" (UID: \"306b0405-fb73-435a-ae73-42a90cd2d2f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4pmtn"
Nov 01 14:21:27.721539 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:27.721539    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/306b0405-fb73-435a-ae73-42a90cd2d2f9-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4pmtn\" (UID: \"306b0405-fb73-435a-ae73-42a90cd2d2f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4pmtn"
Nov 01 14:21:27.721721 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:27.721664    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/306b0405-fb73-435a-ae73-42a90cd2d2f9-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4pmtn\" (UID: \"306b0405-fb73-435a-ae73-42a90cd2d2f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4pmtn"
Nov 01 14:21:27.721721 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:27.721695    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/306b0405-fb73-435a-ae73-42a90cd2d2f9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4pmtn\" (UID: \"306b0405-fb73-435a-ae73-42a90cd2d2f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4pmtn"
Nov 01 14:21:27.721824 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:27.721724    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/306b0405-fb73-435a-ae73-42a90cd2d2f9-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4pmtn\" (UID: \"306b0405-fb73-435a-ae73-42a90cd2d2f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4pmtn"
Nov 01 14:21:27.721824 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:27.721740    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/306b0405-fb73-435a-ae73-42a90cd2d2f9-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4pmtn\" (UID: \"306b0405-fb73-435a-ae73-42a90cd2d2f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4pmtn"
Nov 01 14:21:27.721824 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:27.721760    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/306b0405-fb73-435a-ae73-42a90cd2d2f9-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4pmtn\" (UID: \"306b0405-fb73-435a-ae73-42a90cd2d2f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4pmtn"
Nov 01 14:21:27.721824 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:27.721795    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/306b0405-fb73-435a-ae73-42a90cd2d2f9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4pmtn\" (UID: \"306b0405-fb73-435a-ae73-42a90cd2d2f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4pmtn"
Nov 01 14:21:27.722021 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:27.721860    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/306b0405-fb73-435a-ae73-42a90cd2d2f9-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4pmtn\" (UID: \"306b0405-fb73-435a-ae73-42a90cd2d2f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4pmtn"
Nov 01 14:21:27.722021 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:27.721922    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/306b0405-fb73-435a-ae73-42a90cd2d2f9-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4pmtn\" (UID: \"306b0405-fb73-435a-ae73-42a90cd2d2f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4pmtn"
Nov 01 14:21:27.722122 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:27.722083    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/306b0405-fb73-435a-ae73-42a90cd2d2f9-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4pmtn\" (UID: \"306b0405-fb73-435a-ae73-42a90cd2d2f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4pmtn"
Nov 01 14:21:27.722122 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:27.722112    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/306b0405-fb73-435a-ae73-42a90cd2d2f9-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4pmtn\" (UID: \"306b0405-fb73-435a-ae73-42a90cd2d2f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4pmtn"
Nov 01 14:21:27.722273 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:27.722255    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/306b0405-fb73-435a-ae73-42a90cd2d2f9-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4pmtn\" (UID: \"306b0405-fb73-435a-ae73-42a90cd2d2f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4pmtn"
Nov 01 14:21:27.722567 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:27.722547    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/306b0405-fb73-435a-ae73-42a90cd2d2f9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4pmtn\" (UID: \"306b0405-fb73-435a-ae73-42a90cd2d2f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4pmtn"
Nov 01 14:21:27.723864 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:27.723845    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/306b0405-fb73-435a-ae73-42a90cd2d2f9-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4pmtn\" (UID: \"306b0405-fb73-435a-ae73-42a90cd2d2f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4pmtn"
Nov 01 14:21:27.724108 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:27.724089    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/306b0405-fb73-435a-ae73-42a90cd2d2f9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4pmtn\" (UID: \"306b0405-fb73-435a-ae73-42a90cd2d2f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4pmtn"
Nov 01 14:21:27.774670 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:27.774612    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2kn8\" (UniqueName: \"kubernetes.io/projected/306b0405-fb73-435a-ae73-42a90cd2d2f9-kube-api-access-h2kn8\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4pmtn\" (UID: \"306b0405-fb73-435a-ae73-42a90cd2d2f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4pmtn"
Nov 01 14:21:27.775763 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:27.775740    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-bcw96"]
Nov 01 14:21:27.777661 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:27.777640    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/306b0405-fb73-435a-ae73-42a90cd2d2f9-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-4pmtn\" (UID: \"306b0405-fb73-435a-ae73-42a90cd2d2f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4pmtn"
Nov 01 14:21:27.780029 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:27.780014    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-bcw96"
Nov 01 14:21:27.806523 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:27.806499    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4pmtn"
Nov 01 14:21:27.841002 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:27.840969    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-bcw96"]
Nov 01 14:21:27.861330 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:27.861304    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-jbxgj\""
Nov 01 14:21:27.923445 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:27.923414    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dacd9b6-f9a3-4ae5-969b-c374a28c242a-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-bcw96\" (UID: \"4dacd9b6-f9a3-4ae5-969b-c374a28c242a\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-bcw96"
Nov 01 14:21:27.923445 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:27.923445    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dacd9b6-f9a3-4ae5-969b-c374a28c242a-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-bcw96\" (UID: \"4dacd9b6-f9a3-4ae5-969b-c374a28c242a\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-bcw96"
Nov 01 14:21:27.923633 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:27.923468    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dacd9b6-f9a3-4ae5-969b-c374a28c242a-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-bcw96\" (UID: \"4dacd9b6-f9a3-4ae5-969b-c374a28c242a\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-bcw96"
Nov 01 14:21:27.923633 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:27.923522    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dacd9b6-f9a3-4ae5-969b-c374a28c242a-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-bcw96\" (UID: \"4dacd9b6-f9a3-4ae5-969b-c374a28c242a\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-bcw96"
Nov 01 14:21:27.923633 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:27.923542    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dacd9b6-f9a3-4ae5-969b-c374a28c242a-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-bcw96\" (UID: \"4dacd9b6-f9a3-4ae5-969b-c374a28c242a\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-bcw96"
Nov 01 14:21:27.923633 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:27.923603    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dacd9b6-f9a3-4ae5-969b-c374a28c242a-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-bcw96\" (UID: \"4dacd9b6-f9a3-4ae5-969b-c374a28c242a\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-bcw96"
Nov 01 14:21:27.923762 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:27.923731    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dacd9b6-f9a3-4ae5-969b-c374a28c242a-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-bcw96\" (UID: \"4dacd9b6-f9a3-4ae5-969b-c374a28c242a\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-bcw96"
Nov 01 14:21:27.923800 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:27.923775    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dacd9b6-f9a3-4ae5-969b-c374a28c242a-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-bcw96\" (UID: \"4dacd9b6-f9a3-4ae5-969b-c374a28c242a\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-bcw96"
Nov 01 14:21:27.923832 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:27.923805    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8qkc\" (UniqueName: \"kubernetes.io/projected/4dacd9b6-f9a3-4ae5-969b-c374a28c242a-kube-api-access-l8qkc\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-bcw96\" (UID: \"4dacd9b6-f9a3-4ae5-969b-c374a28c242a\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-bcw96"
Nov 01 14:21:27.977717 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:27.977679    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71znzqm" event={"ID":"97690e5c-e9ca-4ccd-86fa-eb05cea6592a","Type":"ContainerStarted","Data":"81a44116377cd4a440a20684811c6250affb0e12f86aaa569ff49ca68072a3b2"}
Nov 01 14:21:28.024760 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:28.024673    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dacd9b6-f9a3-4ae5-969b-c374a28c242a-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-bcw96\" (UID: \"4dacd9b6-f9a3-4ae5-969b-c374a28c242a\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-bcw96"
Nov 01 14:21:28.024760 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:28.024714    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8qkc\" (UniqueName: \"kubernetes.io/projected/4dacd9b6-f9a3-4ae5-969b-c374a28c242a-kube-api-access-l8qkc\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-bcw96\" (UID: \"4dacd9b6-f9a3-4ae5-969b-c374a28c242a\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-bcw96"
Nov 01 14:21:28.024760 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:28.024738    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dacd9b6-f9a3-4ae5-969b-c374a28c242a-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-bcw96\" (UID: \"4dacd9b6-f9a3-4ae5-969b-c374a28c242a\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-bcw96"
Nov 01 14:21:28.024760 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:28.024752    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dacd9b6-f9a3-4ae5-969b-c374a28c242a-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-bcw96\" (UID: \"4dacd9b6-f9a3-4ae5-969b-c374a28c242a\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-bcw96"
Nov 01 14:21:28.025078 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:28.024876    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dacd9b6-f9a3-4ae5-969b-c374a28c242a-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-bcw96\" (UID: \"4dacd9b6-f9a3-4ae5-969b-c374a28c242a\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-bcw96"
Nov 01 14:21:28.025078 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:28.024922    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dacd9b6-f9a3-4ae5-969b-c374a28c242a-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-bcw96\" (UID: \"4dacd9b6-f9a3-4ae5-969b-c374a28c242a\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-bcw96"
Nov 01 14:21:28.025078 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:28.024957    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dacd9b6-f9a3-4ae5-969b-c374a28c242a-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-bcw96\" (UID: \"4dacd9b6-f9a3-4ae5-969b-c374a28c242a\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-bcw96"
Nov 01 14:21:28.025078 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:28.024996    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dacd9b6-f9a3-4ae5-969b-c374a28c242a-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-bcw96\" (UID: \"4dacd9b6-f9a3-4ae5-969b-c374a28c242a\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-bcw96"
Nov 01 14:21:28.025078 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:28.025054    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dacd9b6-f9a3-4ae5-969b-c374a28c242a-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-bcw96\" (UID: \"4dacd9b6-f9a3-4ae5-969b-c374a28c242a\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-bcw96"
Nov 01 14:21:28.025348 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:28.025184    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dacd9b6-f9a3-4ae5-969b-c374a28c242a-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-bcw96\" (UID: \"4dacd9b6-f9a3-4ae5-969b-c374a28c242a\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-bcw96"
Nov 01 14:21:28.025348 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:28.025295    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dacd9b6-f9a3-4ae5-969b-c374a28c242a-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-bcw96\" (UID: \"4dacd9b6-f9a3-4ae5-969b-c374a28c242a\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-bcw96"
Nov 01 14:21:28.025464 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:28.025403    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dacd9b6-f9a3-4ae5-969b-c374a28c242a-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-bcw96\" (UID: \"4dacd9b6-f9a3-4ae5-969b-c374a28c242a\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-bcw96"
Nov 01 14:21:28.025464 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:28.025424    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dacd9b6-f9a3-4ae5-969b-c374a28c242a-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-bcw96\" (UID: \"4dacd9b6-f9a3-4ae5-969b-c374a28c242a\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-bcw96"
Nov 01 14:21:28.025611 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:28.025591    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dacd9b6-f9a3-4ae5-969b-c374a28c242a-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-bcw96\" (UID: \"4dacd9b6-f9a3-4ae5-969b-c374a28c242a\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-bcw96"
Nov 01 14:21:28.027049 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:28.027027    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dacd9b6-f9a3-4ae5-969b-c374a28c242a-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-bcw96\" (UID: \"4dacd9b6-f9a3-4ae5-969b-c374a28c242a\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-bcw96"
Nov 01 14:21:28.027341 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:28.027324    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dacd9b6-f9a3-4ae5-969b-c374a28c242a-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-bcw96\" (UID: \"4dacd9b6-f9a3-4ae5-969b-c374a28c242a\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-bcw96"
Nov 01 14:21:28.077360 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:28.077331    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dacd9b6-f9a3-4ae5-969b-c374a28c242a-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-bcw96\" (UID: \"4dacd9b6-f9a3-4ae5-969b-c374a28c242a\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-bcw96"
Nov 01 14:21:28.077482 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:28.077404    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8qkc\" (UniqueName: \"kubernetes.io/projected/4dacd9b6-f9a3-4ae5-969b-c374a28c242a-kube-api-access-l8qkc\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-bcw96\" (UID: \"4dacd9b6-f9a3-4ae5-969b-c374a28c242a\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-bcw96"
Nov 01 14:21:28.091896 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:28.091871    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-bcw96"
Nov 01 14:21:28.175998 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:28.175968    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4pmtn"]
Nov 01 14:21:28.181874 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:21:28.181846    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod306b0405_fb73_435a_ae73_42a90cd2d2f9.slice/crio-ef231d7132db018bd7d9384ceecf6497b806ac84f20d1632fdbf6e8a268d89bd WatchSource:0}: Error finding container ef231d7132db018bd7d9384ceecf6497b806ac84f20d1632fdbf6e8a268d89bd: Status 404 returned error can't find the container with id ef231d7132db018bd7d9384ceecf6497b806ac84f20d1632fdbf6e8a268d89bd
Nov 01 14:21:28.459370 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:28.459286    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-bcw96"]
Nov 01 14:21:28.466350 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:21:28.466320    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dacd9b6_f9a3_4ae5_969b_c374a28c242a.slice/crio-19608756f5c94d45898af700c2d0a5ee0b86c9c64869ec41565396a763c63d05 WatchSource:0}: Error finding container 19608756f5c94d45898af700c2d0a5ee0b86c9c64869ec41565396a763c63d05: Status 404 returned error can't find the container with id 19608756f5c94d45898af700c2d0a5ee0b86c9c64869ec41565396a763c63d05
Nov 01 14:21:28.627016 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:28.626980    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqdkk8"]
Nov 01 14:21:28.630765 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:28.630740    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqdkk8"
Nov 01 14:21:28.683590 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:28.683551    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-r2bw2\""
Nov 01 14:21:28.687773 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:28.687735    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqdkk8"]
Nov 01 14:21:28.731770 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:28.731701    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c766b5a4-3a4b-4fe0-ba4e-c440983ff776-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqdkk8\" (UID: \"c766b5a4-3a4b-4fe0-ba4e-c440983ff776\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqdkk8"
Nov 01 14:21:28.731770 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:28.731745    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z57f2\" (UniqueName: \"kubernetes.io/projected/c766b5a4-3a4b-4fe0-ba4e-c440983ff776-kube-api-access-z57f2\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqdkk8\" (UID: \"c766b5a4-3a4b-4fe0-ba4e-c440983ff776\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqdkk8"
Nov 01 14:21:28.731770 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:28.731769    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c766b5a4-3a4b-4fe0-ba4e-c440983ff776-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqdkk8\" (UID: \"c766b5a4-3a4b-4fe0-ba4e-c440983ff776\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqdkk8"
Nov 01 14:21:28.832793 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:28.832756    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c766b5a4-3a4b-4fe0-ba4e-c440983ff776-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqdkk8\" (UID: \"c766b5a4-3a4b-4fe0-ba4e-c440983ff776\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqdkk8"
Nov 01 14:21:28.832982 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:28.832805    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z57f2\" (UniqueName: \"kubernetes.io/projected/c766b5a4-3a4b-4fe0-ba4e-c440983ff776-kube-api-access-z57f2\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqdkk8\" (UID: \"c766b5a4-3a4b-4fe0-ba4e-c440983ff776\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqdkk8"
Nov 01 14:21:28.832982 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:28.832916    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c766b5a4-3a4b-4fe0-ba4e-c440983ff776-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqdkk8\" (UID: \"c766b5a4-3a4b-4fe0-ba4e-c440983ff776\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqdkk8"
Nov 01 14:21:28.833290 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:28.833260    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c766b5a4-3a4b-4fe0-ba4e-c440983ff776-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqdkk8\" (UID: \"c766b5a4-3a4b-4fe0-ba4e-c440983ff776\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqdkk8"
Nov 01 14:21:28.833426 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:28.833289    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c766b5a4-3a4b-4fe0-ba4e-c440983ff776-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqdkk8\" (UID: \"c766b5a4-3a4b-4fe0-ba4e-c440983ff776\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqdkk8"
Nov 01 14:21:28.886301 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:28.886267    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z57f2\" (UniqueName: \"kubernetes.io/projected/c766b5a4-3a4b-4fe0-ba4e-c440983ff776-kube-api-access-z57f2\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqdkk8\" (UID: \"c766b5a4-3a4b-4fe0-ba4e-c440983ff776\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqdkk8"
Nov 01 14:21:28.948598 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:28.948562    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqdkk8"
Nov 01 14:21:28.985347 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:28.985319    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-bcw96" event={"ID":"4dacd9b6-f9a3-4ae5-969b-c374a28c242a","Type":"ContainerStarted","Data":"19608756f5c94d45898af700c2d0a5ee0b86c9c64869ec41565396a763c63d05"}
Nov 01 14:21:28.986986 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:28.986936    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4pmtn" event={"ID":"306b0405-fb73-435a-ae73-42a90cd2d2f9","Type":"ContainerStarted","Data":"ef231d7132db018bd7d9384ceecf6497b806ac84f20d1632fdbf6e8a268d89bd"}
Nov 01 14:21:29.321840 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:29.321750    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqdkk8"]
Nov 01 14:21:29.328485 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:21:29.328454    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc766b5a4_3a4b_4fe0_ba4e_c440983ff776.slice/crio-952a790d81ee0422642604e9ddda38300b02c5042d6c97e7397e29a2aca7daa5 WatchSource:0}: Error finding container 952a790d81ee0422642604e9ddda38300b02c5042d6c97e7397e29a2aca7daa5: Status 404 returned error can't find the container with id 952a790d81ee0422642604e9ddda38300b02c5042d6c97e7397e29a2aca7daa5
Nov 01 14:21:29.993143 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:29.993098    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqdkk8" event={"ID":"c766b5a4-3a4b-4fe0-ba4e-c440983ff776","Type":"ContainerStarted","Data":"952a790d81ee0422642604e9ddda38300b02c5042d6c97e7397e29a2aca7daa5"}
Nov 01 14:21:33.812493 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:33.812447    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 14:21:33.812877 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:33.812544    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 14:21:33.812877 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:33.812590    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 14:21:34.016212 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:34.016158    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-bcw96" event={"ID":"4dacd9b6-f9a3-4ae5-969b-c374a28c242a","Type":"ContainerStarted","Data":"71bbfdaaa9971526e5f7491cac6cd85a0c9a5936ad6d973d4b3965abc9f395e3"}
Nov 01 14:21:34.078136 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:34.078092    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 14:21:34.078281 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:34.078189    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 14:21:34.078281 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:34.078229    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 14:21:34.092795 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:34.092763    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-bcw96"
Nov 01 14:21:34.094517 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:34.094481    4993 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-ai-inference-65dccbd94c-bcw96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Nov 01 14:21:34.094605 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:34.094549    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-bcw96" podUID="4dacd9b6-f9a3-4ae5-969b-c374a28c242a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Nov 01 14:21:34.175726 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:34.175668    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-bcw96" podStartSLOduration=1.8330826199999999 podStartE2EDuration="7.175651553s" totalImagesPullingTime="5.342568933s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:21:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:21:34.175000803 +0000 UTC m=+281.504863561" watchObservedRunningTime="2026-11-01 14:21:34.175651553 +0000 UTC m=+281.505514285"
Nov 01 14:21:35.023143 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:35.023098    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4pmtn" event={"ID":"306b0405-fb73-435a-ae73-42a90cd2d2f9","Type":"ContainerStarted","Data":"40b21ca817d54c994d04a77445c7ab0386d60fd56e09caee8a15bd1ba81ef3d4"}
Nov 01 14:21:35.097003 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:35.096971    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-bcw96"
Nov 01 14:21:35.133241 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:35.133188    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4pmtn" podStartSLOduration=2.239709835 podStartE2EDuration="8.133157981s" totalImagesPullingTime="5.893448146s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:21:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:21:35.131306724 +0000 UTC m=+282.461169451" watchObservedRunningTime="2026-11-01 14:21:35.133157981 +0000 UTC m=+282.463020714"
Nov 01 14:21:35.807406 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:35.807366    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4pmtn"
Nov 01 14:21:35.812845 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:35.812820    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4pmtn"
Nov 01 14:21:36.030209 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:36.029567    4993 generic.go:356] "Generic (PLEG): container finished" podID="97690e5c-e9ca-4ccd-86fa-eb05cea6592a" containerID="08eb49592476c8bb8020729d4a495377efe331a96e5b082a6ee9bf4db9acc241" exitCode=0
Nov 01 14:21:36.030209 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:36.029615    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71znzqm" event={"ID":"97690e5c-e9ca-4ccd-86fa-eb05cea6592a","Type":"ContainerDied","Data":"08eb49592476c8bb8020729d4a495377efe331a96e5b082a6ee9bf4db9acc241"}
Nov 01 14:21:36.030209 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:36.030214    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-bcw96"
Nov 01 14:21:36.030716 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:36.030255    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4pmtn"
Nov 01 14:21:36.031323 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:36.031306    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4pmtn"
Nov 01 14:21:36.031429 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:36.031366    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-bcw96"
Nov 01 14:21:38.039285 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:38.039250    4993 generic.go:356] "Generic (PLEG): container finished" podID="c766b5a4-3a4b-4fe0-ba4e-c440983ff776" containerID="593b37c2017c3ca8de0edcb459d0a26a58116f667ee02f72c8028d548b9a819b" exitCode=0
Nov 01 14:21:38.039741 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:38.039334    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqdkk8" event={"ID":"c766b5a4-3a4b-4fe0-ba4e-c440983ff776","Type":"ContainerDied","Data":"593b37c2017c3ca8de0edcb459d0a26a58116f667ee02f72c8028d548b9a819b"}
Nov 01 14:21:38.041158 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:38.041082    4993 generic.go:356] "Generic (PLEG): container finished" podID="97690e5c-e9ca-4ccd-86fa-eb05cea6592a" containerID="d77b7d89973b6711abded88d1c950c717629c5da3a9fc9ad4b4bb1cd148cb91f" exitCode=0
Nov 01 14:21:38.041235 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:38.041157    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71znzqm" event={"ID":"97690e5c-e9ca-4ccd-86fa-eb05cea6592a","Type":"ContainerDied","Data":"d77b7d89973b6711abded88d1c950c717629c5da3a9fc9ad4b4bb1cd148cb91f"}
Nov 01 14:21:41.057427 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:41.057386    4993 generic.go:356] "Generic (PLEG): container finished" podID="c766b5a4-3a4b-4fe0-ba4e-c440983ff776" containerID="25fb9c138eba49f7ead4f5466480849e4326e846af2e5d4c1f3c3361c6395b21" exitCode=0
Nov 01 14:21:41.057884 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:41.057478    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqdkk8" event={"ID":"c766b5a4-3a4b-4fe0-ba4e-c440983ff776","Type":"ContainerDied","Data":"25fb9c138eba49f7ead4f5466480849e4326e846af2e5d4c1f3c3361c6395b21"}
Nov 01 14:21:45.081672 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:45.081631    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71znzqm" event={"ID":"97690e5c-e9ca-4ccd-86fa-eb05cea6592a","Type":"ContainerStarted","Data":"0c3aea9d2ff79f762cabd3d6d83fe9af277176358d3f14b80c07d5ced9f65ca3"}
Nov 01 14:21:45.190204 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:45.190134    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71znzqm" podStartSLOduration=3.531748094 podStartE2EDuration="19.19011763s" totalImagesPullingTime="15.658369536s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:21:26 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:21:45.188843719 +0000 UTC m=+292.518706456" watchObservedRunningTime="2026-11-01 14:21:45.19011763 +0000 UTC m=+292.519980367"
Nov 01 14:21:46.087373 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:46.087340    4993 generic.go:356] "Generic (PLEG): container finished" podID="c766b5a4-3a4b-4fe0-ba4e-c440983ff776" containerID="e1c3233fba288409dfeb2e02b9a59fcd73f39ecb4acd16088bfee90552b81ba5" exitCode=0
Nov 01 14:21:46.087813 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:46.087436    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqdkk8" event={"ID":"c766b5a4-3a4b-4fe0-ba4e-c440983ff776","Type":"ContainerDied","Data":"e1c3233fba288409dfeb2e02b9a59fcd73f39ecb4acd16088bfee90552b81ba5"}
Nov 01 14:21:46.089282 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:46.089260    4993 generic.go:356] "Generic (PLEG): container finished" podID="97690e5c-e9ca-4ccd-86fa-eb05cea6592a" containerID="0c3aea9d2ff79f762cabd3d6d83fe9af277176358d3f14b80c07d5ced9f65ca3" exitCode=0
Nov 01 14:21:46.089394 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:46.089338    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71znzqm" event={"ID":"97690e5c-e9ca-4ccd-86fa-eb05cea6592a","Type":"ContainerDied","Data":"0c3aea9d2ff79f762cabd3d6d83fe9af277176358d3f14b80c07d5ced9f65ca3"}
Nov 01 14:21:47.226221 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:47.226193    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqdkk8"
Nov 01 14:21:47.255927 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:47.255902    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71znzqm"
Nov 01 14:21:47.304822 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:47.304796    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97690e5c-e9ca-4ccd-86fa-eb05cea6592a-bundle\" (UniqueName: \"kubernetes.io/empty-dir/97690e5c-e9ca-4ccd-86fa-eb05cea6592a-bundle\") pod \"97690e5c-e9ca-4ccd-86fa-eb05cea6592a\" (UID: \"97690e5c-e9ca-4ccd-86fa-eb05cea6592a\") "
Nov 01 14:21:47.304967 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:47.304835    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c766b5a4-3a4b-4fe0-ba4e-c440983ff776-util\" (UniqueName: \"kubernetes.io/empty-dir/c766b5a4-3a4b-4fe0-ba4e-c440983ff776-util\") pod \"c766b5a4-3a4b-4fe0-ba4e-c440983ff776\" (UID: \"c766b5a4-3a4b-4fe0-ba4e-c440983ff776\") "
Nov 01 14:21:47.304967 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:47.304866    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97690e5c-e9ca-4ccd-86fa-eb05cea6592a-util\" (UniqueName: \"kubernetes.io/empty-dir/97690e5c-e9ca-4ccd-86fa-eb05cea6592a-util\") pod \"97690e5c-e9ca-4ccd-86fa-eb05cea6592a\" (UID: \"97690e5c-e9ca-4ccd-86fa-eb05cea6592a\") "
Nov 01 14:21:47.304967 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:47.304901    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97690e5c-e9ca-4ccd-86fa-eb05cea6592a-kube-api-access-hk8bg\" (UniqueName: \"kubernetes.io/projected/97690e5c-e9ca-4ccd-86fa-eb05cea6592a-kube-api-access-hk8bg\") pod \"97690e5c-e9ca-4ccd-86fa-eb05cea6592a\" (UID: \"97690e5c-e9ca-4ccd-86fa-eb05cea6592a\") "
Nov 01 14:21:47.304967 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:47.304932    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c766b5a4-3a4b-4fe0-ba4e-c440983ff776-kube-api-access-z57f2\" (UniqueName: \"kubernetes.io/projected/c766b5a4-3a4b-4fe0-ba4e-c440983ff776-kube-api-access-z57f2\") pod \"c766b5a4-3a4b-4fe0-ba4e-c440983ff776\" (UID: \"c766b5a4-3a4b-4fe0-ba4e-c440983ff776\") "
Nov 01 14:21:47.305223 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:47.304973    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c766b5a4-3a4b-4fe0-ba4e-c440983ff776-bundle\" (UniqueName: \"kubernetes.io/empty-dir/c766b5a4-3a4b-4fe0-ba4e-c440983ff776-bundle\") pod \"c766b5a4-3a4b-4fe0-ba4e-c440983ff776\" (UID: \"c766b5a4-3a4b-4fe0-ba4e-c440983ff776\") "
Nov 01 14:21:47.305620 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:47.305588    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97690e5c-e9ca-4ccd-86fa-eb05cea6592a-bundle" pod "97690e5c-e9ca-4ccd-86fa-eb05cea6592a" (UID: "97690e5c-e9ca-4ccd-86fa-eb05cea6592a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:21:47.305728 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:47.305611    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c766b5a4-3a4b-4fe0-ba4e-c440983ff776-bundle" pod "c766b5a4-3a4b-4fe0-ba4e-c440983ff776" (UID: "c766b5a4-3a4b-4fe0-ba4e-c440983ff776"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:21:47.307487 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:47.307420    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c766b5a4-3a4b-4fe0-ba4e-c440983ff776-kube-api-access-z57f2" pod "c766b5a4-3a4b-4fe0-ba4e-c440983ff776" (UID: "c766b5a4-3a4b-4fe0-ba4e-c440983ff776"). InnerVolumeSpecName "kube-api-access-z57f2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:21:47.307725 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:47.307695    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97690e5c-e9ca-4ccd-86fa-eb05cea6592a-kube-api-access-hk8bg" pod "97690e5c-e9ca-4ccd-86fa-eb05cea6592a" (UID: "97690e5c-e9ca-4ccd-86fa-eb05cea6592a"). InnerVolumeSpecName "kube-api-access-hk8bg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:21:47.309096 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:47.309073    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97690e5c-e9ca-4ccd-86fa-eb05cea6592a-util" pod "97690e5c-e9ca-4ccd-86fa-eb05cea6592a" (UID: "97690e5c-e9ca-4ccd-86fa-eb05cea6592a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:21:47.309904 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:47.309885    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c766b5a4-3a4b-4fe0-ba4e-c440983ff776-util" pod "c766b5a4-3a4b-4fe0-ba4e-c440983ff776" (UID: "c766b5a4-3a4b-4fe0-ba4e-c440983ff776"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:21:47.406383 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:47.406293    4993 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/97690e5c-e9ca-4ccd-86fa-eb05cea6592a-bundle\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:21:47.406383 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:47.406324    4993 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c766b5a4-3a4b-4fe0-ba4e-c440983ff776-util\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:21:47.406383 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:47.406335    4993 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/97690e5c-e9ca-4ccd-86fa-eb05cea6592a-util\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:21:47.406383 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:47.406348    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hk8bg\" (UniqueName: \"kubernetes.io/projected/97690e5c-e9ca-4ccd-86fa-eb05cea6592a-kube-api-access-hk8bg\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:21:47.406383 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:47.406360    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z57f2\" (UniqueName: \"kubernetes.io/projected/c766b5a4-3a4b-4fe0-ba4e-c440983ff776-kube-api-access-z57f2\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:21:47.406383 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:47.406371    4993 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c766b5a4-3a4b-4fe0-ba4e-c440983ff776-bundle\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:21:48.098570 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:48.098532    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqdkk8"
Nov 01 14:21:48.098773 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:48.098535    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqdkk8" event={"ID":"c766b5a4-3a4b-4fe0-ba4e-c440983ff776","Type":"ContainerDied","Data":"952a790d81ee0422642604e9ddda38300b02c5042d6c97e7397e29a2aca7daa5"}
Nov 01 14:21:48.098773 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:48.098658    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="952a790d81ee0422642604e9ddda38300b02c5042d6c97e7397e29a2aca7daa5"
Nov 01 14:21:48.100504 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:48.100479    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71znzqm" event={"ID":"97690e5c-e9ca-4ccd-86fa-eb05cea6592a","Type":"ContainerDied","Data":"81a44116377cd4a440a20684811c6250affb0e12f86aaa569ff49ca68072a3b2"}
Nov 01 14:21:48.100621 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:48.100511    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81a44116377cd4a440a20684811c6250affb0e12f86aaa569ff49ca68072a3b2"
Nov 01 14:21:48.100621 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:48.100524    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71znzqm"
Nov 01 14:21:52.800898 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:52.800867    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xmwtp_bb59051d-c2df-4362-a157-00f14f915e79/console-operator/3.log"
Nov 01 14:21:52.801368 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:52.801107    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xmwtp_bb59051d-c2df-4362-a157-00f14f915e79/console-operator/3.log"
Nov 01 14:21:53.523509 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:53.523474    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-lphxj"]
Nov 01 14:21:53.523818 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:53.523806    4993 cpu_manager.go:414] "containerMap: removing container" podUID="97690e5c-e9ca-4ccd-86fa-eb05cea6592a" containerName="util"
Nov 01 14:21:53.523876 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:53.523820    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97690e5c-e9ca-4ccd-86fa-eb05cea6592a" containerName="util"
Nov 01 14:21:53.523876 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:53.523831    4993 cpu_manager.go:414] "containerMap: removing container" podUID="c766b5a4-3a4b-4fe0-ba4e-c440983ff776" containerName="pull"
Nov 01 14:21:53.523876 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:53.523836    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c766b5a4-3a4b-4fe0-ba4e-c440983ff776" containerName="pull"
Nov 01 14:21:53.523876 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:53.523848    4993 cpu_manager.go:414] "containerMap: removing container" podUID="97690e5c-e9ca-4ccd-86fa-eb05cea6592a" containerName="pull"
Nov 01 14:21:53.523876 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:53.523853    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97690e5c-e9ca-4ccd-86fa-eb05cea6592a" containerName="pull"
Nov 01 14:21:53.523876 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:53.523859    4993 cpu_manager.go:414] "containerMap: removing container" podUID="c766b5a4-3a4b-4fe0-ba4e-c440983ff776" containerName="extract"
Nov 01 14:21:53.523876 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:53.523864    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c766b5a4-3a4b-4fe0-ba4e-c440983ff776" containerName="extract"
Nov 01 14:21:53.523876 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:53.523871    4993 cpu_manager.go:414] "containerMap: removing container" podUID="c766b5a4-3a4b-4fe0-ba4e-c440983ff776" containerName="util"
Nov 01 14:21:53.523876 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:53.523876    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c766b5a4-3a4b-4fe0-ba4e-c440983ff776" containerName="util"
Nov 01 14:21:53.524124 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:53.523898    4993 cpu_manager.go:414] "containerMap: removing container" podUID="97690e5c-e9ca-4ccd-86fa-eb05cea6592a" containerName="extract"
Nov 01 14:21:53.524124 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:53.523903    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97690e5c-e9ca-4ccd-86fa-eb05cea6592a" containerName="extract"
Nov 01 14:21:53.524124 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:53.523957    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="97690e5c-e9ca-4ccd-86fa-eb05cea6592a" containerName="extract"
Nov 01 14:21:53.524124 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:53.523966    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="c766b5a4-3a4b-4fe0-ba4e-c440983ff776" containerName="extract"
Nov 01 14:21:53.528196 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:53.528180    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-lphxj"
Nov 01 14:21:53.577470 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:53.577448    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 01 14:21:53.577715 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:53.577696    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 01 14:21:53.577795 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:53.577700    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 01 14:21:53.578160 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:53.578142    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 01 14:21:53.578245 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:53.578188    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 01 14:21:53.578709 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:53.578696    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-kgzw7\""
Nov 01 14:21:53.656848 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:53.656824    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-lphxj"]
Nov 01 14:21:53.659693 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:53.659652    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8b28697b-968a-4263-9f46-b31c43140548-webhook-cert\") pod \"opendatahub-operator-controller-manager-666cc8ff84-lphxj\" (UID: \"8b28697b-968a-4263-9f46-b31c43140548\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-lphxj"
Nov 01 14:21:53.659693 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:53.659689    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8b28697b-968a-4263-9f46-b31c43140548-apiservice-cert\") pod \"opendatahub-operator-controller-manager-666cc8ff84-lphxj\" (UID: \"8b28697b-968a-4263-9f46-b31c43140548\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-lphxj"
Nov 01 14:21:53.660007 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:53.659718    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnxft\" (UniqueName: \"kubernetes.io/projected/8b28697b-968a-4263-9f46-b31c43140548-kube-api-access-bnxft\") pod \"opendatahub-operator-controller-manager-666cc8ff84-lphxj\" (UID: \"8b28697b-968a-4263-9f46-b31c43140548\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-lphxj"
Nov 01 14:21:53.660007 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:53.659810    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8b28697b-968a-4263-9f46-b31c43140548-manifests\") pod \"opendatahub-operator-controller-manager-666cc8ff84-lphxj\" (UID: \"8b28697b-968a-4263-9f46-b31c43140548\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-lphxj"
Nov 01 14:21:53.660007 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:53.659839    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8b28697b-968a-4263-9f46-b31c43140548-metrics-cert\") pod \"opendatahub-operator-controller-manager-666cc8ff84-lphxj\" (UID: \"8b28697b-968a-4263-9f46-b31c43140548\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-lphxj"
Nov 01 14:21:53.660007 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:53.659854    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8b28697b-968a-4263-9f46-b31c43140548-tmp\") pod \"opendatahub-operator-controller-manager-666cc8ff84-lphxj\" (UID: \"8b28697b-968a-4263-9f46-b31c43140548\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-lphxj"
Nov 01 14:21:53.760547 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:53.760510    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8b28697b-968a-4263-9f46-b31c43140548-manifests\") pod \"opendatahub-operator-controller-manager-666cc8ff84-lphxj\" (UID: \"8b28697b-968a-4263-9f46-b31c43140548\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-lphxj"
Nov 01 14:21:53.760547 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:53.760556    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8b28697b-968a-4263-9f46-b31c43140548-metrics-cert\") pod \"opendatahub-operator-controller-manager-666cc8ff84-lphxj\" (UID: \"8b28697b-968a-4263-9f46-b31c43140548\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-lphxj"
Nov 01 14:21:53.760797 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:53.760577    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8b28697b-968a-4263-9f46-b31c43140548-tmp\") pod \"opendatahub-operator-controller-manager-666cc8ff84-lphxj\" (UID: \"8b28697b-968a-4263-9f46-b31c43140548\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-lphxj"
Nov 01 14:21:53.760797 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:53.760635    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8b28697b-968a-4263-9f46-b31c43140548-webhook-cert\") pod \"opendatahub-operator-controller-manager-666cc8ff84-lphxj\" (UID: \"8b28697b-968a-4263-9f46-b31c43140548\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-lphxj"
Nov 01 14:21:53.760797 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:53.760771    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8b28697b-968a-4263-9f46-b31c43140548-apiservice-cert\") pod \"opendatahub-operator-controller-manager-666cc8ff84-lphxj\" (UID: \"8b28697b-968a-4263-9f46-b31c43140548\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-lphxj"
Nov 01 14:21:53.760955 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:53.760823    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnxft\" (UniqueName: \"kubernetes.io/projected/8b28697b-968a-4263-9f46-b31c43140548-kube-api-access-bnxft\") pod \"opendatahub-operator-controller-manager-666cc8ff84-lphxj\" (UID: \"8b28697b-968a-4263-9f46-b31c43140548\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-lphxj"
Nov 01 14:21:53.760955 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:53.760917    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8b28697b-968a-4263-9f46-b31c43140548-manifests\") pod \"opendatahub-operator-controller-manager-666cc8ff84-lphxj\" (UID: \"8b28697b-968a-4263-9f46-b31c43140548\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-lphxj"
Nov 01 14:21:53.762914 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:53.762888    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8b28697b-968a-4263-9f46-b31c43140548-tmp\") pod \"opendatahub-operator-controller-manager-666cc8ff84-lphxj\" (UID: \"8b28697b-968a-4263-9f46-b31c43140548\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-lphxj"
Nov 01 14:21:53.763130 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:53.763109    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8b28697b-968a-4263-9f46-b31c43140548-apiservice-cert\") pod \"opendatahub-operator-controller-manager-666cc8ff84-lphxj\" (UID: \"8b28697b-968a-4263-9f46-b31c43140548\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-lphxj"
Nov 01 14:21:53.763202 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:53.763134    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8b28697b-968a-4263-9f46-b31c43140548-webhook-cert\") pod \"opendatahub-operator-controller-manager-666cc8ff84-lphxj\" (UID: \"8b28697b-968a-4263-9f46-b31c43140548\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-lphxj"
Nov 01 14:21:53.763202 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:53.763193    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8b28697b-968a-4263-9f46-b31c43140548-metrics-cert\") pod \"opendatahub-operator-controller-manager-666cc8ff84-lphxj\" (UID: \"8b28697b-968a-4263-9f46-b31c43140548\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-lphxj"
Nov 01 14:21:53.818750 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:53.818683    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnxft\" (UniqueName: \"kubernetes.io/projected/8b28697b-968a-4263-9f46-b31c43140548-kube-api-access-bnxft\") pod \"opendatahub-operator-controller-manager-666cc8ff84-lphxj\" (UID: \"8b28697b-968a-4263-9f46-b31c43140548\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-lphxj"
Nov 01 14:21:53.840511 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:53.840484    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-lphxj"
Nov 01 14:21:54.319837 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:54.319792    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-lphxj"]
Nov 01 14:21:54.326114 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:21:54.326080    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b28697b_968a_4263_9f46_b31c43140548.slice/crio-26fadc8c0e7a030752a381c8ad54b83ffde12b30626ceef4d857f9b1cb119f23 WatchSource:0}: Error finding container 26fadc8c0e7a030752a381c8ad54b83ffde12b30626ceef4d857f9b1cb119f23: Status 404 returned error can't find the container with id 26fadc8c0e7a030752a381c8ad54b83ffde12b30626ceef4d857f9b1cb119f23
Nov 01 14:21:54.327778 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:54.327761    4993 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 14:21:55.129230 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:55.129196    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-lphxj" event={"ID":"8b28697b-968a-4263-9f46-b31c43140548","Type":"ContainerStarted","Data":"26fadc8c0e7a030752a381c8ad54b83ffde12b30626ceef4d857f9b1cb119f23"}
Nov 01 14:21:57.501378 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:21:57.501338    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:21:58.144353 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:58.144320    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-lphxj" event={"ID":"8b28697b-968a-4263-9f46-b31c43140548","Type":"ContainerStarted","Data":"e4cc37b44d970344ed88adbb4aec482b21c998c9a2ffad352570ae3fb585e70e"}
Nov 01 14:21:59.150037 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:59.150003    4993 generic.go:356] "Generic (PLEG): container finished" podID="8b28697b-968a-4263-9f46-b31c43140548" containerID="e4cc37b44d970344ed88adbb4aec482b21c998c9a2ffad352570ae3fb585e70e" exitCode=0
Nov 01 14:21:59.150474 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:21:59.150082    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-lphxj" event={"ID":"8b28697b-968a-4263-9f46-b31c43140548","Type":"ContainerDied","Data":"e4cc37b44d970344ed88adbb4aec482b21c998c9a2ffad352570ae3fb585e70e"}
Nov 01 14:22:00.156817 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:00.156781    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-lphxj" event={"ID":"8b28697b-968a-4263-9f46-b31c43140548","Type":"ContainerStarted","Data":"4225634d3c42a06820715ec314d9744a283a83da6be36d248baccbe407238113"}
Nov 01 14:22:00.157246 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:00.156865    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-lphxj"
Nov 01 14:22:00.372976 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:00.372870    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-lphxj" podStartSLOduration=2.461032854 podStartE2EDuration="7.372856756s" totalImagesPullingTime="4.419186257s" totalInitContainerRuntime="492.637645ms" isStatefulPod=false podCreationTimestamp="2026-11-01 14:21:53 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:22:00.326083537 +0000 UTC m=+307.655946273" watchObservedRunningTime="2026-11-01 14:22:00.372856756 +0000 UTC m=+307.702719524"
Nov 01 14:22:11.163514 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:11.163434    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-lphxj"
Nov 01 14:22:18.919734 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:18.919694    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-8dvhs"]
Nov 01 14:22:18.923349 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:18.923324    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-8dvhs"
Nov 01 14:22:18.965537 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:18.965507    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpjrd\" (UniqueName: \"kubernetes.io/projected/9c1bd12c-d218-4715-9239-1c18ce556fe9-kube-api-access-xpjrd\") pod \"authorino-operator-7b7bb4bc77-8dvhs\" (UID: \"9c1bd12c-d218-4715-9239-1c18ce556fe9\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-8dvhs"
Nov 01 14:22:18.980297 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:18.980274    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 01 14:22:19.025662 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:19.025634    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 01 14:22:19.025835 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:19.025739    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-68fq4\""
Nov 01 14:22:19.066333 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:19.066309    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpjrd\" (UniqueName: \"kubernetes.io/projected/9c1bd12c-d218-4715-9239-1c18ce556fe9-kube-api-access-xpjrd\") pod \"authorino-operator-7b7bb4bc77-8dvhs\" (UID: \"9c1bd12c-d218-4715-9239-1c18ce556fe9\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-8dvhs"
Nov 01 14:22:19.120222 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:19.120194    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpjrd\" (UniqueName: \"kubernetes.io/projected/9c1bd12c-d218-4715-9239-1c18ce556fe9-kube-api-access-xpjrd\") pod \"authorino-operator-7b7bb4bc77-8dvhs\" (UID: \"9c1bd12c-d218-4715-9239-1c18ce556fe9\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-8dvhs"
Nov 01 14:22:19.141578 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:19.141551    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-8dvhs"]
Nov 01 14:22:19.234422 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:19.234398    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-8dvhs"
Nov 01 14:22:19.607552 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:19.607481    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-8dvhs"]
Nov 01 14:22:19.635800 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:22:19.635773    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c1bd12c_d218_4715_9239_1c18ce556fe9.slice/crio-ad0767aa404082cd67d310bd39af94f9c58100b98e74003f2f1fa1b43868a88a WatchSource:0}: Error finding container ad0767aa404082cd67d310bd39af94f9c58100b98e74003f2f1fa1b43868a88a: Status 404 returned error can't find the container with id ad0767aa404082cd67d310bd39af94f9c58100b98e74003f2f1fa1b43868a88a
Nov 01 14:22:20.248978 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:20.248944    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-8dvhs" event={"ID":"9c1bd12c-d218-4715-9239-1c18ce556fe9","Type":"ContainerStarted","Data":"ad0767aa404082cd67d310bd39af94f9c58100b98e74003f2f1fa1b43868a88a"}
Nov 01 14:22:23.262713 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:23.262674    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-8dvhs" event={"ID":"9c1bd12c-d218-4715-9239-1c18ce556fe9","Type":"ContainerStarted","Data":"ef904705edc19882ed7f4fdaff7e69b6a484ebb9627df4485014986cad876ab6"}
Nov 01 14:22:23.263118 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:23.262821    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-8dvhs"
Nov 01 14:22:23.370327 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:23.370227    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-8dvhs" podStartSLOduration=1.949893491 podStartE2EDuration="5.370213856s" totalImagesPullingTime="3.420320365s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:22:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:22:23.368631346 +0000 UTC m=+330.698494081" watchObservedRunningTime="2026-11-01 14:22:23.370213856 +0000 UTC m=+330.700076589"
Nov 01 14:22:25.997971 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:25.997920    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cbwwd"]
Nov 01 14:22:26.000391 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:26.000371    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cbwwd"
Nov 01 14:22:26.017984 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:26.017951    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvvbp\" (UniqueName: \"kubernetes.io/projected/c535f9df-00b8-4b69-91b7-e38837ab6802-kube-api-access-xvvbp\") pod \"kuadrant-operator-controller-manager-678d87766f-cbwwd\" (UID: \"c535f9df-00b8-4b69-91b7-e38837ab6802\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cbwwd"
Nov 01 14:22:26.018092 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:26.018006    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c535f9df-00b8-4b69-91b7-e38837ab6802-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-cbwwd\" (UID: \"c535f9df-00b8-4b69-91b7-e38837ab6802\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cbwwd"
Nov 01 14:22:26.049615 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:26.049587    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-wpjdj\""
Nov 01 14:22:26.097232 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:26.097198    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cbwwd"]
Nov 01 14:22:26.119212 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:26.119180    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvvbp\" (UniqueName: \"kubernetes.io/projected/c535f9df-00b8-4b69-91b7-e38837ab6802-kube-api-access-xvvbp\") pod \"kuadrant-operator-controller-manager-678d87766f-cbwwd\" (UID: \"c535f9df-00b8-4b69-91b7-e38837ab6802\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cbwwd"
Nov 01 14:22:26.119360 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:26.119246    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c535f9df-00b8-4b69-91b7-e38837ab6802-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-cbwwd\" (UID: \"c535f9df-00b8-4b69-91b7-e38837ab6802\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cbwwd"
Nov 01 14:22:26.119610 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:26.119591    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c535f9df-00b8-4b69-91b7-e38837ab6802-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-cbwwd\" (UID: \"c535f9df-00b8-4b69-91b7-e38837ab6802\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cbwwd"
Nov 01 14:22:26.173215 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:26.173183    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvvbp\" (UniqueName: \"kubernetes.io/projected/c535f9df-00b8-4b69-91b7-e38837ab6802-kube-api-access-xvvbp\") pod \"kuadrant-operator-controller-manager-678d87766f-cbwwd\" (UID: \"c535f9df-00b8-4b69-91b7-e38837ab6802\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cbwwd"
Nov 01 14:22:26.312062 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:26.311977    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cbwwd"
Nov 01 14:22:26.793900 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:26.793869    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cbwwd"]
Nov 01 14:22:26.800375 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:22:26.800341    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc535f9df_00b8_4b69_91b7_e38837ab6802.slice/crio-c331d43cbbaef25c26c99a730c954ab052751db30a6ce71a0fa8d18e99df05e7 WatchSource:0}: Error finding container c331d43cbbaef25c26c99a730c954ab052751db30a6ce71a0fa8d18e99df05e7: Status 404 returned error can't find the container with id c331d43cbbaef25c26c99a730c954ab052751db30a6ce71a0fa8d18e99df05e7
Nov 01 14:22:27.282074 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:27.282035    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cbwwd" event={"ID":"c535f9df-00b8-4b69-91b7-e38837ab6802","Type":"ContainerStarted","Data":"c331d43cbbaef25c26c99a730c954ab052751db30a6ce71a0fa8d18e99df05e7"}
Nov 01 14:22:27.498657 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:22:27.498623    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:22:30.234368 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:30.234329    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-d8qkk"]
Nov 01 14:22:30.236766 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:30.236745    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-d8qkk"
Nov 01 14:22:30.263563 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:30.263534    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qcrn\" (UniqueName: \"kubernetes.io/projected/e5667cdf-da6e-4930-93c0-f6dfc6910d45-kube-api-access-2qcrn\") pod \"dns-operator-controller-manager-8557cbdb84-d8qkk\" (UID: \"e5667cdf-da6e-4930-93c0-f6dfc6910d45\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-d8qkk"
Nov 01 14:22:30.269194 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:30.269158    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-d8qkk"]
Nov 01 14:22:30.287825 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:30.287777    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Nov 01 14:22:30.348575 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:30.348547    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-dpsf5\""
Nov 01 14:22:30.364931 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:30.364894    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qcrn\" (UniqueName: \"kubernetes.io/projected/e5667cdf-da6e-4930-93c0-f6dfc6910d45-kube-api-access-2qcrn\") pod \"dns-operator-controller-manager-8557cbdb84-d8qkk\" (UID: \"e5667cdf-da6e-4930-93c0-f6dfc6910d45\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-d8qkk"
Nov 01 14:22:30.419008 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:30.418971    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qcrn\" (UniqueName: \"kubernetes.io/projected/e5667cdf-da6e-4930-93c0-f6dfc6910d45-kube-api-access-2qcrn\") pod \"dns-operator-controller-manager-8557cbdb84-d8qkk\" (UID: \"e5667cdf-da6e-4930-93c0-f6dfc6910d45\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-d8qkk"
Nov 01 14:22:30.551953 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:30.551855    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-d8qkk"
Nov 01 14:22:30.983561 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:30.983533    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-d8qkk"]
Nov 01 14:22:30.990709 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:22:30.990678    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5667cdf_da6e_4930_93c0_f6dfc6910d45.slice/crio-df37d67a0093cd537888254fe89580b7999936043d3a38c6ef09ba5a5d95eac7 WatchSource:0}: Error finding container df37d67a0093cd537888254fe89580b7999936043d3a38c6ef09ba5a5d95eac7: Status 404 returned error can't find the container with id df37d67a0093cd537888254fe89580b7999936043d3a38c6ef09ba5a5d95eac7
Nov 01 14:22:31.303002 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:31.302920    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-d8qkk" event={"ID":"e5667cdf-da6e-4930-93c0-f6dfc6910d45","Type":"ContainerStarted","Data":"df37d67a0093cd537888254fe89580b7999936043d3a38c6ef09ba5a5d95eac7"}
Nov 01 14:22:33.315438 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:33.315395    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cbwwd" event={"ID":"c535f9df-00b8-4b69-91b7-e38837ab6802","Type":"ContainerStarted","Data":"3a6f577fa91ede25dd294b7033550c478d16d3cf6580e736ac11600d32bcfb84"}
Nov 01 14:22:33.315926 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:33.315709    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cbwwd"
Nov 01 14:22:33.516231 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:33.516145    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cbwwd" podStartSLOduration=2.980196002 podStartE2EDuration="8.51613319s" totalImagesPullingTime="5.535937188s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:22:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:22:33.476249852 +0000 UTC m=+340.806112586" watchObservedRunningTime="2026-11-01 14:22:33.51613319 +0000 UTC m=+340.845995925"
Nov 01 14:22:33.596092 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:33.596013    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-4ttxb"]
Nov 01 14:22:33.599551 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:33.599526    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-4ttxb"
Nov 01 14:22:33.648225 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:33.648200    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-5d9hp\""
Nov 01 14:22:33.655818 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:33.655789    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Nov 01 14:22:33.655977 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:33.655951    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Nov 01 14:22:33.658113 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:33.658093    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-4ttxb"]
Nov 01 14:22:33.699927 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:33.699896    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn25n\" (UniqueName: \"kubernetes.io/projected/0950e1d1-e5e0-4be4-8254-845cd2968dc2-kube-api-access-tn25n\") pod \"cert-manager-597b95cb84-4ttxb\" (UID: \"0950e1d1-e5e0-4be4-8254-845cd2968dc2\") " pod="cert-manager/cert-manager-597b95cb84-4ttxb"
Nov 01 14:22:33.700089 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:33.699979    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0950e1d1-e5e0-4be4-8254-845cd2968dc2-bound-sa-token\") pod \"cert-manager-597b95cb84-4ttxb\" (UID: \"0950e1d1-e5e0-4be4-8254-845cd2968dc2\") " pod="cert-manager/cert-manager-597b95cb84-4ttxb"
Nov 01 14:22:33.801406 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:33.801368    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn25n\" (UniqueName: \"kubernetes.io/projected/0950e1d1-e5e0-4be4-8254-845cd2968dc2-kube-api-access-tn25n\") pod \"cert-manager-597b95cb84-4ttxb\" (UID: \"0950e1d1-e5e0-4be4-8254-845cd2968dc2\") " pod="cert-manager/cert-manager-597b95cb84-4ttxb"
Nov 01 14:22:33.801406 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:33.801411    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0950e1d1-e5e0-4be4-8254-845cd2968dc2-bound-sa-token\") pod \"cert-manager-597b95cb84-4ttxb\" (UID: \"0950e1d1-e5e0-4be4-8254-845cd2968dc2\") " pod="cert-manager/cert-manager-597b95cb84-4ttxb"
Nov 01 14:22:33.828800 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:33.828765    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f7nb8nm"]
Nov 01 14:22:33.832287 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:33.832258    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f7nb8nm"
Nov 01 14:22:33.862301 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:33.862237    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn25n\" (UniqueName: \"kubernetes.io/projected/0950e1d1-e5e0-4be4-8254-845cd2968dc2-kube-api-access-tn25n\") pod \"cert-manager-597b95cb84-4ttxb\" (UID: \"0950e1d1-e5e0-4be4-8254-845cd2968dc2\") " pod="cert-manager/cert-manager-597b95cb84-4ttxb"
Nov 01 14:22:33.864683 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:33.864654    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0950e1d1-e5e0-4be4-8254-845cd2968dc2-bound-sa-token\") pod \"cert-manager-597b95cb84-4ttxb\" (UID: \"0950e1d1-e5e0-4be4-8254-845cd2968dc2\") " pod="cert-manager/cert-manager-597b95cb84-4ttxb"
Nov 01 14:22:33.881791 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:33.881752    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-bs6j8\""
Nov 01 14:22:33.894687 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:33.894661    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f7nb8nm"]
Nov 01 14:22:33.901848 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:33.901822    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07f0043b-a848-44d4-9cc2-d6de906d2c01-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f7nb8nm\" (UID: \"07f0043b-a848-44d4-9cc2-d6de906d2c01\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f7nb8nm"
Nov 01 14:22:33.901966 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:33.901864    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07f0043b-a848-44d4-9cc2-d6de906d2c01-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f7nb8nm\" (UID: \"07f0043b-a848-44d4-9cc2-d6de906d2c01\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f7nb8nm"
Nov 01 14:22:33.901966 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:33.901889    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jndnh\" (UniqueName: \"kubernetes.io/projected/07f0043b-a848-44d4-9cc2-d6de906d2c01-kube-api-access-jndnh\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f7nb8nm\" (UID: \"07f0043b-a848-44d4-9cc2-d6de906d2c01\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f7nb8nm"
Nov 01 14:22:33.902056 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:33.901974    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07f0043b-a848-44d4-9cc2-d6de906d2c01-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f7nb8nm\" (UID: \"07f0043b-a848-44d4-9cc2-d6de906d2c01\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f7nb8nm"
Nov 01 14:22:33.902056 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:33.902005    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07f0043b-a848-44d4-9cc2-d6de906d2c01-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f7nb8nm\" (UID: \"07f0043b-a848-44d4-9cc2-d6de906d2c01\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f7nb8nm"
Nov 01 14:22:33.902056 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:33.902021    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07f0043b-a848-44d4-9cc2-d6de906d2c01-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f7nb8nm\" (UID: \"07f0043b-a848-44d4-9cc2-d6de906d2c01\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f7nb8nm"
Nov 01 14:22:33.902187 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:33.902098    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07f0043b-a848-44d4-9cc2-d6de906d2c01-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f7nb8nm\" (UID: \"07f0043b-a848-44d4-9cc2-d6de906d2c01\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f7nb8nm"
Nov 01 14:22:33.902306 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:33.902284    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07f0043b-a848-44d4-9cc2-d6de906d2c01-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f7nb8nm\" (UID: \"07f0043b-a848-44d4-9cc2-d6de906d2c01\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f7nb8nm"
Nov 01 14:22:33.902430 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:33.902365    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07f0043b-a848-44d4-9cc2-d6de906d2c01-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f7nb8nm\" (UID: \"07f0043b-a848-44d4-9cc2-d6de906d2c01\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f7nb8nm"
Nov 01 14:22:33.924818 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:33.924789    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-4ttxb"
Nov 01 14:22:34.004309 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:34.003270    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07f0043b-a848-44d4-9cc2-d6de906d2c01-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f7nb8nm\" (UID: \"07f0043b-a848-44d4-9cc2-d6de906d2c01\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f7nb8nm"
Nov 01 14:22:34.004309 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:34.003324    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07f0043b-a848-44d4-9cc2-d6de906d2c01-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f7nb8nm\" (UID: \"07f0043b-a848-44d4-9cc2-d6de906d2c01\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f7nb8nm"
Nov 01 14:22:34.004309 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:34.003360    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07f0043b-a848-44d4-9cc2-d6de906d2c01-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f7nb8nm\" (UID: \"07f0043b-a848-44d4-9cc2-d6de906d2c01\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f7nb8nm"
Nov 01 14:22:34.004309 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:34.003399    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07f0043b-a848-44d4-9cc2-d6de906d2c01-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f7nb8nm\" (UID: \"07f0043b-a848-44d4-9cc2-d6de906d2c01\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f7nb8nm"
Nov 01 14:22:34.004309 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:34.003430    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jndnh\" (UniqueName: \"kubernetes.io/projected/07f0043b-a848-44d4-9cc2-d6de906d2c01-kube-api-access-jndnh\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f7nb8nm\" (UID: \"07f0043b-a848-44d4-9cc2-d6de906d2c01\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f7nb8nm"
Nov 01 14:22:34.004309 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:34.003475    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07f0043b-a848-44d4-9cc2-d6de906d2c01-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f7nb8nm\" (UID: \"07f0043b-a848-44d4-9cc2-d6de906d2c01\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f7nb8nm"
Nov 01 14:22:34.004309 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:34.003521    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07f0043b-a848-44d4-9cc2-d6de906d2c01-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f7nb8nm\" (UID: \"07f0043b-a848-44d4-9cc2-d6de906d2c01\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f7nb8nm"
Nov 01 14:22:34.004309 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:34.003544    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07f0043b-a848-44d4-9cc2-d6de906d2c01-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f7nb8nm\" (UID: \"07f0043b-a848-44d4-9cc2-d6de906d2c01\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f7nb8nm"
Nov 01 14:22:34.004309 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:34.003578    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07f0043b-a848-44d4-9cc2-d6de906d2c01-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f7nb8nm\" (UID: \"07f0043b-a848-44d4-9cc2-d6de906d2c01\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f7nb8nm"
Nov 01 14:22:34.004309 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:34.003938    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07f0043b-a848-44d4-9cc2-d6de906d2c01-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f7nb8nm\" (UID: \"07f0043b-a848-44d4-9cc2-d6de906d2c01\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f7nb8nm"
Nov 01 14:22:34.004309 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:34.004121    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07f0043b-a848-44d4-9cc2-d6de906d2c01-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f7nb8nm\" (UID: \"07f0043b-a848-44d4-9cc2-d6de906d2c01\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f7nb8nm"
Nov 01 14:22:34.004954 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:34.004370    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07f0043b-a848-44d4-9cc2-d6de906d2c01-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f7nb8nm\" (UID: \"07f0043b-a848-44d4-9cc2-d6de906d2c01\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f7nb8nm"
Nov 01 14:22:34.004954 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:34.004625    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07f0043b-a848-44d4-9cc2-d6de906d2c01-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f7nb8nm\" (UID: \"07f0043b-a848-44d4-9cc2-d6de906d2c01\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f7nb8nm"
Nov 01 14:22:34.004954 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:34.004653    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07f0043b-a848-44d4-9cc2-d6de906d2c01-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f7nb8nm\" (UID: \"07f0043b-a848-44d4-9cc2-d6de906d2c01\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f7nb8nm"
Nov 01 14:22:34.007060 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:34.007036    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07f0043b-a848-44d4-9cc2-d6de906d2c01-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f7nb8nm\" (UID: \"07f0043b-a848-44d4-9cc2-d6de906d2c01\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f7nb8nm"
Nov 01 14:22:34.007329 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:34.007311    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07f0043b-a848-44d4-9cc2-d6de906d2c01-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f7nb8nm\" (UID: \"07f0043b-a848-44d4-9cc2-d6de906d2c01\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f7nb8nm"
Nov 01 14:22:34.060814 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:34.060770    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jndnh\" (UniqueName: \"kubernetes.io/projected/07f0043b-a848-44d4-9cc2-d6de906d2c01-kube-api-access-jndnh\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f7nb8nm\" (UID: \"07f0043b-a848-44d4-9cc2-d6de906d2c01\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f7nb8nm"
Nov 01 14:22:34.060964 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:34.060854    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07f0043b-a848-44d4-9cc2-d6de906d2c01-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f7nb8nm\" (UID: \"07f0043b-a848-44d4-9cc2-d6de906d2c01\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f7nb8nm"
Nov 01 14:22:34.145580 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:34.145305    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f7nb8nm"
Nov 01 14:22:34.269788 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:34.269761    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-8dvhs"
Nov 01 14:22:34.297659 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:34.297632    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-4ttxb"]
Nov 01 14:22:34.306282 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:22:34.306255    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0950e1d1_e5e0_4be4_8254_845cd2968dc2.slice/crio-d558b5b3925c8d4698d68f0d3ded89bbcbda15c7547916d790365a07d5a321e3 WatchSource:0}: Error finding container d558b5b3925c8d4698d68f0d3ded89bbcbda15c7547916d790365a07d5a321e3: Status 404 returned error can't find the container with id d558b5b3925c8d4698d68f0d3ded89bbcbda15c7547916d790365a07d5a321e3
Nov 01 14:22:34.323124 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:34.322395    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-4ttxb" event={"ID":"0950e1d1-e5e0-4be4-8254-845cd2968dc2","Type":"ContainerStarted","Data":"d558b5b3925c8d4698d68f0d3ded89bbcbda15c7547916d790365a07d5a321e3"}
Nov 01 14:22:34.525940 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:34.525907    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f7nb8nm"]
Nov 01 14:22:34.532933 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:22:34.532905    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07f0043b_a848_44d4_9cc2_d6de906d2c01.slice/crio-94791b1841989fc7253ed85538c23887da0c90ee819dbcaa6a60988f896bd6e9 WatchSource:0}: Error finding container 94791b1841989fc7253ed85538c23887da0c90ee819dbcaa6a60988f896bd6e9: Status 404 returned error can't find the container with id 94791b1841989fc7253ed85538c23887da0c90ee819dbcaa6a60988f896bd6e9
Nov 01 14:22:34.534955 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:34.534923    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 14:22:34.535018 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:34.534989    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 14:22:34.535050 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:34.535021    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 14:22:35.329058 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:35.329022    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f7nb8nm" event={"ID":"07f0043b-a848-44d4-9cc2-d6de906d2c01","Type":"ContainerStarted","Data":"ebe36de667fce9544f10c4325ad739582d3aede59c393833204b0ebee75bf9c0"}
Nov 01 14:22:35.329538 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:35.329095    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f7nb8nm" event={"ID":"07f0043b-a848-44d4-9cc2-d6de906d2c01","Type":"ContainerStarted","Data":"94791b1841989fc7253ed85538c23887da0c90ee819dbcaa6a60988f896bd6e9"}
Nov 01 14:22:35.330725 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:35.330700    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-d8qkk" event={"ID":"e5667cdf-da6e-4930-93c0-f6dfc6910d45","Type":"ContainerStarted","Data":"841aa11d09bece4ec8485d0e63655e07572a8881405be274f572542f3d39d5c7"}
Nov 01 14:22:35.330855 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:35.330839    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-d8qkk"
Nov 01 14:22:35.457932 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:35.457879    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f7nb8nm" podStartSLOduration=2.457867537 podStartE2EDuration="2.457867537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:22:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:22:35.454214009 +0000 UTC m=+342.784076745" watchObservedRunningTime="2026-11-01 14:22:35.457867537 +0000 UTC m=+342.787730272"
Nov 01 14:22:35.566142 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:35.566079    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-d8qkk" podStartSLOduration=2.26507666 podStartE2EDuration="5.566063203s" totalImagesPullingTime="3.300986543s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:22:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:22:35.565570629 +0000 UTC m=+342.895433364" watchObservedRunningTime="2026-11-01 14:22:35.566063203 +0000 UTC m=+342.895925937"
Nov 01 14:22:36.146074 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:36.146033    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f7nb8nm"
Nov 01 14:22:36.151846 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:36.151819    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f7nb8nm"
Nov 01 14:22:36.338289 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:36.338255    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f7nb8nm"
Nov 01 14:22:36.339192 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:36.339146    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f7nb8nm"
Nov 01 14:22:39.354806 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:39.354768    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-4ttxb" event={"ID":"0950e1d1-e5e0-4be4-8254-845cd2968dc2","Type":"ContainerStarted","Data":"a8464e6806e829a37cebd5db1c6c294c49dfaf0c566d2bd14cc023e048028448"}
Nov 01 14:22:39.460349 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:39.460297    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-4ttxb" podStartSLOduration=2.392720424 podStartE2EDuration="6.460283935s" totalImagesPullingTime="4.067563511s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:22:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:22:39.458946608 +0000 UTC m=+346.788809343" watchObservedRunningTime="2026-11-01 14:22:39.460283935 +0000 UTC m=+346.790146671"
Nov 01 14:22:41.335579 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:41.335543    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-jg9bl"]
Nov 01 14:22:41.338869 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:41.338841    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-jg9bl"
Nov 01 14:22:41.387137 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:41.387101    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-9sr64\""
Nov 01 14:22:41.434698 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:41.434662    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-jg9bl"]
Nov 01 14:22:41.473062 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:41.473025    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhlfw\" (UniqueName: \"kubernetes.io/projected/51623837-6053-4a24-8a54-af87db25a3bf-kube-api-access-qhlfw\") pod \"authorino-57f7f8659f-jg9bl\" (UID: \"51623837-6053-4a24-8a54-af87db25a3bf\") " pod="kuadrant-system/authorino-57f7f8659f-jg9bl"
Nov 01 14:22:41.574421 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:41.574380    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhlfw\" (UniqueName: \"kubernetes.io/projected/51623837-6053-4a24-8a54-af87db25a3bf-kube-api-access-qhlfw\") pod \"authorino-57f7f8659f-jg9bl\" (UID: \"51623837-6053-4a24-8a54-af87db25a3bf\") " pod="kuadrant-system/authorino-57f7f8659f-jg9bl"
Nov 01 14:22:41.627239 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:41.627153    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhlfw\" (UniqueName: \"kubernetes.io/projected/51623837-6053-4a24-8a54-af87db25a3bf-kube-api-access-qhlfw\") pod \"authorino-57f7f8659f-jg9bl\" (UID: \"51623837-6053-4a24-8a54-af87db25a3bf\") " pod="kuadrant-system/authorino-57f7f8659f-jg9bl"
Nov 01 14:22:41.650025 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:41.649998    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-jg9bl"
Nov 01 14:22:42.015826 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:42.015791    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-jg9bl"]
Nov 01 14:22:42.022557 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:22:42.022525    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod51623837_6053_4a24_8a54_af87db25a3bf.slice/crio-d1808efe107f396ff9acc1768ec42d33c3e25469feb691fe1b469df29e5f6a19 WatchSource:0}: Error finding container d1808efe107f396ff9acc1768ec42d33c3e25469feb691fe1b469df29e5f6a19: Status 404 returned error can't find the container with id d1808efe107f396ff9acc1768ec42d33c3e25469feb691fe1b469df29e5f6a19
Nov 01 14:22:42.370382 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:42.370303    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-jg9bl" event={"ID":"51623837-6053-4a24-8a54-af87db25a3bf","Type":"ContainerStarted","Data":"d1808efe107f396ff9acc1768ec42d33c3e25469feb691fe1b469df29e5f6a19"}
Nov 01 14:22:42.961388 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:42.961353    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-gntc9"]
Nov 01 14:22:42.966124 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:42.966104    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-gntc9"
Nov 01 14:22:43.022434 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:43.022403    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-gntc9"]
Nov 01 14:22:43.088596 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:43.088556    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rv46\" (UniqueName: \"kubernetes.io/projected/f403c01e-299d-4a63-8545-65bc96a68ce1-kube-api-access-9rv46\") pod \"authorino-56c6c9ddd5-gntc9\" (UID: \"f403c01e-299d-4a63-8545-65bc96a68ce1\") " pod="kuadrant-system/authorino-56c6c9ddd5-gntc9"
Nov 01 14:22:43.189911 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:43.189876    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rv46\" (UniqueName: \"kubernetes.io/projected/f403c01e-299d-4a63-8545-65bc96a68ce1-kube-api-access-9rv46\") pod \"authorino-56c6c9ddd5-gntc9\" (UID: \"f403c01e-299d-4a63-8545-65bc96a68ce1\") " pod="kuadrant-system/authorino-56c6c9ddd5-gntc9"
Nov 01 14:22:43.242584 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:43.242552    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rv46\" (UniqueName: \"kubernetes.io/projected/f403c01e-299d-4a63-8545-65bc96a68ce1-kube-api-access-9rv46\") pod \"authorino-56c6c9ddd5-gntc9\" (UID: \"f403c01e-299d-4a63-8545-65bc96a68ce1\") " pod="kuadrant-system/authorino-56c6c9ddd5-gntc9"
Nov 01 14:22:43.278572 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:43.278538    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-gntc9"
Nov 01 14:22:43.641357 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:43.641279    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-gntc9"]
Nov 01 14:22:43.647802 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:22:43.647776    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf403c01e_299d_4a63_8545_65bc96a68ce1.slice/crio-409e4cafdfbbc6a017c12ec0aa246d1d14fcfc72afb9e95b35958bb43e9813dd WatchSource:0}: Error finding container 409e4cafdfbbc6a017c12ec0aa246d1d14fcfc72afb9e95b35958bb43e9813dd: Status 404 returned error can't find the container with id 409e4cafdfbbc6a017c12ec0aa246d1d14fcfc72afb9e95b35958bb43e9813dd
Nov 01 14:22:44.325253 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:44.325217    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cbwwd"
Nov 01 14:22:44.380161 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:44.380125    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-gntc9" event={"ID":"f403c01e-299d-4a63-8545-65bc96a68ce1","Type":"ContainerStarted","Data":"409e4cafdfbbc6a017c12ec0aa246d1d14fcfc72afb9e95b35958bb43e9813dd"}
Nov 01 14:22:45.386536 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:45.386487    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-gntc9" event={"ID":"f403c01e-299d-4a63-8545-65bc96a68ce1","Type":"ContainerStarted","Data":"4afb0d832e98fe2a69bb42005625351953972c111a9878dba4f5bcdb09735e95"}
Nov 01 14:22:45.387958 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:45.387926    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-jg9bl" event={"ID":"51623837-6053-4a24-8a54-af87db25a3bf","Type":"ContainerStarted","Data":"15fa83bcdce1c6365842cdd76a5b8549bf93390c9fe0258eaacdf4b4388ff70f"}
Nov 01 14:22:45.502251 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:45.502185    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-gntc9" podStartSLOduration=2.2379829 podStartE2EDuration="3.502148088s" totalImagesPullingTime="1.264165188s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:22:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:22:45.499361397 +0000 UTC m=+352.829224137" watchObservedRunningTime="2026-11-01 14:22:45.502148088 +0000 UTC m=+352.832010822"
Nov 01 14:22:45.533393 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:45.533355    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-jg9bl"]
Nov 01 14:22:45.611417 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:45.608537    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-jg9bl" podStartSLOduration=1.719429079 podStartE2EDuration="4.608512345s" totalImagesPullingTime="2.889083266s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:22:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:22:45.605921379 +0000 UTC m=+352.935784131" watchObservedRunningTime="2026-11-01 14:22:45.608512345 +0000 UTC m=+352.938375082"
Nov 01 14:22:46.339981 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:46.339949    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-d8qkk"
Nov 01 14:22:47.396645 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:47.396606    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-jg9bl" podUID="51623837-6053-4a24-8a54-af87db25a3bf" containerName="authorino" containerID="cri-o://15fa83bcdce1c6365842cdd76a5b8549bf93390c9fe0258eaacdf4b4388ff70f" gracePeriod=30
Nov 01 14:22:47.639843 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:47.639819    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-jg9bl"
Nov 01 14:22:47.732946 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:47.732866    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51623837-6053-4a24-8a54-af87db25a3bf-kube-api-access-qhlfw\" (UniqueName: \"kubernetes.io/projected/51623837-6053-4a24-8a54-af87db25a3bf-kube-api-access-qhlfw\") pod \"51623837-6053-4a24-8a54-af87db25a3bf\" (UID: \"51623837-6053-4a24-8a54-af87db25a3bf\") "
Nov 01 14:22:47.735011 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:47.734986    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51623837-6053-4a24-8a54-af87db25a3bf-kube-api-access-qhlfw" pod "51623837-6053-4a24-8a54-af87db25a3bf" (UID: "51623837-6053-4a24-8a54-af87db25a3bf"). InnerVolumeSpecName "kube-api-access-qhlfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:22:47.833765 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:47.833731    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhlfw\" (UniqueName: \"kubernetes.io/projected/51623837-6053-4a24-8a54-af87db25a3bf-kube-api-access-qhlfw\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:22:48.402363 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:48.402326    4993 generic.go:356] "Generic (PLEG): container finished" podID="51623837-6053-4a24-8a54-af87db25a3bf" containerID="15fa83bcdce1c6365842cdd76a5b8549bf93390c9fe0258eaacdf4b4388ff70f" exitCode=0
Nov 01 14:22:48.402769 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:48.402372    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-jg9bl"
Nov 01 14:22:48.402769 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:48.402374    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-jg9bl" event={"ID":"51623837-6053-4a24-8a54-af87db25a3bf","Type":"ContainerDied","Data":"15fa83bcdce1c6365842cdd76a5b8549bf93390c9fe0258eaacdf4b4388ff70f"}
Nov 01 14:22:48.402769 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:48.402485    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-jg9bl" event={"ID":"51623837-6053-4a24-8a54-af87db25a3bf","Type":"ContainerDied","Data":"d1808efe107f396ff9acc1768ec42d33c3e25469feb691fe1b469df29e5f6a19"}
Nov 01 14:22:48.402769 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:48.402511    4993 scope.go:122] "RemoveContainer" containerID="15fa83bcdce1c6365842cdd76a5b8549bf93390c9fe0258eaacdf4b4388ff70f"
Nov 01 14:22:48.411118 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:48.411096    4993 scope.go:122] "RemoveContainer" containerID="15fa83bcdce1c6365842cdd76a5b8549bf93390c9fe0258eaacdf4b4388ff70f"
Nov 01 14:22:48.411385 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:22:48.411363    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15fa83bcdce1c6365842cdd76a5b8549bf93390c9fe0258eaacdf4b4388ff70f\": container with ID starting with 15fa83bcdce1c6365842cdd76a5b8549bf93390c9fe0258eaacdf4b4388ff70f not found: ID does not exist" containerID="15fa83bcdce1c6365842cdd76a5b8549bf93390c9fe0258eaacdf4b4388ff70f"
Nov 01 14:22:48.411443 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:48.411394    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15fa83bcdce1c6365842cdd76a5b8549bf93390c9fe0258eaacdf4b4388ff70f"} err="failed to get container status \"15fa83bcdce1c6365842cdd76a5b8549bf93390c9fe0258eaacdf4b4388ff70f\": rpc error: code = NotFound desc = could not find container \"15fa83bcdce1c6365842cdd76a5b8549bf93390c9fe0258eaacdf4b4388ff70f\": container with ID starting with 15fa83bcdce1c6365842cdd76a5b8549bf93390c9fe0258eaacdf4b4388ff70f not found: ID does not exist"
Nov 01 14:22:48.552787 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:48.552753    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-jg9bl"]
Nov 01 14:22:48.556635 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:48.556609    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-jg9bl"]
Nov 01 14:22:48.794161 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:48.794129    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51623837-6053-4a24-8a54-af87db25a3bf" path="/var/lib/kubelet/pods/51623837-6053-4a24-8a54-af87db25a3bf/volumes"
Nov 01 14:22:57.498054 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:22:57.498010    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:22:59.858006 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:59.857970    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-2chcp"]
Nov 01 14:22:59.858412 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:59.858347    4993 cpu_manager.go:414] "containerMap: removing container" podUID="51623837-6053-4a24-8a54-af87db25a3bf" containerName="authorino"
Nov 01 14:22:59.858412 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:59.858358    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51623837-6053-4a24-8a54-af87db25a3bf" containerName="authorino"
Nov 01 14:22:59.858480 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:59.858443    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="51623837-6053-4a24-8a54-af87db25a3bf" containerName="authorino"
Nov 01 14:22:59.865073 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:59.865051    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-2chcp"
Nov 01 14:22:59.913912 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:59.913889    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 01 14:22:59.960048 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:59.960020    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 01 14:22:59.960213 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:22:59.960058    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-p5t9t\""
Nov 01 14:23:00.041880 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:00.041850    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/31f198c9-adac-4cd9-8cd9-2551a89aad9c-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-2chcp\" (UID: \"31f198c9-adac-4cd9-8cd9-2551a89aad9c\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-2chcp"
Nov 01 14:23:00.042048 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:00.041887    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/31f198c9-adac-4cd9-8cd9-2551a89aad9c-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-2chcp\" (UID: \"31f198c9-adac-4cd9-8cd9-2551a89aad9c\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-2chcp"
Nov 01 14:23:00.042048 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:00.041924    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/31f198c9-adac-4cd9-8cd9-2551a89aad9c-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-2chcp\" (UID: \"31f198c9-adac-4cd9-8cd9-2551a89aad9c\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-2chcp"
Nov 01 14:23:00.042048 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:00.042023    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m5hx\" (UniqueName: \"kubernetes.io/projected/31f198c9-adac-4cd9-8cd9-2551a89aad9c-kube-api-access-9m5hx\") pod \"kserve-module-controller-manager-7566cf5fc8-2chcp\" (UID: \"31f198c9-adac-4cd9-8cd9-2551a89aad9c\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-2chcp"
Nov 01 14:23:00.053214 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:00.053160    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\""
Nov 01 14:23:00.054907 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:00.054888    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-2chcp"]
Nov 01 14:23:00.143052 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:00.142973    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/31f198c9-adac-4cd9-8cd9-2551a89aad9c-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-2chcp\" (UID: \"31f198c9-adac-4cd9-8cd9-2551a89aad9c\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-2chcp"
Nov 01 14:23:00.143052 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:00.143008    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/31f198c9-adac-4cd9-8cd9-2551a89aad9c-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-2chcp\" (UID: \"31f198c9-adac-4cd9-8cd9-2551a89aad9c\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-2chcp"
Nov 01 14:23:00.143052 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:00.143043    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/31f198c9-adac-4cd9-8cd9-2551a89aad9c-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-2chcp\" (UID: \"31f198c9-adac-4cd9-8cd9-2551a89aad9c\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-2chcp"
Nov 01 14:23:00.143295 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:00.143072    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m5hx\" (UniqueName: \"kubernetes.io/projected/31f198c9-adac-4cd9-8cd9-2551a89aad9c-kube-api-access-9m5hx\") pod \"kserve-module-controller-manager-7566cf5fc8-2chcp\" (UID: \"31f198c9-adac-4cd9-8cd9-2551a89aad9c\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-2chcp"
Nov 01 14:23:00.143400 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:00.143379    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/31f198c9-adac-4cd9-8cd9-2551a89aad9c-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-2chcp\" (UID: \"31f198c9-adac-4cd9-8cd9-2551a89aad9c\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-2chcp"
Nov 01 14:23:00.143435 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:00.143397    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/31f198c9-adac-4cd9-8cd9-2551a89aad9c-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-2chcp\" (UID: \"31f198c9-adac-4cd9-8cd9-2551a89aad9c\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-2chcp"
Nov 01 14:23:00.144068 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:00.144051    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/31f198c9-adac-4cd9-8cd9-2551a89aad9c-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-2chcp\" (UID: \"31f198c9-adac-4cd9-8cd9-2551a89aad9c\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-2chcp"
Nov 01 14:23:00.195866 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:00.195830    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m5hx\" (UniqueName: \"kubernetes.io/projected/31f198c9-adac-4cd9-8cd9-2551a89aad9c-kube-api-access-9m5hx\") pod \"kserve-module-controller-manager-7566cf5fc8-2chcp\" (UID: \"31f198c9-adac-4cd9-8cd9-2551a89aad9c\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-2chcp"
Nov 01 14:23:00.477820 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:00.477732    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-2chcp"
Nov 01 14:23:00.945404 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:00.945372    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-2chcp"]
Nov 01 14:23:00.955829 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:23:00.955794    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31f198c9_adac_4cd9_8cd9_2551a89aad9c.slice/crio-e0fbbe4e51324d85d5bc1f4609f15037435057892c59274fea1ddd411f1f61cb WatchSource:0}: Error finding container e0fbbe4e51324d85d5bc1f4609f15037435057892c59274fea1ddd411f1f61cb: Status 404 returned error can't find the container with id e0fbbe4e51324d85d5bc1f4609f15037435057892c59274fea1ddd411f1f61cb
Nov 01 14:23:01.460827 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:01.460793    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-2chcp" event={"ID":"31f198c9-adac-4cd9-8cd9-2551a89aad9c","Type":"ContainerStarted","Data":"e0fbbe4e51324d85d5bc1f4609f15037435057892c59274fea1ddd411f1f61cb"}
Nov 01 14:23:04.478605 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:04.478565    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-2chcp" event={"ID":"31f198c9-adac-4cd9-8cd9-2551a89aad9c","Type":"ContainerStarted","Data":"301a36497d6bd2c6f1235b076b77ff4619a1e8b6356466bd5f63ba6fe7d16c67"}
Nov 01 14:23:04.479000 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:04.478670    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-2chcp"
Nov 01 14:23:04.634289 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:04.634135    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-2chcp" podStartSLOduration=2.294616814 podStartE2EDuration="5.634122021s" totalImagesPullingTime="3.339505207s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:22:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:23:04.632108283 +0000 UTC m=+371.961971021" watchObservedRunningTime="2026-11-01 14:23:04.634122021 +0000 UTC m=+371.963984757"
Nov 01 14:23:25.484182 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:25.484130    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-2chcp"
Nov 01 14:23:26.547338 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:26.547301    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6fcc8fbfd5-7c246"]
Nov 01 14:23:26.555887 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:26.555864    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6fcc8fbfd5-7c246"
Nov 01 14:23:26.650326 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:26.650281    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 01 14:23:26.650508 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:26.650282    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 01 14:23:26.650508 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:26.650351    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 01 14:23:26.650508 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:26.650372    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-lqgws\""
Nov 01 14:23:26.652266 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:26.652245    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6fcc8fbfd5-7c246"]
Nov 01 14:23:26.691008 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:26.690965    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/406ec2b7-3f8e-40a5-ad35-07a493ba0052-webhook-cert\") pod \"maas-controller-6fcc8fbfd5-7c246\" (UID: \"406ec2b7-3f8e-40a5-ad35-07a493ba0052\") " pod="opendatahub/maas-controller-6fcc8fbfd5-7c246"
Nov 01 14:23:26.691215 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:26.691038    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/406ec2b7-3f8e-40a5-ad35-07a493ba0052-metrics-certs\") pod \"maas-controller-6fcc8fbfd5-7c246\" (UID: \"406ec2b7-3f8e-40a5-ad35-07a493ba0052\") " pod="opendatahub/maas-controller-6fcc8fbfd5-7c246"
Nov 01 14:23:26.691215 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:26.691098    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggf2l\" (UniqueName: \"kubernetes.io/projected/406ec2b7-3f8e-40a5-ad35-07a493ba0052-kube-api-access-ggf2l\") pod \"maas-controller-6fcc8fbfd5-7c246\" (UID: \"406ec2b7-3f8e-40a5-ad35-07a493ba0052\") " pod="opendatahub/maas-controller-6fcc8fbfd5-7c246"
Nov 01 14:23:26.792139 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:26.792109    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/406ec2b7-3f8e-40a5-ad35-07a493ba0052-metrics-certs\") pod \"maas-controller-6fcc8fbfd5-7c246\" (UID: \"406ec2b7-3f8e-40a5-ad35-07a493ba0052\") " pod="opendatahub/maas-controller-6fcc8fbfd5-7c246"
Nov 01 14:23:26.792334 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:26.792153    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggf2l\" (UniqueName: \"kubernetes.io/projected/406ec2b7-3f8e-40a5-ad35-07a493ba0052-kube-api-access-ggf2l\") pod \"maas-controller-6fcc8fbfd5-7c246\" (UID: \"406ec2b7-3f8e-40a5-ad35-07a493ba0052\") " pod="opendatahub/maas-controller-6fcc8fbfd5-7c246"
Nov 01 14:23:26.792334 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:26.792239    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/406ec2b7-3f8e-40a5-ad35-07a493ba0052-webhook-cert\") pod \"maas-controller-6fcc8fbfd5-7c246\" (UID: \"406ec2b7-3f8e-40a5-ad35-07a493ba0052\") " pod="opendatahub/maas-controller-6fcc8fbfd5-7c246"
Nov 01 14:23:26.794729 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:26.794702    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/406ec2b7-3f8e-40a5-ad35-07a493ba0052-metrics-certs\") pod \"maas-controller-6fcc8fbfd5-7c246\" (UID: \"406ec2b7-3f8e-40a5-ad35-07a493ba0052\") " pod="opendatahub/maas-controller-6fcc8fbfd5-7c246"
Nov 01 14:23:26.794848 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:26.794810    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/406ec2b7-3f8e-40a5-ad35-07a493ba0052-webhook-cert\") pod \"maas-controller-6fcc8fbfd5-7c246\" (UID: \"406ec2b7-3f8e-40a5-ad35-07a493ba0052\") " pod="opendatahub/maas-controller-6fcc8fbfd5-7c246"
Nov 01 14:23:26.845146 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:26.845078    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggf2l\" (UniqueName: \"kubernetes.io/projected/406ec2b7-3f8e-40a5-ad35-07a493ba0052-kube-api-access-ggf2l\") pod \"maas-controller-6fcc8fbfd5-7c246\" (UID: \"406ec2b7-3f8e-40a5-ad35-07a493ba0052\") " pod="opendatahub/maas-controller-6fcc8fbfd5-7c246"
Nov 01 14:23:26.869038 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:26.869001    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6fcc8fbfd5-7c246"
Nov 01 14:23:27.235339 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:27.235314    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6fcc8fbfd5-7c246"]
Nov 01 14:23:27.241864 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:23:27.241838    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod406ec2b7_3f8e_40a5_ad35_07a493ba0052.slice/crio-a2d8825eb322c916818a89c618b1bcff74ea86a43a352bc47e3d1ceba2ccf06b WatchSource:0}: Error finding container a2d8825eb322c916818a89c618b1bcff74ea86a43a352bc47e3d1ceba2ccf06b: Status 404 returned error can't find the container with id a2d8825eb322c916818a89c618b1bcff74ea86a43a352bc47e3d1ceba2ccf06b
Nov 01 14:23:27.499896 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:23:27.499810    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:23:27.586186 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:27.586125    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6fcc8fbfd5-7c246" event={"ID":"406ec2b7-3f8e-40a5-ad35-07a493ba0052","Type":"ContainerStarted","Data":"a2d8825eb322c916818a89c618b1bcff74ea86a43a352bc47e3d1ceba2ccf06b"}
Nov 01 14:23:30.604146 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:30.604107    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6fcc8fbfd5-7c246" event={"ID":"406ec2b7-3f8e-40a5-ad35-07a493ba0052","Type":"ContainerStarted","Data":"741bd3d29e8be55b957b826d6c1af966c73c305e46d902920190014b918e2852"}
Nov 01 14:23:30.604563 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:30.604255    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6fcc8fbfd5-7c246"
Nov 01 14:23:30.730251 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:30.730131    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-6fcc8fbfd5-7c246" podStartSLOduration=1.5110153830000002 podStartE2EDuration="4.73010565s" totalImagesPullingTime="3.219090267s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:23:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:23:30.729410617 +0000 UTC m=+398.059273353" watchObservedRunningTime="2026-11-01 14:23:30.73010565 +0000 UTC m=+398.059968384"
Nov 01 14:23:33.585159 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:33.585074    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cmbwq"]
Nov 01 14:23:33.588326 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:33.588300    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cmbwq"
Nov 01 14:23:33.637033 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:33.637004    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\""
Nov 01 14:23:33.637296 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:33.637014    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\""
Nov 01 14:23:33.637296 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:33.637145    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\""
Nov 01 14:23:33.655582 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:33.655553    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/6489d076-e1bd-402d-9d85-4e3bce4123b0-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-cmbwq\" (UID: \"6489d076-e1bd-402d-9d85-4e3bce4123b0\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cmbwq"
Nov 01 14:23:33.655726 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:33.655588    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6489d076-e1bd-402d-9d85-4e3bce4123b0-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-cmbwq\" (UID: \"6489d076-e1bd-402d-9d85-4e3bce4123b0\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cmbwq"
Nov 01 14:23:33.655726 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:33.655611    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6489d076-e1bd-402d-9d85-4e3bce4123b0-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-cmbwq\" (UID: \"6489d076-e1bd-402d-9d85-4e3bce4123b0\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cmbwq"
Nov 01 14:23:33.655726 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:33.655720    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5t7f\" (UniqueName: \"kubernetes.io/projected/6489d076-e1bd-402d-9d85-4e3bce4123b0-kube-api-access-b5t7f\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-cmbwq\" (UID: \"6489d076-e1bd-402d-9d85-4e3bce4123b0\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cmbwq"
Nov 01 14:23:33.683571 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:33.683531    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-2w5kd\""
Nov 01 14:23:33.685583 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:33.685554    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cmbwq"]
Nov 01 14:23:33.757081 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:33.757047    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/6489d076-e1bd-402d-9d85-4e3bce4123b0-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-cmbwq\" (UID: \"6489d076-e1bd-402d-9d85-4e3bce4123b0\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cmbwq"
Nov 01 14:23:33.757309 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:33.757086    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6489d076-e1bd-402d-9d85-4e3bce4123b0-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-cmbwq\" (UID: \"6489d076-e1bd-402d-9d85-4e3bce4123b0\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cmbwq"
Nov 01 14:23:33.757309 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:33.757117    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6489d076-e1bd-402d-9d85-4e3bce4123b0-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-cmbwq\" (UID: \"6489d076-e1bd-402d-9d85-4e3bce4123b0\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cmbwq"
Nov 01 14:23:33.757309 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:33.757161    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5t7f\" (UniqueName: \"kubernetes.io/projected/6489d076-e1bd-402d-9d85-4e3bce4123b0-kube-api-access-b5t7f\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-cmbwq\" (UID: \"6489d076-e1bd-402d-9d85-4e3bce4123b0\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cmbwq"
Nov 01 14:23:33.757309 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:23:33.757219    4993 configmap.go:193] Couldn't get configMap opendatahub/usage-tenancy-proxy-source: configmap "usage-tenancy-proxy-source" not found
Nov 01 14:23:33.757309 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:23:33.757248    4993 secret.go:189] Couldn't get secret opendatahub/usage-logs-tenancy-proxy-tls: secret "usage-logs-tenancy-proxy-tls" not found
Nov 01 14:23:33.757309 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:23:33.757296    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6489d076-e1bd-402d-9d85-4e3bce4123b0-proxy-source podName:6489d076-e1bd-402d-9d85-4e3bce4123b0 nodeName:}" failed. No retries permitted until 2026-11-01 14:23:34.257274758 +0000 UTC m=+401.587137474 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-source" (UniqueName: "kubernetes.io/configmap/6489d076-e1bd-402d-9d85-4e3bce4123b0-proxy-source") pod "usage-logs-tenancy-proxy-9c5d49bcd-cmbwq" (UID: "6489d076-e1bd-402d-9d85-4e3bce4123b0") : configmap "usage-tenancy-proxy-source" not found
Nov 01 14:23:33.757623 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:23:33.757316    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6489d076-e1bd-402d-9d85-4e3bce4123b0-tls-cert podName:6489d076-e1bd-402d-9d85-4e3bce4123b0 nodeName:}" failed. No retries permitted until 2026-11-01 14:23:34.257307349 +0000 UTC m=+401.587170064 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-cert" (UniqueName: "kubernetes.io/secret/6489d076-e1bd-402d-9d85-4e3bce4123b0-tls-cert") pod "usage-logs-tenancy-proxy-9c5d49bcd-cmbwq" (UID: "6489d076-e1bd-402d-9d85-4e3bce4123b0") : secret "usage-logs-tenancy-proxy-tls" not found
Nov 01 14:23:33.757843 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:33.757824    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6489d076-e1bd-402d-9d85-4e3bce4123b0-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-cmbwq\" (UID: \"6489d076-e1bd-402d-9d85-4e3bce4123b0\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cmbwq"
Nov 01 14:23:33.810616 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:33.810587    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5t7f\" (UniqueName: \"kubernetes.io/projected/6489d076-e1bd-402d-9d85-4e3bce4123b0-kube-api-access-b5t7f\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-cmbwq\" (UID: \"6489d076-e1bd-402d-9d85-4e3bce4123b0\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cmbwq"
Nov 01 14:23:34.262922 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:34.262885    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/6489d076-e1bd-402d-9d85-4e3bce4123b0-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-cmbwq\" (UID: \"6489d076-e1bd-402d-9d85-4e3bce4123b0\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cmbwq"
Nov 01 14:23:34.262922 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:34.262928    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6489d076-e1bd-402d-9d85-4e3bce4123b0-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-cmbwq\" (UID: \"6489d076-e1bd-402d-9d85-4e3bce4123b0\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cmbwq"
Nov 01 14:23:34.263964 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:34.263939    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/6489d076-e1bd-402d-9d85-4e3bce4123b0-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-cmbwq\" (UID: \"6489d076-e1bd-402d-9d85-4e3bce4123b0\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cmbwq"
Nov 01 14:23:34.265563 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:34.265539    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6489d076-e1bd-402d-9d85-4e3bce4123b0-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-cmbwq\" (UID: \"6489d076-e1bd-402d-9d85-4e3bce4123b0\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cmbwq"
Nov 01 14:23:34.501453 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:34.501408    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cmbwq"
Nov 01 14:23:34.866384 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:34.866307    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cmbwq"]
Nov 01 14:23:34.872616 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:23:34.872587    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6489d076_e1bd_402d_9d85_4e3bce4123b0.slice/crio-1a96935a62b8d1e1f9e7f069a2ba0112c422f9ffaa7fb12f4472ccd432f7d5ec WatchSource:0}: Error finding container 1a96935a62b8d1e1f9e7f069a2ba0112c422f9ffaa7fb12f4472ccd432f7d5ec: Status 404 returned error can't find the container with id 1a96935a62b8d1e1f9e7f069a2ba0112c422f9ffaa7fb12f4472ccd432f7d5ec
Nov 01 14:23:35.631094 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:35.631058    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cmbwq" event={"ID":"6489d076-e1bd-402d-9d85-4e3bce4123b0","Type":"ContainerStarted","Data":"1a96935a62b8d1e1f9e7f069a2ba0112c422f9ffaa7fb12f4472ccd432f7d5ec"}
Nov 01 14:23:41.612641 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:41.611731    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6fcc8fbfd5-7c246"
Nov 01 14:23:42.453863 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:42.453820    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6fcc8fbfd5-7c246"]
Nov 01 14:23:42.455304 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:42.454890    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-6fcc8fbfd5-7c246" podUID="406ec2b7-3f8e-40a5-ad35-07a493ba0052" containerName="manager" containerID="cri-o://741bd3d29e8be55b957b826d6c1af966c73c305e46d902920190014b918e2852" gracePeriod=30
Nov 01 14:23:42.673940 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:42.673903    4993 generic.go:356] "Generic (PLEG): container finished" podID="406ec2b7-3f8e-40a5-ad35-07a493ba0052" containerID="741bd3d29e8be55b957b826d6c1af966c73c305e46d902920190014b918e2852" exitCode=0
Nov 01 14:23:42.674475 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:42.673985    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6fcc8fbfd5-7c246" event={"ID":"406ec2b7-3f8e-40a5-ad35-07a493ba0052","Type":"ContainerDied","Data":"741bd3d29e8be55b957b826d6c1af966c73c305e46d902920190014b918e2852"}
Nov 01 14:23:43.238767 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:43.238731    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6fcc8fbfd5-7c246"
Nov 01 14:23:43.354641 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:43.354546    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/406ec2b7-3f8e-40a5-ad35-07a493ba0052-kube-api-access-ggf2l\" (UniqueName: \"kubernetes.io/projected/406ec2b7-3f8e-40a5-ad35-07a493ba0052-kube-api-access-ggf2l\") pod \"406ec2b7-3f8e-40a5-ad35-07a493ba0052\" (UID: \"406ec2b7-3f8e-40a5-ad35-07a493ba0052\") "
Nov 01 14:23:43.354812 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:43.354673    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/406ec2b7-3f8e-40a5-ad35-07a493ba0052-webhook-cert\" (UniqueName: \"kubernetes.io/secret/406ec2b7-3f8e-40a5-ad35-07a493ba0052-webhook-cert\") pod \"406ec2b7-3f8e-40a5-ad35-07a493ba0052\" (UID: \"406ec2b7-3f8e-40a5-ad35-07a493ba0052\") "
Nov 01 14:23:43.354812 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:43.354693    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/406ec2b7-3f8e-40a5-ad35-07a493ba0052-metrics-certs\" (UniqueName: \"kubernetes.io/secret/406ec2b7-3f8e-40a5-ad35-07a493ba0052-metrics-certs\") pod \"406ec2b7-3f8e-40a5-ad35-07a493ba0052\" (UID: \"406ec2b7-3f8e-40a5-ad35-07a493ba0052\") "
Nov 01 14:23:43.361204 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:43.357759    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/406ec2b7-3f8e-40a5-ad35-07a493ba0052-metrics-certs" pod "406ec2b7-3f8e-40a5-ad35-07a493ba0052" (UID: "406ec2b7-3f8e-40a5-ad35-07a493ba0052"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:23:43.361204 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:43.357848    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/406ec2b7-3f8e-40a5-ad35-07a493ba0052-webhook-cert" pod "406ec2b7-3f8e-40a5-ad35-07a493ba0052" (UID: "406ec2b7-3f8e-40a5-ad35-07a493ba0052"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:23:43.361204 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:43.359093    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/406ec2b7-3f8e-40a5-ad35-07a493ba0052-kube-api-access-ggf2l" pod "406ec2b7-3f8e-40a5-ad35-07a493ba0052" (UID: "406ec2b7-3f8e-40a5-ad35-07a493ba0052"). InnerVolumeSpecName "kube-api-access-ggf2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:23:43.455970 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:43.455933    4993 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/406ec2b7-3f8e-40a5-ad35-07a493ba0052-webhook-cert\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:23:43.455970 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:43.455966    4993 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/406ec2b7-3f8e-40a5-ad35-07a493ba0052-metrics-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:23:43.456212 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:43.455979    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggf2l\" (UniqueName: \"kubernetes.io/projected/406ec2b7-3f8e-40a5-ad35-07a493ba0052-kube-api-access-ggf2l\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:23:43.682252 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:43.682141    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6fcc8fbfd5-7c246" event={"ID":"406ec2b7-3f8e-40a5-ad35-07a493ba0052","Type":"ContainerDied","Data":"a2d8825eb322c916818a89c618b1bcff74ea86a43a352bc47e3d1ceba2ccf06b"}
Nov 01 14:23:43.682252 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:43.682215    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6fcc8fbfd5-7c246"
Nov 01 14:23:43.682252 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:43.682247    4993 scope.go:122] "RemoveContainer" containerID="741bd3d29e8be55b957b826d6c1af966c73c305e46d902920190014b918e2852"
Nov 01 14:23:43.835676 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:43.835638    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6fcc8fbfd5-7c246"]
Nov 01 14:23:43.881900 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:43.881858    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6fcc8fbfd5-7c246"]
Nov 01 14:23:44.795204 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:44.795149    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="406ec2b7-3f8e-40a5-ad35-07a493ba0052" path="/var/lib/kubelet/pods/406ec2b7-3f8e-40a5-ad35-07a493ba0052/volumes"
Nov 01 14:23:48.713336 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:48.713213    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cmbwq" event={"ID":"6489d076-e1bd-402d-9d85-4e3bce4123b0","Type":"ContainerStarted","Data":"3faa371f9f7e3c0dd029509808e5f3588530d8583c8ee71105baf56825e49b98"}
Nov 01 14:23:48.713777 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:48.713433    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cmbwq"
Nov 01 14:23:48.820477 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:48.820419    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cmbwq" podStartSLOduration=2.395451974 podStartE2EDuration="15.820404538s" totalImagesPullingTime="13.424952564s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:23:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:23:48.818841453 +0000 UTC m=+416.148704190" watchObservedRunningTime="2026-11-01 14:23:48.820404538 +0000 UTC m=+416.150267275"
Nov 01 14:23:57.499010 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:23:57.498969    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:23:59.724946 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:23:59.724905    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cmbwq"
Nov 01 14:24:05.384656 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:05.384613    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-74c9bbb9f8-fftw7"]
Nov 01 14:24:05.385094 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:05.385071    4993 cpu_manager.go:414] "containerMap: removing container" podUID="406ec2b7-3f8e-40a5-ad35-07a493ba0052" containerName="manager"
Nov 01 14:24:05.385143 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:05.385096    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="406ec2b7-3f8e-40a5-ad35-07a493ba0052" containerName="manager"
Nov 01 14:24:05.385228 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:05.385217    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="406ec2b7-3f8e-40a5-ad35-07a493ba0052" containerName="manager"
Nov 01 14:24:05.390874 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:05.390852    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-74c9bbb9f8-fftw7"
Nov 01 14:24:05.439841 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:05.439804    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 14:24:05.481506 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:05.481467    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/870a2497-655d-427d-8ce9-8b0160fc34db-maas-api-tls\") pod \"maas-api-74c9bbb9f8-fftw7\" (UID: \"870a2497-655d-427d-8ce9-8b0160fc34db\") " pod="odh-ai-gateway-infra/maas-api-74c9bbb9f8-fftw7"
Nov 01 14:24:05.481506 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:05.481504    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/870a2497-655d-427d-8ce9-8b0160fc34db-metrics-certs\") pod \"maas-api-74c9bbb9f8-fftw7\" (UID: \"870a2497-655d-427d-8ce9-8b0160fc34db\") " pod="odh-ai-gateway-infra/maas-api-74c9bbb9f8-fftw7"
Nov 01 14:24:05.481718 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:05.481543    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/870a2497-655d-427d-8ce9-8b0160fc34db-openshift-service-ca\") pod \"maas-api-74c9bbb9f8-fftw7\" (UID: \"870a2497-655d-427d-8ce9-8b0160fc34db\") " pod="odh-ai-gateway-infra/maas-api-74c9bbb9f8-fftw7"
Nov 01 14:24:05.481718 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:05.481569    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgwkw\" (UniqueName: \"kubernetes.io/projected/870a2497-655d-427d-8ce9-8b0160fc34db-kube-api-access-kgwkw\") pod \"maas-api-74c9bbb9f8-fftw7\" (UID: \"870a2497-655d-427d-8ce9-8b0160fc34db\") " pod="odh-ai-gateway-infra/maas-api-74c9bbb9f8-fftw7"
Nov 01 14:24:05.486282 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:05.486251    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Nov 01 14:24:05.486445 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:05.486337    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 14:24:05.486445 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:05.486362    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7t7jm\""
Nov 01 14:24:05.486552 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:05.486459    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 01 14:24:05.488804 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:05.488777    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-74c9bbb9f8-fftw7"]
Nov 01 14:24:05.582975 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:05.582938    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/870a2497-655d-427d-8ce9-8b0160fc34db-openshift-service-ca\") pod \"maas-api-74c9bbb9f8-fftw7\" (UID: \"870a2497-655d-427d-8ce9-8b0160fc34db\") " pod="odh-ai-gateway-infra/maas-api-74c9bbb9f8-fftw7"
Nov 01 14:24:05.582975 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:05.582977    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgwkw\" (UniqueName: \"kubernetes.io/projected/870a2497-655d-427d-8ce9-8b0160fc34db-kube-api-access-kgwkw\") pod \"maas-api-74c9bbb9f8-fftw7\" (UID: \"870a2497-655d-427d-8ce9-8b0160fc34db\") " pod="odh-ai-gateway-infra/maas-api-74c9bbb9f8-fftw7"
Nov 01 14:24:05.583265 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:05.583051    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/870a2497-655d-427d-8ce9-8b0160fc34db-maas-api-tls\") pod \"maas-api-74c9bbb9f8-fftw7\" (UID: \"870a2497-655d-427d-8ce9-8b0160fc34db\") " pod="odh-ai-gateway-infra/maas-api-74c9bbb9f8-fftw7"
Nov 01 14:24:05.583265 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:05.583069    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/870a2497-655d-427d-8ce9-8b0160fc34db-metrics-certs\") pod \"maas-api-74c9bbb9f8-fftw7\" (UID: \"870a2497-655d-427d-8ce9-8b0160fc34db\") " pod="odh-ai-gateway-infra/maas-api-74c9bbb9f8-fftw7"
Nov 01 14:24:05.583687 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:05.583658    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/870a2497-655d-427d-8ce9-8b0160fc34db-openshift-service-ca\") pod \"maas-api-74c9bbb9f8-fftw7\" (UID: \"870a2497-655d-427d-8ce9-8b0160fc34db\") " pod="odh-ai-gateway-infra/maas-api-74c9bbb9f8-fftw7"
Nov 01 14:24:05.585557 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:05.585537    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/870a2497-655d-427d-8ce9-8b0160fc34db-metrics-certs\") pod \"maas-api-74c9bbb9f8-fftw7\" (UID: \"870a2497-655d-427d-8ce9-8b0160fc34db\") " pod="odh-ai-gateway-infra/maas-api-74c9bbb9f8-fftw7"
Nov 01 14:24:05.585644 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:05.585573    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/870a2497-655d-427d-8ce9-8b0160fc34db-maas-api-tls\") pod \"maas-api-74c9bbb9f8-fftw7\" (UID: \"870a2497-655d-427d-8ce9-8b0160fc34db\") " pod="odh-ai-gateway-infra/maas-api-74c9bbb9f8-fftw7"
Nov 01 14:24:05.637341 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:05.637274    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgwkw\" (UniqueName: \"kubernetes.io/projected/870a2497-655d-427d-8ce9-8b0160fc34db-kube-api-access-kgwkw\") pod \"maas-api-74c9bbb9f8-fftw7\" (UID: \"870a2497-655d-427d-8ce9-8b0160fc34db\") " pod="odh-ai-gateway-infra/maas-api-74c9bbb9f8-fftw7"
Nov 01 14:24:05.703084 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:05.703049    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-74c9bbb9f8-fftw7"
Nov 01 14:24:06.072333 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:06.072297    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-74c9bbb9f8-fftw7"]
Nov 01 14:24:06.078999 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:24:06.078968    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod870a2497_655d_427d_8ce9_8b0160fc34db.slice/crio-09b8c8a1dd8f6dad2838fc2710b24a671b076fc9959bb8e97946fe0565dae92a WatchSource:0}: Error finding container 09b8c8a1dd8f6dad2838fc2710b24a671b076fc9959bb8e97946fe0565dae92a: Status 404 returned error can't find the container with id 09b8c8a1dd8f6dad2838fc2710b24a671b076fc9959bb8e97946fe0565dae92a
Nov 01 14:24:06.639973 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:06.639937    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-hg8lm"]
Nov 01 14:24:06.644686 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:06.644666    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-hg8lm"
Nov 01 14:24:06.692051 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:06.692011    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlbqq\" (UniqueName: \"kubernetes.io/projected/4a9c6ca9-d1f4-41b1-9496-2e32c99edfce-kube-api-access-xlbqq\") pod \"kserve-controller-manager-776cc66d47-hg8lm\" (UID: \"4a9c6ca9-d1f4-41b1-9496-2e32c99edfce\") " pod="opendatahub/kserve-controller-manager-776cc66d47-hg8lm"
Nov 01 14:24:06.692266 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:06.692081    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4a9c6ca9-d1f4-41b1-9496-2e32c99edfce-cert\") pod \"kserve-controller-manager-776cc66d47-hg8lm\" (UID: \"4a9c6ca9-d1f4-41b1-9496-2e32c99edfce\") " pod="opendatahub/kserve-controller-manager-776cc66d47-hg8lm"
Nov 01 14:24:06.692973 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:06.692950    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-wfdcm\""
Nov 01 14:24:06.693025 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:06.692954    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Nov 01 14:24:06.713088 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:06.713052    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-hg8lm"]
Nov 01 14:24:06.739594 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:06.739551    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-sgznq"]
Nov 01 14:24:06.743117 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:06.743091    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sgznq"
Nov 01 14:24:06.792901 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:06.792865    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps5cn\" (UniqueName: \"kubernetes.io/projected/c3c95345-bbd5-4951-9611-5a95cc49559f-kube-api-access-ps5cn\") pod \"llmisvc-controller-manager-869d5dcc5d-sgznq\" (UID: \"c3c95345-bbd5-4951-9611-5a95cc49559f\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sgznq"
Nov 01 14:24:06.793098 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:06.792930    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c3c95345-bbd5-4951-9611-5a95cc49559f-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-sgznq\" (UID: \"c3c95345-bbd5-4951-9611-5a95cc49559f\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sgznq"
Nov 01 14:24:06.793098 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:06.793026    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4a9c6ca9-d1f4-41b1-9496-2e32c99edfce-cert\") pod \"kserve-controller-manager-776cc66d47-hg8lm\" (UID: \"4a9c6ca9-d1f4-41b1-9496-2e32c99edfce\") " pod="opendatahub/kserve-controller-manager-776cc66d47-hg8lm"
Nov 01 14:24:06.793234 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:06.793188    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlbqq\" (UniqueName: \"kubernetes.io/projected/4a9c6ca9-d1f4-41b1-9496-2e32c99edfce-kube-api-access-xlbqq\") pod \"kserve-controller-manager-776cc66d47-hg8lm\" (UID: \"4a9c6ca9-d1f4-41b1-9496-2e32c99edfce\") " pod="opendatahub/kserve-controller-manager-776cc66d47-hg8lm"
Nov 01 14:24:06.794329 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:06.794306    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Nov 01 14:24:06.794492 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:06.794388    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-ngcfs\""
Nov 01 14:24:06.795956 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:06.795932    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4a9c6ca9-d1f4-41b1-9496-2e32c99edfce-cert\") pod \"kserve-controller-manager-776cc66d47-hg8lm\" (UID: \"4a9c6ca9-d1f4-41b1-9496-2e32c99edfce\") " pod="opendatahub/kserve-controller-manager-776cc66d47-hg8lm"
Nov 01 14:24:06.799026 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:06.798999    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-74c9bbb9f8-fftw7" event={"ID":"870a2497-655d-427d-8ce9-8b0160fc34db","Type":"ContainerStarted","Data":"09b8c8a1dd8f6dad2838fc2710b24a671b076fc9959bb8e97946fe0565dae92a"}
Nov 01 14:24:06.802083 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:06.802062    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-sgznq"]
Nov 01 14:24:06.845776 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:06.845740    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-lr4hb"]
Nov 01 14:24:06.849768 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:06.849742    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-lr4hb"
Nov 01 14:24:06.853018 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:06.852993    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlbqq\" (UniqueName: \"kubernetes.io/projected/4a9c6ca9-d1f4-41b1-9496-2e32c99edfce-kube-api-access-xlbqq\") pod \"kserve-controller-manager-776cc66d47-hg8lm\" (UID: \"4a9c6ca9-d1f4-41b1-9496-2e32c99edfce\") " pod="opendatahub/kserve-controller-manager-776cc66d47-hg8lm"
Nov 01 14:24:06.894823 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:06.894723    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps5cn\" (UniqueName: \"kubernetes.io/projected/c3c95345-bbd5-4951-9611-5a95cc49559f-kube-api-access-ps5cn\") pod \"llmisvc-controller-manager-869d5dcc5d-sgznq\" (UID: \"c3c95345-bbd5-4951-9611-5a95cc49559f\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sgznq"
Nov 01 14:24:06.894823 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:06.894798    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc5xt\" (UniqueName: \"kubernetes.io/projected/6c32c11d-d77f-458d-9cf0-fe262d31aeeb-kube-api-access-nc5xt\") pod \"model-serving-api-75bd89fdfc-lr4hb\" (UID: \"6c32c11d-d77f-458d-9cf0-fe262d31aeeb\") " pod="opendatahub/model-serving-api-75bd89fdfc-lr4hb"
Nov 01 14:24:06.895072 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:06.894880    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c3c95345-bbd5-4951-9611-5a95cc49559f-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-sgznq\" (UID: \"c3c95345-bbd5-4951-9611-5a95cc49559f\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sgznq"
Nov 01 14:24:06.895072 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:06.894999    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c32c11d-d77f-458d-9cf0-fe262d31aeeb-tls-certs\") pod \"model-serving-api-75bd89fdfc-lr4hb\" (UID: \"6c32c11d-d77f-458d-9cf0-fe262d31aeeb\") " pod="opendatahub/model-serving-api-75bd89fdfc-lr4hb"
Nov 01 14:24:06.897992 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:06.897948    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c3c95345-bbd5-4951-9611-5a95cc49559f-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-sgznq\" (UID: \"c3c95345-bbd5-4951-9611-5a95cc49559f\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sgznq"
Nov 01 14:24:06.898455 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:06.898433    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-n46gx\""
Nov 01 14:24:06.898563 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:06.898469    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Nov 01 14:24:06.908914 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:06.908885    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-lr4hb"]
Nov 01 14:24:06.950323 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:06.950286    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps5cn\" (UniqueName: \"kubernetes.io/projected/c3c95345-bbd5-4951-9611-5a95cc49559f-kube-api-access-ps5cn\") pod \"llmisvc-controller-manager-869d5dcc5d-sgznq\" (UID: \"c3c95345-bbd5-4951-9611-5a95cc49559f\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sgznq"
Nov 01 14:24:06.958969 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:06.958926    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-hg8lm"
Nov 01 14:24:06.996130 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:06.996092    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc5xt\" (UniqueName: \"kubernetes.io/projected/6c32c11d-d77f-458d-9cf0-fe262d31aeeb-kube-api-access-nc5xt\") pod \"model-serving-api-75bd89fdfc-lr4hb\" (UID: \"6c32c11d-d77f-458d-9cf0-fe262d31aeeb\") " pod="opendatahub/model-serving-api-75bd89fdfc-lr4hb"
Nov 01 14:24:06.996360 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:06.996154    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c32c11d-d77f-458d-9cf0-fe262d31aeeb-tls-certs\") pod \"model-serving-api-75bd89fdfc-lr4hb\" (UID: \"6c32c11d-d77f-458d-9cf0-fe262d31aeeb\") " pod="opendatahub/model-serving-api-75bd89fdfc-lr4hb"
Nov 01 14:24:06.998757 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:06.998731    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c32c11d-d77f-458d-9cf0-fe262d31aeeb-tls-certs\") pod \"model-serving-api-75bd89fdfc-lr4hb\" (UID: \"6c32c11d-d77f-458d-9cf0-fe262d31aeeb\") " pod="opendatahub/model-serving-api-75bd89fdfc-lr4hb"
Nov 01 14:24:07.049472 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:07.049441    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc5xt\" (UniqueName: \"kubernetes.io/projected/6c32c11d-d77f-458d-9cf0-fe262d31aeeb-kube-api-access-nc5xt\") pod \"model-serving-api-75bd89fdfc-lr4hb\" (UID: \"6c32c11d-d77f-458d-9cf0-fe262d31aeeb\") " pod="opendatahub/model-serving-api-75bd89fdfc-lr4hb"
Nov 01 14:24:07.056338 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:07.056305    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sgznq"
Nov 01 14:24:07.174450 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:07.174362    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-lr4hb"
Nov 01 14:24:07.331701 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:07.331665    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-hg8lm"]
Nov 01 14:24:07.339060 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:24:07.339024    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a9c6ca9_d1f4_41b1_9496_2e32c99edfce.slice/crio-dbd8eb7f00145556325a90f4b743d7fcedb9ed0bb61215bbecf7eaea2ce206f3 WatchSource:0}: Error finding container dbd8eb7f00145556325a90f4b743d7fcedb9ed0bb61215bbecf7eaea2ce206f3: Status 404 returned error can't find the container with id dbd8eb7f00145556325a90f4b743d7fcedb9ed0bb61215bbecf7eaea2ce206f3
Nov 01 14:24:07.435393 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:07.435255    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-sgznq"]
Nov 01 14:24:07.441335 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:24:07.441308    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc3c95345_bbd5_4951_9611_5a95cc49559f.slice/crio-0a66d5060047d8c24105960b497dddad76023a942adfa2f689e61d7740951ee8 WatchSource:0}: Error finding container 0a66d5060047d8c24105960b497dddad76023a942adfa2f689e61d7740951ee8: Status 404 returned error can't find the container with id 0a66d5060047d8c24105960b497dddad76023a942adfa2f689e61d7740951ee8
Nov 01 14:24:07.546111 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:07.546046    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-lr4hb"]
Nov 01 14:24:07.552884 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:24:07.552851    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c32c11d_d77f_458d_9cf0_fe262d31aeeb.slice/crio-3078c23a8b4f5d11c4118707586b61b9253d71e2ff950cc25a770e36d857d2de WatchSource:0}: Error finding container 3078c23a8b4f5d11c4118707586b61b9253d71e2ff950cc25a770e36d857d2de: Status 404 returned error can't find the container with id 3078c23a8b4f5d11c4118707586b61b9253d71e2ff950cc25a770e36d857d2de
Nov 01 14:24:07.806610 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:07.806567    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sgznq" event={"ID":"c3c95345-bbd5-4951-9611-5a95cc49559f","Type":"ContainerStarted","Data":"0a66d5060047d8c24105960b497dddad76023a942adfa2f689e61d7740951ee8"}
Nov 01 14:24:07.808305 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:07.808259    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-hg8lm" event={"ID":"4a9c6ca9-d1f4-41b1-9496-2e32c99edfce","Type":"ContainerStarted","Data":"dbd8eb7f00145556325a90f4b743d7fcedb9ed0bb61215bbecf7eaea2ce206f3"}
Nov 01 14:24:07.809712 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:07.809688    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-lr4hb" event={"ID":"6c32c11d-d77f-458d-9cf0-fe262d31aeeb","Type":"ContainerStarted","Data":"3078c23a8b4f5d11c4118707586b61b9253d71e2ff950cc25a770e36d857d2de"}
Nov 01 14:24:08.819880 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:08.819831    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-74c9bbb9f8-fftw7" event={"ID":"870a2497-655d-427d-8ce9-8b0160fc34db","Type":"ContainerStarted","Data":"eb1849da125d465d940a3bdf0ce23c343616ff3debf4e166e319987d0464efc8"}
Nov 01 14:24:08.820409 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:08.820131    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-74c9bbb9f8-fftw7"
Nov 01 14:24:08.930814 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:08.930746    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-74c9bbb9f8-fftw7" podStartSLOduration=1.88902758 podStartE2EDuration="3.93072589s" totalImagesPullingTime="2.04169831s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:24:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:24:08.926636675 +0000 UTC m=+436.256499413" watchObservedRunningTime="2026-11-01 14:24:08.93072589 +0000 UTC m=+436.260588629"
Nov 01 14:24:10.833085 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:10.833048    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-lr4hb" event={"ID":"6c32c11d-d77f-458d-9cf0-fe262d31aeeb","Type":"ContainerStarted","Data":"648c0d8e8a281ed0d1d802e78af73e284c37fb297d7db8c1cda3ed6f158a9bb8"}
Nov 01 14:24:10.833608 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:10.833149    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-lr4hb"
Nov 01 14:24:10.834803 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:10.834770    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sgznq" event={"ID":"c3c95345-bbd5-4951-9611-5a95cc49559f","Type":"ContainerStarted","Data":"b3ad66ef67bc9ed9c61ecf1356106a60d65e33f6c271c785434c800cd51a1e2f"}
Nov 01 14:24:10.834937 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:10.834828    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sgznq"
Nov 01 14:24:10.836161 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:10.836140    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-hg8lm" event={"ID":"4a9c6ca9-d1f4-41b1-9496-2e32c99edfce","Type":"ContainerStarted","Data":"46c89436309543ebdb491264fa03029e058d9c9cb7e190db0ba545a77341a68e"}
Nov 01 14:24:10.836309 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:10.836290    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-hg8lm"
Nov 01 14:24:10.944740 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:10.944689    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-lr4hb" podStartSLOduration=2.078318499 podStartE2EDuration="4.944676154s" totalImagesPullingTime="2.866357655s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:24:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:24:10.942613768 +0000 UTC m=+438.272476507" watchObservedRunningTime="2026-11-01 14:24:10.944676154 +0000 UTC m=+438.274538890"
Nov 01 14:24:11.052761 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:11.052567    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-hg8lm" podStartSLOduration=2.037334569 podStartE2EDuration="5.052545965s" totalImagesPullingTime="3.015211396s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:24:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:24:11.05061352 +0000 UTC m=+438.380476264" watchObservedRunningTime="2026-11-01 14:24:11.052545965 +0000 UTC m=+438.382408703"
Nov 01 14:24:11.170525 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:11.170383    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sgznq" podStartSLOduration=2.326907301 podStartE2EDuration="5.170363704s" totalImagesPullingTime="2.843456403s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:24:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:24:11.168705939 +0000 UTC m=+438.498568679" watchObservedRunningTime="2026-11-01 14:24:11.170363704 +0000 UTC m=+438.500226440"
Nov 01 14:24:14.832470 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:14.832434    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-74c9bbb9f8-fftw7"
Nov 01 14:24:15.979290 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:15.978225    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d9b94f465-h7lvb"]
Nov 01 14:24:16.089121 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:16.089084    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d9b94f465-h7lvb"]
Nov 01 14:24:16.089121 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:16.089127    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d9b94f465-h7lvb"
Nov 01 14:24:16.205993 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:16.205958    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpplm\" (UniqueName: \"kubernetes.io/projected/7aa3200c-bf3b-410a-a002-9033e60bbfc6-kube-api-access-kpplm\") pod \"authorino-5d9b94f465-h7lvb\" (UID: \"7aa3200c-bf3b-410a-a002-9033e60bbfc6\") " pod="kuadrant-system/authorino-5d9b94f465-h7lvb"
Nov 01 14:24:16.307262 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:16.307227    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpplm\" (UniqueName: \"kubernetes.io/projected/7aa3200c-bf3b-410a-a002-9033e60bbfc6-kube-api-access-kpplm\") pod \"authorino-5d9b94f465-h7lvb\" (UID: \"7aa3200c-bf3b-410a-a002-9033e60bbfc6\") " pod="kuadrant-system/authorino-5d9b94f465-h7lvb"
Nov 01 14:24:16.380559 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:16.378976    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpplm\" (UniqueName: \"kubernetes.io/projected/7aa3200c-bf3b-410a-a002-9033e60bbfc6-kube-api-access-kpplm\") pod \"authorino-5d9b94f465-h7lvb\" (UID: \"7aa3200c-bf3b-410a-a002-9033e60bbfc6\") " pod="kuadrant-system/authorino-5d9b94f465-h7lvb"
Nov 01 14:24:16.411586 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:16.411553    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d9b94f465-h7lvb"]
Nov 01 14:24:16.411759 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:16.411747    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d9b94f465-h7lvb"
Nov 01 14:24:16.438724 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:16.438692    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-687bbd6b9f-md8j6"]
Nov 01 14:24:16.456575 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:16.456551    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bbd6b9f-md8j6"
Nov 01 14:24:16.499269 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:16.499240    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-687bbd6b9f-md8j6"]
Nov 01 14:24:16.512075 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:16.505738    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Nov 01 14:24:16.610672 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:16.610594    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blkkp\" (UniqueName: \"kubernetes.io/projected/1c2096b1-adf7-4291-9423-adbab7aa4449-kube-api-access-blkkp\") pod \"authorino-687bbd6b9f-md8j6\" (UID: \"1c2096b1-adf7-4291-9423-adbab7aa4449\") " pod="kuadrant-system/authorino-687bbd6b9f-md8j6"
Nov 01 14:24:16.610672 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:16.610641    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1c2096b1-adf7-4291-9423-adbab7aa4449-tls-cert\") pod \"authorino-687bbd6b9f-md8j6\" (UID: \"1c2096b1-adf7-4291-9423-adbab7aa4449\") " pod="kuadrant-system/authorino-687bbd6b9f-md8j6"
Nov 01 14:24:16.711715 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:16.711686    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blkkp\" (UniqueName: \"kubernetes.io/projected/1c2096b1-adf7-4291-9423-adbab7aa4449-kube-api-access-blkkp\") pod \"authorino-687bbd6b9f-md8j6\" (UID: \"1c2096b1-adf7-4291-9423-adbab7aa4449\") " pod="kuadrant-system/authorino-687bbd6b9f-md8j6"
Nov 01 14:24:16.711881 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:16.711722    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1c2096b1-adf7-4291-9423-adbab7aa4449-tls-cert\") pod \"authorino-687bbd6b9f-md8j6\" (UID: \"1c2096b1-adf7-4291-9423-adbab7aa4449\") " pod="kuadrant-system/authorino-687bbd6b9f-md8j6"
Nov 01 14:24:16.714033 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:16.714009    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1c2096b1-adf7-4291-9423-adbab7aa4449-tls-cert\") pod \"authorino-687bbd6b9f-md8j6\" (UID: \"1c2096b1-adf7-4291-9423-adbab7aa4449\") " pod="kuadrant-system/authorino-687bbd6b9f-md8j6"
Nov 01 14:24:16.765138 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:16.765110    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blkkp\" (UniqueName: \"kubernetes.io/projected/1c2096b1-adf7-4291-9423-adbab7aa4449-kube-api-access-blkkp\") pod \"authorino-687bbd6b9f-md8j6\" (UID: \"1c2096b1-adf7-4291-9423-adbab7aa4449\") " pod="kuadrant-system/authorino-687bbd6b9f-md8j6"
Nov 01 14:24:16.825227 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:16.825198    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bbd6b9f-md8j6"
Nov 01 14:24:16.834668 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:16.834636    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d9b94f465-h7lvb"]
Nov 01 14:24:16.841789 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:24:16.841760    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7aa3200c_bf3b_410a_a002_9033e60bbfc6.slice/crio-b8572682eedb8bc455f4d53c45155209f31cf3ccf412f30539531f44b6aa4e1b WatchSource:0}: Error finding container b8572682eedb8bc455f4d53c45155209f31cf3ccf412f30539531f44b6aa4e1b: Status 404 returned error can't find the container with id b8572682eedb8bc455f4d53c45155209f31cf3ccf412f30539531f44b6aa4e1b
Nov 01 14:24:16.898695 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:16.898571    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d9b94f465-h7lvb" event={"ID":"7aa3200c-bf3b-410a-a002-9033e60bbfc6","Type":"ContainerStarted","Data":"b8572682eedb8bc455f4d53c45155209f31cf3ccf412f30539531f44b6aa4e1b"}
Nov 01 14:24:17.217332 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:17.217244    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-687bbd6b9f-md8j6"]
Nov 01 14:24:17.222183 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:24:17.222139    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1c2096b1_adf7_4291_9423_adbab7aa4449.slice/crio-daf0f7aa490df89fe4eafcc727f13c4ae5837206e73f19869e4bde6dae7ba0fb WatchSource:0}: Error finding container daf0f7aa490df89fe4eafcc727f13c4ae5837206e73f19869e4bde6dae7ba0fb: Status 404 returned error can't find the container with id daf0f7aa490df89fe4eafcc727f13c4ae5837206e73f19869e4bde6dae7ba0fb
Nov 01 14:24:17.906110 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:17.906067    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-687bbd6b9f-md8j6" event={"ID":"1c2096b1-adf7-4291-9423-adbab7aa4449","Type":"ContainerStarted","Data":"daf0f7aa490df89fe4eafcc727f13c4ae5837206e73f19869e4bde6dae7ba0fb"}
Nov 01 14:24:18.912741 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:18.912692    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d9b94f465-h7lvb" event={"ID":"7aa3200c-bf3b-410a-a002-9033e60bbfc6","Type":"ContainerStarted","Data":"a31754d2d8774f3e6e3aeca7e2eed26e93b3bcb594901a930e1e958ebee32be0"}
Nov 01 14:24:18.913220 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:18.912722    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-5d9b94f465-h7lvb" podUID="7aa3200c-bf3b-410a-a002-9033e60bbfc6" containerName="authorino" containerID="cri-o://a31754d2d8774f3e6e3aeca7e2eed26e93b3bcb594901a930e1e958ebee32be0" gracePeriod=30
Nov 01 14:24:19.018086 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:19.018025    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-5d9b94f465-h7lvb" podStartSLOduration=2.503814912 podStartE2EDuration="4.018011706s" totalImagesPullingTime="1.514196794s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:24:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:24:19.016787687 +0000 UTC m=+446.346650428" watchObservedRunningTime="2026-11-01 14:24:19.018011706 +0000 UTC m=+446.347874442"
Nov 01 14:24:19.205142 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:19.205118    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d9b94f465-h7lvb"
Nov 01 14:24:19.337311 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:19.337270    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7aa3200c-bf3b-410a-a002-9033e60bbfc6-kube-api-access-kpplm\" (UniqueName: \"kubernetes.io/projected/7aa3200c-bf3b-410a-a002-9033e60bbfc6-kube-api-access-kpplm\") pod \"7aa3200c-bf3b-410a-a002-9033e60bbfc6\" (UID: \"7aa3200c-bf3b-410a-a002-9033e60bbfc6\") "
Nov 01 14:24:19.339972 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:19.339936    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aa3200c-bf3b-410a-a002-9033e60bbfc6-kube-api-access-kpplm" pod "7aa3200c-bf3b-410a-a002-9033e60bbfc6" (UID: "7aa3200c-bf3b-410a-a002-9033e60bbfc6"). InnerVolumeSpecName "kube-api-access-kpplm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:24:19.438403 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:19.438325    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpplm\" (UniqueName: \"kubernetes.io/projected/7aa3200c-bf3b-410a-a002-9033e60bbfc6-kube-api-access-kpplm\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:24:19.921259 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:19.921216    4993 generic.go:356] "Generic (PLEG): container finished" podID="7aa3200c-bf3b-410a-a002-9033e60bbfc6" containerID="a31754d2d8774f3e6e3aeca7e2eed26e93b3bcb594901a930e1e958ebee32be0" exitCode=0
Nov 01 14:24:19.921812 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:19.921279    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d9b94f465-h7lvb"
Nov 01 14:24:19.921812 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:19.921310    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d9b94f465-h7lvb" event={"ID":"7aa3200c-bf3b-410a-a002-9033e60bbfc6","Type":"ContainerDied","Data":"a31754d2d8774f3e6e3aeca7e2eed26e93b3bcb594901a930e1e958ebee32be0"}
Nov 01 14:24:19.921812 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:19.921361    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d9b94f465-h7lvb" event={"ID":"7aa3200c-bf3b-410a-a002-9033e60bbfc6","Type":"ContainerDied","Data":"b8572682eedb8bc455f4d53c45155209f31cf3ccf412f30539531f44b6aa4e1b"}
Nov 01 14:24:19.921812 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:19.921384    4993 scope.go:122] "RemoveContainer" containerID="a31754d2d8774f3e6e3aeca7e2eed26e93b3bcb594901a930e1e958ebee32be0"
Nov 01 14:24:19.923545 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:19.923518    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-687bbd6b9f-md8j6" event={"ID":"1c2096b1-adf7-4291-9423-adbab7aa4449","Type":"ContainerStarted","Data":"022618c0a17e939ad2bdd0b0bde51db9b9df807b21758db810af272ed94ec02b"}
Nov 01 14:24:19.933408 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:19.933389    4993 scope.go:122] "RemoveContainer" containerID="a31754d2d8774f3e6e3aeca7e2eed26e93b3bcb594901a930e1e958ebee32be0"
Nov 01 14:24:19.933704 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:24:19.933681    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a31754d2d8774f3e6e3aeca7e2eed26e93b3bcb594901a930e1e958ebee32be0\": container with ID starting with a31754d2d8774f3e6e3aeca7e2eed26e93b3bcb594901a930e1e958ebee32be0 not found: ID does not exist" containerID="a31754d2d8774f3e6e3aeca7e2eed26e93b3bcb594901a930e1e958ebee32be0"
Nov 01 14:24:19.933763 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:19.933718    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a31754d2d8774f3e6e3aeca7e2eed26e93b3bcb594901a930e1e958ebee32be0"} err="failed to get container status \"a31754d2d8774f3e6e3aeca7e2eed26e93b3bcb594901a930e1e958ebee32be0\": rpc error: code = NotFound desc = could not find container \"a31754d2d8774f3e6e3aeca7e2eed26e93b3bcb594901a930e1e958ebee32be0\": container with ID starting with a31754d2d8774f3e6e3aeca7e2eed26e93b3bcb594901a930e1e958ebee32be0 not found: ID does not exist"
Nov 01 14:24:20.076148 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:20.076111    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d9b94f465-h7lvb"]
Nov 01 14:24:20.081139 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:20.081106    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d9b94f465-h7lvb"]
Nov 01 14:24:20.182398 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:20.182299    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-687bbd6b9f-md8j6" podStartSLOduration=2.288616275 podStartE2EDuration="4.182279887s" totalImagesPullingTime="1.893663612s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:24:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:24:20.181987988 +0000 UTC m=+447.511850729" watchObservedRunningTime="2026-11-01 14:24:20.182279887 +0000 UTC m=+447.512142624"
Nov 01 14:24:20.371122 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:20.207751    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-gntc9"]
Nov 01 14:24:20.371122 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:20.207924    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-gntc9" podUID="f403c01e-299d-4a63-8545-65bc96a68ce1" containerName="authorino" containerID="cri-o://4afb0d832e98fe2a69bb42005625351953972c111a9878dba4f5bcdb09735e95" gracePeriod=30
Nov 01 14:24:20.513985 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:20.513956    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-gntc9"
Nov 01 14:24:20.549899 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:20.549871    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f403c01e-299d-4a63-8545-65bc96a68ce1-kube-api-access-9rv46\" (UniqueName: \"kubernetes.io/projected/f403c01e-299d-4a63-8545-65bc96a68ce1-kube-api-access-9rv46\") pod \"f403c01e-299d-4a63-8545-65bc96a68ce1\" (UID: \"f403c01e-299d-4a63-8545-65bc96a68ce1\") "
Nov 01 14:24:20.551911 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:20.551882    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f403c01e-299d-4a63-8545-65bc96a68ce1-kube-api-access-9rv46" pod "f403c01e-299d-4a63-8545-65bc96a68ce1" (UID: "f403c01e-299d-4a63-8545-65bc96a68ce1"). InnerVolumeSpecName "kube-api-access-9rv46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:24:20.650791 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:20.650748    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rv46\" (UniqueName: \"kubernetes.io/projected/f403c01e-299d-4a63-8545-65bc96a68ce1-kube-api-access-9rv46\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:24:20.794751 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:20.794716    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7aa3200c-bf3b-410a-a002-9033e60bbfc6" path="/var/lib/kubelet/pods/7aa3200c-bf3b-410a-a002-9033e60bbfc6/volumes"
Nov 01 14:24:20.930039 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:20.930005    4993 generic.go:356] "Generic (PLEG): container finished" podID="306b0405-fb73-435a-ae73-42a90cd2d2f9" containerID="40b21ca817d54c994d04a77445c7ab0386d60fd56e09caee8a15bd1ba81ef3d4" exitCode=0
Nov 01 14:24:20.930533 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:20.930079    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4pmtn" event={"ID":"306b0405-fb73-435a-ae73-42a90cd2d2f9","Type":"ContainerDied","Data":"40b21ca817d54c994d04a77445c7ab0386d60fd56e09caee8a15bd1ba81ef3d4"}
Nov 01 14:24:20.930603 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:20.930554    4993 scope.go:122] "RemoveContainer" containerID="40b21ca817d54c994d04a77445c7ab0386d60fd56e09caee8a15bd1ba81ef3d4"
Nov 01 14:24:20.931483 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:20.931462    4993 generic.go:356] "Generic (PLEG): container finished" podID="f403c01e-299d-4a63-8545-65bc96a68ce1" containerID="4afb0d832e98fe2a69bb42005625351953972c111a9878dba4f5bcdb09735e95" exitCode=0
Nov 01 14:24:20.931576 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:20.931524    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-gntc9"
Nov 01 14:24:20.931576 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:20.931543    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-gntc9" event={"ID":"f403c01e-299d-4a63-8545-65bc96a68ce1","Type":"ContainerDied","Data":"4afb0d832e98fe2a69bb42005625351953972c111a9878dba4f5bcdb09735e95"}
Nov 01 14:24:20.931576 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:20.931570    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-gntc9" event={"ID":"f403c01e-299d-4a63-8545-65bc96a68ce1","Type":"ContainerDied","Data":"409e4cafdfbbc6a017c12ec0aa246d1d14fcfc72afb9e95b35958bb43e9813dd"}
Nov 01 14:24:20.931686 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:20.931585    4993 scope.go:122] "RemoveContainer" containerID="4afb0d832e98fe2a69bb42005625351953972c111a9878dba4f5bcdb09735e95"
Nov 01 14:24:20.931755 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:20.931721    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 14:24:20.931873 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:20.931805    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 14:24:20.931873 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:20.931850    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 14:24:20.942628 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:20.942597    4993 scope.go:122] "RemoveContainer" containerID="4afb0d832e98fe2a69bb42005625351953972c111a9878dba4f5bcdb09735e95"
Nov 01 14:24:20.942916 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:24:20.942897    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4afb0d832e98fe2a69bb42005625351953972c111a9878dba4f5bcdb09735e95\": container with ID starting with 4afb0d832e98fe2a69bb42005625351953972c111a9878dba4f5bcdb09735e95 not found: ID does not exist" containerID="4afb0d832e98fe2a69bb42005625351953972c111a9878dba4f5bcdb09735e95"
Nov 01 14:24:20.942965 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:20.942923    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4afb0d832e98fe2a69bb42005625351953972c111a9878dba4f5bcdb09735e95"} err="failed to get container status \"4afb0d832e98fe2a69bb42005625351953972c111a9878dba4f5bcdb09735e95\": rpc error: code = NotFound desc = could not find container \"4afb0d832e98fe2a69bb42005625351953972c111a9878dba4f5bcdb09735e95\": container with ID starting with 4afb0d832e98fe2a69bb42005625351953972c111a9878dba4f5bcdb09735e95 not found: ID does not exist"
Nov 01 14:24:21.030351 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:21.030329    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4pmtn"
Nov 01 14:24:21.140592 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:21.140519    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-gntc9"]
Nov 01 14:24:21.144107 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:21.144074    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-gntc9"]
Nov 01 14:24:21.844811 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:21.844783    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-lr4hb"
Nov 01 14:24:21.946106 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:21.946070    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4pmtn" event={"ID":"306b0405-fb73-435a-ae73-42a90cd2d2f9","Type":"ContainerStarted","Data":"2c756d9c8039fac670ca29512bec769a0790f8f0142caf08280d33a905677089"}
Nov 01 14:24:22.794674 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:22.794642    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f403c01e-299d-4a63-8545-65bc96a68ce1" path="/var/lib/kubelet/pods/f403c01e-299d-4a63-8545-65bc96a68ce1/volumes"
Nov 01 14:24:22.806702 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:22.806674    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4pmtn"
Nov 01 14:24:22.807709 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:22.807686    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-4pmtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body=
Nov 01 14:24:22.807824 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:22.807728    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4pmtn" podUID="306b0405-fb73-435a-ae73-42a90cd2d2f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused"
Nov 01 14:24:23.807493 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:23.807458    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-4pmtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body=
Nov 01 14:24:23.807876 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:23.807517    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4pmtn" podUID="306b0405-fb73-435a-ae73-42a90cd2d2f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused"
Nov 01 14:24:24.807070 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:24.807037    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-4pmtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body=
Nov 01 14:24:24.807260 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:24.807090    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4pmtn" podUID="306b0405-fb73-435a-ae73-42a90cd2d2f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused"
Nov 01 14:24:25.669841 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:25.669801    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-74c9bbb9f8-fftw7"]
Nov 01 14:24:25.670648 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:25.670611    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-74c9bbb9f8-fftw7" podUID="870a2497-655d-427d-8ce9-8b0160fc34db" containerName="maas-api" containerID="cri-o://eb1849da125d465d940a3bdf0ce23c343616ff3debf4e166e319987d0464efc8" gracePeriod=30
Nov 01 14:24:25.806833 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:25.806802    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-4pmtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body=
Nov 01 14:24:25.806994 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:25.806866    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4pmtn" podUID="306b0405-fb73-435a-ae73-42a90cd2d2f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused"
Nov 01 14:24:25.940705 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:25.940679    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-74c9bbb9f8-fftw7"
Nov 01 14:24:25.983886 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:25.983850    4993 generic.go:356] "Generic (PLEG): container finished" podID="870a2497-655d-427d-8ce9-8b0160fc34db" containerID="eb1849da125d465d940a3bdf0ce23c343616ff3debf4e166e319987d0464efc8" exitCode=0
Nov 01 14:24:25.984046 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:25.983892    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-74c9bbb9f8-fftw7"
Nov 01 14:24:25.984046 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:25.983929    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-74c9bbb9f8-fftw7" event={"ID":"870a2497-655d-427d-8ce9-8b0160fc34db","Type":"ContainerDied","Data":"eb1849da125d465d940a3bdf0ce23c343616ff3debf4e166e319987d0464efc8"}
Nov 01 14:24:25.984046 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:25.983982    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-74c9bbb9f8-fftw7" event={"ID":"870a2497-655d-427d-8ce9-8b0160fc34db","Type":"ContainerDied","Data":"09b8c8a1dd8f6dad2838fc2710b24a671b076fc9959bb8e97946fe0565dae92a"}
Nov 01 14:24:25.984046 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:25.984006    4993 scope.go:122] "RemoveContainer" containerID="eb1849da125d465d940a3bdf0ce23c343616ff3debf4e166e319987d0464efc8"
Nov 01 14:24:25.993255 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:25.993238    4993 scope.go:122] "RemoveContainer" containerID="eb1849da125d465d940a3bdf0ce23c343616ff3debf4e166e319987d0464efc8"
Nov 01 14:24:25.993526 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:24:25.993507    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb1849da125d465d940a3bdf0ce23c343616ff3debf4e166e319987d0464efc8\": container with ID starting with eb1849da125d465d940a3bdf0ce23c343616ff3debf4e166e319987d0464efc8 not found: ID does not exist" containerID="eb1849da125d465d940a3bdf0ce23c343616ff3debf4e166e319987d0464efc8"
Nov 01 14:24:25.993599 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:25.993534    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb1849da125d465d940a3bdf0ce23c343616ff3debf4e166e319987d0464efc8"} err="failed to get container status \"eb1849da125d465d940a3bdf0ce23c343616ff3debf4e166e319987d0464efc8\": rpc error: code = NotFound desc = could not find container \"eb1849da125d465d940a3bdf0ce23c343616ff3debf4e166e319987d0464efc8\": container with ID starting with eb1849da125d465d940a3bdf0ce23c343616ff3debf4e166e319987d0464efc8 not found: ID does not exist"
Nov 01 14:24:26.013292 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:26.013264    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/870a2497-655d-427d-8ce9-8b0160fc34db-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/870a2497-655d-427d-8ce9-8b0160fc34db-maas-api-tls\") pod \"870a2497-655d-427d-8ce9-8b0160fc34db\" (UID: \"870a2497-655d-427d-8ce9-8b0160fc34db\") "
Nov 01 14:24:26.013403 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:26.013364    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/870a2497-655d-427d-8ce9-8b0160fc34db-metrics-certs\" (UniqueName: \"kubernetes.io/secret/870a2497-655d-427d-8ce9-8b0160fc34db-metrics-certs\") pod \"870a2497-655d-427d-8ce9-8b0160fc34db\" (UID: \"870a2497-655d-427d-8ce9-8b0160fc34db\") "
Nov 01 14:24:26.013403 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:26.013388    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/870a2497-655d-427d-8ce9-8b0160fc34db-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/870a2497-655d-427d-8ce9-8b0160fc34db-openshift-service-ca\") pod \"870a2497-655d-427d-8ce9-8b0160fc34db\" (UID: \"870a2497-655d-427d-8ce9-8b0160fc34db\") "
Nov 01 14:24:26.013477 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:26.013440    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/870a2497-655d-427d-8ce9-8b0160fc34db-kube-api-access-kgwkw\" (UniqueName: \"kubernetes.io/projected/870a2497-655d-427d-8ce9-8b0160fc34db-kube-api-access-kgwkw\") pod \"870a2497-655d-427d-8ce9-8b0160fc34db\" (UID: \"870a2497-655d-427d-8ce9-8b0160fc34db\") "
Nov 01 14:24:26.013781 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:26.013755    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/870a2497-655d-427d-8ce9-8b0160fc34db-openshift-service-ca" pod "870a2497-655d-427d-8ce9-8b0160fc34db" (UID: "870a2497-655d-427d-8ce9-8b0160fc34db"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:24:26.015615 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:26.015482    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/870a2497-655d-427d-8ce9-8b0160fc34db-kube-api-access-kgwkw" pod "870a2497-655d-427d-8ce9-8b0160fc34db" (UID: "870a2497-655d-427d-8ce9-8b0160fc34db"). InnerVolumeSpecName "kube-api-access-kgwkw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:24:26.015615 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:26.015559    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/870a2497-655d-427d-8ce9-8b0160fc34db-maas-api-tls" pod "870a2497-655d-427d-8ce9-8b0160fc34db" (UID: "870a2497-655d-427d-8ce9-8b0160fc34db"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:24:26.015615 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:26.015581    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/870a2497-655d-427d-8ce9-8b0160fc34db-metrics-certs" pod "870a2497-655d-427d-8ce9-8b0160fc34db" (UID: "870a2497-655d-427d-8ce9-8b0160fc34db"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:24:26.114361 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:26.114323    4993 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/870a2497-655d-427d-8ce9-8b0160fc34db-metrics-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:24:26.114361 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:26.114354    4993 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/870a2497-655d-427d-8ce9-8b0160fc34db-openshift-service-ca\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:24:26.114361 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:26.114366    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgwkw\" (UniqueName: \"kubernetes.io/projected/870a2497-655d-427d-8ce9-8b0160fc34db-kube-api-access-kgwkw\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:24:26.114601 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:26.114381    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/870a2497-655d-427d-8ce9-8b0160fc34db-maas-api-tls\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:24:26.395305 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:26.395263    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-74c9bbb9f8-fftw7"]
Nov 01 14:24:26.398752 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:26.398726    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-74c9bbb9f8-fftw7"]
Nov 01 14:24:26.793766 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:26.793734    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="870a2497-655d-427d-8ce9-8b0160fc34db" path="/var/lib/kubelet/pods/870a2497-655d-427d-8ce9-8b0160fc34db/volumes"
Nov 01 14:24:26.807030 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:26.807001    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-4pmtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body=
Nov 01 14:24:26.807148 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:26.807050    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4pmtn" podUID="306b0405-fb73-435a-ae73-42a90cd2d2f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused"
Nov 01 14:24:27.499027 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:24:27.498987    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:24:27.810667 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:27.810593    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4pmtn"
Nov 01 14:24:27.811053 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:27.810970    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4pmtn"
Nov 01 14:24:27.811565 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:27.811542    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4pmtn"
Nov 01 14:24:41.842727 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:41.842689    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sgznq"
Nov 01 14:24:41.846096 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:24:41.846066    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-hg8lm"
Nov 01 14:24:57.499407 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:24:57.499363    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:25:21.641501 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:25:21.641418    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-5csbz"]
Nov 01 14:25:21.641974 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:25:21.641839    4993 cpu_manager.go:414] "containerMap: removing container" podUID="f403c01e-299d-4a63-8545-65bc96a68ce1" containerName="authorino"
Nov 01 14:25:21.641974 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:25:21.641850    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f403c01e-299d-4a63-8545-65bc96a68ce1" containerName="authorino"
Nov 01 14:25:21.641974 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:25:21.641863    4993 cpu_manager.go:414] "containerMap: removing container" podUID="870a2497-655d-427d-8ce9-8b0160fc34db" containerName="maas-api"
Nov 01 14:25:21.641974 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:25:21.641868    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="870a2497-655d-427d-8ce9-8b0160fc34db" containerName="maas-api"
Nov 01 14:25:21.641974 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:25:21.641876    4993 cpu_manager.go:414] "containerMap: removing container" podUID="7aa3200c-bf3b-410a-a002-9033e60bbfc6" containerName="authorino"
Nov 01 14:25:21.641974 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:25:21.641882    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7aa3200c-bf3b-410a-a002-9033e60bbfc6" containerName="authorino"
Nov 01 14:25:21.641974 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:25:21.641960    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="f403c01e-299d-4a63-8545-65bc96a68ce1" containerName="authorino"
Nov 01 14:25:21.641974 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:25:21.641969    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="870a2497-655d-427d-8ce9-8b0160fc34db" containerName="maas-api"
Nov 01 14:25:21.641974 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:25:21.641974    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="7aa3200c-bf3b-410a-a002-9033e60bbfc6" containerName="authorino"
Nov 01 14:25:21.645231 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:25:21.645215    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-5csbz"
Nov 01 14:25:21.703921 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:25:21.703890    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-5csbz"]
Nov 01 14:25:21.721534 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:25:21.721498    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/93dc6178-9868-4299-bad6-0ee49fc5d2ad-cert\") pod \"llmisvc-controller-manager-6466c554d8-5csbz\" (UID: \"93dc6178-9868-4299-bad6-0ee49fc5d2ad\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-5csbz"
Nov 01 14:25:21.721695 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:25:21.721647    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkr6x\" (UniqueName: \"kubernetes.io/projected/93dc6178-9868-4299-bad6-0ee49fc5d2ad-kube-api-access-hkr6x\") pod \"llmisvc-controller-manager-6466c554d8-5csbz\" (UID: \"93dc6178-9868-4299-bad6-0ee49fc5d2ad\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-5csbz"
Nov 01 14:25:21.822099 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:25:21.822068    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkr6x\" (UniqueName: \"kubernetes.io/projected/93dc6178-9868-4299-bad6-0ee49fc5d2ad-kube-api-access-hkr6x\") pod \"llmisvc-controller-manager-6466c554d8-5csbz\" (UID: \"93dc6178-9868-4299-bad6-0ee49fc5d2ad\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-5csbz"
Nov 01 14:25:21.822305 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:25:21.822111    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/93dc6178-9868-4299-bad6-0ee49fc5d2ad-cert\") pod \"llmisvc-controller-manager-6466c554d8-5csbz\" (UID: \"93dc6178-9868-4299-bad6-0ee49fc5d2ad\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-5csbz"
Nov 01 14:25:21.824501 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:25:21.824478    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/93dc6178-9868-4299-bad6-0ee49fc5d2ad-cert\") pod \"llmisvc-controller-manager-6466c554d8-5csbz\" (UID: \"93dc6178-9868-4299-bad6-0ee49fc5d2ad\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-5csbz"
Nov 01 14:25:21.874656 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:25:21.874625    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkr6x\" (UniqueName: \"kubernetes.io/projected/93dc6178-9868-4299-bad6-0ee49fc5d2ad-kube-api-access-hkr6x\") pod \"llmisvc-controller-manager-6466c554d8-5csbz\" (UID: \"93dc6178-9868-4299-bad6-0ee49fc5d2ad\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-5csbz"
Nov 01 14:25:21.957817 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:25:21.957732    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-5csbz"
Nov 01 14:25:22.322970 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:25:22.322937    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-5csbz"]
Nov 01 14:25:22.329312 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:25:22.329283    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93dc6178_9868_4299_bad6_0ee49fc5d2ad.slice/crio-642f8b8803cfbf9e75bf315e7211d697eb3d414a807ed8343f6b6fc5258f9135 WatchSource:0}: Error finding container 642f8b8803cfbf9e75bf315e7211d697eb3d414a807ed8343f6b6fc5258f9135: Status 404 returned error can't find the container with id 642f8b8803cfbf9e75bf315e7211d697eb3d414a807ed8343f6b6fc5258f9135
Nov 01 14:25:23.259185 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:25:23.259129    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-5csbz" event={"ID":"93dc6178-9868-4299-bad6-0ee49fc5d2ad","Type":"ContainerStarted","Data":"8ad31bcd7e385f4a0eb79d019a01eab688904783d3f6a013fba7fdd95deaeae3"}
Nov 01 14:25:23.259574 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:25:23.259196    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-5csbz" event={"ID":"93dc6178-9868-4299-bad6-0ee49fc5d2ad","Type":"ContainerStarted","Data":"642f8b8803cfbf9e75bf315e7211d697eb3d414a807ed8343f6b6fc5258f9135"}
Nov 01 14:25:23.259574 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:25:23.259231    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-5csbz"
Nov 01 14:25:23.367221 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:25:23.367155    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-5csbz" podStartSLOduration=1.542095034 podStartE2EDuration="2.367140494s" totalImagesPullingTime="825.04546ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:25:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:25:23.365353234 +0000 UTC m=+510.695215971" watchObservedRunningTime="2026-11-01 14:25:23.367140494 +0000 UTC m=+510.697003231"
Nov 01 14:25:27.499609 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:25:27.499573    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:25:54.266961 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:25:54.266924    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-5csbz"
Nov 01 14:25:54.419808 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:25:54.419773    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-sgznq"]
Nov 01 14:25:54.420023 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:25:54.419983    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sgznq" podUID="c3c95345-bbd5-4951-9611-5a95cc49559f" containerName="manager" containerID="cri-o://b3ad66ef67bc9ed9c61ecf1356106a60d65e33f6c271c785434c800cd51a1e2f" gracePeriod=30
Nov 01 14:25:54.677079 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:25:54.677055    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sgznq"
Nov 01 14:25:54.712979 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:25:54.712952    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3c95345-bbd5-4951-9611-5a95cc49559f-kube-api-access-ps5cn\" (UniqueName: \"kubernetes.io/projected/c3c95345-bbd5-4951-9611-5a95cc49559f-kube-api-access-ps5cn\") pod \"c3c95345-bbd5-4951-9611-5a95cc49559f\" (UID: \"c3c95345-bbd5-4951-9611-5a95cc49559f\") "
Nov 01 14:25:54.713097 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:25:54.712992    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3c95345-bbd5-4951-9611-5a95cc49559f-cert\" (UniqueName: \"kubernetes.io/secret/c3c95345-bbd5-4951-9611-5a95cc49559f-cert\") pod \"c3c95345-bbd5-4951-9611-5a95cc49559f\" (UID: \"c3c95345-bbd5-4951-9611-5a95cc49559f\") "
Nov 01 14:25:54.714952 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:25:54.714929    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3c95345-bbd5-4951-9611-5a95cc49559f-kube-api-access-ps5cn" pod "c3c95345-bbd5-4951-9611-5a95cc49559f" (UID: "c3c95345-bbd5-4951-9611-5a95cc49559f"). InnerVolumeSpecName "kube-api-access-ps5cn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:25:54.715051 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:25:54.715024    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3c95345-bbd5-4951-9611-5a95cc49559f-cert" pod "c3c95345-bbd5-4951-9611-5a95cc49559f" (UID: "c3c95345-bbd5-4951-9611-5a95cc49559f"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:25:54.814341 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:25:54.814312    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ps5cn\" (UniqueName: \"kubernetes.io/projected/c3c95345-bbd5-4951-9611-5a95cc49559f-kube-api-access-ps5cn\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:25:54.814341 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:25:54.814340    4993 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c3c95345-bbd5-4951-9611-5a95cc49559f-cert\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:25:55.417878 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:25:55.417148    4993 generic.go:356] "Generic (PLEG): container finished" podID="c3c95345-bbd5-4951-9611-5a95cc49559f" containerID="b3ad66ef67bc9ed9c61ecf1356106a60d65e33f6c271c785434c800cd51a1e2f" exitCode=0
Nov 01 14:25:55.417878 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:25:55.417212    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sgznq"
Nov 01 14:25:55.417878 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:25:55.417205    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sgznq" event={"ID":"c3c95345-bbd5-4951-9611-5a95cc49559f","Type":"ContainerDied","Data":"b3ad66ef67bc9ed9c61ecf1356106a60d65e33f6c271c785434c800cd51a1e2f"}
Nov 01 14:25:55.417878 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:25:55.417342    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sgznq" event={"ID":"c3c95345-bbd5-4951-9611-5a95cc49559f","Type":"ContainerDied","Data":"0a66d5060047d8c24105960b497dddad76023a942adfa2f689e61d7740951ee8"}
Nov 01 14:25:55.417878 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:25:55.417376    4993 scope.go:122] "RemoveContainer" containerID="b3ad66ef67bc9ed9c61ecf1356106a60d65e33f6c271c785434c800cd51a1e2f"
Nov 01 14:25:55.428890 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:25:55.428643    4993 scope.go:122] "RemoveContainer" containerID="b3ad66ef67bc9ed9c61ecf1356106a60d65e33f6c271c785434c800cd51a1e2f"
Nov 01 14:25:55.428968 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:25:55.428919    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3ad66ef67bc9ed9c61ecf1356106a60d65e33f6c271c785434c800cd51a1e2f\": container with ID starting with b3ad66ef67bc9ed9c61ecf1356106a60d65e33f6c271c785434c800cd51a1e2f not found: ID does not exist" containerID="b3ad66ef67bc9ed9c61ecf1356106a60d65e33f6c271c785434c800cd51a1e2f"
Nov 01 14:25:55.428968 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:25:55.428946    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3ad66ef67bc9ed9c61ecf1356106a60d65e33f6c271c785434c800cd51a1e2f"} err="failed to get container status \"b3ad66ef67bc9ed9c61ecf1356106a60d65e33f6c271c785434c800cd51a1e2f\": rpc error: code = NotFound desc = could not find container \"b3ad66ef67bc9ed9c61ecf1356106a60d65e33f6c271c785434c800cd51a1e2f\": container with ID starting with b3ad66ef67bc9ed9c61ecf1356106a60d65e33f6c271c785434c800cd51a1e2f not found: ID does not exist"
Nov 01 14:25:55.568319 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:25:55.568289    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-sgznq"]
Nov 01 14:25:55.572076 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:25:55.572049    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-sgznq"]
Nov 01 14:25:56.794510 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:25:56.794467    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3c95345-bbd5-4951-9611-5a95cc49559f" path="/var/lib/kubelet/pods/c3c95345-bbd5-4951-9611-5a95cc49559f/volumes"
Nov 01 14:25:57.499070 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:25:57.499027    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:26:27.499287 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:26:27.499250    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:26:52.834160 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:26:52.834122    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xmwtp_bb59051d-c2df-4362-a157-00f14f915e79/console-operator/3.log"
Nov 01 14:26:52.835069 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:26:52.835043    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xmwtp_bb59051d-c2df-4362-a157-00f14f915e79/console-operator/3.log"
Nov 01 14:26:57.499119 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:26:57.499076    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:27:27.500138 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:27:27.500097    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:27:30.025254 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:27:30.025216    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fvgkd"]
Nov 01 14:27:30.025662 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:27:30.025640    4993 cpu_manager.go:414] "containerMap: removing container" podUID="c3c95345-bbd5-4951-9611-5a95cc49559f" containerName="manager"
Nov 01 14:27:30.025662 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:27:30.025652    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3c95345-bbd5-4951-9611-5a95cc49559f" containerName="manager"
Nov 01 14:27:30.025751 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:27:30.025743    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3c95345-bbd5-4951-9611-5a95cc49559f" containerName="manager"
Nov 01 14:27:30.028912 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:27:30.028895    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fvgkd"
Nov 01 14:27:30.110615 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:27:30.110582    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08df1013-bce6-4645-bba5-736db953ce2b-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-fvgkd\" (UID: \"08df1013-bce6-4645-bba5-736db953ce2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fvgkd"
Nov 01 14:27:30.110806 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:27:30.110623    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08df1013-bce6-4645-bba5-736db953ce2b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-fvgkd\" (UID: \"08df1013-bce6-4645-bba5-736db953ce2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fvgkd"
Nov 01 14:27:30.110806 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:27:30.110664    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08df1013-bce6-4645-bba5-736db953ce2b-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-fvgkd\" (UID: \"08df1013-bce6-4645-bba5-736db953ce2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fvgkd"
Nov 01 14:27:30.110806 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:27:30.110756    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08df1013-bce6-4645-bba5-736db953ce2b-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-fvgkd\" (UID: \"08df1013-bce6-4645-bba5-736db953ce2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fvgkd"
Nov 01 14:27:30.110957 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:27:30.110908    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwmgk\" (UniqueName: \"kubernetes.io/projected/08df1013-bce6-4645-bba5-736db953ce2b-kube-api-access-jwmgk\") pod \"maas-default-gateway-openshift-default-7f5667f86c-fvgkd\" (UID: \"08df1013-bce6-4645-bba5-736db953ce2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fvgkd"
Nov 01 14:27:30.111007 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:27:30.110955    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08df1013-bce6-4645-bba5-736db953ce2b-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-fvgkd\" (UID: \"08df1013-bce6-4645-bba5-736db953ce2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fvgkd"
Nov 01 14:27:30.111105 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:27:30.111078    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08df1013-bce6-4645-bba5-736db953ce2b-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-fvgkd\" (UID: \"08df1013-bce6-4645-bba5-736db953ce2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fvgkd"
Nov 01 14:27:30.111182 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:27:30.111116    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08df1013-bce6-4645-bba5-736db953ce2b-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-fvgkd\" (UID: \"08df1013-bce6-4645-bba5-736db953ce2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fvgkd"
Nov 01 14:27:30.111245 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:27:30.111195    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08df1013-bce6-4645-bba5-736db953ce2b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-fvgkd\" (UID: \"08df1013-bce6-4645-bba5-736db953ce2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fvgkd"
Nov 01 14:27:30.212296 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:27:30.212253    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08df1013-bce6-4645-bba5-736db953ce2b-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-fvgkd\" (UID: \"08df1013-bce6-4645-bba5-736db953ce2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fvgkd"
Nov 01 14:27:30.212296 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:27:30.212299    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08df1013-bce6-4645-bba5-736db953ce2b-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-fvgkd\" (UID: \"08df1013-bce6-4645-bba5-736db953ce2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fvgkd"
Nov 01 14:27:30.212532 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:27:30.212422    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08df1013-bce6-4645-bba5-736db953ce2b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-fvgkd\" (UID: \"08df1013-bce6-4645-bba5-736db953ce2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fvgkd"
Nov 01 14:27:30.212532 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:27:30.212515    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08df1013-bce6-4645-bba5-736db953ce2b-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-fvgkd\" (UID: \"08df1013-bce6-4645-bba5-736db953ce2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fvgkd"
Nov 01 14:27:30.212648 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:27:30.212541    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08df1013-bce6-4645-bba5-736db953ce2b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-fvgkd\" (UID: \"08df1013-bce6-4645-bba5-736db953ce2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fvgkd"
Nov 01 14:27:30.212648 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:27:30.212571    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08df1013-bce6-4645-bba5-736db953ce2b-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-fvgkd\" (UID: \"08df1013-bce6-4645-bba5-736db953ce2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fvgkd"
Nov 01 14:27:30.212648 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:27:30.212597    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08df1013-bce6-4645-bba5-736db953ce2b-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-fvgkd\" (UID: \"08df1013-bce6-4645-bba5-736db953ce2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fvgkd"
Nov 01 14:27:30.212801 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:27:30.212656    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwmgk\" (UniqueName: \"kubernetes.io/projected/08df1013-bce6-4645-bba5-736db953ce2b-kube-api-access-jwmgk\") pod \"maas-default-gateway-openshift-default-7f5667f86c-fvgkd\" (UID: \"08df1013-bce6-4645-bba5-736db953ce2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fvgkd"
Nov 01 14:27:30.212801 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:27:30.212675    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08df1013-bce6-4645-bba5-736db953ce2b-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-fvgkd\" (UID: \"08df1013-bce6-4645-bba5-736db953ce2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fvgkd"
Nov 01 14:27:30.212801 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:27:30.212696    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08df1013-bce6-4645-bba5-736db953ce2b-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-fvgkd\" (UID: \"08df1013-bce6-4645-bba5-736db953ce2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fvgkd"
Nov 01 14:27:30.212801 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:27:30.212731    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08df1013-bce6-4645-bba5-736db953ce2b-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-fvgkd\" (UID: \"08df1013-bce6-4645-bba5-736db953ce2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fvgkd"
Nov 01 14:27:30.212995 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:27:30.212911    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08df1013-bce6-4645-bba5-736db953ce2b-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-fvgkd\" (UID: \"08df1013-bce6-4645-bba5-736db953ce2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fvgkd"
Nov 01 14:27:30.213077 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:27:30.213056    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08df1013-bce6-4645-bba5-736db953ce2b-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-fvgkd\" (UID: \"08df1013-bce6-4645-bba5-736db953ce2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fvgkd"
Nov 01 14:27:30.213160 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:27:30.213142    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08df1013-bce6-4645-bba5-736db953ce2b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-fvgkd\" (UID: \"08df1013-bce6-4645-bba5-736db953ce2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fvgkd"
Nov 01 14:27:30.214937 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:27:30.214915    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08df1013-bce6-4645-bba5-736db953ce2b-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-fvgkd\" (UID: \"08df1013-bce6-4645-bba5-736db953ce2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fvgkd"
Nov 01 14:27:30.215209 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:27:30.215187    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08df1013-bce6-4645-bba5-736db953ce2b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-fvgkd\" (UID: \"08df1013-bce6-4645-bba5-736db953ce2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fvgkd"
Nov 01 14:27:30.237916 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:27:30.237890    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fvgkd"]
Nov 01 14:27:30.265903 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:27:30.265873    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08df1013-bce6-4645-bba5-736db953ce2b-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-fvgkd\" (UID: \"08df1013-bce6-4645-bba5-736db953ce2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fvgkd"
Nov 01 14:27:30.266032 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:27:30.266012    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwmgk\" (UniqueName: \"kubernetes.io/projected/08df1013-bce6-4645-bba5-736db953ce2b-kube-api-access-jwmgk\") pod \"maas-default-gateway-openshift-default-7f5667f86c-fvgkd\" (UID: \"08df1013-bce6-4645-bba5-736db953ce2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fvgkd"
Nov 01 14:27:30.342068 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:27:30.341985    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fvgkd"
Nov 01 14:27:30.765511 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:27:30.765475    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fvgkd"]
Nov 01 14:27:30.771687 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:27:30.771655    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08df1013_bce6_4645_bba5_736db953ce2b.slice/crio-1ad40de17405e109687aeae6694757973ac90767db2990ebe982c9f828ce2198 WatchSource:0}: Error finding container 1ad40de17405e109687aeae6694757973ac90767db2990ebe982c9f828ce2198: Status 404 returned error can't find the container with id 1ad40de17405e109687aeae6694757973ac90767db2990ebe982c9f828ce2198
Nov 01 14:27:30.773844 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:27:30.773805    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 14:27:30.773959 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:27:30.773880    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 14:27:30.773959 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:27:30.773910    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 14:27:30.912354 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:27:30.912311    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fvgkd" event={"ID":"08df1013-bce6-4645-bba5-736db953ce2b","Type":"ContainerStarted","Data":"822f62b290c685dc242f9fd99fa8106970551d2853d7aa42feca3b98f04ceed6"}
Nov 01 14:27:30.912517 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:27:30.912379    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fvgkd" event={"ID":"08df1013-bce6-4645-bba5-736db953ce2b","Type":"ContainerStarted","Data":"1ad40de17405e109687aeae6694757973ac90767db2990ebe982c9f828ce2198"}
Nov 01 14:27:31.023835 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:27:31.023723    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fvgkd" podStartSLOduration=2.023709996 podStartE2EDuration="2.023709996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:27:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:27:31.023335477 +0000 UTC m=+638.353198211" watchObservedRunningTime="2026-11-01 14:27:31.023709996 +0000 UTC m=+638.353572733"
Nov 01 14:27:31.342661 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:27:31.342580    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fvgkd"
Nov 01 14:27:31.344220 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:27:31.344190    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-fvgkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Nov 01 14:27:31.344344 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:27:31.344239    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fvgkd" podUID="08df1013-bce6-4645-bba5-736db953ce2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Nov 01 14:27:31.506007 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:27:31.505974    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-687bbd6b9f-md8j6"]
Nov 01 14:27:31.506212 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:27:31.506190    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-687bbd6b9f-md8j6" podUID="1c2096b1-adf7-4291-9423-adbab7aa4449" containerName="authorino" containerID="cri-o://022618c0a17e939ad2bdd0b0bde51db9b9df807b21758db810af272ed94ec02b" gracePeriod=30
Nov 01 14:27:31.803643 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:27:31.803612    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bbd6b9f-md8j6"
Nov 01 14:27:31.830732 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:27:31.830698    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c2096b1-adf7-4291-9423-adbab7aa4449-tls-cert\" (UniqueName: \"kubernetes.io/secret/1c2096b1-adf7-4291-9423-adbab7aa4449-tls-cert\") pod \"1c2096b1-adf7-4291-9423-adbab7aa4449\" (UID: \"1c2096b1-adf7-4291-9423-adbab7aa4449\") "
Nov 01 14:27:31.830880 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:27:31.830763    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c2096b1-adf7-4291-9423-adbab7aa4449-kube-api-access-blkkp\" (UniqueName: \"kubernetes.io/projected/1c2096b1-adf7-4291-9423-adbab7aa4449-kube-api-access-blkkp\") pod \"1c2096b1-adf7-4291-9423-adbab7aa4449\" (UID: \"1c2096b1-adf7-4291-9423-adbab7aa4449\") "
Nov 01 14:27:31.832745 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:27:31.832716    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c2096b1-adf7-4291-9423-adbab7aa4449-kube-api-access-blkkp" pod "1c2096b1-adf7-4291-9423-adbab7aa4449" (UID: "1c2096b1-adf7-4291-9423-adbab7aa4449"). InnerVolumeSpecName "kube-api-access-blkkp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:27:31.843810 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:27:31.843782    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c2096b1-adf7-4291-9423-adbab7aa4449-tls-cert" pod "1c2096b1-adf7-4291-9423-adbab7aa4449" (UID: "1c2096b1-adf7-4291-9423-adbab7aa4449"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:27:31.918699 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:27:31.918600    4993 generic.go:356] "Generic (PLEG): container finished" podID="1c2096b1-adf7-4291-9423-adbab7aa4449" containerID="022618c0a17e939ad2bdd0b0bde51db9b9df807b21758db810af272ed94ec02b" exitCode=0
Nov 01 14:27:31.918699 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:27:31.918655    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bbd6b9f-md8j6"
Nov 01 14:27:31.918699 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:27:31.918682    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-687bbd6b9f-md8j6" event={"ID":"1c2096b1-adf7-4291-9423-adbab7aa4449","Type":"ContainerDied","Data":"022618c0a17e939ad2bdd0b0bde51db9b9df807b21758db810af272ed94ec02b"}
Nov 01 14:27:31.918977 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:27:31.918730    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-687bbd6b9f-md8j6" event={"ID":"1c2096b1-adf7-4291-9423-adbab7aa4449","Type":"ContainerDied","Data":"daf0f7aa490df89fe4eafcc727f13c4ae5837206e73f19869e4bde6dae7ba0fb"}
Nov 01 14:27:31.918977 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:27:31.918752    4993 scope.go:122] "RemoveContainer" containerID="022618c0a17e939ad2bdd0b0bde51db9b9df807b21758db810af272ed94ec02b"
Nov 01 14:27:31.929594 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:27:31.929571    4993 scope.go:122] "RemoveContainer" containerID="022618c0a17e939ad2bdd0b0bde51db9b9df807b21758db810af272ed94ec02b"
Nov 01 14:27:31.930059 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:27:31.930030    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"022618c0a17e939ad2bdd0b0bde51db9b9df807b21758db810af272ed94ec02b\": container with ID starting with 022618c0a17e939ad2bdd0b0bde51db9b9df807b21758db810af272ed94ec02b not found: ID does not exist" containerID="022618c0a17e939ad2bdd0b0bde51db9b9df807b21758db810af272ed94ec02b"
Nov 01 14:27:31.930157 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:27:31.930068    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"022618c0a17e939ad2bdd0b0bde51db9b9df807b21758db810af272ed94ec02b"} err="failed to get container status \"022618c0a17e939ad2bdd0b0bde51db9b9df807b21758db810af272ed94ec02b\": rpc error: code = NotFound desc = could not find container \"022618c0a17e939ad2bdd0b0bde51db9b9df807b21758db810af272ed94ec02b\": container with ID starting with 022618c0a17e939ad2bdd0b0bde51db9b9df807b21758db810af272ed94ec02b not found: ID does not exist"
Nov 01 14:27:31.931761 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:27:31.931739    4993 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1c2096b1-adf7-4291-9423-adbab7aa4449-tls-cert\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:27:31.931761 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:27:31.931760    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blkkp\" (UniqueName: \"kubernetes.io/projected/1c2096b1-adf7-4291-9423-adbab7aa4449-kube-api-access-blkkp\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:27:32.084472 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:27:32.084430    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-687bbd6b9f-md8j6"]
Nov 01 14:27:32.086136 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:27:32.086110    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-687bbd6b9f-md8j6"]
Nov 01 14:27:32.342874 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:27:32.342838    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-fvgkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Nov 01 14:27:32.343260 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:27:32.342902    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fvgkd" podUID="08df1013-bce6-4645-bba5-736db953ce2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Nov 01 14:27:32.794828 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:27:32.794791    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c2096b1-adf7-4291-9423-adbab7aa4449" path="/var/lib/kubelet/pods/1c2096b1-adf7-4291-9423-adbab7aa4449/volumes"
Nov 01 14:27:33.343054 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:27:33.343021    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-fvgkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Nov 01 14:27:33.343475 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:27:33.343073    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fvgkd" podUID="08df1013-bce6-4645-bba5-736db953ce2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Nov 01 14:27:34.342357 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:27:34.342308    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-fvgkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Nov 01 14:27:34.342550 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:27:34.342401    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fvgkd" podUID="08df1013-bce6-4645-bba5-736db953ce2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Nov 01 14:27:35.342728 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:27:35.342692    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-fvgkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Nov 01 14:27:35.343132 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:27:35.342747    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fvgkd" podUID="08df1013-bce6-4645-bba5-736db953ce2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Nov 01 14:27:36.343109 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:27:36.343077    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-fvgkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Nov 01 14:27:36.343510 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:27:36.343127    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fvgkd" podUID="08df1013-bce6-4645-bba5-736db953ce2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Nov 01 14:27:37.343207 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:27:37.343145    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-fvgkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Nov 01 14:27:37.343702 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:27:37.343233    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fvgkd" podUID="08df1013-bce6-4645-bba5-736db953ce2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Nov 01 14:27:39.015090 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:27:39.015051    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fvgkd"
Nov 01 14:27:39.015838 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:27:39.015815    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fvgkd"
Nov 01 14:27:39.016662 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:27:39.016636    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fvgkd"
Nov 01 14:27:57.499281 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:27:57.499219    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:28:08.275152 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:08.275072    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f7c71759-6b64666d74-9hb2t"]
Nov 01 14:28:08.275657 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:08.275471    4993 cpu_manager.go:414] "containerMap: removing container" podUID="1c2096b1-adf7-4291-9423-adbab7aa4449" containerName="authorino"
Nov 01 14:28:08.275657 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:08.275482    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c2096b1-adf7-4291-9423-adbab7aa4449" containerName="authorino"
Nov 01 14:28:08.275657 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:08.275570    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c2096b1-adf7-4291-9423-adbab7aa4449" containerName="authorino"
Nov 01 14:28:08.278736 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:08.278712    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7c71759-6b64666d74-9hb2t"
Nov 01 14:28:08.301339 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:08.301312    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b48e6b8-b98c-4c82-880b-150630625fe1-metrics-certs\") pod \"maas-api-e2e-ait-f7c71759-6b64666d74-9hb2t\" (UID: \"3b48e6b8-b98c-4c82-880b-150630625fe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7c71759-6b64666d74-9hb2t"
Nov 01 14:28:08.301473 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:08.301351    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj2vz\" (UniqueName: \"kubernetes.io/projected/3b48e6b8-b98c-4c82-880b-150630625fe1-kube-api-access-rj2vz\") pod \"maas-api-e2e-ait-f7c71759-6b64666d74-9hb2t\" (UID: \"3b48e6b8-b98c-4c82-880b-150630625fe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7c71759-6b64666d74-9hb2t"
Nov 01 14:28:08.301473 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:08.301394    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b48e6b8-b98c-4c82-880b-150630625fe1-maas-api-tls\") pod \"maas-api-e2e-ait-f7c71759-6b64666d74-9hb2t\" (UID: \"3b48e6b8-b98c-4c82-880b-150630625fe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7c71759-6b64666d74-9hb2t"
Nov 01 14:28:08.301578 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:08.301555    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b48e6b8-b98c-4c82-880b-150630625fe1-openshift-service-ca\") pod \"maas-api-e2e-ait-f7c71759-6b64666d74-9hb2t\" (UID: \"3b48e6b8-b98c-4c82-880b-150630625fe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7c71759-6b64666d74-9hb2t"
Nov 01 14:28:08.327485 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:08.327451    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-f7c71759-serving-cert\""
Nov 01 14:28:08.327611 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:08.327535    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 14:28:08.373944 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:08.373908    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 14:28:08.374125 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:08.373957    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7t7jm\""
Nov 01 14:28:08.374125 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:08.374018    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 01 14:28:08.376145 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:08.376127    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f7c71759-6b64666d74-9hb2t"]
Nov 01 14:28:08.402447 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:08.402416    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b48e6b8-b98c-4c82-880b-150630625fe1-openshift-service-ca\") pod \"maas-api-e2e-ait-f7c71759-6b64666d74-9hb2t\" (UID: \"3b48e6b8-b98c-4c82-880b-150630625fe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7c71759-6b64666d74-9hb2t"
Nov 01 14:28:08.402641 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:08.402460    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b48e6b8-b98c-4c82-880b-150630625fe1-metrics-certs\") pod \"maas-api-e2e-ait-f7c71759-6b64666d74-9hb2t\" (UID: \"3b48e6b8-b98c-4c82-880b-150630625fe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7c71759-6b64666d74-9hb2t"
Nov 01 14:28:08.402641 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:08.402489    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj2vz\" (UniqueName: \"kubernetes.io/projected/3b48e6b8-b98c-4c82-880b-150630625fe1-kube-api-access-rj2vz\") pod \"maas-api-e2e-ait-f7c71759-6b64666d74-9hb2t\" (UID: \"3b48e6b8-b98c-4c82-880b-150630625fe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7c71759-6b64666d74-9hb2t"
Nov 01 14:28:08.402641 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:08.402525    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b48e6b8-b98c-4c82-880b-150630625fe1-maas-api-tls\") pod \"maas-api-e2e-ait-f7c71759-6b64666d74-9hb2t\" (UID: \"3b48e6b8-b98c-4c82-880b-150630625fe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7c71759-6b64666d74-9hb2t"
Nov 01 14:28:08.402798 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:28:08.402680    4993 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-f7c71759-serving-cert: secret "maas-api-e2e-ait-f7c71759-serving-cert" not found
Nov 01 14:28:08.402798 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:28:08.402744    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b48e6b8-b98c-4c82-880b-150630625fe1-maas-api-tls podName:3b48e6b8-b98c-4c82-880b-150630625fe1 nodeName:}" failed. No retries permitted until 2026-11-01 14:28:08.902723728 +0000 UTC m=+676.232586466 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3b48e6b8-b98c-4c82-880b-150630625fe1-maas-api-tls") pod "maas-api-e2e-ait-f7c71759-6b64666d74-9hb2t" (UID: "3b48e6b8-b98c-4c82-880b-150630625fe1") : secret "maas-api-e2e-ait-f7c71759-serving-cert" not found
Nov 01 14:28:08.403344 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:08.403306    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b48e6b8-b98c-4c82-880b-150630625fe1-openshift-service-ca\") pod \"maas-api-e2e-ait-f7c71759-6b64666d74-9hb2t\" (UID: \"3b48e6b8-b98c-4c82-880b-150630625fe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7c71759-6b64666d74-9hb2t"
Nov 01 14:28:08.405686 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:08.405663    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b48e6b8-b98c-4c82-880b-150630625fe1-metrics-certs\") pod \"maas-api-e2e-ait-f7c71759-6b64666d74-9hb2t\" (UID: \"3b48e6b8-b98c-4c82-880b-150630625fe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7c71759-6b64666d74-9hb2t"
Nov 01 14:28:08.463024 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:08.462972    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj2vz\" (UniqueName: \"kubernetes.io/projected/3b48e6b8-b98c-4c82-880b-150630625fe1-kube-api-access-rj2vz\") pod \"maas-api-e2e-ait-f7c71759-6b64666d74-9hb2t\" (UID: \"3b48e6b8-b98c-4c82-880b-150630625fe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7c71759-6b64666d74-9hb2t"
Nov 01 14:28:08.906863 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:08.906827    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b48e6b8-b98c-4c82-880b-150630625fe1-maas-api-tls\") pod \"maas-api-e2e-ait-f7c71759-6b64666d74-9hb2t\" (UID: \"3b48e6b8-b98c-4c82-880b-150630625fe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7c71759-6b64666d74-9hb2t"
Nov 01 14:28:08.909222 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:08.909182    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b48e6b8-b98c-4c82-880b-150630625fe1-maas-api-tls\") pod \"maas-api-e2e-ait-f7c71759-6b64666d74-9hb2t\" (UID: \"3b48e6b8-b98c-4c82-880b-150630625fe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7c71759-6b64666d74-9hb2t"
Nov 01 14:28:09.191860 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:09.191773    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7c71759-6b64666d74-9hb2t"
Nov 01 14:28:09.557001 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:09.556958    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f7c71759-6b64666d74-9hb2t"]
Nov 01 14:28:09.563772 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:28:09.563746    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b48e6b8_b98c_4c82_880b_150630625fe1.slice/crio-eb8e63a57646bcc53abeaa4978f905ed7df94a2883f6383ba1243592caa51b24 WatchSource:0}: Error finding container eb8e63a57646bcc53abeaa4978f905ed7df94a2883f6383ba1243592caa51b24: Status 404 returned error can't find the container with id eb8e63a57646bcc53abeaa4978f905ed7df94a2883f6383ba1243592caa51b24
Nov 01 14:28:09.565097 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:09.565079    4993 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 14:28:10.114520 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:10.114484    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7c71759-6b64666d74-9hb2t" event={"ID":"3b48e6b8-b98c-4c82-880b-150630625fe1","Type":"ContainerStarted","Data":"eb8e63a57646bcc53abeaa4978f905ed7df94a2883f6383ba1243592caa51b24"}
Nov 01 14:28:11.122214 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:11.122163    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7c71759-6b64666d74-9hb2t" event={"ID":"3b48e6b8-b98c-4c82-880b-150630625fe1","Type":"ContainerStarted","Data":"6c4757b2fdb35cdf384b9c454d67dc826465deb9f14afcec3180e5657651c5c7"}
Nov 01 14:28:11.122659 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:11.122372    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7c71759-6b64666d74-9hb2t"
Nov 01 14:28:11.229442 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:11.229388    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7c71759-6b64666d74-9hb2t" podStartSLOduration=2.129866822 podStartE2EDuration="3.229373594s" totalImagesPullingTime="1.099506772s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:28:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:28:11.228018844 +0000 UTC m=+678.557881582" watchObservedRunningTime="2026-11-01 14:28:11.229373594 +0000 UTC m=+678.559236371"
Nov 01 14:28:15.180413 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:15.180382    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c9fe7b2a-86cc89645f-mcpdq"]
Nov 01 14:28:15.184043 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:15.184020    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-c9fe7b2a-86cc89645f-mcpdq"
Nov 01 14:28:15.266101 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:15.266065    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d97cc8d0-8067-4899-9a62-3046e2a103a7-plugins-config-volume\") pod \"payload-processing-e2e-migrate-c9fe7b2a-86cc89645f-mcpdq\" (UID: \"d97cc8d0-8067-4899-9a62-3046e2a103a7\") " pod="openshift-ingress/payload-processing-e2e-migrate-c9fe7b2a-86cc89645f-mcpdq"
Nov 01 14:28:15.266287 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:15.266105    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lrpn\" (UniqueName: \"kubernetes.io/projected/d97cc8d0-8067-4899-9a62-3046e2a103a7-kube-api-access-4lrpn\") pod \"payload-processing-e2e-migrate-c9fe7b2a-86cc89645f-mcpdq\" (UID: \"d97cc8d0-8067-4899-9a62-3046e2a103a7\") " pod="openshift-ingress/payload-processing-e2e-migrate-c9fe7b2a-86cc89645f-mcpdq"
Nov 01 14:28:15.278645 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:15.278610    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-c9fe7b2a\""
Nov 01 14:28:15.278645 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:15.278608    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-c9fe7b2a-dockercfg-54jk9\""
Nov 01 14:28:15.280961 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:15.280938    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c9fe7b2a-86cc89645f-mcpdq"]
Nov 01 14:28:15.367442 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:15.367397    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d97cc8d0-8067-4899-9a62-3046e2a103a7-plugins-config-volume\") pod \"payload-processing-e2e-migrate-c9fe7b2a-86cc89645f-mcpdq\" (UID: \"d97cc8d0-8067-4899-9a62-3046e2a103a7\") " pod="openshift-ingress/payload-processing-e2e-migrate-c9fe7b2a-86cc89645f-mcpdq"
Nov 01 14:28:15.367442 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:15.367443    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lrpn\" (UniqueName: \"kubernetes.io/projected/d97cc8d0-8067-4899-9a62-3046e2a103a7-kube-api-access-4lrpn\") pod \"payload-processing-e2e-migrate-c9fe7b2a-86cc89645f-mcpdq\" (UID: \"d97cc8d0-8067-4899-9a62-3046e2a103a7\") " pod="openshift-ingress/payload-processing-e2e-migrate-c9fe7b2a-86cc89645f-mcpdq"
Nov 01 14:28:15.368113 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:15.368089    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d97cc8d0-8067-4899-9a62-3046e2a103a7-plugins-config-volume\") pod \"payload-processing-e2e-migrate-c9fe7b2a-86cc89645f-mcpdq\" (UID: \"d97cc8d0-8067-4899-9a62-3046e2a103a7\") " pod="openshift-ingress/payload-processing-e2e-migrate-c9fe7b2a-86cc89645f-mcpdq"
Nov 01 14:28:15.420186 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:15.420146    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lrpn\" (UniqueName: \"kubernetes.io/projected/d97cc8d0-8067-4899-9a62-3046e2a103a7-kube-api-access-4lrpn\") pod \"payload-processing-e2e-migrate-c9fe7b2a-86cc89645f-mcpdq\" (UID: \"d97cc8d0-8067-4899-9a62-3046e2a103a7\") " pod="openshift-ingress/payload-processing-e2e-migrate-c9fe7b2a-86cc89645f-mcpdq"
Nov 01 14:28:15.498100 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:15.498065    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-c9fe7b2a-86cc89645f-mcpdq"
Nov 01 14:28:15.863754 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:15.863676    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c9fe7b2a-86cc89645f-mcpdq"]
Nov 01 14:28:15.870791 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:28:15.870759    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd97cc8d0_8067_4899_9a62_3046e2a103a7.slice/crio-ed763d7bffc33335565983c69091acde4c81f6faa01e3872bf5aa37666f25cc8 WatchSource:0}: Error finding container ed763d7bffc33335565983c69091acde4c81f6faa01e3872bf5aa37666f25cc8: Status 404 returned error can't find the container with id ed763d7bffc33335565983c69091acde4c81f6faa01e3872bf5aa37666f25cc8
Nov 01 14:28:16.147105 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:16.147022    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-c9fe7b2a-86cc89645f-mcpdq" event={"ID":"d97cc8d0-8067-4899-9a62-3046e2a103a7","Type":"ContainerStarted","Data":"ed763d7bffc33335565983c69091acde4c81f6faa01e3872bf5aa37666f25cc8"}
Nov 01 14:28:17.134310 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:17.134280    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7c71759-6b64666d74-9hb2t"
Nov 01 14:28:18.051868 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:18.051832    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0e27521d-5d7bb5dc5f-wdv8b"]
Nov 01 14:28:18.076607 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:18.076573    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e27521d-5d7bb5dc5f-wdv8b"
Nov 01 14:28:18.113015 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:18.112978    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0e27521d-5d7bb5dc5f-wdv8b"]
Nov 01 14:28:18.157544 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:18.157509    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-0e27521d-serving-cert\""
Nov 01 14:28:18.195998 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:18.195888    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da801b51-9486-460c-a6f0-3f94d8999c04-maas-api-tls\") pod \"maas-api-e2e-ait-0e27521d-5d7bb5dc5f-wdv8b\" (UID: \"da801b51-9486-460c-a6f0-3f94d8999c04\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e27521d-5d7bb5dc5f-wdv8b"
Nov 01 14:28:18.195998 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:18.195965    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5rhg\" (UniqueName: \"kubernetes.io/projected/da801b51-9486-460c-a6f0-3f94d8999c04-kube-api-access-c5rhg\") pod \"maas-api-e2e-ait-0e27521d-5d7bb5dc5f-wdv8b\" (UID: \"da801b51-9486-460c-a6f0-3f94d8999c04\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e27521d-5d7bb5dc5f-wdv8b"
Nov 01 14:28:18.196270 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:18.196128    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da801b51-9486-460c-a6f0-3f94d8999c04-metrics-certs\") pod \"maas-api-e2e-ait-0e27521d-5d7bb5dc5f-wdv8b\" (UID: \"da801b51-9486-460c-a6f0-3f94d8999c04\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e27521d-5d7bb5dc5f-wdv8b"
Nov 01 14:28:18.196270 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:18.196232    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da801b51-9486-460c-a6f0-3f94d8999c04-openshift-service-ca\") pod \"maas-api-e2e-ait-0e27521d-5d7bb5dc5f-wdv8b\" (UID: \"da801b51-9486-460c-a6f0-3f94d8999c04\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e27521d-5d7bb5dc5f-wdv8b"
Nov 01 14:28:18.297297 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:18.297257    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da801b51-9486-460c-a6f0-3f94d8999c04-openshift-service-ca\") pod \"maas-api-e2e-ait-0e27521d-5d7bb5dc5f-wdv8b\" (UID: \"da801b51-9486-460c-a6f0-3f94d8999c04\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e27521d-5d7bb5dc5f-wdv8b"
Nov 01 14:28:18.297487 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:18.297328    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da801b51-9486-460c-a6f0-3f94d8999c04-maas-api-tls\") pod \"maas-api-e2e-ait-0e27521d-5d7bb5dc5f-wdv8b\" (UID: \"da801b51-9486-460c-a6f0-3f94d8999c04\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e27521d-5d7bb5dc5f-wdv8b"
Nov 01 14:28:18.297487 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:18.297370    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5rhg\" (UniqueName: \"kubernetes.io/projected/da801b51-9486-460c-a6f0-3f94d8999c04-kube-api-access-c5rhg\") pod \"maas-api-e2e-ait-0e27521d-5d7bb5dc5f-wdv8b\" (UID: \"da801b51-9486-460c-a6f0-3f94d8999c04\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e27521d-5d7bb5dc5f-wdv8b"
Nov 01 14:28:18.297487 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:18.297443    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da801b51-9486-460c-a6f0-3f94d8999c04-metrics-certs\") pod \"maas-api-e2e-ait-0e27521d-5d7bb5dc5f-wdv8b\" (UID: \"da801b51-9486-460c-a6f0-3f94d8999c04\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e27521d-5d7bb5dc5f-wdv8b"
Nov 01 14:28:18.298540 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:28:18.298033    4993 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-0e27521d-serving-cert: secret "maas-api-e2e-ait-0e27521d-serving-cert" not found
Nov 01 14:28:18.298540 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:28:18.298103    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da801b51-9486-460c-a6f0-3f94d8999c04-maas-api-tls podName:da801b51-9486-460c-a6f0-3f94d8999c04 nodeName:}" failed. No retries permitted until 2026-11-01 14:28:18.798083766 +0000 UTC m=+686.127946500 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/da801b51-9486-460c-a6f0-3f94d8999c04-maas-api-tls") pod "maas-api-e2e-ait-0e27521d-5d7bb5dc5f-wdv8b" (UID: "da801b51-9486-460c-a6f0-3f94d8999c04") : secret "maas-api-e2e-ait-0e27521d-serving-cert" not found
Nov 01 14:28:18.298540 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:18.298472    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da801b51-9486-460c-a6f0-3f94d8999c04-openshift-service-ca\") pod \"maas-api-e2e-ait-0e27521d-5d7bb5dc5f-wdv8b\" (UID: \"da801b51-9486-460c-a6f0-3f94d8999c04\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e27521d-5d7bb5dc5f-wdv8b"
Nov 01 14:28:18.301670 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:18.301642    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da801b51-9486-460c-a6f0-3f94d8999c04-metrics-certs\") pod \"maas-api-e2e-ait-0e27521d-5d7bb5dc5f-wdv8b\" (UID: \"da801b51-9486-460c-a6f0-3f94d8999c04\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e27521d-5d7bb5dc5f-wdv8b"
Nov 01 14:28:18.352393 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:18.352324    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5rhg\" (UniqueName: \"kubernetes.io/projected/da801b51-9486-460c-a6f0-3f94d8999c04-kube-api-access-c5rhg\") pod \"maas-api-e2e-ait-0e27521d-5d7bb5dc5f-wdv8b\" (UID: \"da801b51-9486-460c-a6f0-3f94d8999c04\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e27521d-5d7bb5dc5f-wdv8b"
Nov 01 14:28:18.803735 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:18.803698    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da801b51-9486-460c-a6f0-3f94d8999c04-maas-api-tls\") pod \"maas-api-e2e-ait-0e27521d-5d7bb5dc5f-wdv8b\" (UID: \"da801b51-9486-460c-a6f0-3f94d8999c04\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e27521d-5d7bb5dc5f-wdv8b"
Nov 01 14:28:18.806263 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:18.806238    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da801b51-9486-460c-a6f0-3f94d8999c04-maas-api-tls\") pod \"maas-api-e2e-ait-0e27521d-5d7bb5dc5f-wdv8b\" (UID: \"da801b51-9486-460c-a6f0-3f94d8999c04\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e27521d-5d7bb5dc5f-wdv8b"
Nov 01 14:28:18.995223 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:18.995163    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e27521d-5d7bb5dc5f-wdv8b"
Nov 01 14:28:19.168910 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:19.168874    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-c9fe7b2a-86cc89645f-mcpdq" event={"ID":"d97cc8d0-8067-4899-9a62-3046e2a103a7","Type":"ContainerStarted","Data":"75ed000c1797e72bbcab7897f9fc7211b100c2cdfe10212f5cba87910dd3a994"}
Nov 01 14:28:19.169317 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:19.169002    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-c9fe7b2a-86cc89645f-mcpdq"
Nov 01 14:28:19.274519 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:19.274422    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-c9fe7b2a-86cc89645f-mcpdq" podStartSLOduration=1.520540758 podStartE2EDuration="4.274409195s" totalImagesPullingTime="2.753868437s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:28:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:28:19.273799272 +0000 UTC m=+686.603662009" watchObservedRunningTime="2026-11-01 14:28:19.274409195 +0000 UTC m=+686.604271931"
Nov 01 14:28:19.436394 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:19.436356    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0e27521d-5d7bb5dc5f-wdv8b"]
Nov 01 14:28:19.442097 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:28:19.442064    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda801b51_9486_460c_a6f0_3f94d8999c04.slice/crio-98261d42317cac8c1c531cbc5d97b06698edcfa20361fc6116583d6d0d241d52 WatchSource:0}: Error finding container 98261d42317cac8c1c531cbc5d97b06698edcfa20361fc6116583d6d0d241d52: Status 404 returned error can't find the container with id 98261d42317cac8c1c531cbc5d97b06698edcfa20361fc6116583d6d0d241d52
Nov 01 14:28:20.175355 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:20.175303    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e27521d-5d7bb5dc5f-wdv8b" event={"ID":"da801b51-9486-460c-a6f0-3f94d8999c04","Type":"ContainerStarted","Data":"98261d42317cac8c1c531cbc5d97b06698edcfa20361fc6116583d6d0d241d52"}
Nov 01 14:28:21.181939 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:21.181904    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e27521d-5d7bb5dc5f-wdv8b" event={"ID":"da801b51-9486-460c-a6f0-3f94d8999c04","Type":"ContainerStarted","Data":"84225cf13898926e656612a28ecf817494313dbc44500b9ce7e711cee07c64f8"}
Nov 01 14:28:21.182355 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:21.181995    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e27521d-5d7bb5dc5f-wdv8b"
Nov 01 14:28:21.289336 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:21.289282    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e27521d-5d7bb5dc5f-wdv8b" podStartSLOduration=2.404842408 podStartE2EDuration="3.28926648s" totalImagesPullingTime="884.424072ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:28:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:28:21.287061731 +0000 UTC m=+688.616924468" watchObservedRunningTime="2026-11-01 14:28:21.28926648 +0000 UTC m=+688.619129217"
Nov 01 14:28:27.192915 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:27.192885    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e27521d-5d7bb5dc5f-wdv8b"
Nov 01 14:28:27.501052 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:28:27.501015    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:28:30.178876 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:30.178843    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-c9fe7b2a-86cc89645f-mcpdq"
Nov 01 14:28:31.788411 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:31.788375    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c0f3259b-7b67d68778-ncxdk"]
Nov 01 14:28:31.792152 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:31.792133    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c0f3259b-7b67d68778-ncxdk"
Nov 01 14:28:31.840398 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:31.840373    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-c0f3259b-serving-cert\""
Nov 01 14:28:31.888689 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:31.888656    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c0f3259b-7b67d68778-ncxdk"]
Nov 01 14:28:31.937485 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:31.937346    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc2n6\" (UniqueName: \"kubernetes.io/projected/cbb0c8ac-9486-4e9a-820f-bd569b28e75a-kube-api-access-qc2n6\") pod \"maas-api-e2e-derive-c0f3259b-7b67d68778-ncxdk\" (UID: \"cbb0c8ac-9486-4e9a-820f-bd569b28e75a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c0f3259b-7b67d68778-ncxdk"
Nov 01 14:28:31.937485 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:31.937414    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbb0c8ac-9486-4e9a-820f-bd569b28e75a-maas-api-tls\") pod \"maas-api-e2e-derive-c0f3259b-7b67d68778-ncxdk\" (UID: \"cbb0c8ac-9486-4e9a-820f-bd569b28e75a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c0f3259b-7b67d68778-ncxdk"
Nov 01 14:28:31.937770 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:31.937739    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbb0c8ac-9486-4e9a-820f-bd569b28e75a-metrics-certs\") pod \"maas-api-e2e-derive-c0f3259b-7b67d68778-ncxdk\" (UID: \"cbb0c8ac-9486-4e9a-820f-bd569b28e75a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c0f3259b-7b67d68778-ncxdk"
Nov 01 14:28:31.938057 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:31.938028    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbb0c8ac-9486-4e9a-820f-bd569b28e75a-openshift-service-ca\") pod \"maas-api-e2e-derive-c0f3259b-7b67d68778-ncxdk\" (UID: \"cbb0c8ac-9486-4e9a-820f-bd569b28e75a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c0f3259b-7b67d68778-ncxdk"
Nov 01 14:28:32.039327 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:32.039246    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbb0c8ac-9486-4e9a-820f-bd569b28e75a-openshift-service-ca\") pod \"maas-api-e2e-derive-c0f3259b-7b67d68778-ncxdk\" (UID: \"cbb0c8ac-9486-4e9a-820f-bd569b28e75a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c0f3259b-7b67d68778-ncxdk"
Nov 01 14:28:32.039327 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:32.039305    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc2n6\" (UniqueName: \"kubernetes.io/projected/cbb0c8ac-9486-4e9a-820f-bd569b28e75a-kube-api-access-qc2n6\") pod \"maas-api-e2e-derive-c0f3259b-7b67d68778-ncxdk\" (UID: \"cbb0c8ac-9486-4e9a-820f-bd569b28e75a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c0f3259b-7b67d68778-ncxdk"
Nov 01 14:28:32.039327 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:32.039326    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbb0c8ac-9486-4e9a-820f-bd569b28e75a-maas-api-tls\") pod \"maas-api-e2e-derive-c0f3259b-7b67d68778-ncxdk\" (UID: \"cbb0c8ac-9486-4e9a-820f-bd569b28e75a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c0f3259b-7b67d68778-ncxdk"
Nov 01 14:28:32.039601 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:32.039346    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbb0c8ac-9486-4e9a-820f-bd569b28e75a-metrics-certs\") pod \"maas-api-e2e-derive-c0f3259b-7b67d68778-ncxdk\" (UID: \"cbb0c8ac-9486-4e9a-820f-bd569b28e75a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c0f3259b-7b67d68778-ncxdk"
Nov 01 14:28:32.039601 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:28:32.039485    4993 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-c0f3259b-serving-cert: secret "maas-api-e2e-derive-c0f3259b-serving-cert" not found
Nov 01 14:28:32.039601 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:28:32.039566    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbb0c8ac-9486-4e9a-820f-bd569b28e75a-maas-api-tls podName:cbb0c8ac-9486-4e9a-820f-bd569b28e75a nodeName:}" failed. No retries permitted until 2026-11-01 14:28:32.539543143 +0000 UTC m=+699.869405873 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cbb0c8ac-9486-4e9a-820f-bd569b28e75a-maas-api-tls") pod "maas-api-e2e-derive-c0f3259b-7b67d68778-ncxdk" (UID: "cbb0c8ac-9486-4e9a-820f-bd569b28e75a") : secret "maas-api-e2e-derive-c0f3259b-serving-cert" not found
Nov 01 14:28:32.040024 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:32.039999    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbb0c8ac-9486-4e9a-820f-bd569b28e75a-openshift-service-ca\") pod \"maas-api-e2e-derive-c0f3259b-7b67d68778-ncxdk\" (UID: \"cbb0c8ac-9486-4e9a-820f-bd569b28e75a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c0f3259b-7b67d68778-ncxdk"
Nov 01 14:28:32.042103 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:32.042078    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbb0c8ac-9486-4e9a-820f-bd569b28e75a-metrics-certs\") pod \"maas-api-e2e-derive-c0f3259b-7b67d68778-ncxdk\" (UID: \"cbb0c8ac-9486-4e9a-820f-bd569b28e75a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c0f3259b-7b67d68778-ncxdk"
Nov 01 14:28:32.093017 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:32.092986    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc2n6\" (UniqueName: \"kubernetes.io/projected/cbb0c8ac-9486-4e9a-820f-bd569b28e75a-kube-api-access-qc2n6\") pod \"maas-api-e2e-derive-c0f3259b-7b67d68778-ncxdk\" (UID: \"cbb0c8ac-9486-4e9a-820f-bd569b28e75a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c0f3259b-7b67d68778-ncxdk"
Nov 01 14:28:32.543417 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:32.543379    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbb0c8ac-9486-4e9a-820f-bd569b28e75a-maas-api-tls\") pod \"maas-api-e2e-derive-c0f3259b-7b67d68778-ncxdk\" (UID: \"cbb0c8ac-9486-4e9a-820f-bd569b28e75a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c0f3259b-7b67d68778-ncxdk"
Nov 01 14:28:32.545805 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:32.545783    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbb0c8ac-9486-4e9a-820f-bd569b28e75a-maas-api-tls\") pod \"maas-api-e2e-derive-c0f3259b-7b67d68778-ncxdk\" (UID: \"cbb0c8ac-9486-4e9a-820f-bd569b28e75a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c0f3259b-7b67d68778-ncxdk"
Nov 01 14:28:32.706660 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:32.706619    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c0f3259b-7b67d68778-ncxdk"
Nov 01 14:28:33.075134 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:33.075100    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c0f3259b-7b67d68778-ncxdk"]
Nov 01 14:28:33.081661 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:28:33.081632    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbb0c8ac_9486_4e9a_820f_bd569b28e75a.slice/crio-98ee52dbabdee5592ff063a3eaccdcdad44aa529e1c76319fecef3b08ea064d7 WatchSource:0}: Error finding container 98ee52dbabdee5592ff063a3eaccdcdad44aa529e1c76319fecef3b08ea064d7: Status 404 returned error can't find the container with id 98ee52dbabdee5592ff063a3eaccdcdad44aa529e1c76319fecef3b08ea064d7
Nov 01 14:28:33.242179 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:33.242127    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c0f3259b-7b67d68778-ncxdk" event={"ID":"cbb0c8ac-9486-4e9a-820f-bd569b28e75a","Type":"ContainerStarted","Data":"98ee52dbabdee5592ff063a3eaccdcdad44aa529e1c76319fecef3b08ea064d7"}
Nov 01 14:28:34.248542 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:34.248500    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c0f3259b-7b67d68778-ncxdk" event={"ID":"cbb0c8ac-9486-4e9a-820f-bd569b28e75a","Type":"ContainerStarted","Data":"6a23a6ac314f95baf504ba8395297cb36ea30322e82689013a971107e4408d0e"}
Nov 01 14:28:34.248971 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:34.248569    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c0f3259b-7b67d68778-ncxdk"
Nov 01 14:28:34.393374 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:34.393276    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c0f3259b-7b67d68778-ncxdk" podStartSLOduration=2.34792462 podStartE2EDuration="3.393263664s" totalImagesPullingTime="1.045339044s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:28:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:28:34.393035224 +0000 UTC m=+701.722897961" watchObservedRunningTime="2026-11-01 14:28:34.393263664 +0000 UTC m=+701.723126421"
Nov 01 14:28:40.259406 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:28:40.259372    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c0f3259b-7b67d68778-ncxdk"
Nov 01 14:28:57.499750 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:28:57.499711    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:29:09.891659 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:09.891624    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0e27521d-5d7bb5dc5f-wdv8b"]
Nov 01 14:29:09.892049 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:09.891832    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e27521d-5d7bb5dc5f-wdv8b" podUID="da801b51-9486-460c-a6f0-3f94d8999c04" containerName="maas-api" containerID="cri-o://84225cf13898926e656612a28ecf817494313dbc44500b9ce7e711cee07c64f8" gracePeriod=30
Nov 01 14:29:10.029455 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:10.029424    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c9fe7b2a-86cc89645f-mcpdq"]
Nov 01 14:29:10.157335 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:10.157272    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e27521d-5d7bb5dc5f-wdv8b"
Nov 01 14:29:10.298911 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:10.298875    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da801b51-9486-460c-a6f0-3f94d8999c04-kube-api-access-c5rhg\" (UniqueName: \"kubernetes.io/projected/da801b51-9486-460c-a6f0-3f94d8999c04-kube-api-access-c5rhg\") pod \"da801b51-9486-460c-a6f0-3f94d8999c04\" (UID: \"da801b51-9486-460c-a6f0-3f94d8999c04\") "
Nov 01 14:29:10.299098 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:10.298976    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da801b51-9486-460c-a6f0-3f94d8999c04-metrics-certs\" (UniqueName: \"kubernetes.io/secret/da801b51-9486-460c-a6f0-3f94d8999c04-metrics-certs\") pod \"da801b51-9486-460c-a6f0-3f94d8999c04\" (UID: \"da801b51-9486-460c-a6f0-3f94d8999c04\") "
Nov 01 14:29:10.299098 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:10.299066    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da801b51-9486-460c-a6f0-3f94d8999c04-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da801b51-9486-460c-a6f0-3f94d8999c04-openshift-service-ca\") pod \"da801b51-9486-460c-a6f0-3f94d8999c04\" (UID: \"da801b51-9486-460c-a6f0-3f94d8999c04\") "
Nov 01 14:29:10.299220 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:10.299133    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da801b51-9486-460c-a6f0-3f94d8999c04-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da801b51-9486-460c-a6f0-3f94d8999c04-maas-api-tls\") pod \"da801b51-9486-460c-a6f0-3f94d8999c04\" (UID: \"da801b51-9486-460c-a6f0-3f94d8999c04\") "
Nov 01 14:29:10.300276 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:10.300206    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da801b51-9486-460c-a6f0-3f94d8999c04-openshift-service-ca" pod "da801b51-9486-460c-a6f0-3f94d8999c04" (UID: "da801b51-9486-460c-a6f0-3f94d8999c04"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:29:10.303781 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:10.303749    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da801b51-9486-460c-a6f0-3f94d8999c04-kube-api-access-c5rhg" pod "da801b51-9486-460c-a6f0-3f94d8999c04" (UID: "da801b51-9486-460c-a6f0-3f94d8999c04"). InnerVolumeSpecName "kube-api-access-c5rhg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:29:10.304370 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:10.304339    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da801b51-9486-460c-a6f0-3f94d8999c04-maas-api-tls" pod "da801b51-9486-460c-a6f0-3f94d8999c04" (UID: "da801b51-9486-460c-a6f0-3f94d8999c04"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:29:10.305991 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:10.305955    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da801b51-9486-460c-a6f0-3f94d8999c04-metrics-certs" pod "da801b51-9486-460c-a6f0-3f94d8999c04" (UID: "da801b51-9486-460c-a6f0-3f94d8999c04"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:29:10.401158 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:10.401124    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5rhg\" (UniqueName: \"kubernetes.io/projected/da801b51-9486-460c-a6f0-3f94d8999c04-kube-api-access-c5rhg\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:29:10.401158 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:10.401153    4993 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da801b51-9486-460c-a6f0-3f94d8999c04-metrics-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:29:10.401158 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:10.401163    4993 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da801b51-9486-460c-a6f0-3f94d8999c04-openshift-service-ca\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:29:10.401440 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:10.401189    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da801b51-9486-460c-a6f0-3f94d8999c04-maas-api-tls\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:29:10.424295 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:10.424207    4993 generic.go:356] "Generic (PLEG): container finished" podID="da801b51-9486-460c-a6f0-3f94d8999c04" containerID="84225cf13898926e656612a28ecf817494313dbc44500b9ce7e711cee07c64f8" exitCode=0
Nov 01 14:29:10.424295 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:10.424277    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e27521d-5d7bb5dc5f-wdv8b"
Nov 01 14:29:10.424486 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:10.424337    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e27521d-5d7bb5dc5f-wdv8b" event={"ID":"da801b51-9486-460c-a6f0-3f94d8999c04","Type":"ContainerDied","Data":"84225cf13898926e656612a28ecf817494313dbc44500b9ce7e711cee07c64f8"}
Nov 01 14:29:10.424486 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:10.424387    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e27521d-5d7bb5dc5f-wdv8b" event={"ID":"da801b51-9486-460c-a6f0-3f94d8999c04","Type":"ContainerDied","Data":"98261d42317cac8c1c531cbc5d97b06698edcfa20361fc6116583d6d0d241d52"}
Nov 01 14:29:10.424486 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:10.424418    4993 scope.go:122] "RemoveContainer" containerID="84225cf13898926e656612a28ecf817494313dbc44500b9ce7e711cee07c64f8"
Nov 01 14:29:10.435582 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:10.435560    4993 scope.go:122] "RemoveContainer" containerID="84225cf13898926e656612a28ecf817494313dbc44500b9ce7e711cee07c64f8"
Nov 01 14:29:10.435843 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:29:10.435823    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84225cf13898926e656612a28ecf817494313dbc44500b9ce7e711cee07c64f8\": container with ID starting with 84225cf13898926e656612a28ecf817494313dbc44500b9ce7e711cee07c64f8 not found: ID does not exist" containerID="84225cf13898926e656612a28ecf817494313dbc44500b9ce7e711cee07c64f8"
Nov 01 14:29:10.435894 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:10.435853    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84225cf13898926e656612a28ecf817494313dbc44500b9ce7e711cee07c64f8"} err="failed to get container status \"84225cf13898926e656612a28ecf817494313dbc44500b9ce7e711cee07c64f8\": rpc error: code = NotFound desc = could not find container \"84225cf13898926e656612a28ecf817494313dbc44500b9ce7e711cee07c64f8\": container with ID starting with 84225cf13898926e656612a28ecf817494313dbc44500b9ce7e711cee07c64f8 not found: ID does not exist"
Nov 01 14:29:10.579203 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:10.579158    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0e27521d-5d7bb5dc5f-wdv8b"]
Nov 01 14:29:10.588927 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:10.588898    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0e27521d-5d7bb5dc5f-wdv8b"]
Nov 01 14:29:10.796422 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:10.796390    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da801b51-9486-460c-a6f0-3f94d8999c04" path="/var/lib/kubelet/pods/da801b51-9486-460c-a6f0-3f94d8999c04/volumes"
Nov 01 14:29:15.029945 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:15.029901    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-c9fe7b2a-86cc89645f-mcpdq" podUID="d97cc8d0-8067-4899-9a62-3046e2a103a7" containerName="payload-processing" containerID="cri-o://75ed000c1797e72bbcab7897f9fc7211b100c2cdfe10212f5cba87910dd3a994" gracePeriod=25
Nov 01 14:29:15.290942 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:15.290886    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-c9fe7b2a-86cc89645f-mcpdq"
Nov 01 14:29:15.448976 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:15.448944    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d97cc8d0-8067-4899-9a62-3046e2a103a7-kube-api-access-4lrpn\" (UniqueName: \"kubernetes.io/projected/d97cc8d0-8067-4899-9a62-3046e2a103a7-kube-api-access-4lrpn\") pod \"d97cc8d0-8067-4899-9a62-3046e2a103a7\" (UID: \"d97cc8d0-8067-4899-9a62-3046e2a103a7\") "
Nov 01 14:29:15.449140 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:15.449121    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d97cc8d0-8067-4899-9a62-3046e2a103a7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d97cc8d0-8067-4899-9a62-3046e2a103a7-plugins-config-volume\") pod \"d97cc8d0-8067-4899-9a62-3046e2a103a7\" (UID: \"d97cc8d0-8067-4899-9a62-3046e2a103a7\") "
Nov 01 14:29:15.449484 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:15.449462    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d97cc8d0-8067-4899-9a62-3046e2a103a7-plugins-config-volume" pod "d97cc8d0-8067-4899-9a62-3046e2a103a7" (UID: "d97cc8d0-8067-4899-9a62-3046e2a103a7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:29:15.451387 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:15.451361    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d97cc8d0-8067-4899-9a62-3046e2a103a7-kube-api-access-4lrpn" pod "d97cc8d0-8067-4899-9a62-3046e2a103a7" (UID: "d97cc8d0-8067-4899-9a62-3046e2a103a7"). InnerVolumeSpecName "kube-api-access-4lrpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:29:15.451564 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:15.451540    4993 generic.go:356] "Generic (PLEG): container finished" podID="d97cc8d0-8067-4899-9a62-3046e2a103a7" containerID="75ed000c1797e72bbcab7897f9fc7211b100c2cdfe10212f5cba87910dd3a994" exitCode=0
Nov 01 14:29:15.451631 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:15.451591    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-c9fe7b2a-86cc89645f-mcpdq"
Nov 01 14:29:15.451631 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:15.451617    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-c9fe7b2a-86cc89645f-mcpdq" event={"ID":"d97cc8d0-8067-4899-9a62-3046e2a103a7","Type":"ContainerDied","Data":"75ed000c1797e72bbcab7897f9fc7211b100c2cdfe10212f5cba87910dd3a994"}
Nov 01 14:29:15.451730 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:15.451669    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-c9fe7b2a-86cc89645f-mcpdq" event={"ID":"d97cc8d0-8067-4899-9a62-3046e2a103a7","Type":"ContainerDied","Data":"ed763d7bffc33335565983c69091acde4c81f6faa01e3872bf5aa37666f25cc8"}
Nov 01 14:29:15.451730 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:15.451691    4993 scope.go:122] "RemoveContainer" containerID="75ed000c1797e72bbcab7897f9fc7211b100c2cdfe10212f5cba87910dd3a994"
Nov 01 14:29:15.463651 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:15.463635    4993 scope.go:122] "RemoveContainer" containerID="75ed000c1797e72bbcab7897f9fc7211b100c2cdfe10212f5cba87910dd3a994"
Nov 01 14:29:15.463935 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:29:15.463914    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75ed000c1797e72bbcab7897f9fc7211b100c2cdfe10212f5cba87910dd3a994\": container with ID starting with 75ed000c1797e72bbcab7897f9fc7211b100c2cdfe10212f5cba87910dd3a994 not found: ID does not exist" containerID="75ed000c1797e72bbcab7897f9fc7211b100c2cdfe10212f5cba87910dd3a994"
Nov 01 14:29:15.464031 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:15.463942    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75ed000c1797e72bbcab7897f9fc7211b100c2cdfe10212f5cba87910dd3a994"} err="failed to get container status \"75ed000c1797e72bbcab7897f9fc7211b100c2cdfe10212f5cba87910dd3a994\": rpc error: code = NotFound desc = could not find container \"75ed000c1797e72bbcab7897f9fc7211b100c2cdfe10212f5cba87910dd3a994\": container with ID starting with 75ed000c1797e72bbcab7897f9fc7211b100c2cdfe10212f5cba87910dd3a994 not found: ID does not exist"
Nov 01 14:29:15.550115 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:15.550027    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lrpn\" (UniqueName: \"kubernetes.io/projected/d97cc8d0-8067-4899-9a62-3046e2a103a7-kube-api-access-4lrpn\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:29:15.550115 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:15.550058    4993 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d97cc8d0-8067-4899-9a62-3046e2a103a7-plugins-config-volume\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:29:15.613298 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:15.613258    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c9fe7b2a-86cc89645f-mcpdq"]
Nov 01 14:29:15.620496 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:15.620445    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c9fe7b2a-86cc89645f-mcpdq"]
Nov 01 14:29:16.795863 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:16.795828    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d97cc8d0-8067-4899-9a62-3046e2a103a7" path="/var/lib/kubelet/pods/d97cc8d0-8067-4899-9a62-3046e2a103a7/volumes"
Nov 01 14:29:17.091085 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:17.090940    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c0f3259b-7b67d68778-ncxdk"]
Nov 01 14:29:17.091285 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:17.091220    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c0f3259b-7b67d68778-ncxdk" podUID="cbb0c8ac-9486-4e9a-820f-bd569b28e75a" containerName="maas-api" containerID="cri-o://6a23a6ac314f95baf504ba8395297cb36ea30322e82689013a971107e4408d0e" gracePeriod=30
Nov 01 14:29:17.356963 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:17.356900    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c0f3259b-7b67d68778-ncxdk"
Nov 01 14:29:17.469855 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:17.468937    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbb0c8ac-9486-4e9a-820f-bd569b28e75a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbb0c8ac-9486-4e9a-820f-bd569b28e75a-maas-api-tls\") pod \"cbb0c8ac-9486-4e9a-820f-bd569b28e75a\" (UID: \"cbb0c8ac-9486-4e9a-820f-bd569b28e75a\") "
Nov 01 14:29:17.469855 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:17.469022    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbb0c8ac-9486-4e9a-820f-bd569b28e75a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbb0c8ac-9486-4e9a-820f-bd569b28e75a-metrics-certs\") pod \"cbb0c8ac-9486-4e9a-820f-bd569b28e75a\" (UID: \"cbb0c8ac-9486-4e9a-820f-bd569b28e75a\") "
Nov 01 14:29:17.469855 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:17.469075    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbb0c8ac-9486-4e9a-820f-bd569b28e75a-kube-api-access-qc2n6\" (UniqueName: \"kubernetes.io/projected/cbb0c8ac-9486-4e9a-820f-bd569b28e75a-kube-api-access-qc2n6\") pod \"cbb0c8ac-9486-4e9a-820f-bd569b28e75a\" (UID: \"cbb0c8ac-9486-4e9a-820f-bd569b28e75a\") "
Nov 01 14:29:17.469855 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:17.469159    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbb0c8ac-9486-4e9a-820f-bd569b28e75a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbb0c8ac-9486-4e9a-820f-bd569b28e75a-openshift-service-ca\") pod \"cbb0c8ac-9486-4e9a-820f-bd569b28e75a\" (UID: \"cbb0c8ac-9486-4e9a-820f-bd569b28e75a\") "
Nov 01 14:29:17.471120 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:17.471027    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbb0c8ac-9486-4e9a-820f-bd569b28e75a-openshift-service-ca" pod "cbb0c8ac-9486-4e9a-820f-bd569b28e75a" (UID: "cbb0c8ac-9486-4e9a-820f-bd569b28e75a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:29:17.497431 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:17.496631    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbb0c8ac-9486-4e9a-820f-bd569b28e75a-kube-api-access-qc2n6" pod "cbb0c8ac-9486-4e9a-820f-bd569b28e75a" (UID: "cbb0c8ac-9486-4e9a-820f-bd569b28e75a"). InnerVolumeSpecName "kube-api-access-qc2n6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:29:17.502468 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:17.502016    4993 generic.go:356] "Generic (PLEG): container finished" podID="cbb0c8ac-9486-4e9a-820f-bd569b28e75a" containerID="6a23a6ac314f95baf504ba8395297cb36ea30322e82689013a971107e4408d0e" exitCode=0
Nov 01 14:29:17.502468 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:17.502072    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c0f3259b-7b67d68778-ncxdk"
Nov 01 14:29:17.502468 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:17.502222    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c0f3259b-7b67d68778-ncxdk" event={"ID":"cbb0c8ac-9486-4e9a-820f-bd569b28e75a","Type":"ContainerDied","Data":"6a23a6ac314f95baf504ba8395297cb36ea30322e82689013a971107e4408d0e"}
Nov 01 14:29:17.502468 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:17.502271    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c0f3259b-7b67d68778-ncxdk" event={"ID":"cbb0c8ac-9486-4e9a-820f-bd569b28e75a","Type":"ContainerDied","Data":"98ee52dbabdee5592ff063a3eaccdcdad44aa529e1c76319fecef3b08ea064d7"}
Nov 01 14:29:17.502468 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:17.502299    4993 scope.go:122] "RemoveContainer" containerID="6a23a6ac314f95baf504ba8395297cb36ea30322e82689013a971107e4408d0e"
Nov 01 14:29:17.529069 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:17.526334    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbb0c8ac-9486-4e9a-820f-bd569b28e75a-metrics-certs" pod "cbb0c8ac-9486-4e9a-820f-bd569b28e75a" (UID: "cbb0c8ac-9486-4e9a-820f-bd569b28e75a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:29:17.529069 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:17.526429    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbb0c8ac-9486-4e9a-820f-bd569b28e75a-maas-api-tls" pod "cbb0c8ac-9486-4e9a-820f-bd569b28e75a" (UID: "cbb0c8ac-9486-4e9a-820f-bd569b28e75a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:29:17.544472 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:17.544442    4993 scope.go:122] "RemoveContainer" containerID="6a23a6ac314f95baf504ba8395297cb36ea30322e82689013a971107e4408d0e"
Nov 01 14:29:17.544885 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:29:17.544855    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a23a6ac314f95baf504ba8395297cb36ea30322e82689013a971107e4408d0e\": container with ID starting with 6a23a6ac314f95baf504ba8395297cb36ea30322e82689013a971107e4408d0e not found: ID does not exist" containerID="6a23a6ac314f95baf504ba8395297cb36ea30322e82689013a971107e4408d0e"
Nov 01 14:29:17.545006 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:17.544900    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a23a6ac314f95baf504ba8395297cb36ea30322e82689013a971107e4408d0e"} err="failed to get container status \"6a23a6ac314f95baf504ba8395297cb36ea30322e82689013a971107e4408d0e\": rpc error: code = NotFound desc = could not find container \"6a23a6ac314f95baf504ba8395297cb36ea30322e82689013a971107e4408d0e\": container with ID starting with 6a23a6ac314f95baf504ba8395297cb36ea30322e82689013a971107e4408d0e not found: ID does not exist"
Nov 01 14:29:17.570117 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:17.570082    4993 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbb0c8ac-9486-4e9a-820f-bd569b28e75a-metrics-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:29:17.570117 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:17.570111    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qc2n6\" (UniqueName: \"kubernetes.io/projected/cbb0c8ac-9486-4e9a-820f-bd569b28e75a-kube-api-access-qc2n6\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:29:17.570117 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:17.570121    4993 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbb0c8ac-9486-4e9a-820f-bd569b28e75a-openshift-service-ca\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:29:17.570406 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:17.570131    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbb0c8ac-9486-4e9a-820f-bd569b28e75a-maas-api-tls\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:29:17.915772 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:17.915737    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c0f3259b-7b67d68778-ncxdk"]
Nov 01 14:29:17.920479 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:17.920453    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c0f3259b-7b67d68778-ncxdk"]
Nov 01 14:29:18.454879 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:18.454811    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f7c71759-6b64666d74-9hb2t"]
Nov 01 14:29:18.455036 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:18.455007    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7c71759-6b64666d74-9hb2t" podUID="3b48e6b8-b98c-4c82-880b-150630625fe1" containerName="maas-api" containerID="cri-o://6c4757b2fdb35cdf384b9c454d67dc826465deb9f14afcec3180e5657651c5c7" gracePeriod=30
Nov 01 14:29:18.717583 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:18.717525    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7c71759-6b64666d74-9hb2t"
Nov 01 14:29:18.793378 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:18.793344    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbb0c8ac-9486-4e9a-820f-bd569b28e75a" path="/var/lib/kubelet/pods/cbb0c8ac-9486-4e9a-820f-bd569b28e75a/volumes"
Nov 01 14:29:18.882208 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:18.882156    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b48e6b8-b98c-4c82-880b-150630625fe1-kube-api-access-rj2vz\" (UniqueName: \"kubernetes.io/projected/3b48e6b8-b98c-4c82-880b-150630625fe1-kube-api-access-rj2vz\") pod \"3b48e6b8-b98c-4c82-880b-150630625fe1\" (UID: \"3b48e6b8-b98c-4c82-880b-150630625fe1\") "
Nov 01 14:29:18.882359 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:18.882257    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b48e6b8-b98c-4c82-880b-150630625fe1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b48e6b8-b98c-4c82-880b-150630625fe1-openshift-service-ca\") pod \"3b48e6b8-b98c-4c82-880b-150630625fe1\" (UID: \"3b48e6b8-b98c-4c82-880b-150630625fe1\") "
Nov 01 14:29:18.882359 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:18.882289    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b48e6b8-b98c-4c82-880b-150630625fe1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b48e6b8-b98c-4c82-880b-150630625fe1-maas-api-tls\") pod \"3b48e6b8-b98c-4c82-880b-150630625fe1\" (UID: \"3b48e6b8-b98c-4c82-880b-150630625fe1\") "
Nov 01 14:29:18.882359 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:18.882333    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b48e6b8-b98c-4c82-880b-150630625fe1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b48e6b8-b98c-4c82-880b-150630625fe1-metrics-certs\") pod \"3b48e6b8-b98c-4c82-880b-150630625fe1\" (UID: \"3b48e6b8-b98c-4c82-880b-150630625fe1\") "
Nov 01 14:29:18.882634 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:18.882600    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b48e6b8-b98c-4c82-880b-150630625fe1-openshift-service-ca" pod "3b48e6b8-b98c-4c82-880b-150630625fe1" (UID: "3b48e6b8-b98c-4c82-880b-150630625fe1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:29:18.884358 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:18.884332    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b48e6b8-b98c-4c82-880b-150630625fe1-metrics-certs" pod "3b48e6b8-b98c-4c82-880b-150630625fe1" (UID: "3b48e6b8-b98c-4c82-880b-150630625fe1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:29:18.884463 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:18.884427    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b48e6b8-b98c-4c82-880b-150630625fe1-kube-api-access-rj2vz" pod "3b48e6b8-b98c-4c82-880b-150630625fe1" (UID: "3b48e6b8-b98c-4c82-880b-150630625fe1"). InnerVolumeSpecName "kube-api-access-rj2vz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:29:18.884522 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:18.884462    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b48e6b8-b98c-4c82-880b-150630625fe1-maas-api-tls" pod "3b48e6b8-b98c-4c82-880b-150630625fe1" (UID: "3b48e6b8-b98c-4c82-880b-150630625fe1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:29:18.983971 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:18.983937    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rj2vz\" (UniqueName: \"kubernetes.io/projected/3b48e6b8-b98c-4c82-880b-150630625fe1-kube-api-access-rj2vz\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:29:18.983971 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:18.983967    4993 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b48e6b8-b98c-4c82-880b-150630625fe1-openshift-service-ca\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:29:18.984392 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:18.983978    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b48e6b8-b98c-4c82-880b-150630625fe1-maas-api-tls\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:29:18.984392 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:18.983986    4993 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b48e6b8-b98c-4c82-880b-150630625fe1-metrics-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:29:19.518762 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:19.518724    4993 generic.go:356] "Generic (PLEG): container finished" podID="3b48e6b8-b98c-4c82-880b-150630625fe1" containerID="6c4757b2fdb35cdf384b9c454d67dc826465deb9f14afcec3180e5657651c5c7" exitCode=0
Nov 01 14:29:19.518927 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:19.518777    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7c71759-6b64666d74-9hb2t"
Nov 01 14:29:19.518927 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:19.518803    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7c71759-6b64666d74-9hb2t" event={"ID":"3b48e6b8-b98c-4c82-880b-150630625fe1","Type":"ContainerDied","Data":"6c4757b2fdb35cdf384b9c454d67dc826465deb9f14afcec3180e5657651c5c7"}
Nov 01 14:29:19.518927 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:19.518858    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f7c71759-6b64666d74-9hb2t" event={"ID":"3b48e6b8-b98c-4c82-880b-150630625fe1","Type":"ContainerDied","Data":"eb8e63a57646bcc53abeaa4978f905ed7df94a2883f6383ba1243592caa51b24"}
Nov 01 14:29:19.518927 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:19.518878    4993 scope.go:122] "RemoveContainer" containerID="6c4757b2fdb35cdf384b9c454d67dc826465deb9f14afcec3180e5657651c5c7"
Nov 01 14:29:19.528001 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:19.527975    4993 scope.go:122] "RemoveContainer" containerID="6c4757b2fdb35cdf384b9c454d67dc826465deb9f14afcec3180e5657651c5c7"
Nov 01 14:29:19.528264 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:29:19.528245    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c4757b2fdb35cdf384b9c454d67dc826465deb9f14afcec3180e5657651c5c7\": container with ID starting with 6c4757b2fdb35cdf384b9c454d67dc826465deb9f14afcec3180e5657651c5c7 not found: ID does not exist" containerID="6c4757b2fdb35cdf384b9c454d67dc826465deb9f14afcec3180e5657651c5c7"
Nov 01 14:29:19.528326 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:19.528272    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c4757b2fdb35cdf384b9c454d67dc826465deb9f14afcec3180e5657651c5c7"} err="failed to get container status \"6c4757b2fdb35cdf384b9c454d67dc826465deb9f14afcec3180e5657651c5c7\": rpc error: code = NotFound desc = could not find container \"6c4757b2fdb35cdf384b9c454d67dc826465deb9f14afcec3180e5657651c5c7\": container with ID starting with 6c4757b2fdb35cdf384b9c454d67dc826465deb9f14afcec3180e5657651c5c7 not found: ID does not exist"
Nov 01 14:29:19.669680 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:19.669651    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f7c71759-6b64666d74-9hb2t"]
Nov 01 14:29:19.715957 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:19.715925    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f7c71759-6b64666d74-9hb2t"]
Nov 01 14:29:20.794321 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:20.794288    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b48e6b8-b98c-4c82-880b-150630625fe1" path="/var/lib/kubelet/pods/3b48e6b8-b98c-4c82-880b-150630625fe1/volumes"
Nov 01 14:29:25.049292 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:25.049258    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-007270-7bf55ff5fc-88krj"]
Nov 01 14:29:25.049701 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:25.049686    4993 cpu_manager.go:414] "containerMap: removing container" podUID="d97cc8d0-8067-4899-9a62-3046e2a103a7" containerName="payload-processing"
Nov 01 14:29:25.049742 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:25.049703    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d97cc8d0-8067-4899-9a62-3046e2a103a7" containerName="payload-processing"
Nov 01 14:29:25.049742 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:25.049714    4993 cpu_manager.go:414] "containerMap: removing container" podUID="da801b51-9486-460c-a6f0-3f94d8999c04" containerName="maas-api"
Nov 01 14:29:25.049742 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:25.049719    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da801b51-9486-460c-a6f0-3f94d8999c04" containerName="maas-api"
Nov 01 14:29:25.049742 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:25.049738    4993 cpu_manager.go:414] "containerMap: removing container" podUID="3b48e6b8-b98c-4c82-880b-150630625fe1" containerName="maas-api"
Nov 01 14:29:25.049855 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:25.049744    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b48e6b8-b98c-4c82-880b-150630625fe1" containerName="maas-api"
Nov 01 14:29:25.049855 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:25.049767    4993 cpu_manager.go:414] "containerMap: removing container" podUID="cbb0c8ac-9486-4e9a-820f-bd569b28e75a" containerName="maas-api"
Nov 01 14:29:25.049855 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:25.049774    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbb0c8ac-9486-4e9a-820f-bd569b28e75a" containerName="maas-api"
Nov 01 14:29:25.049855 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:25.049833    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b48e6b8-b98c-4c82-880b-150630625fe1" containerName="maas-api"
Nov 01 14:29:25.049855 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:25.049847    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbb0c8ac-9486-4e9a-820f-bd569b28e75a" containerName="maas-api"
Nov 01 14:29:25.049999 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:25.049856    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="d97cc8d0-8067-4899-9a62-3046e2a103a7" containerName="payload-processing"
Nov 01 14:29:25.049999 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:25.049863    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="da801b51-9486-460c-a6f0-3f94d8999c04" containerName="maas-api"
Nov 01 14:29:25.053225 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:25.053198    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-007270-7bf55ff5fc-88krj"
Nov 01 14:29:25.107138 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:25.107110    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 14:29:25.108223 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:25.108198    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-007270-serving-cert\""
Nov 01 14:29:25.135768 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:25.135734    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea582656-480c-4e92-868d-c1f36225e433-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-007270-7bf55ff5fc-88krj\" (UID: \"ea582656-480c-4e92-868d-c1f36225e433\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-007270-7bf55ff5fc-88krj"
Nov 01 14:29:25.135924 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:25.135776    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2qvv\" (UniqueName: \"kubernetes.io/projected/ea582656-480c-4e92-868d-c1f36225e433-kube-api-access-m2qvv\") pod \"maas-api-e2e-worker-w2-007270-7bf55ff5fc-88krj\" (UID: \"ea582656-480c-4e92-868d-c1f36225e433\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-007270-7bf55ff5fc-88krj"
Nov 01 14:29:25.135924 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:25.135880    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea582656-480c-4e92-868d-c1f36225e433-maas-api-tls\") pod \"maas-api-e2e-worker-w2-007270-7bf55ff5fc-88krj\" (UID: \"ea582656-480c-4e92-868d-c1f36225e433\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-007270-7bf55ff5fc-88krj"
Nov 01 14:29:25.136038 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:25.135957    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea582656-480c-4e92-868d-c1f36225e433-metrics-certs\") pod \"maas-api-e2e-worker-w2-007270-7bf55ff5fc-88krj\" (UID: \"ea582656-480c-4e92-868d-c1f36225e433\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-007270-7bf55ff5fc-88krj"
Nov 01 14:29:25.153601 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:25.153576    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 14:29:25.153601 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:25.153574    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 01 14:29:25.154427 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:25.154407    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7t7jm\""
Nov 01 14:29:25.156662 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:25.156641    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-007270-7bf55ff5fc-88krj"]
Nov 01 14:29:25.236748 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:25.236719    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea582656-480c-4e92-868d-c1f36225e433-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-007270-7bf55ff5fc-88krj\" (UID: \"ea582656-480c-4e92-868d-c1f36225e433\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-007270-7bf55ff5fc-88krj"
Nov 01 14:29:25.236903 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:25.236754    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2qvv\" (UniqueName: \"kubernetes.io/projected/ea582656-480c-4e92-868d-c1f36225e433-kube-api-access-m2qvv\") pod \"maas-api-e2e-worker-w2-007270-7bf55ff5fc-88krj\" (UID: \"ea582656-480c-4e92-868d-c1f36225e433\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-007270-7bf55ff5fc-88krj"
Nov 01 14:29:25.236903 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:25.236802    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea582656-480c-4e92-868d-c1f36225e433-maas-api-tls\") pod \"maas-api-e2e-worker-w2-007270-7bf55ff5fc-88krj\" (UID: \"ea582656-480c-4e92-868d-c1f36225e433\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-007270-7bf55ff5fc-88krj"
Nov 01 14:29:25.236903 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:25.236851    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea582656-480c-4e92-868d-c1f36225e433-metrics-certs\") pod \"maas-api-e2e-worker-w2-007270-7bf55ff5fc-88krj\" (UID: \"ea582656-480c-4e92-868d-c1f36225e433\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-007270-7bf55ff5fc-88krj"
Nov 01 14:29:25.237021 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:29:25.236939    4993 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w2-007270-serving-cert: secret "maas-api-e2e-worker-w2-007270-serving-cert" not found
Nov 01 14:29:25.237021 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:29:25.237002    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea582656-480c-4e92-868d-c1f36225e433-maas-api-tls podName:ea582656-480c-4e92-868d-c1f36225e433 nodeName:}" failed. No retries permitted until 2026-11-01 14:29:25.736985255 +0000 UTC m=+753.066847969 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ea582656-480c-4e92-868d-c1f36225e433-maas-api-tls") pod "maas-api-e2e-worker-w2-007270-7bf55ff5fc-88krj" (UID: "ea582656-480c-4e92-868d-c1f36225e433") : secret "maas-api-e2e-worker-w2-007270-serving-cert" not found
Nov 01 14:29:25.237412 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:25.237393    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea582656-480c-4e92-868d-c1f36225e433-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-007270-7bf55ff5fc-88krj\" (UID: \"ea582656-480c-4e92-868d-c1f36225e433\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-007270-7bf55ff5fc-88krj"
Nov 01 14:29:25.239194 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:25.239155    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea582656-480c-4e92-868d-c1f36225e433-metrics-certs\") pod \"maas-api-e2e-worker-w2-007270-7bf55ff5fc-88krj\" (UID: \"ea582656-480c-4e92-868d-c1f36225e433\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-007270-7bf55ff5fc-88krj"
Nov 01 14:29:25.291539 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:25.291501    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2qvv\" (UniqueName: \"kubernetes.io/projected/ea582656-480c-4e92-868d-c1f36225e433-kube-api-access-m2qvv\") pod \"maas-api-e2e-worker-w2-007270-7bf55ff5fc-88krj\" (UID: \"ea582656-480c-4e92-868d-c1f36225e433\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-007270-7bf55ff5fc-88krj"
Nov 01 14:29:25.742534 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:25.742493    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea582656-480c-4e92-868d-c1f36225e433-maas-api-tls\") pod \"maas-api-e2e-worker-w2-007270-7bf55ff5fc-88krj\" (UID: \"ea582656-480c-4e92-868d-c1f36225e433\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-007270-7bf55ff5fc-88krj"
Nov 01 14:29:25.744892 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:25.744866    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea582656-480c-4e92-868d-c1f36225e433-maas-api-tls\") pod \"maas-api-e2e-worker-w2-007270-7bf55ff5fc-88krj\" (UID: \"ea582656-480c-4e92-868d-c1f36225e433\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-007270-7bf55ff5fc-88krj"
Nov 01 14:29:25.965789 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:25.965755    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-007270-7bf55ff5fc-88krj"
Nov 01 14:29:26.349473 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:26.349391    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-007270-7bf55ff5fc-88krj"]
Nov 01 14:29:26.355817 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:29:26.355788    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea582656_480c_4e92_868d_c1f36225e433.slice/crio-9f74ef8749865c7d51b96f699dffb082b41b7d6b97ac36e5a11dc52faefb94e0 WatchSource:0}: Error finding container 9f74ef8749865c7d51b96f699dffb082b41b7d6b97ac36e5a11dc52faefb94e0: Status 404 returned error can't find the container with id 9f74ef8749865c7d51b96f699dffb082b41b7d6b97ac36e5a11dc52faefb94e0
Nov 01 14:29:26.556163 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:26.556125    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-007270-7bf55ff5fc-88krj" event={"ID":"ea582656-480c-4e92-868d-c1f36225e433","Type":"ContainerStarted","Data":"9f74ef8749865c7d51b96f699dffb082b41b7d6b97ac36e5a11dc52faefb94e0"}
Nov 01 14:29:27.498853 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:29:27.498820    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:29:27.562974 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:27.562889    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-007270-7bf55ff5fc-88krj" event={"ID":"ea582656-480c-4e92-868d-c1f36225e433","Type":"ContainerStarted","Data":"5816b8dc1eaad31d3aa4c76157985729dd1cfc85d6aae5e3545bda160df0f5d8"}
Nov 01 14:29:27.563112 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:27.562999    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-007270-7bf55ff5fc-88krj"
Nov 01 14:29:27.671762 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:27.671712    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-007270-7bf55ff5fc-88krj" podStartSLOduration=1.724130073 podStartE2EDuration="2.671698806s" totalImagesPullingTime="947.568733ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:29:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:29:27.670908936 +0000 UTC m=+755.000771672" watchObservedRunningTime="2026-11-01 14:29:27.671698806 +0000 UTC m=+755.001561541"
Nov 01 14:29:33.574274 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:33.574194    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-007270-7bf55ff5fc-88krj"
Nov 01 14:29:39.883282 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:39.883247    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-1f8e2c-7bcc77fd5d-8lxsw"]
Nov 01 14:29:39.886890 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:39.886874    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1f8e2c-7bcc77fd5d-8lxsw"
Nov 01 14:29:39.938733 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:39.938709    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-1f8e2c-serving-cert\""
Nov 01 14:29:39.974902 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:39.974869    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac70b22c-c696-4be4-84cc-593406d0cda6-metrics-certs\") pod \"maas-api-e2e-worker-w0-1f8e2c-7bcc77fd5d-8lxsw\" (UID: \"ac70b22c-c696-4be4-84cc-593406d0cda6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1f8e2c-7bcc77fd5d-8lxsw"
Nov 01 14:29:39.974902 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:39.974906    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac70b22c-c696-4be4-84cc-593406d0cda6-maas-api-tls\") pod \"maas-api-e2e-worker-w0-1f8e2c-7bcc77fd5d-8lxsw\" (UID: \"ac70b22c-c696-4be4-84cc-593406d0cda6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1f8e2c-7bcc77fd5d-8lxsw"
Nov 01 14:29:39.975084 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:39.974994    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwcq6\" (UniqueName: \"kubernetes.io/projected/ac70b22c-c696-4be4-84cc-593406d0cda6-kube-api-access-rwcq6\") pod \"maas-api-e2e-worker-w0-1f8e2c-7bcc77fd5d-8lxsw\" (UID: \"ac70b22c-c696-4be4-84cc-593406d0cda6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1f8e2c-7bcc77fd5d-8lxsw"
Nov 01 14:29:39.975152 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:39.975138    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac70b22c-c696-4be4-84cc-593406d0cda6-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-1f8e2c-7bcc77fd5d-8lxsw\" (UID: \"ac70b22c-c696-4be4-84cc-593406d0cda6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1f8e2c-7bcc77fd5d-8lxsw"
Nov 01 14:29:39.985331 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:39.985299    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-1f8e2c-7bcc77fd5d-8lxsw"]
Nov 01 14:29:40.075690 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:40.075658    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac70b22c-c696-4be4-84cc-593406d0cda6-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-1f8e2c-7bcc77fd5d-8lxsw\" (UID: \"ac70b22c-c696-4be4-84cc-593406d0cda6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1f8e2c-7bcc77fd5d-8lxsw"
Nov 01 14:29:40.075852 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:40.075720    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac70b22c-c696-4be4-84cc-593406d0cda6-metrics-certs\") pod \"maas-api-e2e-worker-w0-1f8e2c-7bcc77fd5d-8lxsw\" (UID: \"ac70b22c-c696-4be4-84cc-593406d0cda6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1f8e2c-7bcc77fd5d-8lxsw"
Nov 01 14:29:40.075852 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:40.075748    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac70b22c-c696-4be4-84cc-593406d0cda6-maas-api-tls\") pod \"maas-api-e2e-worker-w0-1f8e2c-7bcc77fd5d-8lxsw\" (UID: \"ac70b22c-c696-4be4-84cc-593406d0cda6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1f8e2c-7bcc77fd5d-8lxsw"
Nov 01 14:29:40.075852 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:40.075800    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwcq6\" (UniqueName: \"kubernetes.io/projected/ac70b22c-c696-4be4-84cc-593406d0cda6-kube-api-access-rwcq6\") pod \"maas-api-e2e-worker-w0-1f8e2c-7bcc77fd5d-8lxsw\" (UID: \"ac70b22c-c696-4be4-84cc-593406d0cda6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1f8e2c-7bcc77fd5d-8lxsw"
Nov 01 14:29:40.076004 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:29:40.075881    4993 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-1f8e2c-serving-cert: secret "maas-api-e2e-worker-w0-1f8e2c-serving-cert" not found
Nov 01 14:29:40.076004 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:29:40.075974    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac70b22c-c696-4be4-84cc-593406d0cda6-maas-api-tls podName:ac70b22c-c696-4be4-84cc-593406d0cda6 nodeName:}" failed. No retries permitted until 2026-11-01 14:29:40.575954891 +0000 UTC m=+767.905817606 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ac70b22c-c696-4be4-84cc-593406d0cda6-maas-api-tls") pod "maas-api-e2e-worker-w0-1f8e2c-7bcc77fd5d-8lxsw" (UID: "ac70b22c-c696-4be4-84cc-593406d0cda6") : secret "maas-api-e2e-worker-w0-1f8e2c-serving-cert" not found
Nov 01 14:29:40.076364 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:40.076341    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac70b22c-c696-4be4-84cc-593406d0cda6-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-1f8e2c-7bcc77fd5d-8lxsw\" (UID: \"ac70b22c-c696-4be4-84cc-593406d0cda6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1f8e2c-7bcc77fd5d-8lxsw"
Nov 01 14:29:40.078278 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:40.078245    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac70b22c-c696-4be4-84cc-593406d0cda6-metrics-certs\") pod \"maas-api-e2e-worker-w0-1f8e2c-7bcc77fd5d-8lxsw\" (UID: \"ac70b22c-c696-4be4-84cc-593406d0cda6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1f8e2c-7bcc77fd5d-8lxsw"
Nov 01 14:29:40.131734 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:40.131701    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwcq6\" (UniqueName: \"kubernetes.io/projected/ac70b22c-c696-4be4-84cc-593406d0cda6-kube-api-access-rwcq6\") pod \"maas-api-e2e-worker-w0-1f8e2c-7bcc77fd5d-8lxsw\" (UID: \"ac70b22c-c696-4be4-84cc-593406d0cda6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1f8e2c-7bcc77fd5d-8lxsw"
Nov 01 14:29:40.580378 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:40.580342    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac70b22c-c696-4be4-84cc-593406d0cda6-maas-api-tls\") pod \"maas-api-e2e-worker-w0-1f8e2c-7bcc77fd5d-8lxsw\" (UID: \"ac70b22c-c696-4be4-84cc-593406d0cda6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1f8e2c-7bcc77fd5d-8lxsw"
Nov 01 14:29:40.582754 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:40.582724    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac70b22c-c696-4be4-84cc-593406d0cda6-maas-api-tls\") pod \"maas-api-e2e-worker-w0-1f8e2c-7bcc77fd5d-8lxsw\" (UID: \"ac70b22c-c696-4be4-84cc-593406d0cda6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1f8e2c-7bcc77fd5d-8lxsw"
Nov 01 14:29:40.800135 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:40.800101    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1f8e2c-7bcc77fd5d-8lxsw"
Nov 01 14:29:41.179676 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:41.179588    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-1f8e2c-7bcc77fd5d-8lxsw"]
Nov 01 14:29:41.186072 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:29:41.186043    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac70b22c_c696_4be4_84cc_593406d0cda6.slice/crio-c4e2d56a5a31ef22c78187e3b91b6c0e0742cee298ce0c8a75d441e6c174853a WatchSource:0}: Error finding container c4e2d56a5a31ef22c78187e3b91b6c0e0742cee298ce0c8a75d441e6c174853a: Status 404 returned error can't find the container with id c4e2d56a5a31ef22c78187e3b91b6c0e0742cee298ce0c8a75d441e6c174853a
Nov 01 14:29:41.636080 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:41.636045    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1f8e2c-7bcc77fd5d-8lxsw" event={"ID":"ac70b22c-c696-4be4-84cc-593406d0cda6","Type":"ContainerStarted","Data":"c4e2d56a5a31ef22c78187e3b91b6c0e0742cee298ce0c8a75d441e6c174853a"}
Nov 01 14:29:42.643205 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:42.643146    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1f8e2c-7bcc77fd5d-8lxsw" event={"ID":"ac70b22c-c696-4be4-84cc-593406d0cda6","Type":"ContainerStarted","Data":"3a0ea2e2d87f9b08e4d0796456f943824fb95c18e7b8e9bb038b6ea6c1abfbcb"}
Nov 01 14:29:42.643592 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:42.643236    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1f8e2c-7bcc77fd5d-8lxsw"
Nov 01 14:29:42.755926 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:42.755862    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1f8e2c-7bcc77fd5d-8lxsw" podStartSLOduration=2.765943399 podStartE2EDuration="3.755846344s" totalImagesPullingTime="989.902945ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:29:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:29:42.75443164 +0000 UTC m=+770.084294378" watchObservedRunningTime="2026-11-01 14:29:42.755846344 +0000 UTC m=+770.085709080"
Nov 01 14:29:48.653300 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:48.653269    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1f8e2c-7bcc77fd5d-8lxsw"
Nov 01 14:29:54.772788 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:54.772751    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-00a787-65c44bcd97-thn6r"]
Nov 01 14:29:54.777836 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:54.777809    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-00a787-65c44bcd97-thn6r"
Nov 01 14:29:54.826333 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:54.826298    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-00a787-serving-cert\""
Nov 01 14:29:54.875746 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:54.875710    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-00a787-65c44bcd97-thn6r"]
Nov 01 14:29:54.915749 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:54.915709    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0da21f81-aebc-40fd-9991-60e5631cf2d5-metrics-certs\") pod \"maas-api-e2e-worker-w1-00a787-65c44bcd97-thn6r\" (UID: \"0da21f81-aebc-40fd-9991-60e5631cf2d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-00a787-65c44bcd97-thn6r"
Nov 01 14:29:54.915936 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:54.915767    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2pq5\" (UniqueName: \"kubernetes.io/projected/0da21f81-aebc-40fd-9991-60e5631cf2d5-kube-api-access-h2pq5\") pod \"maas-api-e2e-worker-w1-00a787-65c44bcd97-thn6r\" (UID: \"0da21f81-aebc-40fd-9991-60e5631cf2d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-00a787-65c44bcd97-thn6r"
Nov 01 14:29:54.915936 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:54.915841    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0da21f81-aebc-40fd-9991-60e5631cf2d5-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-00a787-65c44bcd97-thn6r\" (UID: \"0da21f81-aebc-40fd-9991-60e5631cf2d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-00a787-65c44bcd97-thn6r"
Nov 01 14:29:54.915936 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:54.915883    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0da21f81-aebc-40fd-9991-60e5631cf2d5-maas-api-tls\") pod \"maas-api-e2e-worker-w1-00a787-65c44bcd97-thn6r\" (UID: \"0da21f81-aebc-40fd-9991-60e5631cf2d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-00a787-65c44bcd97-thn6r"
Nov 01 14:29:55.017034 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:55.016996    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0da21f81-aebc-40fd-9991-60e5631cf2d5-maas-api-tls\") pod \"maas-api-e2e-worker-w1-00a787-65c44bcd97-thn6r\" (UID: \"0da21f81-aebc-40fd-9991-60e5631cf2d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-00a787-65c44bcd97-thn6r"
Nov 01 14:29:55.017245 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:55.017067    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0da21f81-aebc-40fd-9991-60e5631cf2d5-metrics-certs\") pod \"maas-api-e2e-worker-w1-00a787-65c44bcd97-thn6r\" (UID: \"0da21f81-aebc-40fd-9991-60e5631cf2d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-00a787-65c44bcd97-thn6r"
Nov 01 14:29:55.017245 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:55.017090    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2pq5\" (UniqueName: \"kubernetes.io/projected/0da21f81-aebc-40fd-9991-60e5631cf2d5-kube-api-access-h2pq5\") pod \"maas-api-e2e-worker-w1-00a787-65c44bcd97-thn6r\" (UID: \"0da21f81-aebc-40fd-9991-60e5631cf2d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-00a787-65c44bcd97-thn6r"
Nov 01 14:29:55.017245 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:55.017139    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0da21f81-aebc-40fd-9991-60e5631cf2d5-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-00a787-65c44bcd97-thn6r\" (UID: \"0da21f81-aebc-40fd-9991-60e5631cf2d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-00a787-65c44bcd97-thn6r"
Nov 01 14:29:55.017245 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:29:55.017150    4993 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w1-00a787-serving-cert: secret "maas-api-e2e-worker-w1-00a787-serving-cert" not found
Nov 01 14:29:55.017441 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:29:55.017258    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0da21f81-aebc-40fd-9991-60e5631cf2d5-maas-api-tls podName:0da21f81-aebc-40fd-9991-60e5631cf2d5 nodeName:}" failed. No retries permitted until 2026-11-01 14:29:55.517241216 +0000 UTC m=+782.847103931 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0da21f81-aebc-40fd-9991-60e5631cf2d5-maas-api-tls") pod "maas-api-e2e-worker-w1-00a787-65c44bcd97-thn6r" (UID: "0da21f81-aebc-40fd-9991-60e5631cf2d5") : secret "maas-api-e2e-worker-w1-00a787-serving-cert" not found
Nov 01 14:29:55.017873 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:55.017855    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0da21f81-aebc-40fd-9991-60e5631cf2d5-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-00a787-65c44bcd97-thn6r\" (UID: \"0da21f81-aebc-40fd-9991-60e5631cf2d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-00a787-65c44bcd97-thn6r"
Nov 01 14:29:55.019840 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:55.019806    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0da21f81-aebc-40fd-9991-60e5631cf2d5-metrics-certs\") pod \"maas-api-e2e-worker-w1-00a787-65c44bcd97-thn6r\" (UID: \"0da21f81-aebc-40fd-9991-60e5631cf2d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-00a787-65c44bcd97-thn6r"
Nov 01 14:29:55.075491 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:55.075419    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2pq5\" (UniqueName: \"kubernetes.io/projected/0da21f81-aebc-40fd-9991-60e5631cf2d5-kube-api-access-h2pq5\") pod \"maas-api-e2e-worker-w1-00a787-65c44bcd97-thn6r\" (UID: \"0da21f81-aebc-40fd-9991-60e5631cf2d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-00a787-65c44bcd97-thn6r"
Nov 01 14:29:55.521119 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:55.521087    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0da21f81-aebc-40fd-9991-60e5631cf2d5-maas-api-tls\") pod \"maas-api-e2e-worker-w1-00a787-65c44bcd97-thn6r\" (UID: \"0da21f81-aebc-40fd-9991-60e5631cf2d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-00a787-65c44bcd97-thn6r"
Nov 01 14:29:55.523650 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:55.523622    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0da21f81-aebc-40fd-9991-60e5631cf2d5-maas-api-tls\") pod \"maas-api-e2e-worker-w1-00a787-65c44bcd97-thn6r\" (UID: \"0da21f81-aebc-40fd-9991-60e5631cf2d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-00a787-65c44bcd97-thn6r"
Nov 01 14:29:55.690617 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:55.690586    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-00a787-65c44bcd97-thn6r"
Nov 01 14:29:56.077550 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:56.077469    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-00a787-65c44bcd97-thn6r"]
Nov 01 14:29:56.084151 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:29:56.084116    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0da21f81_aebc_40fd_9991_60e5631cf2d5.slice/crio-f101720edea263ec66474c0a4f6d88dd5626235f8916f0de545549d85a2834b7 WatchSource:0}: Error finding container f101720edea263ec66474c0a4f6d88dd5626235f8916f0de545549d85a2834b7: Status 404 returned error can't find the container with id f101720edea263ec66474c0a4f6d88dd5626235f8916f0de545549d85a2834b7
Nov 01 14:29:56.718437 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:56.718399    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-00a787-65c44bcd97-thn6r" event={"ID":"0da21f81-aebc-40fd-9991-60e5631cf2d5","Type":"ContainerStarted","Data":"f101720edea263ec66474c0a4f6d88dd5626235f8916f0de545549d85a2834b7"}
Nov 01 14:29:57.500569 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:29:57.500537    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:29:57.725606 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:57.725570    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-00a787-65c44bcd97-thn6r" event={"ID":"0da21f81-aebc-40fd-9991-60e5631cf2d5","Type":"ContainerStarted","Data":"ec2dd19821ba88da2d66c7e7f473b6f1dc56b9dbd2b98f1514093b28aef08eb8"}
Nov 01 14:29:57.725787 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:57.725678    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-00a787-65c44bcd97-thn6r"
Nov 01 14:29:57.836981 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:57.836881    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-00a787-65c44bcd97-thn6r" podStartSLOduration=2.99553447 podStartE2EDuration="3.836868783s" totalImagesPullingTime="841.334313ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:29:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:29:57.83512521 +0000 UTC m=+785.164987946" watchObservedRunningTime="2026-11-01 14:29:57.836868783 +0000 UTC m=+785.166731519"
Nov 01 14:29:58.606486 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:58.606456    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-007270/simulator-w2-007270-kserve-5bf8844f79-c7v9v"]
Nov 01 14:29:58.610388 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:58.610366    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-007270/simulator-w2-007270-kserve-5bf8844f79-c7v9v"
Nov 01 14:29:58.707960 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:58.707921    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-007270\"/\"openshift-service-ca.crt\""
Nov 01 14:29:58.707960 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:58.707946    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-007270\"/\"default-dockercfg-9t7qz\""
Nov 01 14:29:58.708146 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:58.708050    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-007270\"/\"simulator-w2-007270-kserve-self-signed-certs\""
Nov 01 14:29:58.708146 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:58.708129    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-007270\"/\"kube-root-ca.crt\""
Nov 01 14:29:58.710155 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:58.710134    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-007270/simulator-w2-007270-kserve-5bf8844f79-c7v9v"]
Nov 01 14:29:58.751813 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:58.751785    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ae81a81-e763-40ac-8f5e-4bd38f9a3e68-dshm\") pod \"simulator-w2-007270-kserve-5bf8844f79-c7v9v\" (UID: \"2ae81a81-e763-40ac-8f5e-4bd38f9a3e68\") " pod="e2e-models-e2e-worker-w2-007270/simulator-w2-007270-kserve-5bf8844f79-c7v9v"
Nov 01 14:29:58.751956 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:58.751818    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ae81a81-e763-40ac-8f5e-4bd38f9a3e68-tmp-dir\") pod \"simulator-w2-007270-kserve-5bf8844f79-c7v9v\" (UID: \"2ae81a81-e763-40ac-8f5e-4bd38f9a3e68\") " pod="e2e-models-e2e-worker-w2-007270/simulator-w2-007270-kserve-5bf8844f79-c7v9v"
Nov 01 14:29:58.751956 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:58.751839    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cx68x\" (UniqueName: \"kubernetes.io/projected/2ae81a81-e763-40ac-8f5e-4bd38f9a3e68-kube-api-access-cx68x\") pod \"simulator-w2-007270-kserve-5bf8844f79-c7v9v\" (UID: \"2ae81a81-e763-40ac-8f5e-4bd38f9a3e68\") " pod="e2e-models-e2e-worker-w2-007270/simulator-w2-007270-kserve-5bf8844f79-c7v9v"
Nov 01 14:29:58.752052 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:58.751961    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ae81a81-e763-40ac-8f5e-4bd38f9a3e68-model-cache\") pod \"simulator-w2-007270-kserve-5bf8844f79-c7v9v\" (UID: \"2ae81a81-e763-40ac-8f5e-4bd38f9a3e68\") " pod="e2e-models-e2e-worker-w2-007270/simulator-w2-007270-kserve-5bf8844f79-c7v9v"
Nov 01 14:29:58.752052 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:58.752003    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ae81a81-e763-40ac-8f5e-4bd38f9a3e68-tls-certs\") pod \"simulator-w2-007270-kserve-5bf8844f79-c7v9v\" (UID: \"2ae81a81-e763-40ac-8f5e-4bd38f9a3e68\") " pod="e2e-models-e2e-worker-w2-007270/simulator-w2-007270-kserve-5bf8844f79-c7v9v"
Nov 01 14:29:58.752153 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:58.752136    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ae81a81-e763-40ac-8f5e-4bd38f9a3e68-home\") pod \"simulator-w2-007270-kserve-5bf8844f79-c7v9v\" (UID: \"2ae81a81-e763-40ac-8f5e-4bd38f9a3e68\") " pod="e2e-models-e2e-worker-w2-007270/simulator-w2-007270-kserve-5bf8844f79-c7v9v"
Nov 01 14:29:58.853601 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:58.853559    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ae81a81-e763-40ac-8f5e-4bd38f9a3e68-home\") pod \"simulator-w2-007270-kserve-5bf8844f79-c7v9v\" (UID: \"2ae81a81-e763-40ac-8f5e-4bd38f9a3e68\") " pod="e2e-models-e2e-worker-w2-007270/simulator-w2-007270-kserve-5bf8844f79-c7v9v"
Nov 01 14:29:58.853601 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:58.853608    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ae81a81-e763-40ac-8f5e-4bd38f9a3e68-dshm\") pod \"simulator-w2-007270-kserve-5bf8844f79-c7v9v\" (UID: \"2ae81a81-e763-40ac-8f5e-4bd38f9a3e68\") " pod="e2e-models-e2e-worker-w2-007270/simulator-w2-007270-kserve-5bf8844f79-c7v9v"
Nov 01 14:29:58.853863 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:58.853643    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ae81a81-e763-40ac-8f5e-4bd38f9a3e68-tmp-dir\") pod \"simulator-w2-007270-kserve-5bf8844f79-c7v9v\" (UID: \"2ae81a81-e763-40ac-8f5e-4bd38f9a3e68\") " pod="e2e-models-e2e-worker-w2-007270/simulator-w2-007270-kserve-5bf8844f79-c7v9v"
Nov 01 14:29:58.853863 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:58.853661    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx68x\" (UniqueName: \"kubernetes.io/projected/2ae81a81-e763-40ac-8f5e-4bd38f9a3e68-kube-api-access-cx68x\") pod \"simulator-w2-007270-kserve-5bf8844f79-c7v9v\" (UID: \"2ae81a81-e763-40ac-8f5e-4bd38f9a3e68\") " pod="e2e-models-e2e-worker-w2-007270/simulator-w2-007270-kserve-5bf8844f79-c7v9v"
Nov 01 14:29:58.853863 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:58.853730    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ae81a81-e763-40ac-8f5e-4bd38f9a3e68-model-cache\") pod \"simulator-w2-007270-kserve-5bf8844f79-c7v9v\" (UID: \"2ae81a81-e763-40ac-8f5e-4bd38f9a3e68\") " pod="e2e-models-e2e-worker-w2-007270/simulator-w2-007270-kserve-5bf8844f79-c7v9v"
Nov 01 14:29:58.853863 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:58.853772    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ae81a81-e763-40ac-8f5e-4bd38f9a3e68-tls-certs\") pod \"simulator-w2-007270-kserve-5bf8844f79-c7v9v\" (UID: \"2ae81a81-e763-40ac-8f5e-4bd38f9a3e68\") " pod="e2e-models-e2e-worker-w2-007270/simulator-w2-007270-kserve-5bf8844f79-c7v9v"
Nov 01 14:29:58.854099 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:58.854072    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ae81a81-e763-40ac-8f5e-4bd38f9a3e68-home\") pod \"simulator-w2-007270-kserve-5bf8844f79-c7v9v\" (UID: \"2ae81a81-e763-40ac-8f5e-4bd38f9a3e68\") " pod="e2e-models-e2e-worker-w2-007270/simulator-w2-007270-kserve-5bf8844f79-c7v9v"
Nov 01 14:29:58.854202 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:58.854164    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ae81a81-e763-40ac-8f5e-4bd38f9a3e68-tmp-dir\") pod \"simulator-w2-007270-kserve-5bf8844f79-c7v9v\" (UID: \"2ae81a81-e763-40ac-8f5e-4bd38f9a3e68\") " pod="e2e-models-e2e-worker-w2-007270/simulator-w2-007270-kserve-5bf8844f79-c7v9v"
Nov 01 14:29:58.854326 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:58.854306    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ae81a81-e763-40ac-8f5e-4bd38f9a3e68-model-cache\") pod \"simulator-w2-007270-kserve-5bf8844f79-c7v9v\" (UID: \"2ae81a81-e763-40ac-8f5e-4bd38f9a3e68\") " pod="e2e-models-e2e-worker-w2-007270/simulator-w2-007270-kserve-5bf8844f79-c7v9v"
Nov 01 14:29:58.856018 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:58.855990    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ae81a81-e763-40ac-8f5e-4bd38f9a3e68-dshm\") pod \"simulator-w2-007270-kserve-5bf8844f79-c7v9v\" (UID: \"2ae81a81-e763-40ac-8f5e-4bd38f9a3e68\") " pod="e2e-models-e2e-worker-w2-007270/simulator-w2-007270-kserve-5bf8844f79-c7v9v"
Nov 01 14:29:58.856583 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:58.856544    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ae81a81-e763-40ac-8f5e-4bd38f9a3e68-tls-certs\") pod \"simulator-w2-007270-kserve-5bf8844f79-c7v9v\" (UID: \"2ae81a81-e763-40ac-8f5e-4bd38f9a3e68\") " pod="e2e-models-e2e-worker-w2-007270/simulator-w2-007270-kserve-5bf8844f79-c7v9v"
Nov 01 14:29:58.909253 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:58.909221    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx68x\" (UniqueName: \"kubernetes.io/projected/2ae81a81-e763-40ac-8f5e-4bd38f9a3e68-kube-api-access-cx68x\") pod \"simulator-w2-007270-kserve-5bf8844f79-c7v9v\" (UID: \"2ae81a81-e763-40ac-8f5e-4bd38f9a3e68\") " pod="e2e-models-e2e-worker-w2-007270/simulator-w2-007270-kserve-5bf8844f79-c7v9v"
Nov 01 14:29:58.921776 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:58.921745    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-007270/simulator-w2-007270-kserve-5bf8844f79-c7v9v"
Nov 01 14:29:59.294470 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:59.294436    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-007270/simulator-w2-007270-kserve-5bf8844f79-c7v9v"]
Nov 01 14:29:59.300633 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:29:59.300594    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2ae81a81_e763_40ac_8f5e_4bd38f9a3e68.slice/crio-6dca24665904dca4595aaecb558fec9634ccac1ceda21aff7a7787099046d6c0 WatchSource:0}: Error finding container 6dca24665904dca4595aaecb558fec9634ccac1ceda21aff7a7787099046d6c0: Status 404 returned error can't find the container with id 6dca24665904dca4595aaecb558fec9634ccac1ceda21aff7a7787099046d6c0
Nov 01 14:29:59.738361 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:29:59.738319    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-007270/simulator-w2-007270-kserve-5bf8844f79-c7v9v" event={"ID":"2ae81a81-e763-40ac-8f5e-4bd38f9a3e68","Type":"ContainerStarted","Data":"6dca24665904dca4595aaecb558fec9634ccac1ceda21aff7a7787099046d6c0"}
Nov 01 14:30:00.195999 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:00.195917    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892390-bbwlq"]
Nov 01 14:30:00.199866 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:00.199840    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892390-bbwlq"
Nov 01 14:30:00.248955 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:00.248922    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\""
Nov 01 14:30:00.269465 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:00.269432    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2f22\" (UniqueName: \"kubernetes.io/projected/ed6fda13-6178-4c29-a753-a4f8d8d4d3b3-kube-api-access-f2f22\") pod \"maas-api-key-cleanup-e2e-worker-w4-536a87-29892390-bbwlq\" (UID: \"ed6fda13-6178-4c29-a753-a4f8d8d4d3b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892390-bbwlq"
Nov 01 14:30:00.269673 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:00.269645    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed6fda13-6178-4c29-a753-a4f8d8d4d3b3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-536a87-29892390-bbwlq\" (UID: \"ed6fda13-6178-4c29-a753-a4f8d8d4d3b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892390-bbwlq"
Nov 01 14:30:00.370665 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:00.370612    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed6fda13-6178-4c29-a753-a4f8d8d4d3b3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-536a87-29892390-bbwlq\" (UID: \"ed6fda13-6178-4c29-a753-a4f8d8d4d3b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892390-bbwlq"
Nov 01 14:30:00.370833 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:00.370725    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2f22\" (UniqueName: \"kubernetes.io/projected/ed6fda13-6178-4c29-a753-a4f8d8d4d3b3-kube-api-access-f2f22\") pod \"maas-api-key-cleanup-e2e-worker-w4-536a87-29892390-bbwlq\" (UID: \"ed6fda13-6178-4c29-a753-a4f8d8d4d3b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892390-bbwlq"
Nov 01 14:30:00.371333 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:00.371307    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed6fda13-6178-4c29-a753-a4f8d8d4d3b3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-536a87-29892390-bbwlq\" (UID: \"ed6fda13-6178-4c29-a753-a4f8d8d4d3b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892390-bbwlq"
Nov 01 14:30:00.399795 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:00.399710    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892390-bbwlq"]
Nov 01 14:30:00.432293 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:00.432259    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2f22\" (UniqueName: \"kubernetes.io/projected/ed6fda13-6178-4c29-a753-a4f8d8d4d3b3-kube-api-access-f2f22\") pod \"maas-api-key-cleanup-e2e-worker-w4-536a87-29892390-bbwlq\" (UID: \"ed6fda13-6178-4c29-a753-a4f8d8d4d3b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892390-bbwlq"
Nov 01 14:30:00.515394 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:00.515352    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892390-bbwlq"
Nov 01 14:30:00.906240 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:00.905759    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892390-bbwlq"]
Nov 01 14:30:00.923926 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:30:00.923887    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded6fda13_6178_4c29_a753_a4f8d8d4d3b3.slice/crio-aea2151c47330aac384e70e781263c7273db87fc1310a895716566f4a7471e50 WatchSource:0}: Error finding container aea2151c47330aac384e70e781263c7273db87fc1310a895716566f4a7471e50: Status 404 returned error can't find the container with id aea2151c47330aac384e70e781263c7273db87fc1310a895716566f4a7471e50
Nov 01 14:30:01.750851 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:01.750811    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892390-bbwlq" event={"ID":"ed6fda13-6178-4c29-a753-a4f8d8d4d3b3","Type":"ContainerStarted","Data":"aea2151c47330aac384e70e781263c7273db87fc1310a895716566f4a7471e50"}
Nov 01 14:30:01.752320 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:01.752292    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-007270/simulator-w2-007270-kserve-5bf8844f79-c7v9v" event={"ID":"2ae81a81-e763-40ac-8f5e-4bd38f9a3e68","Type":"ContainerStarted","Data":"09998d05d6ddb8e95f6dad6a29b774ff321d60f92a894e18aa4bd1da2ec938ae"}
Nov 01 14:30:01.860252 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:01.860200    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-007270/simulator-w2-007270-kserve-5bf8844f79-c7v9v" podStartSLOduration=2.227225668 podStartE2EDuration="3.860185655s" totalImagesPullingTime="1.632959987s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:29:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:30:01.859243545 +0000 UTC m=+789.189106277" watchObservedRunningTime="2026-11-01 14:30:01.860185655 +0000 UTC m=+789.190048384"
Nov 01 14:30:01.921847 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:01.921808    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-007270/simulator-w2-007270-kserve-5bf8844f79-c7v9v"
Nov 01 14:30:03.739099 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:03.738213    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-00a787-65c44bcd97-thn6r"
Nov 01 14:30:05.778055 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:05.778019    4993 generic.go:356] "Generic (PLEG): container finished" podID="ed6fda13-6178-4c29-a753-a4f8d8d4d3b3" containerID="5f26e1a20cb310172fd9162f8a42b5264c9fbc043ff79cfd3556d0e12690e428" exitCode=60
Nov 01 14:30:05.778500 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:05.778107    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892390-bbwlq" event={"ID":"ed6fda13-6178-4c29-a753-a4f8d8d4d3b3","Type":"ContainerDied","Data":"5f26e1a20cb310172fd9162f8a42b5264c9fbc043ff79cfd3556d0e12690e428"}
Nov 01 14:30:05.778500 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:05.778338    4993 scope.go:122] "RemoveContainer" containerID="5f26e1a20cb310172fd9162f8a42b5264c9fbc043ff79cfd3556d0e12690e428"
Nov 01 14:30:06.786145 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:06.786111    4993 generic.go:356] "Generic (PLEG): container finished" podID="ed6fda13-6178-4c29-a753-a4f8d8d4d3b3" containerID="60d1a854dee1d52cb9894c7655c5e9384bb7f9d7096a577c4049d1e47f54721b" exitCode=60
Nov 01 14:30:06.786595 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:06.786236    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892390-bbwlq" event={"ID":"ed6fda13-6178-4c29-a753-a4f8d8d4d3b3","Type":"ContainerDied","Data":"60d1a854dee1d52cb9894c7655c5e9384bb7f9d7096a577c4049d1e47f54721b"}
Nov 01 14:30:06.786595 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:06.786309    4993 scope.go:122] "RemoveContainer" containerID="5f26e1a20cb310172fd9162f8a42b5264c9fbc043ff79cfd3556d0e12690e428"
Nov 01 14:30:06.786595 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:06.786577    4993 scope.go:122] "RemoveContainer" containerID="60d1a854dee1d52cb9894c7655c5e9384bb7f9d7096a577c4049d1e47f54721b"
Nov 01 14:30:06.786864 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:30:06.786839    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-536a87-29892390-bbwlq_odh-ai-gateway-infra(ed6fda13-6178-4c29-a753-a4f8d8d4d3b3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892390-bbwlq" podUID="ed6fda13-6178-4c29-a753-a4f8d8d4d3b3"
Nov 01 14:30:07.792581 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:07.792550    4993 scope.go:122] "RemoveContainer" containerID="60d1a854dee1d52cb9894c7655c5e9384bb7f9d7096a577c4049d1e47f54721b"
Nov 01 14:30:07.793011 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:30:07.792793    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-536a87-29892390-bbwlq_odh-ai-gateway-infra(ed6fda13-6178-4c29-a753-a4f8d8d4d3b3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892390-bbwlq" podUID="ed6fda13-6178-4c29-a753-a4f8d8d4d3b3"
Nov 01 14:30:08.922125 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:08.922092    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-007270/simulator-w2-007270-kserve-5bf8844f79-c7v9v"
Nov 01 14:30:08.934505 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:08.934480    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-007270/simulator-w2-007270-kserve-5bf8844f79-c7v9v"
Nov 01 14:30:09.813265 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:09.813236    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-007270/simulator-w2-007270-kserve-5bf8844f79-c7v9v"
Nov 01 14:30:18.787710 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:18.787682    4993 scope.go:122] "RemoveContainer" containerID="60d1a854dee1d52cb9894c7655c5e9384bb7f9d7096a577c4049d1e47f54721b"
Nov 01 14:30:19.851621 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:19.851590    4993 generic.go:356] "Generic (PLEG): container finished" podID="ed6fda13-6178-4c29-a753-a4f8d8d4d3b3" containerID="22b1ff91287abd3f2522964c0c6833a8ac8c2083db84314f6ca3d5060b7bc084" exitCode=60
Nov 01 14:30:19.852222 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:19.851667    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892390-bbwlq" event={"ID":"ed6fda13-6178-4c29-a753-a4f8d8d4d3b3","Type":"ContainerDied","Data":"22b1ff91287abd3f2522964c0c6833a8ac8c2083db84314f6ca3d5060b7bc084"}
Nov 01 14:30:19.852222 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:19.851747    4993 scope.go:122] "RemoveContainer" containerID="60d1a854dee1d52cb9894c7655c5e9384bb7f9d7096a577c4049d1e47f54721b"
Nov 01 14:30:19.852222 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:19.852038    4993 scope.go:122] "RemoveContainer" containerID="22b1ff91287abd3f2522964c0c6833a8ac8c2083db84314f6ca3d5060b7bc084"
Nov 01 14:30:19.852375 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:30:19.852304    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-536a87-29892390-bbwlq_odh-ai-gateway-infra(ed6fda13-6178-4c29-a753-a4f8d8d4d3b3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892390-bbwlq" podUID="ed6fda13-6178-4c29-a753-a4f8d8d4d3b3"
Nov 01 14:30:20.969218 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:20.969184    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892390-bbwlq"]
Nov 01 14:30:21.110699 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:21.110671    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892390-bbwlq"
Nov 01 14:30:21.174722 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:21.174688    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed6fda13-6178-4c29-a753-a4f8d8d4d3b3-kube-api-access-f2f22\" (UniqueName: \"kubernetes.io/projected/ed6fda13-6178-4c29-a753-a4f8d8d4d3b3-kube-api-access-f2f22\") pod \"ed6fda13-6178-4c29-a753-a4f8d8d4d3b3\" (UID: \"ed6fda13-6178-4c29-a753-a4f8d8d4d3b3\") "
Nov 01 14:30:21.174903 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:21.174798    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed6fda13-6178-4c29-a753-a4f8d8d4d3b3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed6fda13-6178-4c29-a753-a4f8d8d4d3b3-maas-api-service-ca\") pod \"ed6fda13-6178-4c29-a753-a4f8d8d4d3b3\" (UID: \"ed6fda13-6178-4c29-a753-a4f8d8d4d3b3\") "
Nov 01 14:30:21.175209 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:21.175184    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed6fda13-6178-4c29-a753-a4f8d8d4d3b3-maas-api-service-ca" pod "ed6fda13-6178-4c29-a753-a4f8d8d4d3b3" (UID: "ed6fda13-6178-4c29-a753-a4f8d8d4d3b3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:30:21.176734 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:21.176714    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed6fda13-6178-4c29-a753-a4f8d8d4d3b3-kube-api-access-f2f22" pod "ed6fda13-6178-4c29-a753-a4f8d8d4d3b3" (UID: "ed6fda13-6178-4c29-a753-a4f8d8d4d3b3"). InnerVolumeSpecName "kube-api-access-f2f22". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:30:21.275769 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:21.275736    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2f22\" (UniqueName: \"kubernetes.io/projected/ed6fda13-6178-4c29-a753-a4f8d8d4d3b3-kube-api-access-f2f22\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:30:21.275769 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:21.275762    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed6fda13-6178-4c29-a753-a4f8d8d4d3b3-maas-api-service-ca\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:30:21.866399 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:21.866370    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892390-bbwlq"
Nov 01 14:30:21.866571 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:21.866410    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892390-bbwlq" event={"ID":"ed6fda13-6178-4c29-a753-a4f8d8d4d3b3","Type":"ContainerDied","Data":"aea2151c47330aac384e70e781263c7273db87fc1310a895716566f4a7471e50"}
Nov 01 14:30:21.866571 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:21.866480    4993 scope.go:122] "RemoveContainer" containerID="22b1ff91287abd3f2522964c0c6833a8ac8c2083db84314f6ca3d5060b7bc084"
Nov 01 14:30:22.019025 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:22.018989    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892390-bbwlq"]
Nov 01 14:30:22.065138 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:22.065102    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892390-bbwlq"]
Nov 01 14:30:22.793842 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:22.793810    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed6fda13-6178-4c29-a753-a4f8d8d4d3b3" path="/var/lib/kubelet/pods/ed6fda13-6178-4c29-a753-a4f8d8d4d3b3/volumes"
Nov 01 14:30:27.500531 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:30:27.500494    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:30:35.166059 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:35.166026    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-1f8e2c/simulator-w0-1f8e2c-kserve-755bf645d-zf8t4"]
Nov 01 14:30:35.166590 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:35.166529    4993 cpu_manager.go:414] "containerMap: removing container" podUID="ed6fda13-6178-4c29-a753-a4f8d8d4d3b3" containerName="cleanup"
Nov 01 14:30:35.166590 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:35.166547    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed6fda13-6178-4c29-a753-a4f8d8d4d3b3" containerName="cleanup"
Nov 01 14:30:35.166590 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:35.166558    4993 cpu_manager.go:414] "containerMap: removing container" podUID="ed6fda13-6178-4c29-a753-a4f8d8d4d3b3" containerName="cleanup"
Nov 01 14:30:35.166590 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:35.166564    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed6fda13-6178-4c29-a753-a4f8d8d4d3b3" containerName="cleanup"
Nov 01 14:30:35.166801 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:35.166708    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed6fda13-6178-4c29-a753-a4f8d8d4d3b3" containerName="cleanup"
Nov 01 14:30:35.166801 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:35.166722    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed6fda13-6178-4c29-a753-a4f8d8d4d3b3" containerName="cleanup"
Nov 01 14:30:35.175812 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:35.175789    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1f8e2c/simulator-w0-1f8e2c-kserve-755bf645d-zf8t4"
Nov 01 14:30:35.251341 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:35.251306    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-1f8e2c\"/\"openshift-service-ca.crt\""
Nov 01 14:30:35.251341 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:35.251331    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-1f8e2c\"/\"default-dockercfg-pfpf5\""
Nov 01 14:30:35.251531 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:35.251364    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-1f8e2c\"/\"kube-root-ca.crt\""
Nov 01 14:30:35.272122 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:35.272091    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-1f8e2c\"/\"simulator-w0-1f8e2c-kserve-self-signed-certs\""
Nov 01 14:30:35.274248 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:35.274225    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1f8e2c/simulator-w0-1f8e2c-kserve-755bf645d-zf8t4"]
Nov 01 14:30:35.311400 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:35.311367    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbc61e12-d326-4d31-be81-88fb641337c7-home\") pod \"simulator-w0-1f8e2c-kserve-755bf645d-zf8t4\" (UID: \"bbc61e12-d326-4d31-be81-88fb641337c7\") " pod="e2e-models-e2e-worker-w0-1f8e2c/simulator-w0-1f8e2c-kserve-755bf645d-zf8t4"
Nov 01 14:30:35.311589 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:35.311426    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbc61e12-d326-4d31-be81-88fb641337c7-tmp-dir\") pod \"simulator-w0-1f8e2c-kserve-755bf645d-zf8t4\" (UID: \"bbc61e12-d326-4d31-be81-88fb641337c7\") " pod="e2e-models-e2e-worker-w0-1f8e2c/simulator-w0-1f8e2c-kserve-755bf645d-zf8t4"
Nov 01 14:30:35.311589 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:35.311551    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbc61e12-d326-4d31-be81-88fb641337c7-tls-certs\") pod \"simulator-w0-1f8e2c-kserve-755bf645d-zf8t4\" (UID: \"bbc61e12-d326-4d31-be81-88fb641337c7\") " pod="e2e-models-e2e-worker-w0-1f8e2c/simulator-w0-1f8e2c-kserve-755bf645d-zf8t4"
Nov 01 14:30:35.311698 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:35.311680    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbc61e12-d326-4d31-be81-88fb641337c7-model-cache\") pod \"simulator-w0-1f8e2c-kserve-755bf645d-zf8t4\" (UID: \"bbc61e12-d326-4d31-be81-88fb641337c7\") " pod="e2e-models-e2e-worker-w0-1f8e2c/simulator-w0-1f8e2c-kserve-755bf645d-zf8t4"
Nov 01 14:30:35.311736 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:35.311723    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbc61e12-d326-4d31-be81-88fb641337c7-dshm\") pod \"simulator-w0-1f8e2c-kserve-755bf645d-zf8t4\" (UID: \"bbc61e12-d326-4d31-be81-88fb641337c7\") " pod="e2e-models-e2e-worker-w0-1f8e2c/simulator-w0-1f8e2c-kserve-755bf645d-zf8t4"
Nov 01 14:30:35.311787 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:35.311753    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvt7c\" (UniqueName: \"kubernetes.io/projected/bbc61e12-d326-4d31-be81-88fb641337c7-kube-api-access-fvt7c\") pod \"simulator-w0-1f8e2c-kserve-755bf645d-zf8t4\" (UID: \"bbc61e12-d326-4d31-be81-88fb641337c7\") " pod="e2e-models-e2e-worker-w0-1f8e2c/simulator-w0-1f8e2c-kserve-755bf645d-zf8t4"
Nov 01 14:30:35.412644 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:35.412610    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbc61e12-d326-4d31-be81-88fb641337c7-home\") pod \"simulator-w0-1f8e2c-kserve-755bf645d-zf8t4\" (UID: \"bbc61e12-d326-4d31-be81-88fb641337c7\") " pod="e2e-models-e2e-worker-w0-1f8e2c/simulator-w0-1f8e2c-kserve-755bf645d-zf8t4"
Nov 01 14:30:35.412814 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:35.412657    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbc61e12-d326-4d31-be81-88fb641337c7-tmp-dir\") pod \"simulator-w0-1f8e2c-kserve-755bf645d-zf8t4\" (UID: \"bbc61e12-d326-4d31-be81-88fb641337c7\") " pod="e2e-models-e2e-worker-w0-1f8e2c/simulator-w0-1f8e2c-kserve-755bf645d-zf8t4"
Nov 01 14:30:35.412814 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:35.412677    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbc61e12-d326-4d31-be81-88fb641337c7-tls-certs\") pod \"simulator-w0-1f8e2c-kserve-755bf645d-zf8t4\" (UID: \"bbc61e12-d326-4d31-be81-88fb641337c7\") " pod="e2e-models-e2e-worker-w0-1f8e2c/simulator-w0-1f8e2c-kserve-755bf645d-zf8t4"
Nov 01 14:30:35.412814 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:35.412720    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbc61e12-d326-4d31-be81-88fb641337c7-model-cache\") pod \"simulator-w0-1f8e2c-kserve-755bf645d-zf8t4\" (UID: \"bbc61e12-d326-4d31-be81-88fb641337c7\") " pod="e2e-models-e2e-worker-w0-1f8e2c/simulator-w0-1f8e2c-kserve-755bf645d-zf8t4"
Nov 01 14:30:35.412814 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:35.412743    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbc61e12-d326-4d31-be81-88fb641337c7-dshm\") pod \"simulator-w0-1f8e2c-kserve-755bf645d-zf8t4\" (UID: \"bbc61e12-d326-4d31-be81-88fb641337c7\") " pod="e2e-models-e2e-worker-w0-1f8e2c/simulator-w0-1f8e2c-kserve-755bf645d-zf8t4"
Nov 01 14:30:35.412814 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:35.412759    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvt7c\" (UniqueName: \"kubernetes.io/projected/bbc61e12-d326-4d31-be81-88fb641337c7-kube-api-access-fvt7c\") pod \"simulator-w0-1f8e2c-kserve-755bf645d-zf8t4\" (UID: \"bbc61e12-d326-4d31-be81-88fb641337c7\") " pod="e2e-models-e2e-worker-w0-1f8e2c/simulator-w0-1f8e2c-kserve-755bf645d-zf8t4"
Nov 01 14:30:35.413112 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:35.413086    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbc61e12-d326-4d31-be81-88fb641337c7-tmp-dir\") pod \"simulator-w0-1f8e2c-kserve-755bf645d-zf8t4\" (UID: \"bbc61e12-d326-4d31-be81-88fb641337c7\") " pod="e2e-models-e2e-worker-w0-1f8e2c/simulator-w0-1f8e2c-kserve-755bf645d-zf8t4"
Nov 01 14:30:35.413215 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:35.413101    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbc61e12-d326-4d31-be81-88fb641337c7-model-cache\") pod \"simulator-w0-1f8e2c-kserve-755bf645d-zf8t4\" (UID: \"bbc61e12-d326-4d31-be81-88fb641337c7\") " pod="e2e-models-e2e-worker-w0-1f8e2c/simulator-w0-1f8e2c-kserve-755bf645d-zf8t4"
Nov 01 14:30:35.413215 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:35.413144    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbc61e12-d326-4d31-be81-88fb641337c7-home\") pod \"simulator-w0-1f8e2c-kserve-755bf645d-zf8t4\" (UID: \"bbc61e12-d326-4d31-be81-88fb641337c7\") " pod="e2e-models-e2e-worker-w0-1f8e2c/simulator-w0-1f8e2c-kserve-755bf645d-zf8t4"
Nov 01 14:30:35.415079 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:35.415048    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbc61e12-d326-4d31-be81-88fb641337c7-dshm\") pod \"simulator-w0-1f8e2c-kserve-755bf645d-zf8t4\" (UID: \"bbc61e12-d326-4d31-be81-88fb641337c7\") " pod="e2e-models-e2e-worker-w0-1f8e2c/simulator-w0-1f8e2c-kserve-755bf645d-zf8t4"
Nov 01 14:30:35.415207 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:35.415189    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbc61e12-d326-4d31-be81-88fb641337c7-tls-certs\") pod \"simulator-w0-1f8e2c-kserve-755bf645d-zf8t4\" (UID: \"bbc61e12-d326-4d31-be81-88fb641337c7\") " pod="e2e-models-e2e-worker-w0-1f8e2c/simulator-w0-1f8e2c-kserve-755bf645d-zf8t4"
Nov 01 14:30:35.486206 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:35.486158    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvt7c\" (UniqueName: \"kubernetes.io/projected/bbc61e12-d326-4d31-be81-88fb641337c7-kube-api-access-fvt7c\") pod \"simulator-w0-1f8e2c-kserve-755bf645d-zf8t4\" (UID: \"bbc61e12-d326-4d31-be81-88fb641337c7\") " pod="e2e-models-e2e-worker-w0-1f8e2c/simulator-w0-1f8e2c-kserve-755bf645d-zf8t4"
Nov 01 14:30:35.488907 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:35.488840    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1f8e2c/simulator-w0-1f8e2c-kserve-755bf645d-zf8t4"
Nov 01 14:30:35.859127 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:35.859046    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1f8e2c/simulator-w0-1f8e2c-kserve-755bf645d-zf8t4"]
Nov 01 14:30:35.864986 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:30:35.864960    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbbc61e12_d326_4d31_be81_88fb641337c7.slice/crio-8d118730bca35b480da1ff596463fa71ca83a0a7255d8ddd32b74074aaad18bb WatchSource:0}: Error finding container 8d118730bca35b480da1ff596463fa71ca83a0a7255d8ddd32b74074aaad18bb: Status 404 returned error can't find the container with id 8d118730bca35b480da1ff596463fa71ca83a0a7255d8ddd32b74074aaad18bb
Nov 01 14:30:35.946932 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:35.946897    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1f8e2c/simulator-w0-1f8e2c-kserve-755bf645d-zf8t4" event={"ID":"bbc61e12-d326-4d31-be81-88fb641337c7","Type":"ContainerStarted","Data":"b6bce2c001544b5a381a1bd23a0fb0699996dd3fcbf4e57c232d7815539ecfad"}
Nov 01 14:30:35.947070 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:35.946941    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1f8e2c/simulator-w0-1f8e2c-kserve-755bf645d-zf8t4" event={"ID":"bbc61e12-d326-4d31-be81-88fb641337c7","Type":"ContainerStarted","Data":"8d118730bca35b480da1ff596463fa71ca83a0a7255d8ddd32b74074aaad18bb"}
Nov 01 14:30:36.053511 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:36.053450    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-1f8e2c/simulator-w0-1f8e2c-kserve-755bf645d-zf8t4" podStartSLOduration=1.053434508 podStartE2EDuration="1.053434508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:30:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:30:36.052523984 +0000 UTC m=+823.382386722" watchObservedRunningTime="2026-11-01 14:30:36.053434508 +0000 UTC m=+823.383297243"
Nov 01 14:30:36.489509 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:36.489463    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f8e2c/simulator-w0-1f8e2c-kserve-755bf645d-zf8t4"
Nov 01 14:30:45.489712 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:45.489674    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-1f8e2c/simulator-w0-1f8e2c-kserve-755bf645d-zf8t4"
Nov 01 14:30:45.506578 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:45.506549    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-1f8e2c/simulator-w0-1f8e2c-kserve-755bf645d-zf8t4"
Nov 01 14:30:46.007609 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:46.007578    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f8e2c/simulator-w0-1f8e2c-kserve-755bf645d-zf8t4"
Nov 01 14:30:57.500847 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:30:57.500803    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:30:59.809691 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:59.809651    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-00a787/simulator-w1-00a787-kserve-5bdcf4b567-4xsn7"]
Nov 01 14:30:59.810277 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:59.810261    4993 cpu_manager.go:414] "containerMap: removing container" podUID="ed6fda13-6178-4c29-a753-a4f8d8d4d3b3" containerName="cleanup"
Nov 01 14:30:59.810360 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:59.810279    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed6fda13-6178-4c29-a753-a4f8d8d4d3b3" containerName="cleanup"
Nov 01 14:30:59.810413 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:59.810396    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed6fda13-6178-4c29-a753-a4f8d8d4d3b3" containerName="cleanup"
Nov 01 14:30:59.815473 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:59.815451    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-00a787/simulator-w1-00a787-kserve-5bdcf4b567-4xsn7"
Nov 01 14:30:59.847037 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:59.847009    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b8fc4b9-5b38-4d33-8f56-c8913e7e403f-dshm\") pod \"simulator-w1-00a787-kserve-5bdcf4b567-4xsn7\" (UID: \"6b8fc4b9-5b38-4d33-8f56-c8913e7e403f\") " pod="e2e-models-e2e-worker-w1-00a787/simulator-w1-00a787-kserve-5bdcf4b567-4xsn7"
Nov 01 14:30:59.847361 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:59.847236    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnd8g\" (UniqueName: \"kubernetes.io/projected/6b8fc4b9-5b38-4d33-8f56-c8913e7e403f-kube-api-access-gnd8g\") pod \"simulator-w1-00a787-kserve-5bdcf4b567-4xsn7\" (UID: \"6b8fc4b9-5b38-4d33-8f56-c8913e7e403f\") " pod="e2e-models-e2e-worker-w1-00a787/simulator-w1-00a787-kserve-5bdcf4b567-4xsn7"
Nov 01 14:30:59.847361 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:59.847352    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b8fc4b9-5b38-4d33-8f56-c8913e7e403f-home\") pod \"simulator-w1-00a787-kserve-5bdcf4b567-4xsn7\" (UID: \"6b8fc4b9-5b38-4d33-8f56-c8913e7e403f\") " pod="e2e-models-e2e-worker-w1-00a787/simulator-w1-00a787-kserve-5bdcf4b567-4xsn7"
Nov 01 14:30:59.847490 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:59.847402    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b8fc4b9-5b38-4d33-8f56-c8913e7e403f-model-cache\") pod \"simulator-w1-00a787-kserve-5bdcf4b567-4xsn7\" (UID: \"6b8fc4b9-5b38-4d33-8f56-c8913e7e403f\") " pod="e2e-models-e2e-worker-w1-00a787/simulator-w1-00a787-kserve-5bdcf4b567-4xsn7"
Nov 01 14:30:59.847490 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:59.847467    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b8fc4b9-5b38-4d33-8f56-c8913e7e403f-tmp-dir\") pod \"simulator-w1-00a787-kserve-5bdcf4b567-4xsn7\" (UID: \"6b8fc4b9-5b38-4d33-8f56-c8913e7e403f\") " pod="e2e-models-e2e-worker-w1-00a787/simulator-w1-00a787-kserve-5bdcf4b567-4xsn7"
Nov 01 14:30:59.847601 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:59.847561    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b8fc4b9-5b38-4d33-8f56-c8913e7e403f-tls-certs\") pod \"simulator-w1-00a787-kserve-5bdcf4b567-4xsn7\" (UID: \"6b8fc4b9-5b38-4d33-8f56-c8913e7e403f\") " pod="e2e-models-e2e-worker-w1-00a787/simulator-w1-00a787-kserve-5bdcf4b567-4xsn7"
Nov 01 14:30:59.910283 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:59.910220    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-00a787\"/\"openshift-service-ca.crt\""
Nov 01 14:30:59.910410 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:59.910315    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-00a787\"/\"default-dockercfg-czjqm\""
Nov 01 14:30:59.910410 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:59.910314    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-00a787\"/\"kube-root-ca.crt\""
Nov 01 14:30:59.910410 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:59.910372    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-00a787\"/\"simulator-w1-00a787-kserve-self-signed-certs\""
Nov 01 14:30:59.912496 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:59.912475    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-00a787/simulator-w1-00a787-kserve-5bdcf4b567-4xsn7"]
Nov 01 14:30:59.948801 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:59.948760    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b8fc4b9-5b38-4d33-8f56-c8913e7e403f-tmp-dir\") pod \"simulator-w1-00a787-kserve-5bdcf4b567-4xsn7\" (UID: \"6b8fc4b9-5b38-4d33-8f56-c8913e7e403f\") " pod="e2e-models-e2e-worker-w1-00a787/simulator-w1-00a787-kserve-5bdcf4b567-4xsn7"
Nov 01 14:30:59.948999 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:59.948824    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b8fc4b9-5b38-4d33-8f56-c8913e7e403f-tls-certs\") pod \"simulator-w1-00a787-kserve-5bdcf4b567-4xsn7\" (UID: \"6b8fc4b9-5b38-4d33-8f56-c8913e7e403f\") " pod="e2e-models-e2e-worker-w1-00a787/simulator-w1-00a787-kserve-5bdcf4b567-4xsn7"
Nov 01 14:30:59.948999 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:59.948864    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b8fc4b9-5b38-4d33-8f56-c8913e7e403f-dshm\") pod \"simulator-w1-00a787-kserve-5bdcf4b567-4xsn7\" (UID: \"6b8fc4b9-5b38-4d33-8f56-c8913e7e403f\") " pod="e2e-models-e2e-worker-w1-00a787/simulator-w1-00a787-kserve-5bdcf4b567-4xsn7"
Nov 01 14:30:59.948999 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:59.948978    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnd8g\" (UniqueName: \"kubernetes.io/projected/6b8fc4b9-5b38-4d33-8f56-c8913e7e403f-kube-api-access-gnd8g\") pod \"simulator-w1-00a787-kserve-5bdcf4b567-4xsn7\" (UID: \"6b8fc4b9-5b38-4d33-8f56-c8913e7e403f\") " pod="e2e-models-e2e-worker-w1-00a787/simulator-w1-00a787-kserve-5bdcf4b567-4xsn7"
Nov 01 14:30:59.949147 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:59.949033    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b8fc4b9-5b38-4d33-8f56-c8913e7e403f-home\") pod \"simulator-w1-00a787-kserve-5bdcf4b567-4xsn7\" (UID: \"6b8fc4b9-5b38-4d33-8f56-c8913e7e403f\") " pod="e2e-models-e2e-worker-w1-00a787/simulator-w1-00a787-kserve-5bdcf4b567-4xsn7"
Nov 01 14:30:59.949698 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:59.949631    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b8fc4b9-5b38-4d33-8f56-c8913e7e403f-home\") pod \"simulator-w1-00a787-kserve-5bdcf4b567-4xsn7\" (UID: \"6b8fc4b9-5b38-4d33-8f56-c8913e7e403f\") " pod="e2e-models-e2e-worker-w1-00a787/simulator-w1-00a787-kserve-5bdcf4b567-4xsn7"
Nov 01 14:30:59.950009 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:59.949982    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b8fc4b9-5b38-4d33-8f56-c8913e7e403f-tmp-dir\") pod \"simulator-w1-00a787-kserve-5bdcf4b567-4xsn7\" (UID: \"6b8fc4b9-5b38-4d33-8f56-c8913e7e403f\") " pod="e2e-models-e2e-worker-w1-00a787/simulator-w1-00a787-kserve-5bdcf4b567-4xsn7"
Nov 01 14:30:59.950157 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:59.950108    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b8fc4b9-5b38-4d33-8f56-c8913e7e403f-model-cache\") pod \"simulator-w1-00a787-kserve-5bdcf4b567-4xsn7\" (UID: \"6b8fc4b9-5b38-4d33-8f56-c8913e7e403f\") " pod="e2e-models-e2e-worker-w1-00a787/simulator-w1-00a787-kserve-5bdcf4b567-4xsn7"
Nov 01 14:30:59.950585 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:59.950507    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b8fc4b9-5b38-4d33-8f56-c8913e7e403f-model-cache\") pod \"simulator-w1-00a787-kserve-5bdcf4b567-4xsn7\" (UID: \"6b8fc4b9-5b38-4d33-8f56-c8913e7e403f\") " pod="e2e-models-e2e-worker-w1-00a787/simulator-w1-00a787-kserve-5bdcf4b567-4xsn7"
Nov 01 14:30:59.951730 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:59.951709    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b8fc4b9-5b38-4d33-8f56-c8913e7e403f-dshm\") pod \"simulator-w1-00a787-kserve-5bdcf4b567-4xsn7\" (UID: \"6b8fc4b9-5b38-4d33-8f56-c8913e7e403f\") " pod="e2e-models-e2e-worker-w1-00a787/simulator-w1-00a787-kserve-5bdcf4b567-4xsn7"
Nov 01 14:30:59.952489 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:30:59.952470    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b8fc4b9-5b38-4d33-8f56-c8913e7e403f-tls-certs\") pod \"simulator-w1-00a787-kserve-5bdcf4b567-4xsn7\" (UID: \"6b8fc4b9-5b38-4d33-8f56-c8913e7e403f\") " pod="e2e-models-e2e-worker-w1-00a787/simulator-w1-00a787-kserve-5bdcf4b567-4xsn7"
Nov 01 14:31:00.002545 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:31:00.002510    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnd8g\" (UniqueName: \"kubernetes.io/projected/6b8fc4b9-5b38-4d33-8f56-c8913e7e403f-kube-api-access-gnd8g\") pod \"simulator-w1-00a787-kserve-5bdcf4b567-4xsn7\" (UID: \"6b8fc4b9-5b38-4d33-8f56-c8913e7e403f\") " pod="e2e-models-e2e-worker-w1-00a787/simulator-w1-00a787-kserve-5bdcf4b567-4xsn7"
Nov 01 14:31:00.153753 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:31:00.153668    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-00a787/simulator-w1-00a787-kserve-5bdcf4b567-4xsn7"
Nov 01 14:31:00.520892 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:31:00.520848    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-00a787/simulator-w1-00a787-kserve-5bdcf4b567-4xsn7"]
Nov 01 14:31:00.526984 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:31:00.526958    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6b8fc4b9_5b38_4d33_8f56_c8913e7e403f.slice/crio-8c168e65c7d637449c6b7c7899a61ad7d17b049ab08f03b442cbac9d1ccf6806 WatchSource:0}: Error finding container 8c168e65c7d637449c6b7c7899a61ad7d17b049ab08f03b442cbac9d1ccf6806: Status 404 returned error can't find the container with id 8c168e65c7d637449c6b7c7899a61ad7d17b049ab08f03b442cbac9d1ccf6806
Nov 01 14:31:01.082795 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:31:01.082749    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-00a787/simulator-w1-00a787-kserve-5bdcf4b567-4xsn7" event={"ID":"6b8fc4b9-5b38-4d33-8f56-c8913e7e403f","Type":"ContainerStarted","Data":"ae7fe7833bf1c036a266b82ea50175d7d93c5fccb7c95a7b935f77a5e853624d"}
Nov 01 14:31:01.083269 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:31:01.082804    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-00a787/simulator-w1-00a787-kserve-5bdcf4b567-4xsn7" event={"ID":"6b8fc4b9-5b38-4d33-8f56-c8913e7e403f","Type":"ContainerStarted","Data":"8c168e65c7d637449c6b7c7899a61ad7d17b049ab08f03b442cbac9d1ccf6806"}
Nov 01 14:31:01.154351 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:31:01.154317    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-00a787/simulator-w1-00a787-kserve-5bdcf4b567-4xsn7"
Nov 01 14:31:01.189190 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:31:01.189112    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-00a787/simulator-w1-00a787-kserve-5bdcf4b567-4xsn7" podStartSLOduration=2.189096159 podStartE2EDuration="2.189096159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:30:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:31:01.188266163 +0000 UTC m=+848.518128902" watchObservedRunningTime="2026-11-01 14:31:01.189096159 +0000 UTC m=+848.518958924"
Nov 01 14:31:10.154290 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:31:10.154205    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-00a787/simulator-w1-00a787-kserve-5bdcf4b567-4xsn7"
Nov 01 14:31:10.166563 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:31:10.166536    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-00a787/simulator-w1-00a787-kserve-5bdcf4b567-4xsn7"
Nov 01 14:31:11.149234 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:31:11.149197    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-00a787/simulator-w1-00a787-kserve-5bdcf4b567-4xsn7"
Nov 01 14:31:27.499051 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:31:27.499015    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:31:52.870662 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:31:52.870632    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xmwtp_bb59051d-c2df-4362-a157-00f14f915e79/console-operator/3.log"
Nov 01 14:31:52.874104 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:31:52.874076    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xmwtp_bb59051d-c2df-4362-a157-00f14f915e79/console-operator/3.log"
Nov 01 14:31:57.499391 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:31:57.499354    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:31:59.082873 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:31:59.082837    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-536a87/premium-w4-536a87-kserve-74c6fdbc8-sf89f"]
Nov 01 14:31:59.086961 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:31:59.086934    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-536a87/premium-w4-536a87-kserve-74c6fdbc8-sf89f"
Nov 01 14:31:59.130895 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:31:59.130855    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a720a76-488a-4a65-98ad-31ca84fa7842-dshm\") pod \"premium-w4-536a87-kserve-74c6fdbc8-sf89f\" (UID: \"8a720a76-488a-4a65-98ad-31ca84fa7842\") " pod="e2e-models-e2e-worker-w4-536a87/premium-w4-536a87-kserve-74c6fdbc8-sf89f"
Nov 01 14:31:59.131078 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:31:59.130935    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9qjj\" (UniqueName: \"kubernetes.io/projected/8a720a76-488a-4a65-98ad-31ca84fa7842-kube-api-access-d9qjj\") pod \"premium-w4-536a87-kserve-74c6fdbc8-sf89f\" (UID: \"8a720a76-488a-4a65-98ad-31ca84fa7842\") " pod="e2e-models-e2e-worker-w4-536a87/premium-w4-536a87-kserve-74c6fdbc8-sf89f"
Nov 01 14:31:59.131147 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:31:59.131121    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a720a76-488a-4a65-98ad-31ca84fa7842-tmp-dir\") pod \"premium-w4-536a87-kserve-74c6fdbc8-sf89f\" (UID: \"8a720a76-488a-4a65-98ad-31ca84fa7842\") " pod="e2e-models-e2e-worker-w4-536a87/premium-w4-536a87-kserve-74c6fdbc8-sf89f"
Nov 01 14:31:59.131222 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:31:59.131177    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a720a76-488a-4a65-98ad-31ca84fa7842-model-cache\") pod \"premium-w4-536a87-kserve-74c6fdbc8-sf89f\" (UID: \"8a720a76-488a-4a65-98ad-31ca84fa7842\") " pod="e2e-models-e2e-worker-w4-536a87/premium-w4-536a87-kserve-74c6fdbc8-sf89f"
Nov 01 14:31:59.131280 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:31:59.131233    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a720a76-488a-4a65-98ad-31ca84fa7842-tls-certs\") pod \"premium-w4-536a87-kserve-74c6fdbc8-sf89f\" (UID: \"8a720a76-488a-4a65-98ad-31ca84fa7842\") " pod="e2e-models-e2e-worker-w4-536a87/premium-w4-536a87-kserve-74c6fdbc8-sf89f"
Nov 01 14:31:59.131280 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:31:59.131257    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a720a76-488a-4a65-98ad-31ca84fa7842-home\") pod \"premium-w4-536a87-kserve-74c6fdbc8-sf89f\" (UID: \"8a720a76-488a-4a65-98ad-31ca84fa7842\") " pod="e2e-models-e2e-worker-w4-536a87/premium-w4-536a87-kserve-74c6fdbc8-sf89f"
Nov 01 14:31:59.151261 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:31:59.151226    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-536a87\"/\"default-dockercfg-glbls\""
Nov 01 14:31:59.181650 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:31:59.181600    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-536a87\"/\"openshift-service-ca.crt\""
Nov 01 14:31:59.181650 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:31:59.181618    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-536a87\"/\"kube-root-ca.crt\""
Nov 01 14:31:59.181650 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:31:59.181633    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-536a87\"/\"premium-w4-536a87-kserve-self-signed-certs\""
Nov 01 14:31:59.183906 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:31:59.183886    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-536a87/premium-w4-536a87-kserve-74c6fdbc8-sf89f"]
Nov 01 14:31:59.232108 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:31:59.232078    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9qjj\" (UniqueName: \"kubernetes.io/projected/8a720a76-488a-4a65-98ad-31ca84fa7842-kube-api-access-d9qjj\") pod \"premium-w4-536a87-kserve-74c6fdbc8-sf89f\" (UID: \"8a720a76-488a-4a65-98ad-31ca84fa7842\") " pod="e2e-models-e2e-worker-w4-536a87/premium-w4-536a87-kserve-74c6fdbc8-sf89f"
Nov 01 14:31:59.232257 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:31:59.232148    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a720a76-488a-4a65-98ad-31ca84fa7842-tmp-dir\") pod \"premium-w4-536a87-kserve-74c6fdbc8-sf89f\" (UID: \"8a720a76-488a-4a65-98ad-31ca84fa7842\") " pod="e2e-models-e2e-worker-w4-536a87/premium-w4-536a87-kserve-74c6fdbc8-sf89f"
Nov 01 14:31:59.232257 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:31:59.232195    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a720a76-488a-4a65-98ad-31ca84fa7842-model-cache\") pod \"premium-w4-536a87-kserve-74c6fdbc8-sf89f\" (UID: \"8a720a76-488a-4a65-98ad-31ca84fa7842\") " pod="e2e-models-e2e-worker-w4-536a87/premium-w4-536a87-kserve-74c6fdbc8-sf89f"
Nov 01 14:31:59.232257 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:31:59.232242    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a720a76-488a-4a65-98ad-31ca84fa7842-tls-certs\") pod \"premium-w4-536a87-kserve-74c6fdbc8-sf89f\" (UID: \"8a720a76-488a-4a65-98ad-31ca84fa7842\") " pod="e2e-models-e2e-worker-w4-536a87/premium-w4-536a87-kserve-74c6fdbc8-sf89f"
Nov 01 14:31:59.232402 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:31:59.232264    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a720a76-488a-4a65-98ad-31ca84fa7842-home\") pod \"premium-w4-536a87-kserve-74c6fdbc8-sf89f\" (UID: \"8a720a76-488a-4a65-98ad-31ca84fa7842\") " pod="e2e-models-e2e-worker-w4-536a87/premium-w4-536a87-kserve-74c6fdbc8-sf89f"
Nov 01 14:31:59.232402 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:31:59.232296    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a720a76-488a-4a65-98ad-31ca84fa7842-dshm\") pod \"premium-w4-536a87-kserve-74c6fdbc8-sf89f\" (UID: \"8a720a76-488a-4a65-98ad-31ca84fa7842\") " pod="e2e-models-e2e-worker-w4-536a87/premium-w4-536a87-kserve-74c6fdbc8-sf89f"
Nov 01 14:31:59.232637 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:31:59.232609    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a720a76-488a-4a65-98ad-31ca84fa7842-tmp-dir\") pod \"premium-w4-536a87-kserve-74c6fdbc8-sf89f\" (UID: \"8a720a76-488a-4a65-98ad-31ca84fa7842\") " pod="e2e-models-e2e-worker-w4-536a87/premium-w4-536a87-kserve-74c6fdbc8-sf89f"
Nov 01 14:31:59.232740 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:31:59.232622    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a720a76-488a-4a65-98ad-31ca84fa7842-model-cache\") pod \"premium-w4-536a87-kserve-74c6fdbc8-sf89f\" (UID: \"8a720a76-488a-4a65-98ad-31ca84fa7842\") " pod="e2e-models-e2e-worker-w4-536a87/premium-w4-536a87-kserve-74c6fdbc8-sf89f"
Nov 01 14:31:59.232740 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:31:59.232671    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a720a76-488a-4a65-98ad-31ca84fa7842-home\") pod \"premium-w4-536a87-kserve-74c6fdbc8-sf89f\" (UID: \"8a720a76-488a-4a65-98ad-31ca84fa7842\") " pod="e2e-models-e2e-worker-w4-536a87/premium-w4-536a87-kserve-74c6fdbc8-sf89f"
Nov 01 14:31:59.234624 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:31:59.234593    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a720a76-488a-4a65-98ad-31ca84fa7842-dshm\") pod \"premium-w4-536a87-kserve-74c6fdbc8-sf89f\" (UID: \"8a720a76-488a-4a65-98ad-31ca84fa7842\") " pod="e2e-models-e2e-worker-w4-536a87/premium-w4-536a87-kserve-74c6fdbc8-sf89f"
Nov 01 14:31:59.234861 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:31:59.234842    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a720a76-488a-4a65-98ad-31ca84fa7842-tls-certs\") pod \"premium-w4-536a87-kserve-74c6fdbc8-sf89f\" (UID: \"8a720a76-488a-4a65-98ad-31ca84fa7842\") " pod="e2e-models-e2e-worker-w4-536a87/premium-w4-536a87-kserve-74c6fdbc8-sf89f"
Nov 01 14:31:59.284982 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:31:59.284953    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9qjj\" (UniqueName: \"kubernetes.io/projected/8a720a76-488a-4a65-98ad-31ca84fa7842-kube-api-access-d9qjj\") pod \"premium-w4-536a87-kserve-74c6fdbc8-sf89f\" (UID: \"8a720a76-488a-4a65-98ad-31ca84fa7842\") " pod="e2e-models-e2e-worker-w4-536a87/premium-w4-536a87-kserve-74c6fdbc8-sf89f"
Nov 01 14:31:59.398260 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:31:59.398147    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-536a87/premium-w4-536a87-kserve-74c6fdbc8-sf89f"
Nov 01 14:31:59.791857 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:31:59.791822    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-536a87/premium-w4-536a87-kserve-74c6fdbc8-sf89f"]
Nov 01 14:31:59.797482 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:31:59.797458    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8a720a76_488a_4a65_98ad_31ca84fa7842.slice/crio-a5dd081048f8953ac160f157e50966ada50a13d66abf577500f38a6329849bbc WatchSource:0}: Error finding container a5dd081048f8953ac160f157e50966ada50a13d66abf577500f38a6329849bbc: Status 404 returned error can't find the container with id a5dd081048f8953ac160f157e50966ada50a13d66abf577500f38a6329849bbc
Nov 01 14:32:00.381639 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:32:00.381603    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-536a87/premium-w4-536a87-kserve-74c6fdbc8-sf89f" event={"ID":"8a720a76-488a-4a65-98ad-31ca84fa7842","Type":"ContainerStarted","Data":"f3959116fe019c5c61eaac8ca70d68bed208bfa32a91dbf4d6f0c3dad66f46ef"}
Nov 01 14:32:00.381639 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:32:00.381645    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-536a87/premium-w4-536a87-kserve-74c6fdbc8-sf89f" event={"ID":"8a720a76-488a-4a65-98ad-31ca84fa7842","Type":"ContainerStarted","Data":"a5dd081048f8953ac160f157e50966ada50a13d66abf577500f38a6329849bbc"}
Nov 01 14:32:00.399049 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:32:00.399024    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-536a87/premium-w4-536a87-kserve-74c6fdbc8-sf89f"
Nov 01 14:32:09.399334 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:32:09.399298    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-536a87/premium-w4-536a87-kserve-74c6fdbc8-sf89f"
Nov 01 14:32:09.411510 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:32:09.411483    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-536a87/premium-w4-536a87-kserve-74c6fdbc8-sf89f"
Nov 01 14:32:09.440726 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:32:09.440697    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-536a87/premium-w4-536a87-kserve-74c6fdbc8-sf89f"
Nov 01 14:32:09.518975 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:32:09.518921    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-536a87/premium-w4-536a87-kserve-74c6fdbc8-sf89f" podStartSLOduration=10.518909125 podStartE2EDuration="10.518909125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:31:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:32:00.487253022 +0000 UTC m=+907.817115758" watchObservedRunningTime="2026-11-01 14:32:09.518909125 +0000 UTC m=+916.848771905"
Nov 01 14:32:27.499782 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:32:27.499738    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:32:57.499998 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:32:57.499959    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:33:26.330968 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:26.330923    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-1f8e2c/e2e-conflict-a-a12d4d90-kserve-556bbf6448-7r5xm"]
Nov 01 14:33:26.334892 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:26.334867    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1f8e2c/e2e-conflict-a-a12d4d90-kserve-556bbf6448-7r5xm"
Nov 01 14:33:26.411033 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:26.410992    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/643af801-f12c-4c20-b4a6-246733fbb2b0-tmp-dir\") pod \"e2e-conflict-a-a12d4d90-kserve-556bbf6448-7r5xm\" (UID: \"643af801-f12c-4c20-b4a6-246733fbb2b0\") " pod="e2e-models-e2e-worker-w0-1f8e2c/e2e-conflict-a-a12d4d90-kserve-556bbf6448-7r5xm"
Nov 01 14:33:26.411033 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:26.411038    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/643af801-f12c-4c20-b4a6-246733fbb2b0-home\") pod \"e2e-conflict-a-a12d4d90-kserve-556bbf6448-7r5xm\" (UID: \"643af801-f12c-4c20-b4a6-246733fbb2b0\") " pod="e2e-models-e2e-worker-w0-1f8e2c/e2e-conflict-a-a12d4d90-kserve-556bbf6448-7r5xm"
Nov 01 14:33:26.411347 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:26.411233    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r72x\" (UniqueName: \"kubernetes.io/projected/643af801-f12c-4c20-b4a6-246733fbb2b0-kube-api-access-7r72x\") pod \"e2e-conflict-a-a12d4d90-kserve-556bbf6448-7r5xm\" (UID: \"643af801-f12c-4c20-b4a6-246733fbb2b0\") " pod="e2e-models-e2e-worker-w0-1f8e2c/e2e-conflict-a-a12d4d90-kserve-556bbf6448-7r5xm"
Nov 01 14:33:26.411347 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:26.411337    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/643af801-f12c-4c20-b4a6-246733fbb2b0-dshm\") pod \"e2e-conflict-a-a12d4d90-kserve-556bbf6448-7r5xm\" (UID: \"643af801-f12c-4c20-b4a6-246733fbb2b0\") " pod="e2e-models-e2e-worker-w0-1f8e2c/e2e-conflict-a-a12d4d90-kserve-556bbf6448-7r5xm"
Nov 01 14:33:26.411468 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:26.411446    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/643af801-f12c-4c20-b4a6-246733fbb2b0-model-cache\") pod \"e2e-conflict-a-a12d4d90-kserve-556bbf6448-7r5xm\" (UID: \"643af801-f12c-4c20-b4a6-246733fbb2b0\") " pod="e2e-models-e2e-worker-w0-1f8e2c/e2e-conflict-a-a12d4d90-kserve-556bbf6448-7r5xm"
Nov 01 14:33:26.411546 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:26.411529    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/643af801-f12c-4c20-b4a6-246733fbb2b0-tls-certs\") pod \"e2e-conflict-a-a12d4d90-kserve-556bbf6448-7r5xm\" (UID: \"643af801-f12c-4c20-b4a6-246733fbb2b0\") " pod="e2e-models-e2e-worker-w0-1f8e2c/e2e-conflict-a-a12d4d90-kserve-556bbf6448-7r5xm"
Nov 01 14:33:26.429498 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:26.429461    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-1f8e2c\"/\"e2e-conflict-a-a12d4d90-kserve-self-signed-certs\""
Nov 01 14:33:26.433250 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:26.433215    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1f8e2c/e2e-conflict-a-a12d4d90-kserve-556bbf6448-7r5xm"]
Nov 01 14:33:26.513139 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:26.513099    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/643af801-f12c-4c20-b4a6-246733fbb2b0-model-cache\") pod \"e2e-conflict-a-a12d4d90-kserve-556bbf6448-7r5xm\" (UID: \"643af801-f12c-4c20-b4a6-246733fbb2b0\") " pod="e2e-models-e2e-worker-w0-1f8e2c/e2e-conflict-a-a12d4d90-kserve-556bbf6448-7r5xm"
Nov 01 14:33:26.513347 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:26.513198    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/643af801-f12c-4c20-b4a6-246733fbb2b0-tls-certs\") pod \"e2e-conflict-a-a12d4d90-kserve-556bbf6448-7r5xm\" (UID: \"643af801-f12c-4c20-b4a6-246733fbb2b0\") " pod="e2e-models-e2e-worker-w0-1f8e2c/e2e-conflict-a-a12d4d90-kserve-556bbf6448-7r5xm"
Nov 01 14:33:26.513347 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:26.513259    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/643af801-f12c-4c20-b4a6-246733fbb2b0-tmp-dir\") pod \"e2e-conflict-a-a12d4d90-kserve-556bbf6448-7r5xm\" (UID: \"643af801-f12c-4c20-b4a6-246733fbb2b0\") " pod="e2e-models-e2e-worker-w0-1f8e2c/e2e-conflict-a-a12d4d90-kserve-556bbf6448-7r5xm"
Nov 01 14:33:26.513347 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:26.513291    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/643af801-f12c-4c20-b4a6-246733fbb2b0-home\") pod \"e2e-conflict-a-a12d4d90-kserve-556bbf6448-7r5xm\" (UID: \"643af801-f12c-4c20-b4a6-246733fbb2b0\") " pod="e2e-models-e2e-worker-w0-1f8e2c/e2e-conflict-a-a12d4d90-kserve-556bbf6448-7r5xm"
Nov 01 14:33:26.513347 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:26.513339    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r72x\" (UniqueName: \"kubernetes.io/projected/643af801-f12c-4c20-b4a6-246733fbb2b0-kube-api-access-7r72x\") pod \"e2e-conflict-a-a12d4d90-kserve-556bbf6448-7r5xm\" (UID: \"643af801-f12c-4c20-b4a6-246733fbb2b0\") " pod="e2e-models-e2e-worker-w0-1f8e2c/e2e-conflict-a-a12d4d90-kserve-556bbf6448-7r5xm"
Nov 01 14:33:26.513576 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:26.513405    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/643af801-f12c-4c20-b4a6-246733fbb2b0-dshm\") pod \"e2e-conflict-a-a12d4d90-kserve-556bbf6448-7r5xm\" (UID: \"643af801-f12c-4c20-b4a6-246733fbb2b0\") " pod="e2e-models-e2e-worker-w0-1f8e2c/e2e-conflict-a-a12d4d90-kserve-556bbf6448-7r5xm"
Nov 01 14:33:26.513639 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:26.513582    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/643af801-f12c-4c20-b4a6-246733fbb2b0-model-cache\") pod \"e2e-conflict-a-a12d4d90-kserve-556bbf6448-7r5xm\" (UID: \"643af801-f12c-4c20-b4a6-246733fbb2b0\") " pod="e2e-models-e2e-worker-w0-1f8e2c/e2e-conflict-a-a12d4d90-kserve-556bbf6448-7r5xm"
Nov 01 14:33:26.513695 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:26.513665    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/643af801-f12c-4c20-b4a6-246733fbb2b0-home\") pod \"e2e-conflict-a-a12d4d90-kserve-556bbf6448-7r5xm\" (UID: \"643af801-f12c-4c20-b4a6-246733fbb2b0\") " pod="e2e-models-e2e-worker-w0-1f8e2c/e2e-conflict-a-a12d4d90-kserve-556bbf6448-7r5xm"
Nov 01 14:33:26.514062 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:26.514036    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/643af801-f12c-4c20-b4a6-246733fbb2b0-tmp-dir\") pod \"e2e-conflict-a-a12d4d90-kserve-556bbf6448-7r5xm\" (UID: \"643af801-f12c-4c20-b4a6-246733fbb2b0\") " pod="e2e-models-e2e-worker-w0-1f8e2c/e2e-conflict-a-a12d4d90-kserve-556bbf6448-7r5xm"
Nov 01 14:33:26.515626 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:26.515605    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/643af801-f12c-4c20-b4a6-246733fbb2b0-dshm\") pod \"e2e-conflict-a-a12d4d90-kserve-556bbf6448-7r5xm\" (UID: \"643af801-f12c-4c20-b4a6-246733fbb2b0\") " pod="e2e-models-e2e-worker-w0-1f8e2c/e2e-conflict-a-a12d4d90-kserve-556bbf6448-7r5xm"
Nov 01 14:33:26.516149 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:26.516127    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/643af801-f12c-4c20-b4a6-246733fbb2b0-tls-certs\") pod \"e2e-conflict-a-a12d4d90-kserve-556bbf6448-7r5xm\" (UID: \"643af801-f12c-4c20-b4a6-246733fbb2b0\") " pod="e2e-models-e2e-worker-w0-1f8e2c/e2e-conflict-a-a12d4d90-kserve-556bbf6448-7r5xm"
Nov 01 14:33:26.567378 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:26.567338    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r72x\" (UniqueName: \"kubernetes.io/projected/643af801-f12c-4c20-b4a6-246733fbb2b0-kube-api-access-7r72x\") pod \"e2e-conflict-a-a12d4d90-kserve-556bbf6448-7r5xm\" (UID: \"643af801-f12c-4c20-b4a6-246733fbb2b0\") " pod="e2e-models-e2e-worker-w0-1f8e2c/e2e-conflict-a-a12d4d90-kserve-556bbf6448-7r5xm"
Nov 01 14:33:26.647445 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:26.647335    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1f8e2c/e2e-conflict-a-a12d4d90-kserve-556bbf6448-7r5xm"
Nov 01 14:33:27.046842 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:27.046801    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1f8e2c/e2e-conflict-a-a12d4d90-kserve-556bbf6448-7r5xm"]
Nov 01 14:33:27.053333 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:33:27.053307    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod643af801_f12c_4c20_b4a6_246733fbb2b0.slice/crio-55f7312addabace942143a167b5821b349ba58da56e33364befa2963dd925db5 WatchSource:0}: Error finding container 55f7312addabace942143a167b5821b349ba58da56e33364befa2963dd925db5: Status 404 returned error can't find the container with id 55f7312addabace942143a167b5821b349ba58da56e33364befa2963dd925db5
Nov 01 14:33:27.501234 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:33:27.501195    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:33:27.876586 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:27.876477    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1f8e2c/e2e-conflict-a-a12d4d90-kserve-556bbf6448-7r5xm" event={"ID":"643af801-f12c-4c20-b4a6-246733fbb2b0","Type":"ContainerStarted","Data":"ca766737225c781ff59293c0f6ecb973aab0a4a5e7b9a3d8a8c9806e291e5d95"}
Nov 01 14:33:27.876586 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:27.876538    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1f8e2c/e2e-conflict-a-a12d4d90-kserve-556bbf6448-7r5xm" event={"ID":"643af801-f12c-4c20-b4a6-246733fbb2b0","Type":"ContainerStarted","Data":"55f7312addabace942143a167b5821b349ba58da56e33364befa2963dd925db5"}
Nov 01 14:33:27.984116 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:27.984050    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-1f8e2c/e2e-conflict-a-a12d4d90-kserve-556bbf6448-7r5xm" podStartSLOduration=1.984030585 podStartE2EDuration="1.984030585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:33:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:33:27.982241835 +0000 UTC m=+995.312104571" watchObservedRunningTime="2026-11-01 14:33:27.984030585 +0000 UTC m=+995.313893323"
Nov 01 14:33:28.648425 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:28.648387    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f8e2c/e2e-conflict-a-a12d4d90-kserve-556bbf6448-7r5xm"
Nov 01 14:33:34.536644 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:34.536600    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx"]
Nov 01 14:33:34.540562 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:34.540538    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx"
Nov 01 14:33:34.591297 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:34.591261    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-5b0dcc-openshift-default-dockercfg-4d7r2\""
Nov 01 14:33:34.602795 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:34.602760    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3b714f6-ea83-4197-8587-1407bbf7907a-istio-data\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx\" (UID: \"c3b714f6-ea83-4197-8587-1407bbf7907a\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx"
Nov 01 14:33:34.603045 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:34.603019    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3b714f6-ea83-4197-8587-1407bbf7907a-istio-token\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx\" (UID: \"c3b714f6-ea83-4197-8587-1407bbf7907a\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx"
Nov 01 14:33:34.603238 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:34.603218    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3b714f6-ea83-4197-8587-1407bbf7907a-istio-envoy\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx\" (UID: \"c3b714f6-ea83-4197-8587-1407bbf7907a\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx"
Nov 01 14:33:34.603469 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:34.603447    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6l2j\" (UniqueName: \"kubernetes.io/projected/c3b714f6-ea83-4197-8587-1407bbf7907a-kube-api-access-x6l2j\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx\" (UID: \"c3b714f6-ea83-4197-8587-1407bbf7907a\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx"
Nov 01 14:33:34.603663 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:34.603644    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3b714f6-ea83-4197-8587-1407bbf7907a-workload-certs\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx\" (UID: \"c3b714f6-ea83-4197-8587-1407bbf7907a\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx"
Nov 01 14:33:34.603819 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:34.603802    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3b714f6-ea83-4197-8587-1407bbf7907a-istio-podinfo\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx\" (UID: \"c3b714f6-ea83-4197-8587-1407bbf7907a\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx"
Nov 01 14:33:34.603943 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:34.603927    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3b714f6-ea83-4197-8587-1407bbf7907a-credential-socket\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx\" (UID: \"c3b714f6-ea83-4197-8587-1407bbf7907a\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx"
Nov 01 14:33:34.604054 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:34.604038    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3b714f6-ea83-4197-8587-1407bbf7907a-istiod-ca-cert\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx\" (UID: \"c3b714f6-ea83-4197-8587-1407bbf7907a\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx"
Nov 01 14:33:34.604201 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:34.604183    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3b714f6-ea83-4197-8587-1407bbf7907a-workload-socket\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx\" (UID: \"c3b714f6-ea83-4197-8587-1407bbf7907a\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx"
Nov 01 14:33:34.649203 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:34.649155    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx"]
Nov 01 14:33:34.721710 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:34.721673    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3b714f6-ea83-4197-8587-1407bbf7907a-workload-certs\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx\" (UID: \"c3b714f6-ea83-4197-8587-1407bbf7907a\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx"
Nov 01 14:33:34.721987 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:34.721964    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3b714f6-ea83-4197-8587-1407bbf7907a-istio-podinfo\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx\" (UID: \"c3b714f6-ea83-4197-8587-1407bbf7907a\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx"
Nov 01 14:33:34.722104 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:34.722090    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3b714f6-ea83-4197-8587-1407bbf7907a-credential-socket\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx\" (UID: \"c3b714f6-ea83-4197-8587-1407bbf7907a\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx"
Nov 01 14:33:34.722293 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:34.722277    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3b714f6-ea83-4197-8587-1407bbf7907a-istiod-ca-cert\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx\" (UID: \"c3b714f6-ea83-4197-8587-1407bbf7907a\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx"
Nov 01 14:33:34.722413 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:34.722399    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3b714f6-ea83-4197-8587-1407bbf7907a-workload-socket\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx\" (UID: \"c3b714f6-ea83-4197-8587-1407bbf7907a\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx"
Nov 01 14:33:34.722517 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:34.722505    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3b714f6-ea83-4197-8587-1407bbf7907a-istio-data\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx\" (UID: \"c3b714f6-ea83-4197-8587-1407bbf7907a\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx"
Nov 01 14:33:34.722622 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:34.722601    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3b714f6-ea83-4197-8587-1407bbf7907a-istio-token\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx\" (UID: \"c3b714f6-ea83-4197-8587-1407bbf7907a\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx"
Nov 01 14:33:34.722760 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:34.722748    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3b714f6-ea83-4197-8587-1407bbf7907a-istio-envoy\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx\" (UID: \"c3b714f6-ea83-4197-8587-1407bbf7907a\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx"
Nov 01 14:33:34.722872 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:34.722857    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6l2j\" (UniqueName: \"kubernetes.io/projected/c3b714f6-ea83-4197-8587-1407bbf7907a-kube-api-access-x6l2j\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx\" (UID: \"c3b714f6-ea83-4197-8587-1407bbf7907a\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx"
Nov 01 14:33:34.725201 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:34.723355    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3b714f6-ea83-4197-8587-1407bbf7907a-istiod-ca-cert\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx\" (UID: \"c3b714f6-ea83-4197-8587-1407bbf7907a\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx"
Nov 01 14:33:34.725201 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:34.723643    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3b714f6-ea83-4197-8587-1407bbf7907a-credential-socket\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx\" (UID: \"c3b714f6-ea83-4197-8587-1407bbf7907a\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx"
Nov 01 14:33:34.725201 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:34.722638    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3b714f6-ea83-4197-8587-1407bbf7907a-workload-certs\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx\" (UID: \"c3b714f6-ea83-4197-8587-1407bbf7907a\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx"
Nov 01 14:33:34.725201 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:34.724128    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3b714f6-ea83-4197-8587-1407bbf7907a-istio-data\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx\" (UID: \"c3b714f6-ea83-4197-8587-1407bbf7907a\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx"
Nov 01 14:33:34.725201 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:34.724334    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3b714f6-ea83-4197-8587-1407bbf7907a-workload-socket\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx\" (UID: \"c3b714f6-ea83-4197-8587-1407bbf7907a\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx"
Nov 01 14:33:34.727747 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:34.727657    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3b714f6-ea83-4197-8587-1407bbf7907a-istio-envoy\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx\" (UID: \"c3b714f6-ea83-4197-8587-1407bbf7907a\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx"
Nov 01 14:33:34.728124 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:34.728105    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3b714f6-ea83-4197-8587-1407bbf7907a-istio-podinfo\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx\" (UID: \"c3b714f6-ea83-4197-8587-1407bbf7907a\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx"
Nov 01 14:33:34.805475 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:34.804941    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3b714f6-ea83-4197-8587-1407bbf7907a-istio-token\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx\" (UID: \"c3b714f6-ea83-4197-8587-1407bbf7907a\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx"
Nov 01 14:33:34.810246 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:34.810212    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6l2j\" (UniqueName: \"kubernetes.io/projected/c3b714f6-ea83-4197-8587-1407bbf7907a-kube-api-access-x6l2j\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx\" (UID: \"c3b714f6-ea83-4197-8587-1407bbf7907a\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx"
Nov 01 14:33:34.863192 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:34.858335    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx"
Nov 01 14:33:35.374914 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:35.374874    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx"]
Nov 01 14:33:35.405514 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:35.404062    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 14:33:35.405514 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:35.404149    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 14:33:35.405514 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:35.404233    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 14:33:35.938804 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:35.938767    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx" event={"ID":"c3b714f6-ea83-4197-8587-1407bbf7907a","Type":"ContainerStarted","Data":"6c0c9edbd0c327e92f5929626048fdbc2f861312ff98a9d55da1a2750e368a7e"}
Nov 01 14:33:35.939213 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:35.938830    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx" event={"ID":"c3b714f6-ea83-4197-8587-1407bbf7907a","Type":"ContainerStarted","Data":"a6f82c9bd79b72962ecca9fa95d46463710be9f65766240d7f2b01beb6464e0d"}
Nov 01 14:33:36.050135 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:36.050084    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx" podStartSLOduration=2.0500691 podStartE2EDuration="2.0500691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:33:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:33:36.049156209 +0000 UTC m=+1003.379018945" watchObservedRunningTime="2026-11-01 14:33:36.0500691 +0000 UTC m=+1003.379931836"
Nov 01 14:33:36.647768 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:36.647732    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-1f8e2c/e2e-conflict-a-a12d4d90-kserve-556bbf6448-7r5xm"
Nov 01 14:33:36.697389 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:36.697357    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-1f8e2c/e2e-conflict-a-a12d4d90-kserve-556bbf6448-7r5xm"
Nov 01 14:33:36.863572 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:36.863536    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx"
Nov 01 14:33:36.865834 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:36.865801    4993 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Nov 01 14:33:36.866111 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:36.866083    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx" podUID="c3b714f6-ea83-4197-8587-1407bbf7907a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Nov 01 14:33:36.994861 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:36.994822    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f8e2c/e2e-conflict-a-a12d4d90-kserve-556bbf6448-7r5xm"
Nov 01 14:33:37.873193 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:37.872695    4993 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Nov 01 14:33:37.873193 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:37.872759    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx" podUID="c3b714f6-ea83-4197-8587-1407bbf7907a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Nov 01 14:33:38.871193 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:38.868585    4993 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Nov 01 14:33:38.871193 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:38.868654    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx" podUID="c3b714f6-ea83-4197-8587-1407bbf7907a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Nov 01 14:33:39.874193 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:39.870583    4993 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Nov 01 14:33:39.874193 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:39.870656    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx" podUID="c3b714f6-ea83-4197-8587-1407bbf7907a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Nov 01 14:33:40.870683 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:40.870638    4993 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Nov 01 14:33:40.879226 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:40.870914    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx" podUID="c3b714f6-ea83-4197-8587-1407bbf7907a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Nov 01 14:33:41.863246 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:41.863209    4993 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Nov 01 14:33:41.863450 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:41.863261    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx" podUID="c3b714f6-ea83-4197-8587-1407bbf7907a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Nov 01 14:33:42.867029 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:42.866990    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx"
Nov 01 14:33:42.867456 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:42.867360    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx"
Nov 01 14:33:42.868045 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:42.868024    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx"
Nov 01 14:33:57.500428 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:33:57.500380    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:33:57.530368 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:57.530344    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-5b0dcc/distinct-w3-5b0dcc-kserve-57cb5cb5cf-msjth"]
Nov 01 14:33:57.533963 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:57.533930    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-w3-5b0dcc-kserve-57cb5cb5cf-msjth"
Nov 01 14:33:57.586049 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:57.586018    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3ac5585-dc23-445d-89fc-b68f90dac95a-tmp-dir\") pod \"distinct-w3-5b0dcc-kserve-57cb5cb5cf-msjth\" (UID: \"e3ac5585-dc23-445d-89fc-b68f90dac95a\") " pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-w3-5b0dcc-kserve-57cb5cb5cf-msjth"
Nov 01 14:33:57.586224 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:57.586062    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3ac5585-dc23-445d-89fc-b68f90dac95a-home\") pod \"distinct-w3-5b0dcc-kserve-57cb5cb5cf-msjth\" (UID: \"e3ac5585-dc23-445d-89fc-b68f90dac95a\") " pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-w3-5b0dcc-kserve-57cb5cb5cf-msjth"
Nov 01 14:33:57.586224 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:57.586140    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3ac5585-dc23-445d-89fc-b68f90dac95a-dshm\") pod \"distinct-w3-5b0dcc-kserve-57cb5cb5cf-msjth\" (UID: \"e3ac5585-dc23-445d-89fc-b68f90dac95a\") " pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-w3-5b0dcc-kserve-57cb5cb5cf-msjth"
Nov 01 14:33:57.586224 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:57.586220    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pb98\" (UniqueName: \"kubernetes.io/projected/e3ac5585-dc23-445d-89fc-b68f90dac95a-kube-api-access-4pb98\") pod \"distinct-w3-5b0dcc-kserve-57cb5cb5cf-msjth\" (UID: \"e3ac5585-dc23-445d-89fc-b68f90dac95a\") " pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-w3-5b0dcc-kserve-57cb5cb5cf-msjth"
Nov 01 14:33:57.586414 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:57.586247    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3ac5585-dc23-445d-89fc-b68f90dac95a-tls-certs\") pod \"distinct-w3-5b0dcc-kserve-57cb5cb5cf-msjth\" (UID: \"e3ac5585-dc23-445d-89fc-b68f90dac95a\") " pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-w3-5b0dcc-kserve-57cb5cb5cf-msjth"
Nov 01 14:33:57.586414 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:57.586349    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3ac5585-dc23-445d-89fc-b68f90dac95a-model-cache\") pod \"distinct-w3-5b0dcc-kserve-57cb5cb5cf-msjth\" (UID: \"e3ac5585-dc23-445d-89fc-b68f90dac95a\") " pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-w3-5b0dcc-kserve-57cb5cb5cf-msjth"
Nov 01 14:33:57.628755 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:57.628702    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-5b0dcc\"/\"openshift-service-ca.crt\""
Nov 01 14:33:57.628755 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:57.628720    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-5b0dcc\"/\"kube-root-ca.crt\""
Nov 01 14:33:57.628755 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:57.628702    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-5b0dcc\"/\"default-dockercfg-nqggd\""
Nov 01 14:33:57.628991 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:57.628772    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-5b0dcc\"/\"distinct-w3-5b0dcc-kserve-self-signed-certs\""
Nov 01 14:33:57.632507 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:57.632479    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-5b0dcc/distinct-w3-5b0dcc-kserve-57cb5cb5cf-msjth"]
Nov 01 14:33:57.687432 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:57.687394    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3ac5585-dc23-445d-89fc-b68f90dac95a-home\") pod \"distinct-w3-5b0dcc-kserve-57cb5cb5cf-msjth\" (UID: \"e3ac5585-dc23-445d-89fc-b68f90dac95a\") " pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-w3-5b0dcc-kserve-57cb5cb5cf-msjth"
Nov 01 14:33:57.687605 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:57.687475    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3ac5585-dc23-445d-89fc-b68f90dac95a-dshm\") pod \"distinct-w3-5b0dcc-kserve-57cb5cb5cf-msjth\" (UID: \"e3ac5585-dc23-445d-89fc-b68f90dac95a\") " pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-w3-5b0dcc-kserve-57cb5cb5cf-msjth"
Nov 01 14:33:57.687605 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:57.687520    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pb98\" (UniqueName: \"kubernetes.io/projected/e3ac5585-dc23-445d-89fc-b68f90dac95a-kube-api-access-4pb98\") pod \"distinct-w3-5b0dcc-kserve-57cb5cb5cf-msjth\" (UID: \"e3ac5585-dc23-445d-89fc-b68f90dac95a\") " pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-w3-5b0dcc-kserve-57cb5cb5cf-msjth"
Nov 01 14:33:57.687727 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:57.687646    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3ac5585-dc23-445d-89fc-b68f90dac95a-tls-certs\") pod \"distinct-w3-5b0dcc-kserve-57cb5cb5cf-msjth\" (UID: \"e3ac5585-dc23-445d-89fc-b68f90dac95a\") " pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-w3-5b0dcc-kserve-57cb5cb5cf-msjth"
Nov 01 14:33:57.687803 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:57.687782    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3ac5585-dc23-445d-89fc-b68f90dac95a-model-cache\") pod \"distinct-w3-5b0dcc-kserve-57cb5cb5cf-msjth\" (UID: \"e3ac5585-dc23-445d-89fc-b68f90dac95a\") " pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-w3-5b0dcc-kserve-57cb5cb5cf-msjth"
Nov 01 14:33:57.687803 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:57.687794    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3ac5585-dc23-445d-89fc-b68f90dac95a-home\") pod \"distinct-w3-5b0dcc-kserve-57cb5cb5cf-msjth\" (UID: \"e3ac5585-dc23-445d-89fc-b68f90dac95a\") " pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-w3-5b0dcc-kserve-57cb5cb5cf-msjth"
Nov 01 14:33:57.687930 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:57.687872    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3ac5585-dc23-445d-89fc-b68f90dac95a-tmp-dir\") pod \"distinct-w3-5b0dcc-kserve-57cb5cb5cf-msjth\" (UID: \"e3ac5585-dc23-445d-89fc-b68f90dac95a\") " pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-w3-5b0dcc-kserve-57cb5cb5cf-msjth"
Nov 01 14:33:57.688074 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:57.688054    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3ac5585-dc23-445d-89fc-b68f90dac95a-model-cache\") pod \"distinct-w3-5b0dcc-kserve-57cb5cb5cf-msjth\" (UID: \"e3ac5585-dc23-445d-89fc-b68f90dac95a\") " pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-w3-5b0dcc-kserve-57cb5cb5cf-msjth"
Nov 01 14:33:57.688179 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:57.688135    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3ac5585-dc23-445d-89fc-b68f90dac95a-tmp-dir\") pod \"distinct-w3-5b0dcc-kserve-57cb5cb5cf-msjth\" (UID: \"e3ac5585-dc23-445d-89fc-b68f90dac95a\") " pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-w3-5b0dcc-kserve-57cb5cb5cf-msjth"
Nov 01 14:33:57.689682 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:57.689658    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3ac5585-dc23-445d-89fc-b68f90dac95a-dshm\") pod \"distinct-w3-5b0dcc-kserve-57cb5cb5cf-msjth\" (UID: \"e3ac5585-dc23-445d-89fc-b68f90dac95a\") " pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-w3-5b0dcc-kserve-57cb5cb5cf-msjth"
Nov 01 14:33:57.690036 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:57.690017    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3ac5585-dc23-445d-89fc-b68f90dac95a-tls-certs\") pod \"distinct-w3-5b0dcc-kserve-57cb5cb5cf-msjth\" (UID: \"e3ac5585-dc23-445d-89fc-b68f90dac95a\") " pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-w3-5b0dcc-kserve-57cb5cb5cf-msjth"
Nov 01 14:33:57.742260 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:57.742223    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pb98\" (UniqueName: \"kubernetes.io/projected/e3ac5585-dc23-445d-89fc-b68f90dac95a-kube-api-access-4pb98\") pod \"distinct-w3-5b0dcc-kserve-57cb5cb5cf-msjth\" (UID: \"e3ac5585-dc23-445d-89fc-b68f90dac95a\") " pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-w3-5b0dcc-kserve-57cb5cb5cf-msjth"
Nov 01 14:33:57.847675 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:57.847582    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-w3-5b0dcc-kserve-57cb5cb5cf-msjth"
Nov 01 14:33:58.248952 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:58.248916    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-5b0dcc/distinct-w3-5b0dcc-kserve-57cb5cb5cf-msjth"]
Nov 01 14:33:58.255200 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:33:58.255159    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode3ac5585_dc23_445d_89fc_b68f90dac95a.slice/crio-b06defadfe07e00f98dba3f1dba806e84dc1f7baf682bced39bef9f598cf0001 WatchSource:0}: Error finding container b06defadfe07e00f98dba3f1dba806e84dc1f7baf682bced39bef9f598cf0001: Status 404 returned error can't find the container with id b06defadfe07e00f98dba3f1dba806e84dc1f7baf682bced39bef9f598cf0001
Nov 01 14:33:59.098520 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:59.098477    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-w3-5b0dcc-kserve-57cb5cb5cf-msjth" event={"ID":"e3ac5585-dc23-445d-89fc-b68f90dac95a","Type":"ContainerStarted","Data":"08ec96cbfd6ef12925ec063c35ae354ff867b41b162fc4c4fb4166a67a6be41f"}
Nov 01 14:33:59.098932 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:59.098533    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-w3-5b0dcc-kserve-57cb5cb5cf-msjth" event={"ID":"e3ac5585-dc23-445d-89fc-b68f90dac95a","Type":"ContainerStarted","Data":"b06defadfe07e00f98dba3f1dba806e84dc1f7baf682bced39bef9f598cf0001"}
Nov 01 14:33:59.207502 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:59.207438    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-w3-5b0dcc-kserve-57cb5cb5cf-msjth" podStartSLOduration=2.207419585 podStartE2EDuration="2.207419585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:33:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:33:59.20710582 +0000 UTC m=+1026.536968557" watchObservedRunningTime="2026-11-01 14:33:59.207419585 +0000 UTC m=+1026.537282323"
Nov 01 14:33:59.848078 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:33:59.848041    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-w3-5b0dcc-kserve-57cb5cb5cf-msjth"
Nov 01 14:34:03.086770 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:03.086679    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hk2fv"]
Nov 01 14:34:03.090770 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:03.090746    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hk2fv"
Nov 01 14:34:03.136593 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:03.136555    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hk2fv\" (UID: \"4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hk2fv"
Nov 01 14:34:03.136593 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:03.136597    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hk2fv\" (UID: \"4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hk2fv"
Nov 01 14:34:03.136822 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:03.136638    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hk2fv\" (UID: \"4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hk2fv"
Nov 01 14:34:03.136822 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:03.136733    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hk2fv\" (UID: \"4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hk2fv"
Nov 01 14:34:03.136822 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:03.136791    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hk2fv\" (UID: \"4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hk2fv"
Nov 01 14:34:03.136982 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:03.136859    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hk2fv\" (UID: \"4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hk2fv"
Nov 01 14:34:03.136982 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:03.136894    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hk2fv\" (UID: \"4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hk2fv"
Nov 01 14:34:03.136982 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:03.136912    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hk2fv\" (UID: \"4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hk2fv"
Nov 01 14:34:03.136982 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:03.136938    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6vxp\" (UniqueName: \"kubernetes.io/projected/4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81-kube-api-access-w6vxp\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hk2fv\" (UID: \"4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hk2fv"
Nov 01 14:34:03.201037 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:03.201003    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hk2fv"]
Nov 01 14:34:03.237819 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:03.237786    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hk2fv\" (UID: \"4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hk2fv"
Nov 01 14:34:03.238008 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:03.237836    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hk2fv\" (UID: \"4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hk2fv"
Nov 01 14:34:03.238008 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:03.237880    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hk2fv\" (UID: \"4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hk2fv"
Nov 01 14:34:03.238008 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:03.237900    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hk2fv\" (UID: \"4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hk2fv"
Nov 01 14:34:03.238008 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:03.237926    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hk2fv\" (UID: \"4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hk2fv"
Nov 01 14:34:03.238008 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:03.237944    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hk2fv\" (UID: \"4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hk2fv"
Nov 01 14:34:03.238303 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:03.238147    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6vxp\" (UniqueName: \"kubernetes.io/projected/4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81-kube-api-access-w6vxp\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hk2fv\" (UID: \"4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hk2fv"
Nov 01 14:34:03.238303 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:03.238279    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hk2fv\" (UID: \"4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hk2fv"
Nov 01 14:34:03.238482 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:03.238334    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hk2fv\" (UID: \"4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hk2fv"
Nov 01 14:34:03.238482 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:03.238355    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hk2fv\" (UID: \"4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hk2fv"
Nov 01 14:34:03.238482 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:03.238372    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hk2fv\" (UID: \"4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hk2fv"
Nov 01 14:34:03.238482 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:03.238279    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hk2fv\" (UID: \"4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hk2fv"
Nov 01 14:34:03.238699 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:03.238638    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hk2fv\" (UID: \"4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hk2fv"
Nov 01 14:34:03.238742 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:03.238693    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hk2fv\" (UID: \"4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hk2fv"
Nov 01 14:34:03.240207 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:03.240186    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hk2fv\" (UID: \"4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hk2fv"
Nov 01 14:34:03.240726 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:03.240701    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hk2fv\" (UID: \"4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hk2fv"
Nov 01 14:34:03.291472 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:03.291443    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hk2fv\" (UID: \"4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hk2fv"
Nov 01 14:34:03.291648 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:03.291623    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6vxp\" (UniqueName: \"kubernetes.io/projected/4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81-kube-api-access-w6vxp\") pod \"maas-default-gateway-openshift-default-7f5667f86c-hk2fv\" (UID: \"4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hk2fv"
Nov 01 14:34:03.404740 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:03.404648    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hk2fv"
Nov 01 14:34:03.831471 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:03.831437    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hk2fv"]
Nov 01 14:34:03.837463 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:34:03.837438    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d0df8f2_5aa9_4dd1_b3c0_33844c92cd81.slice/crio-2839545b6d4539dac9acdc2cb19091d9835f7ee93a01b18a0d742322f6032f98 WatchSource:0}: Error finding container 2839545b6d4539dac9acdc2cb19091d9835f7ee93a01b18a0d742322f6032f98: Status 404 returned error can't find the container with id 2839545b6d4539dac9acdc2cb19091d9835f7ee93a01b18a0d742322f6032f98
Nov 01 14:34:03.839760 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:03.839729    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 14:34:03.839833 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:03.839793    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 14:34:03.839833 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:03.839823    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 14:34:04.125104 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:04.125010    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hk2fv" event={"ID":"4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81","Type":"ContainerStarted","Data":"3fce53d3d877e579e59648fdae1b5e8072d2da4814fb0331671432d09a5c1805"}
Nov 01 14:34:04.125104 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:04.125069    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hk2fv" event={"ID":"4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81","Type":"ContainerStarted","Data":"2839545b6d4539dac9acdc2cb19091d9835f7ee93a01b18a0d742322f6032f98"}
Nov 01 14:34:04.234677 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:04.234619    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hk2fv" podStartSLOduration=1.234599335 podStartE2EDuration="1.234599335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:34:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:34:04.232954814 +0000 UTC m=+1031.562817551" watchObservedRunningTime="2026-11-01 14:34:04.234599335 +0000 UTC m=+1031.564462141"
Nov 01 14:34:04.405060 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:04.404974    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hk2fv"
Nov 01 14:34:04.406663 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:04.406640    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-hk2fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Nov 01 14:34:04.406722 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:04.406692    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hk2fv" podUID="4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Nov 01 14:34:05.405898 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:05.405862    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-hk2fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Nov 01 14:34:05.406313 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:05.405922    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hk2fv" podUID="4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Nov 01 14:34:06.405624 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:06.405581    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-hk2fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Nov 01 14:34:06.405830 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:06.405654    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hk2fv" podUID="4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Nov 01 14:34:07.405389 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:07.405348    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-hk2fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Nov 01 14:34:07.405784 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:07.405425    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hk2fv" podUID="4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Nov 01 14:34:07.848666 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:07.848627    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-w3-5b0dcc-kserve-57cb5cb5cf-msjth"
Nov 01 14:34:07.885222 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:07.885156    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-w3-5b0dcc-kserve-57cb5cb5cf-msjth"
Nov 01 14:34:08.190285 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:08.190193    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-w3-5b0dcc-kserve-57cb5cb5cf-msjth"
Nov 01 14:34:08.405508 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:08.405473    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-hk2fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Nov 01 14:34:08.405967 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:08.405545    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hk2fv" podUID="4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Nov 01 14:34:09.404991 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:09.404952    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-hk2fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Nov 01 14:34:09.405162 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:09.405011    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hk2fv" podUID="4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Nov 01 14:34:10.405417 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:10.405379    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-hk2fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Nov 01 14:34:10.405796 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:10.405448    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hk2fv" podUID="4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Nov 01 14:34:11.405342 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:11.405302    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-hk2fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Nov 01 14:34:11.405539 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:11.405373    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hk2fv" podUID="4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Nov 01 14:34:12.480053 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:12.480019    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hk2fv"
Nov 01 14:34:12.480917 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:12.480825    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hk2fv"
Nov 01 14:34:12.481583 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:12.481556    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hk2fv"
Nov 01 14:34:13.339122 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:13.339086    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-00a787/distinct-w1-00a787-kserve-dbd65bf4f-lbcwn"]
Nov 01 14:34:13.343161 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:13.343138    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-00a787/distinct-w1-00a787-kserve-dbd65bf4f-lbcwn"
Nov 01 14:34:13.372833 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:13.372804    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1c97a57-0e89-4370-bc30-c75368d340f9-dshm\") pod \"distinct-w1-00a787-kserve-dbd65bf4f-lbcwn\" (UID: \"e1c97a57-0e89-4370-bc30-c75368d340f9\") " pod="e2e-models-e2e-worker-w1-00a787/distinct-w1-00a787-kserve-dbd65bf4f-lbcwn"
Nov 01 14:34:13.372995 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:13.372879    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cv5n\" (UniqueName: \"kubernetes.io/projected/e1c97a57-0e89-4370-bc30-c75368d340f9-kube-api-access-6cv5n\") pod \"distinct-w1-00a787-kserve-dbd65bf4f-lbcwn\" (UID: \"e1c97a57-0e89-4370-bc30-c75368d340f9\") " pod="e2e-models-e2e-worker-w1-00a787/distinct-w1-00a787-kserve-dbd65bf4f-lbcwn"
Nov 01 14:34:13.373041 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:13.373011    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1c97a57-0e89-4370-bc30-c75368d340f9-tls-certs\") pod \"distinct-w1-00a787-kserve-dbd65bf4f-lbcwn\" (UID: \"e1c97a57-0e89-4370-bc30-c75368d340f9\") " pod="e2e-models-e2e-worker-w1-00a787/distinct-w1-00a787-kserve-dbd65bf4f-lbcwn"
Nov 01 14:34:13.373090 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:13.373058    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1c97a57-0e89-4370-bc30-c75368d340f9-home\") pod \"distinct-w1-00a787-kserve-dbd65bf4f-lbcwn\" (UID: \"e1c97a57-0e89-4370-bc30-c75368d340f9\") " pod="e2e-models-e2e-worker-w1-00a787/distinct-w1-00a787-kserve-dbd65bf4f-lbcwn"
Nov 01 14:34:13.373125 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:13.373108    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1c97a57-0e89-4370-bc30-c75368d340f9-model-cache\") pod \"distinct-w1-00a787-kserve-dbd65bf4f-lbcwn\" (UID: \"e1c97a57-0e89-4370-bc30-c75368d340f9\") " pod="e2e-models-e2e-worker-w1-00a787/distinct-w1-00a787-kserve-dbd65bf4f-lbcwn"
Nov 01 14:34:13.373160 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:13.373140    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1c97a57-0e89-4370-bc30-c75368d340f9-tmp-dir\") pod \"distinct-w1-00a787-kserve-dbd65bf4f-lbcwn\" (UID: \"e1c97a57-0e89-4370-bc30-c75368d340f9\") " pod="e2e-models-e2e-worker-w1-00a787/distinct-w1-00a787-kserve-dbd65bf4f-lbcwn"
Nov 01 14:34:13.391684 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:13.391655    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-00a787\"/\"distinct-w1-00a787-kserve-self-signed-certs\""
Nov 01 14:34:13.402072 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:13.402043    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-00a787/distinct-w1-00a787-kserve-dbd65bf4f-lbcwn"]
Nov 01 14:34:13.474247 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:13.474209    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1c97a57-0e89-4370-bc30-c75368d340f9-tmp-dir\") pod \"distinct-w1-00a787-kserve-dbd65bf4f-lbcwn\" (UID: \"e1c97a57-0e89-4370-bc30-c75368d340f9\") " pod="e2e-models-e2e-worker-w1-00a787/distinct-w1-00a787-kserve-dbd65bf4f-lbcwn"
Nov 01 14:34:13.474434 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:13.474302    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1c97a57-0e89-4370-bc30-c75368d340f9-dshm\") pod \"distinct-w1-00a787-kserve-dbd65bf4f-lbcwn\" (UID: \"e1c97a57-0e89-4370-bc30-c75368d340f9\") " pod="e2e-models-e2e-worker-w1-00a787/distinct-w1-00a787-kserve-dbd65bf4f-lbcwn"
Nov 01 14:34:13.474434 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:13.474350    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cv5n\" (UniqueName: \"kubernetes.io/projected/e1c97a57-0e89-4370-bc30-c75368d340f9-kube-api-access-6cv5n\") pod \"distinct-w1-00a787-kserve-dbd65bf4f-lbcwn\" (UID: \"e1c97a57-0e89-4370-bc30-c75368d340f9\") " pod="e2e-models-e2e-worker-w1-00a787/distinct-w1-00a787-kserve-dbd65bf4f-lbcwn"
Nov 01 14:34:13.474434 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:13.474414    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1c97a57-0e89-4370-bc30-c75368d340f9-tls-certs\") pod \"distinct-w1-00a787-kserve-dbd65bf4f-lbcwn\" (UID: \"e1c97a57-0e89-4370-bc30-c75368d340f9\") " pod="e2e-models-e2e-worker-w1-00a787/distinct-w1-00a787-kserve-dbd65bf4f-lbcwn"
Nov 01 14:34:13.474587 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:13.474442    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1c97a57-0e89-4370-bc30-c75368d340f9-home\") pod \"distinct-w1-00a787-kserve-dbd65bf4f-lbcwn\" (UID: \"e1c97a57-0e89-4370-bc30-c75368d340f9\") " pod="e2e-models-e2e-worker-w1-00a787/distinct-w1-00a787-kserve-dbd65bf4f-lbcwn"
Nov 01 14:34:13.474587 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:13.474462    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1c97a57-0e89-4370-bc30-c75368d340f9-model-cache\") pod \"distinct-w1-00a787-kserve-dbd65bf4f-lbcwn\" (UID: \"e1c97a57-0e89-4370-bc30-c75368d340f9\") " pod="e2e-models-e2e-worker-w1-00a787/distinct-w1-00a787-kserve-dbd65bf4f-lbcwn"
Nov 01 14:34:13.474700 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:13.474627    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1c97a57-0e89-4370-bc30-c75368d340f9-tmp-dir\") pod \"distinct-w1-00a787-kserve-dbd65bf4f-lbcwn\" (UID: \"e1c97a57-0e89-4370-bc30-c75368d340f9\") " pod="e2e-models-e2e-worker-w1-00a787/distinct-w1-00a787-kserve-dbd65bf4f-lbcwn"
Nov 01 14:34:13.474866 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:13.474847    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1c97a57-0e89-4370-bc30-c75368d340f9-model-cache\") pod \"distinct-w1-00a787-kserve-dbd65bf4f-lbcwn\" (UID: \"e1c97a57-0e89-4370-bc30-c75368d340f9\") " pod="e2e-models-e2e-worker-w1-00a787/distinct-w1-00a787-kserve-dbd65bf4f-lbcwn"
Nov 01 14:34:13.474923 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:13.474855    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1c97a57-0e89-4370-bc30-c75368d340f9-home\") pod \"distinct-w1-00a787-kserve-dbd65bf4f-lbcwn\" (UID: \"e1c97a57-0e89-4370-bc30-c75368d340f9\") " pod="e2e-models-e2e-worker-w1-00a787/distinct-w1-00a787-kserve-dbd65bf4f-lbcwn"
Nov 01 14:34:13.477318 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:13.477290    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1c97a57-0e89-4370-bc30-c75368d340f9-tls-certs\") pod \"distinct-w1-00a787-kserve-dbd65bf4f-lbcwn\" (UID: \"e1c97a57-0e89-4370-bc30-c75368d340f9\") " pod="e2e-models-e2e-worker-w1-00a787/distinct-w1-00a787-kserve-dbd65bf4f-lbcwn"
Nov 01 14:34:13.477449 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:13.477323    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1c97a57-0e89-4370-bc30-c75368d340f9-dshm\") pod \"distinct-w1-00a787-kserve-dbd65bf4f-lbcwn\" (UID: \"e1c97a57-0e89-4370-bc30-c75368d340f9\") " pod="e2e-models-e2e-worker-w1-00a787/distinct-w1-00a787-kserve-dbd65bf4f-lbcwn"
Nov 01 14:34:13.527822 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:13.527791    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cv5n\" (UniqueName: \"kubernetes.io/projected/e1c97a57-0e89-4370-bc30-c75368d340f9-kube-api-access-6cv5n\") pod \"distinct-w1-00a787-kserve-dbd65bf4f-lbcwn\" (UID: \"e1c97a57-0e89-4370-bc30-c75368d340f9\") " pod="e2e-models-e2e-worker-w1-00a787/distinct-w1-00a787-kserve-dbd65bf4f-lbcwn"
Nov 01 14:34:13.657563 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:13.657475    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-00a787/distinct-w1-00a787-kserve-dbd65bf4f-lbcwn"
Nov 01 14:34:14.064695 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:14.064659    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-00a787/distinct-w1-00a787-kserve-dbd65bf4f-lbcwn"]
Nov 01 14:34:14.072784 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:34:14.072755    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode1c97a57_0e89_4370_bc30_c75368d340f9.slice/crio-8b5917d7895cdde55cc565fc0429bc9c0e67da83610f8388bb429f14048abdd9 WatchSource:0}: Error finding container 8b5917d7895cdde55cc565fc0429bc9c0e67da83610f8388bb429f14048abdd9: Status 404 returned error can't find the container with id 8b5917d7895cdde55cc565fc0429bc9c0e67da83610f8388bb429f14048abdd9
Nov 01 14:34:14.222530 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:14.222487    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-00a787/distinct-w1-00a787-kserve-dbd65bf4f-lbcwn" event={"ID":"e1c97a57-0e89-4370-bc30-c75368d340f9","Type":"ContainerStarted","Data":"581910a8db537325e9bf269cc934ccfaf9d759c6989acb3257fe69f698851061"}
Nov 01 14:34:14.222704 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:14.222541    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-00a787/distinct-w1-00a787-kserve-dbd65bf4f-lbcwn" event={"ID":"e1c97a57-0e89-4370-bc30-c75368d340f9","Type":"ContainerStarted","Data":"8b5917d7895cdde55cc565fc0429bc9c0e67da83610f8388bb429f14048abdd9"}
Nov 01 14:34:14.330096 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:14.330000    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-00a787/distinct-w1-00a787-kserve-dbd65bf4f-lbcwn" podStartSLOduration=1.329988185 podStartE2EDuration="1.329988185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:34:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:34:14.327743172 +0000 UTC m=+1041.657605908" watchObservedRunningTime="2026-11-01 14:34:14.329988185 +0000 UTC m=+1041.659850921"
Nov 01 14:34:14.657817 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:14.657734    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-00a787/distinct-w1-00a787-kserve-dbd65bf4f-lbcwn"
Nov 01 14:34:23.658306 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:23.658276    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-00a787/distinct-w1-00a787-kserve-dbd65bf4f-lbcwn"
Nov 01 14:34:23.671067 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:23.671041    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-00a787/distinct-w1-00a787-kserve-dbd65bf4f-lbcwn"
Nov 01 14:34:24.329944 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:24.329910    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-00a787/distinct-w1-00a787-kserve-dbd65bf4f-lbcwn"
Nov 01 14:34:27.503995 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:34:27.503956    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:34:41.071903 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:41.071866    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm"]
Nov 01 14:34:41.076010 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:41.075986    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm"
Nov 01 14:34:41.126954 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:41.126923    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-00a787-openshift-default-dockercfg-v7d89\""
Nov 01 14:34:41.132470 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:41.132442    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db204f9-b7a9-4b31-ab49-a4406d2b879d-credential-socket\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm\" (UID: \"6db204f9-b7a9-4b31-ab49-a4406d2b879d\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm"
Nov 01 14:34:41.132587 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:41.132473    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6db204f9-b7a9-4b31-ab49-a4406d2b879d-workload-certs\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm\" (UID: \"6db204f9-b7a9-4b31-ab49-a4406d2b879d\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm"
Nov 01 14:34:41.132630 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:41.132602    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6db204f9-b7a9-4b31-ab49-a4406d2b879d-istio-podinfo\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm\" (UID: \"6db204f9-b7a9-4b31-ab49-a4406d2b879d\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm"
Nov 01 14:34:41.132666 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:41.132629    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6db204f9-b7a9-4b31-ab49-a4406d2b879d-istio-envoy\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm\" (UID: \"6db204f9-b7a9-4b31-ab49-a4406d2b879d\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm"
Nov 01 14:34:41.132710 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:41.132689    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db204f9-b7a9-4b31-ab49-a4406d2b879d-workload-socket\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm\" (UID: \"6db204f9-b7a9-4b31-ab49-a4406d2b879d\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm"
Nov 01 14:34:41.132761 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:41.132747    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6db204f9-b7a9-4b31-ab49-a4406d2b879d-istiod-ca-cert\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm\" (UID: \"6db204f9-b7a9-4b31-ab49-a4406d2b879d\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm"
Nov 01 14:34:41.132808 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:41.132793    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6db204f9-b7a9-4b31-ab49-a4406d2b879d-istio-token\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm\" (UID: \"6db204f9-b7a9-4b31-ab49-a4406d2b879d\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm"
Nov 01 14:34:41.132851 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:41.132839    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6db204f9-b7a9-4b31-ab49-a4406d2b879d-istio-data\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm\" (UID: \"6db204f9-b7a9-4b31-ab49-a4406d2b879d\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm"
Nov 01 14:34:41.132886 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:41.132865    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7qgv\" (UniqueName: \"kubernetes.io/projected/6db204f9-b7a9-4b31-ab49-a4406d2b879d-kube-api-access-c7qgv\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm\" (UID: \"6db204f9-b7a9-4b31-ab49-a4406d2b879d\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm"
Nov 01 14:34:41.182836 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:41.182798    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm"]
Nov 01 14:34:41.234024 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:41.233989    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6db204f9-b7a9-4b31-ab49-a4406d2b879d-istio-data\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm\" (UID: \"6db204f9-b7a9-4b31-ab49-a4406d2b879d\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm"
Nov 01 14:34:41.234220 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:41.234030    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7qgv\" (UniqueName: \"kubernetes.io/projected/6db204f9-b7a9-4b31-ab49-a4406d2b879d-kube-api-access-c7qgv\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm\" (UID: \"6db204f9-b7a9-4b31-ab49-a4406d2b879d\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm"
Nov 01 14:34:41.234220 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:41.234111    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db204f9-b7a9-4b31-ab49-a4406d2b879d-credential-socket\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm\" (UID: \"6db204f9-b7a9-4b31-ab49-a4406d2b879d\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm"
Nov 01 14:34:41.234220 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:41.234135    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6db204f9-b7a9-4b31-ab49-a4406d2b879d-workload-certs\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm\" (UID: \"6db204f9-b7a9-4b31-ab49-a4406d2b879d\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm"
Nov 01 14:34:41.234397 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:41.234221    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6db204f9-b7a9-4b31-ab49-a4406d2b879d-istio-podinfo\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm\" (UID: \"6db204f9-b7a9-4b31-ab49-a4406d2b879d\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm"
Nov 01 14:34:41.234397 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:41.234269    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6db204f9-b7a9-4b31-ab49-a4406d2b879d-istio-envoy\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm\" (UID: \"6db204f9-b7a9-4b31-ab49-a4406d2b879d\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm"
Nov 01 14:34:41.234397 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:41.234298    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db204f9-b7a9-4b31-ab49-a4406d2b879d-workload-socket\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm\" (UID: \"6db204f9-b7a9-4b31-ab49-a4406d2b879d\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm"
Nov 01 14:34:41.234397 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:41.234334    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6db204f9-b7a9-4b31-ab49-a4406d2b879d-istiod-ca-cert\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm\" (UID: \"6db204f9-b7a9-4b31-ab49-a4406d2b879d\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm"
Nov 01 14:34:41.234397 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:41.234391    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6db204f9-b7a9-4b31-ab49-a4406d2b879d-istio-token\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm\" (UID: \"6db204f9-b7a9-4b31-ab49-a4406d2b879d\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm"
Nov 01 14:34:41.234656 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:41.234498    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6db204f9-b7a9-4b31-ab49-a4406d2b879d-istio-data\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm\" (UID: \"6db204f9-b7a9-4b31-ab49-a4406d2b879d\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm"
Nov 01 14:34:41.234656 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:41.234535    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db204f9-b7a9-4b31-ab49-a4406d2b879d-credential-socket\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm\" (UID: \"6db204f9-b7a9-4b31-ab49-a4406d2b879d\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm"
Nov 01 14:34:41.234656 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:41.234582    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6db204f9-b7a9-4b31-ab49-a4406d2b879d-workload-certs\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm\" (UID: \"6db204f9-b7a9-4b31-ab49-a4406d2b879d\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm"
Nov 01 14:34:41.234838 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:41.234818    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db204f9-b7a9-4b31-ab49-a4406d2b879d-workload-socket\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm\" (UID: \"6db204f9-b7a9-4b31-ab49-a4406d2b879d\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm"
Nov 01 14:34:41.235050 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:41.235033    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6db204f9-b7a9-4b31-ab49-a4406d2b879d-istiod-ca-cert\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm\" (UID: \"6db204f9-b7a9-4b31-ab49-a4406d2b879d\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm"
Nov 01 14:34:41.236520 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:41.236500    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6db204f9-b7a9-4b31-ab49-a4406d2b879d-istio-envoy\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm\" (UID: \"6db204f9-b7a9-4b31-ab49-a4406d2b879d\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm"
Nov 01 14:34:41.236670 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:41.236652    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6db204f9-b7a9-4b31-ab49-a4406d2b879d-istio-podinfo\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm\" (UID: \"6db204f9-b7a9-4b31-ab49-a4406d2b879d\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm"
Nov 01 14:34:41.287083 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:41.287048    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6db204f9-b7a9-4b31-ab49-a4406d2b879d-istio-token\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm\" (UID: \"6db204f9-b7a9-4b31-ab49-a4406d2b879d\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm"
Nov 01 14:34:41.287266 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:41.287197    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7qgv\" (UniqueName: \"kubernetes.io/projected/6db204f9-b7a9-4b31-ab49-a4406d2b879d-kube-api-access-c7qgv\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm\" (UID: \"6db204f9-b7a9-4b31-ab49-a4406d2b879d\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm"
Nov 01 14:34:41.389813 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:41.389726    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm"
Nov 01 14:34:41.814147 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:41.814110    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm"]
Nov 01 14:34:41.820640 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:34:41.820610    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6db204f9_b7a9_4b31_ab49_a4406d2b879d.slice/crio-87744fe1032d0f5747e2e2315e014584cbf7361fecb51df6dc98fcf19b4e99a8 WatchSource:0}: Error finding container 87744fe1032d0f5747e2e2315e014584cbf7361fecb51df6dc98fcf19b4e99a8: Status 404 returned error can't find the container with id 87744fe1032d0f5747e2e2315e014584cbf7361fecb51df6dc98fcf19b4e99a8
Nov 01 14:34:41.823025 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:41.822991    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 14:34:41.823102 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:41.823063    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 14:34:41.823102 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:41.823096    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 14:34:42.423708 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:42.423662    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm" event={"ID":"6db204f9-b7a9-4b31-ab49-a4406d2b879d","Type":"ContainerStarted","Data":"1722a1329d811379e417698f11af469d4516c2c279a21625d24741ba481002db"}
Nov 01 14:34:42.424129 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:42.423725    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm" event={"ID":"6db204f9-b7a9-4b31-ab49-a4406d2b879d","Type":"ContainerStarted","Data":"87744fe1032d0f5747e2e2315e014584cbf7361fecb51df6dc98fcf19b4e99a8"}
Nov 01 14:34:42.535999 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:42.535938    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm" podStartSLOduration=1.535922614 podStartE2EDuration="1.535922614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:34:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:34:42.533281541 +0000 UTC m=+1069.863144277" watchObservedRunningTime="2026-11-01 14:34:42.535922614 +0000 UTC m=+1069.865785350"
Nov 01 14:34:43.390881 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:43.390842    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm"
Nov 01 14:34:43.392407 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:43.392377    4993 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Nov 01 14:34:43.392546 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:43.392440    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm" podUID="6db204f9-b7a9-4b31-ab49-a4406d2b879d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Nov 01 14:34:44.391062 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:44.391021    4993 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Nov 01 14:34:44.391543 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:44.391148    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm" podUID="6db204f9-b7a9-4b31-ab49-a4406d2b879d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Nov 01 14:34:45.390250 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:45.390209    4993 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Nov 01 14:34:45.390461 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:45.390284    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm" podUID="6db204f9-b7a9-4b31-ab49-a4406d2b879d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Nov 01 14:34:46.391030 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:46.390992    4993 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Nov 01 14:34:46.391473 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:46.391083    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm" podUID="6db204f9-b7a9-4b31-ab49-a4406d2b879d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Nov 01 14:34:47.390741 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:47.390706    4993 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Nov 01 14:34:47.390917 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:47.390784    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm" podUID="6db204f9-b7a9-4b31-ab49-a4406d2b879d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Nov 01 14:34:48.390792 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:48.390747    4993 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Nov 01 14:34:48.391210 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:48.390826    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm" podUID="6db204f9-b7a9-4b31-ab49-a4406d2b879d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Nov 01 14:34:49.394813 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:49.394781    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm"
Nov 01 14:34:49.395289 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:49.395087    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm"
Nov 01 14:34:49.395761 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:49.395744    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm"
Nov 01 14:34:57.501051 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:34:57.501007    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:34:59.923963 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:59.923928    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n"]
Nov 01 14:34:59.928793 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:59.928769    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n"
Nov 01 14:34:59.961658 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:59.961628    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34a09630-08a4-4cf5-a72d-6931a77c3ea7-istio-token\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n\" (UID: \"34a09630-08a4-4cf5-a72d-6931a77c3ea7\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n"
Nov 01 14:34:59.961805 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:59.961681    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34a09630-08a4-4cf5-a72d-6931a77c3ea7-istio-podinfo\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n\" (UID: \"34a09630-08a4-4cf5-a72d-6931a77c3ea7\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n"
Nov 01 14:34:59.961805 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:59.961784    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34a09630-08a4-4cf5-a72d-6931a77c3ea7-workload-socket\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n\" (UID: \"34a09630-08a4-4cf5-a72d-6931a77c3ea7\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n"
Nov 01 14:34:59.961919 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:59.961817    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34a09630-08a4-4cf5-a72d-6931a77c3ea7-istio-envoy\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n\" (UID: \"34a09630-08a4-4cf5-a72d-6931a77c3ea7\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n"
Nov 01 14:34:59.961974 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:59.961950    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34a09630-08a4-4cf5-a72d-6931a77c3ea7-istio-data\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n\" (UID: \"34a09630-08a4-4cf5-a72d-6931a77c3ea7\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n"
Nov 01 14:34:59.962061 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:59.962040    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34a09630-08a4-4cf5-a72d-6931a77c3ea7-workload-certs\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n\" (UID: \"34a09630-08a4-4cf5-a72d-6931a77c3ea7\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n"
Nov 01 14:34:59.962108 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:59.962074    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq5lb\" (UniqueName: \"kubernetes.io/projected/34a09630-08a4-4cf5-a72d-6931a77c3ea7-kube-api-access-zq5lb\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n\" (UID: \"34a09630-08a4-4cf5-a72d-6931a77c3ea7\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n"
Nov 01 14:34:59.962108 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:59.962096    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34a09630-08a4-4cf5-a72d-6931a77c3ea7-credential-socket\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n\" (UID: \"34a09630-08a4-4cf5-a72d-6931a77c3ea7\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n"
Nov 01 14:34:59.962258 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:59.962242    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34a09630-08a4-4cf5-a72d-6931a77c3ea7-istiod-ca-cert\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n\" (UID: \"34a09630-08a4-4cf5-a72d-6931a77c3ea7\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n"
Nov 01 14:34:59.977426 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:34:59.977405    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-1f8e2c-openshift-default-dockercfg-cq5wp\""
Nov 01 14:35:00.034567 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:00.034533    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n"]
Nov 01 14:35:00.063276 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:00.063245    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34a09630-08a4-4cf5-a72d-6931a77c3ea7-workload-certs\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n\" (UID: \"34a09630-08a4-4cf5-a72d-6931a77c3ea7\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n"
Nov 01 14:35:00.063451 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:00.063284    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq5lb\" (UniqueName: \"kubernetes.io/projected/34a09630-08a4-4cf5-a72d-6931a77c3ea7-kube-api-access-zq5lb\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n\" (UID: \"34a09630-08a4-4cf5-a72d-6931a77c3ea7\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n"
Nov 01 14:35:00.063451 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:00.063305    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34a09630-08a4-4cf5-a72d-6931a77c3ea7-credential-socket\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n\" (UID: \"34a09630-08a4-4cf5-a72d-6931a77c3ea7\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n"
Nov 01 14:35:00.063451 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:00.063357    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34a09630-08a4-4cf5-a72d-6931a77c3ea7-istiod-ca-cert\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n\" (UID: \"34a09630-08a4-4cf5-a72d-6931a77c3ea7\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n"
Nov 01 14:35:00.063451 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:00.063392    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34a09630-08a4-4cf5-a72d-6931a77c3ea7-istio-token\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n\" (UID: \"34a09630-08a4-4cf5-a72d-6931a77c3ea7\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n"
Nov 01 14:35:00.063451 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:00.063440    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34a09630-08a4-4cf5-a72d-6931a77c3ea7-istio-podinfo\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n\" (UID: \"34a09630-08a4-4cf5-a72d-6931a77c3ea7\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n"
Nov 01 14:35:00.064128 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:00.063500    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34a09630-08a4-4cf5-a72d-6931a77c3ea7-workload-socket\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n\" (UID: \"34a09630-08a4-4cf5-a72d-6931a77c3ea7\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n"
Nov 01 14:35:00.064128 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:00.063543    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34a09630-08a4-4cf5-a72d-6931a77c3ea7-istio-envoy\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n\" (UID: \"34a09630-08a4-4cf5-a72d-6931a77c3ea7\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n"
Nov 01 14:35:00.064128 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:00.063615    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34a09630-08a4-4cf5-a72d-6931a77c3ea7-istio-data\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n\" (UID: \"34a09630-08a4-4cf5-a72d-6931a77c3ea7\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n"
Nov 01 14:35:00.064128 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:00.063737    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34a09630-08a4-4cf5-a72d-6931a77c3ea7-workload-certs\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n\" (UID: \"34a09630-08a4-4cf5-a72d-6931a77c3ea7\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n"
Nov 01 14:35:00.064128 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:00.063957    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34a09630-08a4-4cf5-a72d-6931a77c3ea7-workload-socket\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n\" (UID: \"34a09630-08a4-4cf5-a72d-6931a77c3ea7\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n"
Nov 01 14:35:00.064128 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:00.063977    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34a09630-08a4-4cf5-a72d-6931a77c3ea7-istio-data\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n\" (UID: \"34a09630-08a4-4cf5-a72d-6931a77c3ea7\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n"
Nov 01 14:35:00.064474 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:00.064132    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34a09630-08a4-4cf5-a72d-6931a77c3ea7-istiod-ca-cert\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n\" (UID: \"34a09630-08a4-4cf5-a72d-6931a77c3ea7\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n"
Nov 01 14:35:00.064474 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:00.064363    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34a09630-08a4-4cf5-a72d-6931a77c3ea7-credential-socket\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n\" (UID: \"34a09630-08a4-4cf5-a72d-6931a77c3ea7\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n"
Nov 01 14:35:00.066294 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:00.066265    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34a09630-08a4-4cf5-a72d-6931a77c3ea7-istio-envoy\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n\" (UID: \"34a09630-08a4-4cf5-a72d-6931a77c3ea7\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n"
Nov 01 14:35:00.066426 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:00.066316    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34a09630-08a4-4cf5-a72d-6931a77c3ea7-istio-podinfo\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n\" (UID: \"34a09630-08a4-4cf5-a72d-6931a77c3ea7\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n"
Nov 01 14:35:00.116329 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:00.116300    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34a09630-08a4-4cf5-a72d-6931a77c3ea7-istio-token\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n\" (UID: \"34a09630-08a4-4cf5-a72d-6931a77c3ea7\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n"
Nov 01 14:35:00.116476 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:00.116356    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq5lb\" (UniqueName: \"kubernetes.io/projected/34a09630-08a4-4cf5-a72d-6931a77c3ea7-kube-api-access-zq5lb\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n\" (UID: \"34a09630-08a4-4cf5-a72d-6931a77c3ea7\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n"
Nov 01 14:35:00.242004 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:00.241971    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n"
Nov 01 14:35:00.667258 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:00.667157    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n"]
Nov 01 14:35:00.672654 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:35:00.672620    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34a09630_08a4_4cf5_a72d_6931a77c3ea7.slice/crio-a33cf881997206a2ede0ec23e97098a22eea3c7c52cf750f0f8f85b61b18863c WatchSource:0}: Error finding container a33cf881997206a2ede0ec23e97098a22eea3c7c52cf750f0f8f85b61b18863c: Status 404 returned error can't find the container with id a33cf881997206a2ede0ec23e97098a22eea3c7c52cf750f0f8f85b61b18863c
Nov 01 14:35:00.674993 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:00.674959    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 14:35:00.675080 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:00.675023    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 14:35:00.675127 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:00.675083    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 14:35:01.539966 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:01.539917    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n" event={"ID":"34a09630-08a4-4cf5-a72d-6931a77c3ea7","Type":"ContainerStarted","Data":"0f30b63febac6fd9293dbe241feb50545820316df7880497d551d6ad37b8b726"}
Nov 01 14:35:01.540441 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:01.540027    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n" event={"ID":"34a09630-08a4-4cf5-a72d-6931a77c3ea7","Type":"ContainerStarted","Data":"a33cf881997206a2ede0ec23e97098a22eea3c7c52cf750f0f8f85b61b18863c"}
Nov 01 14:35:02.242568 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:02.242528    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n"
Nov 01 14:35:02.244475 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:02.244444    4993 patch_prober.go:29] interesting pod/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Nov 01 14:35:02.244625 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:02.244529    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n" podUID="34a09630-08a4-4cf5-a72d-6931a77c3ea7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Nov 01 14:35:03.242715 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:03.242678    4993 patch_prober.go:29] interesting pod/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Nov 01 14:35:03.243227 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:03.242756    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n" podUID="34a09630-08a4-4cf5-a72d-6931a77c3ea7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Nov 01 14:35:04.242584 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:04.242548    4993 patch_prober.go:29] interesting pod/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Nov 01 14:35:04.242765 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:04.242602    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n" podUID="34a09630-08a4-4cf5-a72d-6931a77c3ea7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Nov 01 14:35:05.242972 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:05.242933    4993 patch_prober.go:29] interesting pod/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Nov 01 14:35:05.243346 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:05.242986    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n" podUID="34a09630-08a4-4cf5-a72d-6931a77c3ea7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Nov 01 14:35:06.242853 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:06.242813    4993 patch_prober.go:29] interesting pod/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Nov 01 14:35:06.243003 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:06.242870    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n" podUID="34a09630-08a4-4cf5-a72d-6931a77c3ea7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Nov 01 14:35:07.242853 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:07.242812    4993 patch_prober.go:29] interesting pod/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Nov 01 14:35:07.243059 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:07.242871    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n" podUID="34a09630-08a4-4cf5-a72d-6931a77c3ea7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Nov 01 14:35:08.246787 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:08.246757    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n"
Nov 01 14:35:08.247263 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:08.247195    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n"
Nov 01 14:35:08.247794 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:08.247774    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n"
Nov 01 14:35:08.447881 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:08.447819    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n" podStartSLOduration=9.447802151 podStartE2EDuration="9.447802151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:34:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:35:01.747706568 +0000 UTC m=+1089.077569306" watchObservedRunningTime="2026-11-01 14:35:08.447802151 +0000 UTC m=+1095.777664888"
Nov 01 14:35:18.732334 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:18.732298    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-00a787/distinct-2-w1-00a787-kserve-56f5f9dc7d-bhxxr"]
Nov 01 14:35:18.736821 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:18.736795    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-00a787/distinct-2-w1-00a787-kserve-56f5f9dc7d-bhxxr"
Nov 01 14:35:18.814140 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:18.814101    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0180369-b893-47c4-a683-ad9b6da82101-tls-certs\") pod \"distinct-2-w1-00a787-kserve-56f5f9dc7d-bhxxr\" (UID: \"c0180369-b893-47c4-a683-ad9b6da82101\") " pod="e2e-models-e2e-worker-w1-00a787/distinct-2-w1-00a787-kserve-56f5f9dc7d-bhxxr"
Nov 01 14:35:18.814322 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:18.814250    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0180369-b893-47c4-a683-ad9b6da82101-dshm\") pod \"distinct-2-w1-00a787-kserve-56f5f9dc7d-bhxxr\" (UID: \"c0180369-b893-47c4-a683-ad9b6da82101\") " pod="e2e-models-e2e-worker-w1-00a787/distinct-2-w1-00a787-kserve-56f5f9dc7d-bhxxr"
Nov 01 14:35:18.814375 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:18.814320    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0180369-b893-47c4-a683-ad9b6da82101-model-cache\") pod \"distinct-2-w1-00a787-kserve-56f5f9dc7d-bhxxr\" (UID: \"c0180369-b893-47c4-a683-ad9b6da82101\") " pod="e2e-models-e2e-worker-w1-00a787/distinct-2-w1-00a787-kserve-56f5f9dc7d-bhxxr"
Nov 01 14:35:18.814418 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:18.814398    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0180369-b893-47c4-a683-ad9b6da82101-tmp-dir\") pod \"distinct-2-w1-00a787-kserve-56f5f9dc7d-bhxxr\" (UID: \"c0180369-b893-47c4-a683-ad9b6da82101\") " pod="e2e-models-e2e-worker-w1-00a787/distinct-2-w1-00a787-kserve-56f5f9dc7d-bhxxr"
Nov 01 14:35:18.814522 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:18.814507    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0180369-b893-47c4-a683-ad9b6da82101-home\") pod \"distinct-2-w1-00a787-kserve-56f5f9dc7d-bhxxr\" (UID: \"c0180369-b893-47c4-a683-ad9b6da82101\") " pod="e2e-models-e2e-worker-w1-00a787/distinct-2-w1-00a787-kserve-56f5f9dc7d-bhxxr"
Nov 01 14:35:18.814600 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:18.814582    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f5kx\" (UniqueName: \"kubernetes.io/projected/c0180369-b893-47c4-a683-ad9b6da82101-kube-api-access-7f5kx\") pod \"distinct-2-w1-00a787-kserve-56f5f9dc7d-bhxxr\" (UID: \"c0180369-b893-47c4-a683-ad9b6da82101\") " pod="e2e-models-e2e-worker-w1-00a787/distinct-2-w1-00a787-kserve-56f5f9dc7d-bhxxr"
Nov 01 14:35:18.831018 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:18.830978    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-00a787\"/\"distinct-2-w1-00a787-kserve-self-signed-certs\""
Nov 01 14:35:18.833644 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:18.833615    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-00a787/distinct-2-w1-00a787-kserve-56f5f9dc7d-bhxxr"]
Nov 01 14:35:18.916161 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:18.916122    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0180369-b893-47c4-a683-ad9b6da82101-home\") pod \"distinct-2-w1-00a787-kserve-56f5f9dc7d-bhxxr\" (UID: \"c0180369-b893-47c4-a683-ad9b6da82101\") " pod="e2e-models-e2e-worker-w1-00a787/distinct-2-w1-00a787-kserve-56f5f9dc7d-bhxxr"
Nov 01 14:35:18.916379 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:18.916218    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f5kx\" (UniqueName: \"kubernetes.io/projected/c0180369-b893-47c4-a683-ad9b6da82101-kube-api-access-7f5kx\") pod \"distinct-2-w1-00a787-kserve-56f5f9dc7d-bhxxr\" (UID: \"c0180369-b893-47c4-a683-ad9b6da82101\") " pod="e2e-models-e2e-worker-w1-00a787/distinct-2-w1-00a787-kserve-56f5f9dc7d-bhxxr"
Nov 01 14:35:18.916379 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:18.916301    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0180369-b893-47c4-a683-ad9b6da82101-tls-certs\") pod \"distinct-2-w1-00a787-kserve-56f5f9dc7d-bhxxr\" (UID: \"c0180369-b893-47c4-a683-ad9b6da82101\") " pod="e2e-models-e2e-worker-w1-00a787/distinct-2-w1-00a787-kserve-56f5f9dc7d-bhxxr"
Nov 01 14:35:18.916379 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:18.916363    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0180369-b893-47c4-a683-ad9b6da82101-dshm\") pod \"distinct-2-w1-00a787-kserve-56f5f9dc7d-bhxxr\" (UID: \"c0180369-b893-47c4-a683-ad9b6da82101\") " pod="e2e-models-e2e-worker-w1-00a787/distinct-2-w1-00a787-kserve-56f5f9dc7d-bhxxr"
Nov 01 14:35:18.916550 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:18.916425    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0180369-b893-47c4-a683-ad9b6da82101-model-cache\") pod \"distinct-2-w1-00a787-kserve-56f5f9dc7d-bhxxr\" (UID: \"c0180369-b893-47c4-a683-ad9b6da82101\") " pod="e2e-models-e2e-worker-w1-00a787/distinct-2-w1-00a787-kserve-56f5f9dc7d-bhxxr"
Nov 01 14:35:18.916550 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:18.916465    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0180369-b893-47c4-a683-ad9b6da82101-tmp-dir\") pod \"distinct-2-w1-00a787-kserve-56f5f9dc7d-bhxxr\" (UID: \"c0180369-b893-47c4-a683-ad9b6da82101\") " pod="e2e-models-e2e-worker-w1-00a787/distinct-2-w1-00a787-kserve-56f5f9dc7d-bhxxr"
Nov 01 14:35:18.916660 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:18.916610    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0180369-b893-47c4-a683-ad9b6da82101-home\") pod \"distinct-2-w1-00a787-kserve-56f5f9dc7d-bhxxr\" (UID: \"c0180369-b893-47c4-a683-ad9b6da82101\") " pod="e2e-models-e2e-worker-w1-00a787/distinct-2-w1-00a787-kserve-56f5f9dc7d-bhxxr"
Nov 01 14:35:18.916848 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:18.916821    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0180369-b893-47c4-a683-ad9b6da82101-model-cache\") pod \"distinct-2-w1-00a787-kserve-56f5f9dc7d-bhxxr\" (UID: \"c0180369-b893-47c4-a683-ad9b6da82101\") " pod="e2e-models-e2e-worker-w1-00a787/distinct-2-w1-00a787-kserve-56f5f9dc7d-bhxxr"
Nov 01 14:35:18.916916 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:18.916833    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0180369-b893-47c4-a683-ad9b6da82101-tmp-dir\") pod \"distinct-2-w1-00a787-kserve-56f5f9dc7d-bhxxr\" (UID: \"c0180369-b893-47c4-a683-ad9b6da82101\") " pod="e2e-models-e2e-worker-w1-00a787/distinct-2-w1-00a787-kserve-56f5f9dc7d-bhxxr"
Nov 01 14:35:18.918686 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:18.918658    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0180369-b893-47c4-a683-ad9b6da82101-dshm\") pod \"distinct-2-w1-00a787-kserve-56f5f9dc7d-bhxxr\" (UID: \"c0180369-b893-47c4-a683-ad9b6da82101\") " pod="e2e-models-e2e-worker-w1-00a787/distinct-2-w1-00a787-kserve-56f5f9dc7d-bhxxr"
Nov 01 14:35:18.918943 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:18.918923    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0180369-b893-47c4-a683-ad9b6da82101-tls-certs\") pod \"distinct-2-w1-00a787-kserve-56f5f9dc7d-bhxxr\" (UID: \"c0180369-b893-47c4-a683-ad9b6da82101\") " pod="e2e-models-e2e-worker-w1-00a787/distinct-2-w1-00a787-kserve-56f5f9dc7d-bhxxr"
Nov 01 14:35:18.969339 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:18.969305    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f5kx\" (UniqueName: \"kubernetes.io/projected/c0180369-b893-47c4-a683-ad9b6da82101-kube-api-access-7f5kx\") pod \"distinct-2-w1-00a787-kserve-56f5f9dc7d-bhxxr\" (UID: \"c0180369-b893-47c4-a683-ad9b6da82101\") " pod="e2e-models-e2e-worker-w1-00a787/distinct-2-w1-00a787-kserve-56f5f9dc7d-bhxxr"
Nov 01 14:35:19.049391 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:19.049359    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-00a787/distinct-2-w1-00a787-kserve-56f5f9dc7d-bhxxr"
Nov 01 14:35:19.472459 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:19.472384    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-00a787/distinct-2-w1-00a787-kserve-56f5f9dc7d-bhxxr"]
Nov 01 14:35:19.478291 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:35:19.478262    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc0180369_b893_47c4_a683_ad9b6da82101.slice/crio-ba76b3d8f4fe38d5ecd52eff5b9d5014fd38585ed71464754c9649c4703379b7 WatchSource:0}: Error finding container ba76b3d8f4fe38d5ecd52eff5b9d5014fd38585ed71464754c9649c4703379b7: Status 404 returned error can't find the container with id ba76b3d8f4fe38d5ecd52eff5b9d5014fd38585ed71464754c9649c4703379b7
Nov 01 14:35:19.682704 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:19.682645    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-00a787/distinct-2-w1-00a787-kserve-56f5f9dc7d-bhxxr" event={"ID":"c0180369-b893-47c4-a683-ad9b6da82101","Type":"ContainerStarted","Data":"97665b126c6408fa7d4fcd310e7991e451940b112dac9caa4c2a1763f6043a73"}
Nov 01 14:35:19.682894 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:19.682719    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-00a787/distinct-2-w1-00a787-kserve-56f5f9dc7d-bhxxr" event={"ID":"c0180369-b893-47c4-a683-ad9b6da82101","Type":"ContainerStarted","Data":"ba76b3d8f4fe38d5ecd52eff5b9d5014fd38585ed71464754c9649c4703379b7"}
Nov 01 14:35:19.790607 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:19.790558    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-00a787/distinct-2-w1-00a787-kserve-56f5f9dc7d-bhxxr" podStartSLOduration=1.790544529 podStartE2EDuration="1.790544529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:35:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:35:19.788074719 +0000 UTC m=+1107.117937469" watchObservedRunningTime="2026-11-01 14:35:19.790544529 +0000 UTC m=+1107.120407268"
Nov 01 14:35:20.050585 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:20.050511    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-00a787/distinct-2-w1-00a787-kserve-56f5f9dc7d-bhxxr"
Nov 01 14:35:27.500435 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:35:27.500399    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:35:29.056433 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:29.056395    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-00a787/distinct-2-w1-00a787-kserve-56f5f9dc7d-bhxxr"
Nov 01 14:35:29.078636 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:29.078597    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-00a787/distinct-2-w1-00a787-kserve-56f5f9dc7d-bhxxr"
Nov 01 14:35:29.769699 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:29.769666    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-00a787/distinct-2-w1-00a787-kserve-56f5f9dc7d-bhxxr"
Nov 01 14:35:56.664030 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:56.663998    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx"]
Nov 01 14:35:56.668319 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:56.668291    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx"
Nov 01 14:35:56.817003 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:56.816960    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20e06dcd-6062-4dfa-9294-388c801fa8f7-istio-data\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx\" (UID: \"20e06dcd-6062-4dfa-9294-388c801fa8f7\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx"
Nov 01 14:35:56.817211 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:56.817073    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p56r\" (UniqueName: \"kubernetes.io/projected/20e06dcd-6062-4dfa-9294-388c801fa8f7-kube-api-access-6p56r\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx\" (UID: \"20e06dcd-6062-4dfa-9294-388c801fa8f7\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx"
Nov 01 14:35:56.817267 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:56.817213    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20e06dcd-6062-4dfa-9294-388c801fa8f7-istiod-ca-cert\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx\" (UID: \"20e06dcd-6062-4dfa-9294-388c801fa8f7\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx"
Nov 01 14:35:56.817307 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:56.817269    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20e06dcd-6062-4dfa-9294-388c801fa8f7-istio-envoy\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx\" (UID: \"20e06dcd-6062-4dfa-9294-388c801fa8f7\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx"
Nov 01 14:35:56.817356 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:56.817340    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20e06dcd-6062-4dfa-9294-388c801fa8f7-istio-podinfo\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx\" (UID: \"20e06dcd-6062-4dfa-9294-388c801fa8f7\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx"
Nov 01 14:35:56.817392 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:56.817377    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20e06dcd-6062-4dfa-9294-388c801fa8f7-workload-socket\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx\" (UID: \"20e06dcd-6062-4dfa-9294-388c801fa8f7\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx"
Nov 01 14:35:56.817508 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:56.817488    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20e06dcd-6062-4dfa-9294-388c801fa8f7-istio-token\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx\" (UID: \"20e06dcd-6062-4dfa-9294-388c801fa8f7\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx"
Nov 01 14:35:56.817584 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:56.817523    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20e06dcd-6062-4dfa-9294-388c801fa8f7-workload-certs\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx\" (UID: \"20e06dcd-6062-4dfa-9294-388c801fa8f7\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx"
Nov 01 14:35:56.817584 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:56.817565    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20e06dcd-6062-4dfa-9294-388c801fa8f7-credential-socket\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx\" (UID: \"20e06dcd-6062-4dfa-9294-388c801fa8f7\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx"
Nov 01 14:35:56.880611 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:56.880566    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx"]
Nov 01 14:35:56.918495 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:56.918406    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20e06dcd-6062-4dfa-9294-388c801fa8f7-istio-data\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx\" (UID: \"20e06dcd-6062-4dfa-9294-388c801fa8f7\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx"
Nov 01 14:35:56.918495 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:56.918465    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p56r\" (UniqueName: \"kubernetes.io/projected/20e06dcd-6062-4dfa-9294-388c801fa8f7-kube-api-access-6p56r\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx\" (UID: \"20e06dcd-6062-4dfa-9294-388c801fa8f7\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx"
Nov 01 14:35:56.918735 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:56.918532    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20e06dcd-6062-4dfa-9294-388c801fa8f7-istiod-ca-cert\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx\" (UID: \"20e06dcd-6062-4dfa-9294-388c801fa8f7\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx"
Nov 01 14:35:56.918735 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:56.918592    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20e06dcd-6062-4dfa-9294-388c801fa8f7-istio-envoy\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx\" (UID: \"20e06dcd-6062-4dfa-9294-388c801fa8f7\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx"
Nov 01 14:35:56.918735 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:56.918628    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20e06dcd-6062-4dfa-9294-388c801fa8f7-istio-podinfo\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx\" (UID: \"20e06dcd-6062-4dfa-9294-388c801fa8f7\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx"
Nov 01 14:35:56.918735 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:56.918668    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20e06dcd-6062-4dfa-9294-388c801fa8f7-workload-socket\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx\" (UID: \"20e06dcd-6062-4dfa-9294-388c801fa8f7\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx"
Nov 01 14:35:56.918949 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:56.918788    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20e06dcd-6062-4dfa-9294-388c801fa8f7-istio-token\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx\" (UID: \"20e06dcd-6062-4dfa-9294-388c801fa8f7\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx"
Nov 01 14:35:56.918949 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:56.918817    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20e06dcd-6062-4dfa-9294-388c801fa8f7-workload-certs\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx\" (UID: \"20e06dcd-6062-4dfa-9294-388c801fa8f7\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx"
Nov 01 14:35:56.918949 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:56.918856    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20e06dcd-6062-4dfa-9294-388c801fa8f7-credential-socket\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx\" (UID: \"20e06dcd-6062-4dfa-9294-388c801fa8f7\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx"
Nov 01 14:35:56.918949 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:56.918882    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20e06dcd-6062-4dfa-9294-388c801fa8f7-istio-data\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx\" (UID: \"20e06dcd-6062-4dfa-9294-388c801fa8f7\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx"
Nov 01 14:35:56.919248 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:56.919211    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20e06dcd-6062-4dfa-9294-388c801fa8f7-credential-socket\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx\" (UID: \"20e06dcd-6062-4dfa-9294-388c801fa8f7\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx"
Nov 01 14:35:56.919378 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:56.919350    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20e06dcd-6062-4dfa-9294-388c801fa8f7-istiod-ca-cert\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx\" (UID: \"20e06dcd-6062-4dfa-9294-388c801fa8f7\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx"
Nov 01 14:35:56.919502 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:56.919479    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20e06dcd-6062-4dfa-9294-388c801fa8f7-workload-certs\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx\" (UID: \"20e06dcd-6062-4dfa-9294-388c801fa8f7\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx"
Nov 01 14:35:56.919677 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:56.919517    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20e06dcd-6062-4dfa-9294-388c801fa8f7-workload-socket\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx\" (UID: \"20e06dcd-6062-4dfa-9294-388c801fa8f7\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx"
Nov 01 14:35:56.921551 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:56.921528    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20e06dcd-6062-4dfa-9294-388c801fa8f7-istio-podinfo\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx\" (UID: \"20e06dcd-6062-4dfa-9294-388c801fa8f7\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx"
Nov 01 14:35:56.921657 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:56.921633    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20e06dcd-6062-4dfa-9294-388c801fa8f7-istio-envoy\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx\" (UID: \"20e06dcd-6062-4dfa-9294-388c801fa8f7\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx"
Nov 01 14:35:56.971954 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:56.971916    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20e06dcd-6062-4dfa-9294-388c801fa8f7-istio-token\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx\" (UID: \"20e06dcd-6062-4dfa-9294-388c801fa8f7\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx"
Nov 01 14:35:56.971954 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:56.971942    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p56r\" (UniqueName: \"kubernetes.io/projected/20e06dcd-6062-4dfa-9294-388c801fa8f7-kube-api-access-6p56r\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx\" (UID: \"20e06dcd-6062-4dfa-9294-388c801fa8f7\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx"
Nov 01 14:35:56.981940 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:56.981896    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx"
Nov 01 14:35:57.412129 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:57.412091    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx"]
Nov 01 14:35:57.416243 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:35:57.416212    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20e06dcd_6062_4dfa_9294_388c801fa8f7.slice/crio-04a9ec64ad8e48a40b22e036cf484f7c6651dca9fabb8f01edd2a578d0b4235b WatchSource:0}: Error finding container 04a9ec64ad8e48a40b22e036cf484f7c6651dca9fabb8f01edd2a578d0b4235b: Status 404 returned error can't find the container with id 04a9ec64ad8e48a40b22e036cf484f7c6651dca9fabb8f01edd2a578d0b4235b
Nov 01 14:35:57.418668 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:57.418629    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 14:35:57.418767 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:57.418696    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 14:35:57.418767 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:57.418726    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 14:35:57.503394 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:35:57.503362    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:35:57.947283 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:57.947225    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx" event={"ID":"20e06dcd-6062-4dfa-9294-388c801fa8f7","Type":"ContainerStarted","Data":"21dd3ee3e960a36e2c49c599548586c522b594d17eb4e857d272c7144291c3d1"}
Nov 01 14:35:57.947756 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:57.947301    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx" event={"ID":"20e06dcd-6062-4dfa-9294-388c801fa8f7","Type":"ContainerStarted","Data":"04a9ec64ad8e48a40b22e036cf484f7c6651dca9fabb8f01edd2a578d0b4235b"}
Nov 01 14:35:57.982391 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:57.982348    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx"
Nov 01 14:35:58.078747 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:58.078666    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx" podStartSLOduration=2.078647882 podStartE2EDuration="2.078647882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:35:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:35:58.077500975 +0000 UTC m=+1145.407363709" watchObservedRunningTime="2026-11-01 14:35:58.078647882 +0000 UTC m=+1145.408510622"
Nov 01 14:35:58.984318 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:58.984281    4993 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Nov 01 14:35:58.984813 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:58.984338    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx" podUID="20e06dcd-6062-4dfa-9294-388c801fa8f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Nov 01 14:35:59.989548 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:59.989509    4993 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Nov 01 14:35:59.990066 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:35:59.989604    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx" podUID="20e06dcd-6062-4dfa-9294-388c801fa8f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Nov 01 14:36:00.983054 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:00.983013    4993 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Nov 01 14:36:00.983258 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:00.983082    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx" podUID="20e06dcd-6062-4dfa-9294-388c801fa8f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Nov 01 14:36:01.983629 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:01.983593    4993 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Nov 01 14:36:01.984031 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:01.983663    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx" podUID="20e06dcd-6062-4dfa-9294-388c801fa8f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Nov 01 14:36:02.983473 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:02.983434    4993 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Nov 01 14:36:02.983659 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:02.983494    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx" podUID="20e06dcd-6062-4dfa-9294-388c801fa8f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Nov 01 14:36:03.983514 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:03.983474    4993 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Nov 01 14:36:03.983737 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:03.983531    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx" podUID="20e06dcd-6062-4dfa-9294-388c801fa8f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Nov 01 14:36:04.987655 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:04.987622    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx"
Nov 01 14:36:04.988120 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:04.987953    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx"
Nov 01 14:36:04.988760 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:04.988743    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx"
Nov 01 14:36:18.060849 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:18.060806    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1f8e2c/e2e-conflict-a-a12d4d90-kserve-556bbf6448-7r5xm"]
Nov 01 14:36:27.504344 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:36:27.504305    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:36:33.089884 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:33.089836    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-1f8e2c/e2e-conflict-a-a12d4d90-kserve-556bbf6448-7r5xm" podUID="643af801-f12c-4c20-b4a6-246733fbb2b0" containerName="main" containerID="cri-o://ca766737225c781ff59293c0f6ecb973aab0a4a5e7b9a3d8a8c9806e291e5d95" gracePeriod=45
Nov 01 14:36:33.502619 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:33.502577    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1f8e2c/e2e-conflict-a-a12d4d90-kserve-556bbf6448-7r5xm"
Nov 01 14:36:33.592199 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:33.588032    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/643af801-f12c-4c20-b4a6-246733fbb2b0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/643af801-f12c-4c20-b4a6-246733fbb2b0-tmp-dir\") pod \"643af801-f12c-4c20-b4a6-246733fbb2b0\" (UID: \"643af801-f12c-4c20-b4a6-246733fbb2b0\") "
Nov 01 14:36:33.592199 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:33.588146    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/643af801-f12c-4c20-b4a6-246733fbb2b0-kube-api-access-7r72x\" (UniqueName: \"kubernetes.io/projected/643af801-f12c-4c20-b4a6-246733fbb2b0-kube-api-access-7r72x\") pod \"643af801-f12c-4c20-b4a6-246733fbb2b0\" (UID: \"643af801-f12c-4c20-b4a6-246733fbb2b0\") "
Nov 01 14:36:33.592199 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:33.588248    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/643af801-f12c-4c20-b4a6-246733fbb2b0-home\" (UniqueName: \"kubernetes.io/empty-dir/643af801-f12c-4c20-b4a6-246733fbb2b0-home\") pod \"643af801-f12c-4c20-b4a6-246733fbb2b0\" (UID: \"643af801-f12c-4c20-b4a6-246733fbb2b0\") "
Nov 01 14:36:33.592199 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:33.588309    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/643af801-f12c-4c20-b4a6-246733fbb2b0-tls-certs\" (UniqueName: \"kubernetes.io/secret/643af801-f12c-4c20-b4a6-246733fbb2b0-tls-certs\") pod \"643af801-f12c-4c20-b4a6-246733fbb2b0\" (UID: \"643af801-f12c-4c20-b4a6-246733fbb2b0\") "
Nov 01 14:36:33.592199 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:33.588357    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/643af801-f12c-4c20-b4a6-246733fbb2b0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/643af801-f12c-4c20-b4a6-246733fbb2b0-model-cache\") pod \"643af801-f12c-4c20-b4a6-246733fbb2b0\" (UID: \"643af801-f12c-4c20-b4a6-246733fbb2b0\") "
Nov 01 14:36:33.592199 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:33.588446    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/643af801-f12c-4c20-b4a6-246733fbb2b0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/643af801-f12c-4c20-b4a6-246733fbb2b0-dshm\") pod \"643af801-f12c-4c20-b4a6-246733fbb2b0\" (UID: \"643af801-f12c-4c20-b4a6-246733fbb2b0\") "
Nov 01 14:36:33.592199 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:33.589651    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/643af801-f12c-4c20-b4a6-246733fbb2b0-tmp-dir" pod "643af801-f12c-4c20-b4a6-246733fbb2b0" (UID: "643af801-f12c-4c20-b4a6-246733fbb2b0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:36:33.592199 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:33.590263    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/643af801-f12c-4c20-b4a6-246733fbb2b0-home" pod "643af801-f12c-4c20-b4a6-246733fbb2b0" (UID: "643af801-f12c-4c20-b4a6-246733fbb2b0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:36:33.592199 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:33.590952    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/643af801-f12c-4c20-b4a6-246733fbb2b0-model-cache" pod "643af801-f12c-4c20-b4a6-246733fbb2b0" (UID: "643af801-f12c-4c20-b4a6-246733fbb2b0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:36:33.606291 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:33.606233    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/643af801-f12c-4c20-b4a6-246733fbb2b0-kube-api-access-7r72x" pod "643af801-f12c-4c20-b4a6-246733fbb2b0" (UID: "643af801-f12c-4c20-b4a6-246733fbb2b0"). InnerVolumeSpecName "kube-api-access-7r72x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:36:33.614496 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:33.614427    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/643af801-f12c-4c20-b4a6-246733fbb2b0-dshm" pod "643af801-f12c-4c20-b4a6-246733fbb2b0" (UID: "643af801-f12c-4c20-b4a6-246733fbb2b0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:36:33.625691 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:33.624939    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/643af801-f12c-4c20-b4a6-246733fbb2b0-tls-certs" pod "643af801-f12c-4c20-b4a6-246733fbb2b0" (UID: "643af801-f12c-4c20-b4a6-246733fbb2b0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:36:33.694189 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:33.693846    4993 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/643af801-f12c-4c20-b4a6-246733fbb2b0-dshm\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:36:33.694189 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:33.693882    4993 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/643af801-f12c-4c20-b4a6-246733fbb2b0-tmp-dir\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:36:33.694189 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:33.693899    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7r72x\" (UniqueName: \"kubernetes.io/projected/643af801-f12c-4c20-b4a6-246733fbb2b0-kube-api-access-7r72x\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:36:33.694189 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:33.693916    4993 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/643af801-f12c-4c20-b4a6-246733fbb2b0-home\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:36:33.694189 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:33.693932    4993 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/643af801-f12c-4c20-b4a6-246733fbb2b0-tls-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:36:33.694189 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:33.693945    4993 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/643af801-f12c-4c20-b4a6-246733fbb2b0-model-cache\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:36:34.230193 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:34.229953    4993 generic.go:356] "Generic (PLEG): container finished" podID="643af801-f12c-4c20-b4a6-246733fbb2b0" containerID="ca766737225c781ff59293c0f6ecb973aab0a4a5e7b9a3d8a8c9806e291e5d95" exitCode=0
Nov 01 14:36:34.230193 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:34.230116    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1f8e2c/e2e-conflict-a-a12d4d90-kserve-556bbf6448-7r5xm" event={"ID":"643af801-f12c-4c20-b4a6-246733fbb2b0","Type":"ContainerDied","Data":"ca766737225c781ff59293c0f6ecb973aab0a4a5e7b9a3d8a8c9806e291e5d95"}
Nov 01 14:36:34.230193 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:34.230195    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1f8e2c/e2e-conflict-a-a12d4d90-kserve-556bbf6448-7r5xm" event={"ID":"643af801-f12c-4c20-b4a6-246733fbb2b0","Type":"ContainerDied","Data":"55f7312addabace942143a167b5821b349ba58da56e33364befa2963dd925db5"}
Nov 01 14:36:34.230854 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:34.230241    4993 scope.go:122] "RemoveContainer" containerID="ca766737225c781ff59293c0f6ecb973aab0a4a5e7b9a3d8a8c9806e291e5d95"
Nov 01 14:36:34.230854 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:34.230444    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1f8e2c/e2e-conflict-a-a12d4d90-kserve-556bbf6448-7r5xm"
Nov 01 14:36:34.245337 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:34.245308    4993 scope.go:122] "RemoveContainer" containerID="ca766737225c781ff59293c0f6ecb973aab0a4a5e7b9a3d8a8c9806e291e5d95"
Nov 01 14:36:34.245756 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:36:34.245736    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca766737225c781ff59293c0f6ecb973aab0a4a5e7b9a3d8a8c9806e291e5d95\": container with ID starting with ca766737225c781ff59293c0f6ecb973aab0a4a5e7b9a3d8a8c9806e291e5d95 not found: ID does not exist" containerID="ca766737225c781ff59293c0f6ecb973aab0a4a5e7b9a3d8a8c9806e291e5d95"
Nov 01 14:36:34.245806 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:34.245768    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca766737225c781ff59293c0f6ecb973aab0a4a5e7b9a3d8a8c9806e291e5d95"} err="failed to get container status \"ca766737225c781ff59293c0f6ecb973aab0a4a5e7b9a3d8a8c9806e291e5d95\": rpc error: code = NotFound desc = could not find container \"ca766737225c781ff59293c0f6ecb973aab0a4a5e7b9a3d8a8c9806e291e5d95\": container with ID starting with ca766737225c781ff59293c0f6ecb973aab0a4a5e7b9a3d8a8c9806e291e5d95 not found: ID does not exist"
Nov 01 14:36:34.425957 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:34.413550    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1f8e2c/e2e-conflict-a-a12d4d90-kserve-556bbf6448-7r5xm"]
Nov 01 14:36:34.439366 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:34.439322    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-1f8e2c/e2e-conflict-a-a12d4d90-kserve-556bbf6448-7r5xm"]
Nov 01 14:36:34.827664 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:34.827617    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="643af801-f12c-4c20-b4a6-246733fbb2b0" path="/var/lib/kubelet/pods/643af801-f12c-4c20-b4a6-246733fbb2b0/volumes"
Nov 01 14:36:35.923211 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:35.923151    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l"]
Nov 01 14:36:35.924051 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:35.924023    4993 cpu_manager.go:414] "containerMap: removing container" podUID="643af801-f12c-4c20-b4a6-246733fbb2b0" containerName="main"
Nov 01 14:36:35.924051 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:35.924049    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="643af801-f12c-4c20-b4a6-246733fbb2b0" containerName="main"
Nov 01 14:36:35.924275 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:35.924259    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="643af801-f12c-4c20-b4a6-246733fbb2b0" containerName="main"
Nov 01 14:36:35.929491 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:35.929470    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l"
Nov 01 14:36:36.031025 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:36.030988    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l"]
Nov 01 14:36:36.041091 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:36.041050    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bb22946-1d01-4745-a1e3-519f213c333c-istio-envoy\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l\" (UID: \"5bb22946-1d01-4745-a1e3-519f213c333c\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l"
Nov 01 14:36:36.041315 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:36.041102    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bb22946-1d01-4745-a1e3-519f213c333c-istio-token\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l\" (UID: \"5bb22946-1d01-4745-a1e3-519f213c333c\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l"
Nov 01 14:36:36.041315 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:36.041234    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb22946-1d01-4745-a1e3-519f213c333c-workload-socket\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l\" (UID: \"5bb22946-1d01-4745-a1e3-519f213c333c\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l"
Nov 01 14:36:36.041315 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:36.041263    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bb22946-1d01-4745-a1e3-519f213c333c-istio-data\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l\" (UID: \"5bb22946-1d01-4745-a1e3-519f213c333c\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l"
Nov 01 14:36:36.041315 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:36.041296    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bb22946-1d01-4745-a1e3-519f213c333c-workload-certs\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l\" (UID: \"5bb22946-1d01-4745-a1e3-519f213c333c\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l"
Nov 01 14:36:36.041545 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:36.041320    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bb22946-1d01-4745-a1e3-519f213c333c-istiod-ca-cert\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l\" (UID: \"5bb22946-1d01-4745-a1e3-519f213c333c\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l"
Nov 01 14:36:36.041545 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:36.041350    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bb22946-1d01-4745-a1e3-519f213c333c-istio-podinfo\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l\" (UID: \"5bb22946-1d01-4745-a1e3-519f213c333c\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l"
Nov 01 14:36:36.041545 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:36.041393    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb22946-1d01-4745-a1e3-519f213c333c-credential-socket\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l\" (UID: \"5bb22946-1d01-4745-a1e3-519f213c333c\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l"
Nov 01 14:36:36.041545 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:36.041435    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4lrc\" (UniqueName: \"kubernetes.io/projected/5bb22946-1d01-4745-a1e3-519f213c333c-kube-api-access-m4lrc\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l\" (UID: \"5bb22946-1d01-4745-a1e3-519f213c333c\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l"
Nov 01 14:36:36.142759 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:36.142723    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bb22946-1d01-4745-a1e3-519f213c333c-istio-envoy\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l\" (UID: \"5bb22946-1d01-4745-a1e3-519f213c333c\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l"
Nov 01 14:36:36.142759 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:36.142763    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bb22946-1d01-4745-a1e3-519f213c333c-istio-token\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l\" (UID: \"5bb22946-1d01-4745-a1e3-519f213c333c\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l"
Nov 01 14:36:36.142993 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:36.142803    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb22946-1d01-4745-a1e3-519f213c333c-workload-socket\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l\" (UID: \"5bb22946-1d01-4745-a1e3-519f213c333c\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l"
Nov 01 14:36:36.142993 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:36.142822    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bb22946-1d01-4745-a1e3-519f213c333c-istio-data\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l\" (UID: \"5bb22946-1d01-4745-a1e3-519f213c333c\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l"
Nov 01 14:36:36.142993 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:36.142847    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bb22946-1d01-4745-a1e3-519f213c333c-workload-certs\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l\" (UID: \"5bb22946-1d01-4745-a1e3-519f213c333c\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l"
Nov 01 14:36:36.142993 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:36.142958    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bb22946-1d01-4745-a1e3-519f213c333c-istiod-ca-cert\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l\" (UID: \"5bb22946-1d01-4745-a1e3-519f213c333c\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l"
Nov 01 14:36:36.143218 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:36.143004    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bb22946-1d01-4745-a1e3-519f213c333c-istio-podinfo\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l\" (UID: \"5bb22946-1d01-4745-a1e3-519f213c333c\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l"
Nov 01 14:36:36.143218 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:36.143068    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb22946-1d01-4745-a1e3-519f213c333c-credential-socket\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l\" (UID: \"5bb22946-1d01-4745-a1e3-519f213c333c\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l"
Nov 01 14:36:36.143218 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:36.143128    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4lrc\" (UniqueName: \"kubernetes.io/projected/5bb22946-1d01-4745-a1e3-519f213c333c-kube-api-access-m4lrc\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l\" (UID: \"5bb22946-1d01-4745-a1e3-519f213c333c\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l"
Nov 01 14:36:36.143458 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:36.143431    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bb22946-1d01-4745-a1e3-519f213c333c-istio-data\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l\" (UID: \"5bb22946-1d01-4745-a1e3-519f213c333c\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l"
Nov 01 14:36:36.143700 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:36.143669    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb22946-1d01-4745-a1e3-519f213c333c-workload-socket\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l\" (UID: \"5bb22946-1d01-4745-a1e3-519f213c333c\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l"
Nov 01 14:36:36.143889 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:36.143865    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bb22946-1d01-4745-a1e3-519f213c333c-workload-certs\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l\" (UID: \"5bb22946-1d01-4745-a1e3-519f213c333c\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l"
Nov 01 14:36:36.144000 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:36.143891    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bb22946-1d01-4745-a1e3-519f213c333c-istiod-ca-cert\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l\" (UID: \"5bb22946-1d01-4745-a1e3-519f213c333c\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l"
Nov 01 14:36:36.144063 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:36.144017    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb22946-1d01-4745-a1e3-519f213c333c-credential-socket\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l\" (UID: \"5bb22946-1d01-4745-a1e3-519f213c333c\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l"
Nov 01 14:36:36.145348 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:36.145326    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bb22946-1d01-4745-a1e3-519f213c333c-istio-envoy\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l\" (UID: \"5bb22946-1d01-4745-a1e3-519f213c333c\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l"
Nov 01 14:36:36.146063 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:36.146045    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bb22946-1d01-4745-a1e3-519f213c333c-istio-podinfo\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l\" (UID: \"5bb22946-1d01-4745-a1e3-519f213c333c\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l"
Nov 01 14:36:36.196831 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:36.196767    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bb22946-1d01-4745-a1e3-519f213c333c-istio-token\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l\" (UID: \"5bb22946-1d01-4745-a1e3-519f213c333c\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l"
Nov 01 14:36:36.196962 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:36.196846    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4lrc\" (UniqueName: \"kubernetes.io/projected/5bb22946-1d01-4745-a1e3-519f213c333c-kube-api-access-m4lrc\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l\" (UID: \"5bb22946-1d01-4745-a1e3-519f213c333c\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l"
Nov 01 14:36:36.250324 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:36.250292    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l"
Nov 01 14:36:36.681454 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:36.681413    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l"]
Nov 01 14:36:36.688340 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:36:36.688307    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bb22946_1d01_4745_a1e3_519f213c333c.slice/crio-d15d8e43191f7e895bce675847ce9cf3f7d2ca727e73882d0a5ef44d24c46169 WatchSource:0}: Error finding container d15d8e43191f7e895bce675847ce9cf3f7d2ca727e73882d0a5ef44d24c46169: Status 404 returned error can't find the container with id d15d8e43191f7e895bce675847ce9cf3f7d2ca727e73882d0a5ef44d24c46169
Nov 01 14:36:36.690772 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:36.690740    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 14:36:36.690869 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:36.690811    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 14:36:36.690869 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:36.690849    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 14:36:37.249744 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:37.249707    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l" event={"ID":"5bb22946-1d01-4745-a1e3-519f213c333c","Type":"ContainerStarted","Data":"92579e6a99b638be7a0253edf0a8aa2f462cd552ff72624764336db62129aacc"}
Nov 01 14:36:37.250155 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:37.249763    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l" event={"ID":"5bb22946-1d01-4745-a1e3-519f213c333c","Type":"ContainerStarted","Data":"d15d8e43191f7e895bce675847ce9cf3f7d2ca727e73882d0a5ef44d24c46169"}
Nov 01 14:36:37.251007 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:37.250988    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l"
Nov 01 14:36:37.272192 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:37.253082    4993 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Nov 01 14:36:37.272192 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:37.253132    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l" podUID="5bb22946-1d01-4745-a1e3-519f213c333c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Nov 01 14:36:37.383126 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:37.383055    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l" podStartSLOduration=2.38303485 podStartE2EDuration="2.38303485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:36:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:36:37.361335039 +0000 UTC m=+1184.691197776" watchObservedRunningTime="2026-11-01 14:36:37.38303485 +0000 UTC m=+1184.712897588"
Nov 01 14:36:38.250744 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:38.250708    4993 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Nov 01 14:36:38.251183 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:38.250768    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l" podUID="5bb22946-1d01-4745-a1e3-519f213c333c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Nov 01 14:36:39.250704 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:39.250667    4993 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Nov 01 14:36:39.250883 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:39.250726    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l" podUID="5bb22946-1d01-4745-a1e3-519f213c333c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Nov 01 14:36:40.268192 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:40.263202    4993 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Nov 01 14:36:40.268192 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:40.263263    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l" podUID="5bb22946-1d01-4745-a1e3-519f213c333c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Nov 01 14:36:41.251183 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:41.251128    4993 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Nov 01 14:36:41.251392 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:41.251233    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l" podUID="5bb22946-1d01-4745-a1e3-519f213c333c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Nov 01 14:36:42.007995 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:42.007954    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-drj66"]
Nov 01 14:36:42.011894 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:42.011869    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-drj66"
Nov 01 14:36:42.026976 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:42.026949    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de2b3fa1-8060-4944-b43a-1d9501fa8cda-istio-envoy\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-drj66\" (UID: \"de2b3fa1-8060-4944-b43a-1d9501fa8cda\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-drj66"
Nov 01 14:36:42.027101 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:42.026991    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de2b3fa1-8060-4944-b43a-1d9501fa8cda-istiod-ca-cert\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-drj66\" (UID: \"de2b3fa1-8060-4944-b43a-1d9501fa8cda\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-drj66"
Nov 01 14:36:42.027101 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:42.027061    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de2b3fa1-8060-4944-b43a-1d9501fa8cda-istio-podinfo\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-drj66\" (UID: \"de2b3fa1-8060-4944-b43a-1d9501fa8cda\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-drj66"
Nov 01 14:36:42.027246 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:42.027107    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de2b3fa1-8060-4944-b43a-1d9501fa8cda-istio-token\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-drj66\" (UID: \"de2b3fa1-8060-4944-b43a-1d9501fa8cda\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-drj66"
Nov 01 14:36:42.027308 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:42.027244    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de2b3fa1-8060-4944-b43a-1d9501fa8cda-workload-certs\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-drj66\" (UID: \"de2b3fa1-8060-4944-b43a-1d9501fa8cda\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-drj66"
Nov 01 14:36:42.027363 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:42.027324    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f8qd\" (UniqueName: \"kubernetes.io/projected/de2b3fa1-8060-4944-b43a-1d9501fa8cda-kube-api-access-2f8qd\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-drj66\" (UID: \"de2b3fa1-8060-4944-b43a-1d9501fa8cda\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-drj66"
Nov 01 14:36:42.027417 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:42.027373    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de2b3fa1-8060-4944-b43a-1d9501fa8cda-istio-data\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-drj66\" (UID: \"de2b3fa1-8060-4944-b43a-1d9501fa8cda\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-drj66"
Nov 01 14:36:42.027417 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:42.027393    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de2b3fa1-8060-4944-b43a-1d9501fa8cda-workload-socket\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-drj66\" (UID: \"de2b3fa1-8060-4944-b43a-1d9501fa8cda\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-drj66"
Nov 01 14:36:42.027523 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:42.027466    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de2b3fa1-8060-4944-b43a-1d9501fa8cda-credential-socket\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-drj66\" (UID: \"de2b3fa1-8060-4944-b43a-1d9501fa8cda\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-drj66"
Nov 01 14:36:42.114227 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:42.114190    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-drj66"]
Nov 01 14:36:42.128918 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:42.128881    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de2b3fa1-8060-4944-b43a-1d9501fa8cda-istio-podinfo\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-drj66\" (UID: \"de2b3fa1-8060-4944-b43a-1d9501fa8cda\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-drj66"
Nov 01 14:36:42.129090 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:42.128992    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de2b3fa1-8060-4944-b43a-1d9501fa8cda-istio-token\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-drj66\" (UID: \"de2b3fa1-8060-4944-b43a-1d9501fa8cda\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-drj66"
Nov 01 14:36:42.129157 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:42.129121    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de2b3fa1-8060-4944-b43a-1d9501fa8cda-workload-certs\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-drj66\" (UID: \"de2b3fa1-8060-4944-b43a-1d9501fa8cda\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-drj66"
Nov 01 14:36:42.129245 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:42.129207    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f8qd\" (UniqueName: \"kubernetes.io/projected/de2b3fa1-8060-4944-b43a-1d9501fa8cda-kube-api-access-2f8qd\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-drj66\" (UID: \"de2b3fa1-8060-4944-b43a-1d9501fa8cda\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-drj66"
Nov 01 14:36:42.129304 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:42.129244    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de2b3fa1-8060-4944-b43a-1d9501fa8cda-istio-data\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-drj66\" (UID: \"de2b3fa1-8060-4944-b43a-1d9501fa8cda\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-drj66"
Nov 01 14:36:42.129304 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:42.129273    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de2b3fa1-8060-4944-b43a-1d9501fa8cda-workload-socket\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-drj66\" (UID: \"de2b3fa1-8060-4944-b43a-1d9501fa8cda\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-drj66"
Nov 01 14:36:42.129403 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:42.129334    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de2b3fa1-8060-4944-b43a-1d9501fa8cda-credential-socket\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-drj66\" (UID: \"de2b3fa1-8060-4944-b43a-1d9501fa8cda\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-drj66"
Nov 01 14:36:42.129403 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:42.129371    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de2b3fa1-8060-4944-b43a-1d9501fa8cda-istio-envoy\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-drj66\" (UID: \"de2b3fa1-8060-4944-b43a-1d9501fa8cda\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-drj66"
Nov 01 14:36:42.129499 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:42.129416    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de2b3fa1-8060-4944-b43a-1d9501fa8cda-istiod-ca-cert\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-drj66\" (UID: \"de2b3fa1-8060-4944-b43a-1d9501fa8cda\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-drj66"
Nov 01 14:36:42.129708 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:42.129671    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de2b3fa1-8060-4944-b43a-1d9501fa8cda-workload-certs\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-drj66\" (UID: \"de2b3fa1-8060-4944-b43a-1d9501fa8cda\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-drj66"
Nov 01 14:36:42.129876 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:42.129853    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de2b3fa1-8060-4944-b43a-1d9501fa8cda-workload-socket\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-drj66\" (UID: \"de2b3fa1-8060-4944-b43a-1d9501fa8cda\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-drj66"
Nov 01 14:36:42.130206 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:42.130183    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de2b3fa1-8060-4944-b43a-1d9501fa8cda-istiod-ca-cert\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-drj66\" (UID: \"de2b3fa1-8060-4944-b43a-1d9501fa8cda\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-drj66"
Nov 01 14:36:42.130311 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:42.130275    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de2b3fa1-8060-4944-b43a-1d9501fa8cda-credential-socket\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-drj66\" (UID: \"de2b3fa1-8060-4944-b43a-1d9501fa8cda\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-drj66"
Nov 01 14:36:42.130516 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:42.130494    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de2b3fa1-8060-4944-b43a-1d9501fa8cda-istio-data\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-drj66\" (UID: \"de2b3fa1-8060-4944-b43a-1d9501fa8cda\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-drj66"
Nov 01 14:36:42.132728 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:42.132695    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de2b3fa1-8060-4944-b43a-1d9501fa8cda-istio-envoy\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-drj66\" (UID: \"de2b3fa1-8060-4944-b43a-1d9501fa8cda\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-drj66"
Nov 01 14:36:42.132833 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:42.132780    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de2b3fa1-8060-4944-b43a-1d9501fa8cda-istio-podinfo\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-drj66\" (UID: \"de2b3fa1-8060-4944-b43a-1d9501fa8cda\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-drj66"
Nov 01 14:36:42.187114 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:42.187080    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f8qd\" (UniqueName: \"kubernetes.io/projected/de2b3fa1-8060-4944-b43a-1d9501fa8cda-kube-api-access-2f8qd\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-drj66\" (UID: \"de2b3fa1-8060-4944-b43a-1d9501fa8cda\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-drj66"
Nov 01 14:36:42.191185 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:42.188584    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de2b3fa1-8060-4944-b43a-1d9501fa8cda-istio-token\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-drj66\" (UID: \"de2b3fa1-8060-4944-b43a-1d9501fa8cda\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-drj66"
Nov 01 14:36:42.250790 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:42.250757    4993 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Nov 01 14:36:42.250941 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:42.250810    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l" podUID="5bb22946-1d01-4745-a1e3-519f213c333c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Nov 01 14:36:42.327774 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:42.327684    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-drj66"
Nov 01 14:36:42.786911 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:42.786780    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-drj66"]
Nov 01 14:36:42.790425 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:36:42.790387    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde2b3fa1_8060_4944_b43a_1d9501fa8cda.slice/crio-77bd80675c9e09a1a2b3e5092b463bb1955d19924a7984520eab16e3a0e7d56b WatchSource:0}: Error finding container 77bd80675c9e09a1a2b3e5092b463bb1955d19924a7984520eab16e3a0e7d56b: Status 404 returned error can't find the container with id 77bd80675c9e09a1a2b3e5092b463bb1955d19924a7984520eab16e3a0e7d56b
Nov 01 14:36:42.802199 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:42.802134    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 14:36:42.802457 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:42.802425    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 14:36:42.802642 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:42.802609    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 14:36:43.250820 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:43.250779    4993 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Nov 01 14:36:43.251310 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:43.250841    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l" podUID="5bb22946-1d01-4745-a1e3-519f213c333c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Nov 01 14:36:43.314697 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:43.314647    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-drj66" event={"ID":"de2b3fa1-8060-4944-b43a-1d9501fa8cda","Type":"ContainerStarted","Data":"a1a01cae9973a2f7065735c885ffaab8307168ec1b37bdc544b2e61bedc4b194"}
Nov 01 14:36:43.314881 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:43.314715    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-drj66" event={"ID":"de2b3fa1-8060-4944-b43a-1d9501fa8cda","Type":"ContainerStarted","Data":"77bd80675c9e09a1a2b3e5092b463bb1955d19924a7984520eab16e3a0e7d56b"}
Nov 01 14:36:43.329189 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:43.328456    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-drj66"
Nov 01 14:36:43.332026 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:43.331992    4993 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-drj66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Nov 01 14:36:43.332197 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:43.332062    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-drj66" podUID="de2b3fa1-8060-4944-b43a-1d9501fa8cda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Nov 01 14:36:44.274194 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:44.272516    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l"
Nov 01 14:36:44.274194 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:44.273631    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l"
Nov 01 14:36:44.278191 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:44.275353    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l"
Nov 01 14:36:44.328525 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:44.328483    4993 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-drj66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Nov 01 14:36:44.328693 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:44.328545    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-drj66" podUID="de2b3fa1-8060-4944-b43a-1d9501fa8cda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Nov 01 14:36:44.485494 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:44.485435    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-drj66" podStartSLOduration=3.485421554 podStartE2EDuration="3.485421554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:36:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:36:43.467414357 +0000 UTC m=+1190.797277094" watchObservedRunningTime="2026-11-01 14:36:44.485421554 +0000 UTC m=+1191.815284291"
Nov 01 14:36:45.328130 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:45.328097    4993 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-drj66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Nov 01 14:36:45.328640 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:45.328154    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-drj66" podUID="de2b3fa1-8060-4944-b43a-1d9501fa8cda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Nov 01 14:36:46.327959 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:46.327923    4993 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-drj66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Nov 01 14:36:46.328135 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:46.327984    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-drj66" podUID="de2b3fa1-8060-4944-b43a-1d9501fa8cda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Nov 01 14:36:47.328994 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:47.328960    4993 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-drj66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Nov 01 14:36:47.329485 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:47.329013    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-drj66" podUID="de2b3fa1-8060-4944-b43a-1d9501fa8cda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Nov 01 14:36:48.328519 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:48.328471    4993 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-drj66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Nov 01 14:36:48.328739 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:48.328557    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-drj66" podUID="de2b3fa1-8060-4944-b43a-1d9501fa8cda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Nov 01 14:36:49.352952 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:49.352913    4993 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-drj66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Nov 01 14:36:49.353604 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:49.353559    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-drj66" podUID="de2b3fa1-8060-4944-b43a-1d9501fa8cda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Nov 01 14:36:50.336564 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:50.336523    4993 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-drj66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Nov 01 14:36:50.336822 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:50.336789    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-drj66" podUID="de2b3fa1-8060-4944-b43a-1d9501fa8cda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Nov 01 14:36:50.459200 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:50.456305    4993 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 14:36:50.459200 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:50.456433    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm" podUID="6db204f9-b7a9-4b31-ab49-a4406d2b879d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 14:36:51.058051 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:51.058013    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr"]
Nov 01 14:36:51.062279 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:51.062259    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr"
Nov 01 14:36:51.143589 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:51.143555    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8-workload-socket\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr\" (UID: \"0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr"
Nov 01 14:36:51.143741 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:51.143599    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8-istio-envoy\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr\" (UID: \"0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr"
Nov 01 14:36:51.143741 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:51.143713    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8-istio-data\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr\" (UID: \"0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr"
Nov 01 14:36:51.143838 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:51.143754    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8-istiod-ca-cert\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr\" (UID: \"0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr"
Nov 01 14:36:51.143912 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:51.143882    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdzrz\" (UniqueName: \"kubernetes.io/projected/0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8-kube-api-access-zdzrz\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr\" (UID: \"0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr"
Nov 01 14:36:51.143954 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:51.143932    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8-workload-certs\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr\" (UID: \"0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr"
Nov 01 14:36:51.144068 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:51.144053    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8-istio-token\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr\" (UID: \"0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr"
Nov 01 14:36:51.144101 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:51.144088    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8-istio-podinfo\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr\" (UID: \"0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr"
Nov 01 14:36:51.144252 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:51.144232    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8-credential-socket\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr\" (UID: \"0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr"
Nov 01 14:36:51.182282 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:51.182242    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr"]
Nov 01 14:36:51.245239 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:51.245200    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8-workload-socket\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr\" (UID: \"0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr"
Nov 01 14:36:51.245434 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:51.245257    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8-istio-envoy\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr\" (UID: \"0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr"
Nov 01 14:36:51.245434 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:51.245302    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8-istio-data\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr\" (UID: \"0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr"
Nov 01 14:36:51.245434 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:51.245331    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8-istiod-ca-cert\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr\" (UID: \"0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr"
Nov 01 14:36:51.245434 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:51.245396    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdzrz\" (UniqueName: \"kubernetes.io/projected/0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8-kube-api-access-zdzrz\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr\" (UID: \"0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr"
Nov 01 14:36:51.245434 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:51.245422    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8-workload-certs\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr\" (UID: \"0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr"
Nov 01 14:36:51.245690 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:51.245505    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8-istio-token\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr\" (UID: \"0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr"
Nov 01 14:36:51.245690 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:51.245536    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8-istio-podinfo\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr\" (UID: \"0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr"
Nov 01 14:36:51.245690 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:51.245629    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8-credential-socket\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr\" (UID: \"0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr"
Nov 01 14:36:51.245834 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:51.245724    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8-workload-socket\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr\" (UID: \"0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr"
Nov 01 14:36:51.246087 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:51.246059    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8-istio-data\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr\" (UID: \"0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr"
Nov 01 14:36:51.246370 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:51.246340    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8-istiod-ca-cert\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr\" (UID: \"0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr"
Nov 01 14:36:51.246482 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:51.246400    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8-credential-socket\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr\" (UID: \"0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr"
Nov 01 14:36:51.246660 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:51.246636    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8-workload-certs\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr\" (UID: \"0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr"
Nov 01 14:36:51.270224 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:51.248980    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8-istio-envoy\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr\" (UID: \"0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr"
Nov 01 14:36:51.270224 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:51.249273    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8-istio-podinfo\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr\" (UID: \"0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr"
Nov 01 14:36:51.299480 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:51.299445    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8-istio-token\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr\" (UID: \"0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr"
Nov 01 14:36:51.299665 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:51.299622    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdzrz\" (UniqueName: \"kubernetes.io/projected/0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8-kube-api-access-zdzrz\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr\" (UID: \"0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr"
Nov 01 14:36:51.381110 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:51.381022    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr"
Nov 01 14:36:51.816967 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:51.816928    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr"]
Nov 01 14:36:51.862515 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:36:51.862448    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c2f4202_6c87_4721_bc5e_5fb7eb6fa6a8.slice/crio-7be62e690055f257532b935e5c16414e4f1e0eb90e6705fc17d14b6cc516a6d5 WatchSource:0}: Error finding container 7be62e690055f257532b935e5c16414e4f1e0eb90e6705fc17d14b6cc516a6d5: Status 404 returned error can't find the container with id 7be62e690055f257532b935e5c16414e4f1e0eb90e6705fc17d14b6cc516a6d5
Nov 01 14:36:51.865705 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:51.865652    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 14:36:51.865825 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:51.865754    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 14:36:51.865825 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:51.865796    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 14:36:52.300197 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:52.299475    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-drj66"
Nov 01 14:36:52.300509 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:52.300489    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-drj66"
Nov 01 14:36:52.301262 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:52.301242    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-drj66"
Nov 01 14:36:52.416889 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:52.416849    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr" event={"ID":"0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8","Type":"ContainerStarted","Data":"91e56a8579f5bb611a79d73a592f92672f8a575458be3ff73967704425a366b4"}
Nov 01 14:36:52.417052 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:52.416914    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr" event={"ID":"0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8","Type":"ContainerStarted","Data":"7be62e690055f257532b935e5c16414e4f1e0eb90e6705fc17d14b6cc516a6d5"}
Nov 01 14:36:52.920936 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:52.920903    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xmwtp_bb59051d-c2df-4362-a157-00f14f915e79/console-operator/3.log"
Nov 01 14:36:52.921386 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:52.921132    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xmwtp_bb59051d-c2df-4362-a157-00f14f915e79/console-operator/3.log"
Nov 01 14:36:53.384282 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:53.384234    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr"
Nov 01 14:36:53.385346 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:53.385313    4993 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Nov 01 14:36:53.385477 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:53.385373    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr" podUID="0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Nov 01 14:36:54.382195 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:54.381627    4993 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Nov 01 14:36:54.382195 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:54.381696    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr" podUID="0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Nov 01 14:36:55.381931 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:55.381894    4993 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Nov 01 14:36:55.382115 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:55.381951    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr" podUID="0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Nov 01 14:36:56.382628 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:56.382579    4993 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Nov 01 14:36:56.383087 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:56.382664    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr" podUID="0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Nov 01 14:36:57.381571 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:57.381535    4993 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Nov 01 14:36:57.381796 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:57.381597    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr" podUID="0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Nov 01 14:36:57.503252 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:36:57.503160    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:36:58.382035 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:58.381991    4993 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Nov 01 14:36:58.382289 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:58.382062    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr" podUID="0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Nov 01 14:36:59.385489 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:59.385455    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr"
Nov 01 14:36:59.385883 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:59.385727    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr"
Nov 01 14:36:59.386522 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:59.386504    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr"
Nov 01 14:36:59.542582 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:36:59.542527    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr" podStartSLOduration=8.542512297 podStartE2EDuration="8.542512297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:36:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:36:52.692594291 +0000 UTC m=+1200.022457028" watchObservedRunningTime="2026-11-01 14:36:59.542512297 +0000 UTC m=+1206.872375033"
Nov 01 14:37:10.603838 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:10.603763    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-1f8e2c/distinct-w0-1f8e2c-kserve-6657447954-rpml6"]
Nov 01 14:37:10.607750 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:10.607728    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1f8e2c/distinct-w0-1f8e2c-kserve-6657447954-rpml6"
Nov 01 14:37:10.670987 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:10.670946    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39e0e8e2-87d8-4332-bb5d-86f3849861be-model-cache\") pod \"distinct-w0-1f8e2c-kserve-6657447954-rpml6\" (UID: \"39e0e8e2-87d8-4332-bb5d-86f3849861be\") " pod="e2e-models-e2e-worker-w0-1f8e2c/distinct-w0-1f8e2c-kserve-6657447954-rpml6"
Nov 01 14:37:10.671188 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:10.671020    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39e0e8e2-87d8-4332-bb5d-86f3849861be-tls-certs\") pod \"distinct-w0-1f8e2c-kserve-6657447954-rpml6\" (UID: \"39e0e8e2-87d8-4332-bb5d-86f3849861be\") " pod="e2e-models-e2e-worker-w0-1f8e2c/distinct-w0-1f8e2c-kserve-6657447954-rpml6"
Nov 01 14:37:10.671188 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:10.671075    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39e0e8e2-87d8-4332-bb5d-86f3849861be-tmp-dir\") pod \"distinct-w0-1f8e2c-kserve-6657447954-rpml6\" (UID: \"39e0e8e2-87d8-4332-bb5d-86f3849861be\") " pod="e2e-models-e2e-worker-w0-1f8e2c/distinct-w0-1f8e2c-kserve-6657447954-rpml6"
Nov 01 14:37:10.671188 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:10.671155    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39e0e8e2-87d8-4332-bb5d-86f3849861be-home\") pod \"distinct-w0-1f8e2c-kserve-6657447954-rpml6\" (UID: \"39e0e8e2-87d8-4332-bb5d-86f3849861be\") " pod="e2e-models-e2e-worker-w0-1f8e2c/distinct-w0-1f8e2c-kserve-6657447954-rpml6"
Nov 01 14:37:10.671391 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:10.671201    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39e0e8e2-87d8-4332-bb5d-86f3849861be-dshm\") pod \"distinct-w0-1f8e2c-kserve-6657447954-rpml6\" (UID: \"39e0e8e2-87d8-4332-bb5d-86f3849861be\") " pod="e2e-models-e2e-worker-w0-1f8e2c/distinct-w0-1f8e2c-kserve-6657447954-rpml6"
Nov 01 14:37:10.671391 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:10.671258    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhxcf\" (UniqueName: \"kubernetes.io/projected/39e0e8e2-87d8-4332-bb5d-86f3849861be-kube-api-access-fhxcf\") pod \"distinct-w0-1f8e2c-kserve-6657447954-rpml6\" (UID: \"39e0e8e2-87d8-4332-bb5d-86f3849861be\") " pod="e2e-models-e2e-worker-w0-1f8e2c/distinct-w0-1f8e2c-kserve-6657447954-rpml6"
Nov 01 14:37:10.702269 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:10.702236    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-1f8e2c\"/\"distinct-w0-1f8e2c-kserve-self-signed-certs\""
Nov 01 14:37:10.706796 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:10.706762    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1f8e2c/distinct-w0-1f8e2c-kserve-6657447954-rpml6"]
Nov 01 14:37:10.773637 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:10.773594    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39e0e8e2-87d8-4332-bb5d-86f3849861be-tls-certs\") pod \"distinct-w0-1f8e2c-kserve-6657447954-rpml6\" (UID: \"39e0e8e2-87d8-4332-bb5d-86f3849861be\") " pod="e2e-models-e2e-worker-w0-1f8e2c/distinct-w0-1f8e2c-kserve-6657447954-rpml6"
Nov 01 14:37:10.773835 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:10.773692    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39e0e8e2-87d8-4332-bb5d-86f3849861be-tmp-dir\") pod \"distinct-w0-1f8e2c-kserve-6657447954-rpml6\" (UID: \"39e0e8e2-87d8-4332-bb5d-86f3849861be\") " pod="e2e-models-e2e-worker-w0-1f8e2c/distinct-w0-1f8e2c-kserve-6657447954-rpml6"
Nov 01 14:37:10.773835 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:10.773828    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39e0e8e2-87d8-4332-bb5d-86f3849861be-home\") pod \"distinct-w0-1f8e2c-kserve-6657447954-rpml6\" (UID: \"39e0e8e2-87d8-4332-bb5d-86f3849861be\") " pod="e2e-models-e2e-worker-w0-1f8e2c/distinct-w0-1f8e2c-kserve-6657447954-rpml6"
Nov 01 14:37:10.773970 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:10.773852    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39e0e8e2-87d8-4332-bb5d-86f3849861be-dshm\") pod \"distinct-w0-1f8e2c-kserve-6657447954-rpml6\" (UID: \"39e0e8e2-87d8-4332-bb5d-86f3849861be\") " pod="e2e-models-e2e-worker-w0-1f8e2c/distinct-w0-1f8e2c-kserve-6657447954-rpml6"
Nov 01 14:37:10.773970 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:10.773929    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhxcf\" (UniqueName: \"kubernetes.io/projected/39e0e8e2-87d8-4332-bb5d-86f3849861be-kube-api-access-fhxcf\") pod \"distinct-w0-1f8e2c-kserve-6657447954-rpml6\" (UID: \"39e0e8e2-87d8-4332-bb5d-86f3849861be\") " pod="e2e-models-e2e-worker-w0-1f8e2c/distinct-w0-1f8e2c-kserve-6657447954-rpml6"
Nov 01 14:37:10.774085 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:10.773986    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39e0e8e2-87d8-4332-bb5d-86f3849861be-model-cache\") pod \"distinct-w0-1f8e2c-kserve-6657447954-rpml6\" (UID: \"39e0e8e2-87d8-4332-bb5d-86f3849861be\") " pod="e2e-models-e2e-worker-w0-1f8e2c/distinct-w0-1f8e2c-kserve-6657447954-rpml6"
Nov 01 14:37:10.774589 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:10.774562    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39e0e8e2-87d8-4332-bb5d-86f3849861be-model-cache\") pod \"distinct-w0-1f8e2c-kserve-6657447954-rpml6\" (UID: \"39e0e8e2-87d8-4332-bb5d-86f3849861be\") " pod="e2e-models-e2e-worker-w0-1f8e2c/distinct-w0-1f8e2c-kserve-6657447954-rpml6"
Nov 01 14:37:10.774923 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:10.774893    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39e0e8e2-87d8-4332-bb5d-86f3849861be-home\") pod \"distinct-w0-1f8e2c-kserve-6657447954-rpml6\" (UID: \"39e0e8e2-87d8-4332-bb5d-86f3849861be\") " pod="e2e-models-e2e-worker-w0-1f8e2c/distinct-w0-1f8e2c-kserve-6657447954-rpml6"
Nov 01 14:37:10.775249 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:10.775223    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39e0e8e2-87d8-4332-bb5d-86f3849861be-tmp-dir\") pod \"distinct-w0-1f8e2c-kserve-6657447954-rpml6\" (UID: \"39e0e8e2-87d8-4332-bb5d-86f3849861be\") " pod="e2e-models-e2e-worker-w0-1f8e2c/distinct-w0-1f8e2c-kserve-6657447954-rpml6"
Nov 01 14:37:10.778978 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:10.778952    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39e0e8e2-87d8-4332-bb5d-86f3849861be-dshm\") pod \"distinct-w0-1f8e2c-kserve-6657447954-rpml6\" (UID: \"39e0e8e2-87d8-4332-bb5d-86f3849861be\") " pod="e2e-models-e2e-worker-w0-1f8e2c/distinct-w0-1f8e2c-kserve-6657447954-rpml6"
Nov 01 14:37:10.779304 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:10.779279    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39e0e8e2-87d8-4332-bb5d-86f3849861be-tls-certs\") pod \"distinct-w0-1f8e2c-kserve-6657447954-rpml6\" (UID: \"39e0e8e2-87d8-4332-bb5d-86f3849861be\") " pod="e2e-models-e2e-worker-w0-1f8e2c/distinct-w0-1f8e2c-kserve-6657447954-rpml6"
Nov 01 14:37:10.828117 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:10.828077    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhxcf\" (UniqueName: \"kubernetes.io/projected/39e0e8e2-87d8-4332-bb5d-86f3849861be-kube-api-access-fhxcf\") pod \"distinct-w0-1f8e2c-kserve-6657447954-rpml6\" (UID: \"39e0e8e2-87d8-4332-bb5d-86f3849861be\") " pod="e2e-models-e2e-worker-w0-1f8e2c/distinct-w0-1f8e2c-kserve-6657447954-rpml6"
Nov 01 14:37:10.921151 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:10.921047    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1f8e2c/distinct-w0-1f8e2c-kserve-6657447954-rpml6"
Nov 01 14:37:11.304004 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:11.303973    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1f8e2c/distinct-w0-1f8e2c-kserve-6657447954-rpml6"]
Nov 01 14:37:11.310272 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:37:11.310244    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod39e0e8e2_87d8_4332_bb5d_86f3849861be.slice/crio-c2e65a6e39cf76cc65001d016172eb64442eef4b91e6816b6bba3792e53cbaca WatchSource:0}: Error finding container c2e65a6e39cf76cc65001d016172eb64442eef4b91e6816b6bba3792e53cbaca: Status 404 returned error can't find the container with id c2e65a6e39cf76cc65001d016172eb64442eef4b91e6816b6bba3792e53cbaca
Nov 01 14:37:11.553361 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:11.553328    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1f8e2c/distinct-w0-1f8e2c-kserve-6657447954-rpml6" event={"ID":"39e0e8e2-87d8-4332-bb5d-86f3849861be","Type":"ContainerStarted","Data":"bede0709c95e1ff913eb37843cb9a91670f153f524b63cbb214fea3f29dc4866"}
Nov 01 14:37:11.553361 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:11.553368    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1f8e2c/distinct-w0-1f8e2c-kserve-6657447954-rpml6" event={"ID":"39e0e8e2-87d8-4332-bb5d-86f3849861be","Type":"ContainerStarted","Data":"c2e65a6e39cf76cc65001d016172eb64442eef4b91e6816b6bba3792e53cbaca"}
Nov 01 14:37:11.661859 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:11.661755    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-1f8e2c/distinct-w0-1f8e2c-kserve-6657447954-rpml6" podStartSLOduration=1.6617399769999999 podStartE2EDuration="1.661739977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:37:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:37:11.660110567 +0000 UTC m=+1218.989973306" watchObservedRunningTime="2026-11-01 14:37:11.661739977 +0000 UTC m=+1218.991602713"
Nov 01 14:37:11.921675 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:11.921594    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f8e2c/distinct-w0-1f8e2c-kserve-6657447954-rpml6"
Nov 01 14:37:12.819250 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:12.819221    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-5h4tl-w5zwd"]
Nov 01 14:37:12.822977 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:12.822954    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-5h4tl-w5zwd"
Nov 01 14:37:12.904512 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:12.904470    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e6e13755-98b2-495b-9bd0-a1dd39b880df-archives-path\") pod \"periodic-gathering-5h4tl-w5zwd\" (UID: \"e6e13755-98b2-495b-9bd0-a1dd39b880df\") " pod="openshift-insights/periodic-gathering-5h4tl-w5zwd"
Nov 01 14:37:12.904714 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:12.904655    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6e13755-98b2-495b-9bd0-a1dd39b880df-service-ca-bundle\") pod \"periodic-gathering-5h4tl-w5zwd\" (UID: \"e6e13755-98b2-495b-9bd0-a1dd39b880df\") " pod="openshift-insights/periodic-gathering-5h4tl-w5zwd"
Nov 01 14:37:12.904850 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:12.904827    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-256jv\" (UniqueName: \"kubernetes.io/projected/e6e13755-98b2-495b-9bd0-a1dd39b880df-kube-api-access-256jv\") pod \"periodic-gathering-5h4tl-w5zwd\" (UID: \"e6e13755-98b2-495b-9bd0-a1dd39b880df\") " pod="openshift-insights/periodic-gathering-5h4tl-w5zwd"
Nov 01 14:37:12.904931 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:12.904915    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6e13755-98b2-495b-9bd0-a1dd39b880df-trusted-ca-bundle\") pod \"periodic-gathering-5h4tl-w5zwd\" (UID: \"e6e13755-98b2-495b-9bd0-a1dd39b880df\") " pod="openshift-insights/periodic-gathering-5h4tl-w5zwd"
Nov 01 14:37:12.917440 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:12.917401    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-55h6z\""
Nov 01 14:37:12.920346 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:12.920321    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-5h4tl-w5zwd"]
Nov 01 14:37:13.006681 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:13.006644    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6e13755-98b2-495b-9bd0-a1dd39b880df-trusted-ca-bundle\") pod \"periodic-gathering-5h4tl-w5zwd\" (UID: \"e6e13755-98b2-495b-9bd0-a1dd39b880df\") " pod="openshift-insights/periodic-gathering-5h4tl-w5zwd"
Nov 01 14:37:13.006855 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:13.006752    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e6e13755-98b2-495b-9bd0-a1dd39b880df-archives-path\") pod \"periodic-gathering-5h4tl-w5zwd\" (UID: \"e6e13755-98b2-495b-9bd0-a1dd39b880df\") " pod="openshift-insights/periodic-gathering-5h4tl-w5zwd"
Nov 01 14:37:13.006855 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:13.006794    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6e13755-98b2-495b-9bd0-a1dd39b880df-service-ca-bundle\") pod \"periodic-gathering-5h4tl-w5zwd\" (UID: \"e6e13755-98b2-495b-9bd0-a1dd39b880df\") " pod="openshift-insights/periodic-gathering-5h4tl-w5zwd"
Nov 01 14:37:13.006974 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:13.006868    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-256jv\" (UniqueName: \"kubernetes.io/projected/e6e13755-98b2-495b-9bd0-a1dd39b880df-kube-api-access-256jv\") pod \"periodic-gathering-5h4tl-w5zwd\" (UID: \"e6e13755-98b2-495b-9bd0-a1dd39b880df\") " pod="openshift-insights/periodic-gathering-5h4tl-w5zwd"
Nov 01 14:37:13.007313 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:13.007284    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e6e13755-98b2-495b-9bd0-a1dd39b880df-archives-path\") pod \"periodic-gathering-5h4tl-w5zwd\" (UID: \"e6e13755-98b2-495b-9bd0-a1dd39b880df\") " pod="openshift-insights/periodic-gathering-5h4tl-w5zwd"
Nov 01 14:37:13.007617 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:13.007596    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6e13755-98b2-495b-9bd0-a1dd39b880df-service-ca-bundle\") pod \"periodic-gathering-5h4tl-w5zwd\" (UID: \"e6e13755-98b2-495b-9bd0-a1dd39b880df\") " pod="openshift-insights/periodic-gathering-5h4tl-w5zwd"
Nov 01 14:37:13.007661 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:13.007602    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6e13755-98b2-495b-9bd0-a1dd39b880df-trusted-ca-bundle\") pod \"periodic-gathering-5h4tl-w5zwd\" (UID: \"e6e13755-98b2-495b-9bd0-a1dd39b880df\") " pod="openshift-insights/periodic-gathering-5h4tl-w5zwd"
Nov 01 14:37:13.059615 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:13.059577    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-256jv\" (UniqueName: \"kubernetes.io/projected/e6e13755-98b2-495b-9bd0-a1dd39b880df-kube-api-access-256jv\") pod \"periodic-gathering-5h4tl-w5zwd\" (UID: \"e6e13755-98b2-495b-9bd0-a1dd39b880df\") " pod="openshift-insights/periodic-gathering-5h4tl-w5zwd"
Nov 01 14:37:13.135386 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:13.135295    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-5h4tl-w5zwd"
Nov 01 14:37:13.504696 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:13.504666    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-5h4tl-w5zwd"]
Nov 01 14:37:13.510454 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:37:13.510426    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6e13755_98b2_495b_9bd0_a1dd39b880df.slice/crio-9ee13766b67d31e4b22299e771783004ca9dc5626034349a8ba2ecb3740bc1ec WatchSource:0}: Error finding container 9ee13766b67d31e4b22299e771783004ca9dc5626034349a8ba2ecb3740bc1ec: Status 404 returned error can't find the container with id 9ee13766b67d31e4b22299e771783004ca9dc5626034349a8ba2ecb3740bc1ec
Nov 01 14:37:13.566871 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:13.566839    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-5h4tl-w5zwd" event={"ID":"e6e13755-98b2-495b-9bd0-a1dd39b880df","Type":"ContainerStarted","Data":"9ee13766b67d31e4b22299e771783004ca9dc5626034349a8ba2ecb3740bc1ec"}
Nov 01 14:37:14.574998 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:14.574958    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-5h4tl-w5zwd" event={"ID":"e6e13755-98b2-495b-9bd0-a1dd39b880df","Type":"ContainerStarted","Data":"6289ebdc14f802c76d2475c3cb7d75bf28eeb140b2bed8c5eec02969bf759bc6"}
Nov 01 14:37:14.681972 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:14.681903    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-5h4tl-w5zwd" podStartSLOduration=2.681889627 podStartE2EDuration="2.681889627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:37:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:37:14.680069471 +0000 UTC m=+1222.009932209" watchObservedRunningTime="2026-11-01 14:37:14.681889627 +0000 UTC m=+1222.011752342"
Nov 01 14:37:20.921845 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:20.921799    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-1f8e2c/distinct-w0-1f8e2c-kserve-6657447954-rpml6"
Nov 01 14:37:20.942928 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:20.942896    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-1f8e2c/distinct-w0-1f8e2c-kserve-6657447954-rpml6"
Nov 01 14:37:22.208192 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:22.198320    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-00a787/simulator-w1-00a787-kserve-5bdcf4b567-4xsn7" podUID="6b8fc4b9-5b38-4d33-8f56-c8913e7e403f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 14:37:22.247854 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:22.247809    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-w3-5b0dcc-kserve-57cb5cb5cf-msjth" podUID="e3ac5585-dc23-445d-89fc-b68f90dac95a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 14:37:22.387273 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:22.387224    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-00a787/distinct-w1-00a787-kserve-dbd65bf4f-lbcwn" podUID="e1c97a57-0e89-4370-bc30-c75368d340f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 14:37:22.493341 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:22.493309    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f8e2c/distinct-w0-1f8e2c-kserve-6657447954-rpml6"
Nov 01 14:37:22.806580 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:22.806442    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-007270/simulator-w2-007270-kserve-5bf8844f79-c7v9v" podUID="2ae81a81-e763-40ac-8f5e-4bd38f9a3e68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 14:37:23.005305 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:23.005251    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f8e2c/simulator-w0-1f8e2c-kserve-755bf645d-zf8t4" podUID="bbc61e12-d326-4d31-be81-88fb641337c7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 14:37:23.353113 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:23.353076    4993 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-drj66 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 14:37:23.353773 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:23.353732    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-drj66" podUID="de2b3fa1-8060-4944-b43a-1d9501fa8cda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 14:37:27.501816 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:37:27.501771    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:37:41.890998 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:41.890952    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-007270/simulator-w2-007270-kserve-5bf8844f79-c7v9v" podUID="2ae81a81-e763-40ac-8f5e-4bd38f9a3e68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 14:37:41.930301 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:41.930250    4993 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1f8e2c/distinct-w0-1f8e2c-kserve-6657447954-rpml6" podUID="39e0e8e2-87d8-4332-bb5d-86f3849861be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": context deadline exceeded"
Nov 01 14:37:42.059265 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:42.059219    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f8e2c/simulator-w0-1f8e2c-kserve-755bf645d-zf8t4" podUID="bbc61e12-d326-4d31-be81-88fb641337c7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 14:37:42.137336 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:42.137283    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-00a787/simulator-w1-00a787-kserve-5bdcf4b567-4xsn7" podUID="6b8fc4b9-5b38-4d33-8f56-c8913e7e403f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 14:37:45.977726 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:45.977684    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld"]
Nov 01 14:37:45.986121 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:45.986086    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld"
Nov 01 14:37:46.018399 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:46.018316    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc"]
Nov 01 14:37:46.024236 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:46.024207    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc"
Nov 01 14:37:46.060324 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:46.060295    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld"]
Nov 01 14:37:46.061326 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:46.061302    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-007270-openshift-default-dockercfg-78vkt\""
Nov 01 14:37:46.102929 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:46.102889    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4df5d536-0c2d-43ae-9e86-61e2b2d64ee6-workload-certs\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld\" (UID: \"4df5d536-0c2d-43ae-9e86-61e2b2d64ee6\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld"
Nov 01 14:37:46.103116 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:46.102958    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fefc13c7-831b-45aa-b73c-69814cc7e5b4-istio-envoy\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc\" (UID: \"fefc13c7-831b-45aa-b73c-69814cc7e5b4\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc"
Nov 01 14:37:46.103116 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:46.102984    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fefc13c7-831b-45aa-b73c-69814cc7e5b4-workload-socket\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc\" (UID: \"fefc13c7-831b-45aa-b73c-69814cc7e5b4\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc"
Nov 01 14:37:46.103116 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:46.103008    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fefc13c7-831b-45aa-b73c-69814cc7e5b4-istio-podinfo\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc\" (UID: \"fefc13c7-831b-45aa-b73c-69814cc7e5b4\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc"
Nov 01 14:37:46.103116 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:46.103029    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4df5d536-0c2d-43ae-9e86-61e2b2d64ee6-istio-data\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld\" (UID: \"4df5d536-0c2d-43ae-9e86-61e2b2d64ee6\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld"
Nov 01 14:37:46.103116 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:46.103055    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fefc13c7-831b-45aa-b73c-69814cc7e5b4-workload-certs\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc\" (UID: \"fefc13c7-831b-45aa-b73c-69814cc7e5b4\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc"
Nov 01 14:37:46.103116 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:46.103077    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4df5d536-0c2d-43ae-9e86-61e2b2d64ee6-istio-envoy\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld\" (UID: \"4df5d536-0c2d-43ae-9e86-61e2b2d64ee6\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld"
Nov 01 14:37:46.103472 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:46.103122    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4df5d536-0c2d-43ae-9e86-61e2b2d64ee6-istio-token\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld\" (UID: \"4df5d536-0c2d-43ae-9e86-61e2b2d64ee6\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld"
Nov 01 14:37:46.103472 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:46.103220    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4df5d536-0c2d-43ae-9e86-61e2b2d64ee6-workload-socket\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld\" (UID: \"4df5d536-0c2d-43ae-9e86-61e2b2d64ee6\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld"
Nov 01 14:37:46.103472 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:46.103258    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqljm\" (UniqueName: \"kubernetes.io/projected/fefc13c7-831b-45aa-b73c-69814cc7e5b4-kube-api-access-kqljm\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc\" (UID: \"fefc13c7-831b-45aa-b73c-69814cc7e5b4\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc"
Nov 01 14:37:46.103472 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:46.103312    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mjs8\" (UniqueName: \"kubernetes.io/projected/4df5d536-0c2d-43ae-9e86-61e2b2d64ee6-kube-api-access-7mjs8\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld\" (UID: \"4df5d536-0c2d-43ae-9e86-61e2b2d64ee6\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld"
Nov 01 14:37:46.103472 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:46.103346    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4df5d536-0c2d-43ae-9e86-61e2b2d64ee6-istiod-ca-cert\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld\" (UID: \"4df5d536-0c2d-43ae-9e86-61e2b2d64ee6\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld"
Nov 01 14:37:46.103472 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:46.103381    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fefc13c7-831b-45aa-b73c-69814cc7e5b4-istio-data\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc\" (UID: \"fefc13c7-831b-45aa-b73c-69814cc7e5b4\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc"
Nov 01 14:37:46.103472 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:46.103435    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fefc13c7-831b-45aa-b73c-69814cc7e5b4-istio-token\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc\" (UID: \"fefc13c7-831b-45aa-b73c-69814cc7e5b4\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc"
Nov 01 14:37:46.103472 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:46.103463    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4df5d536-0c2d-43ae-9e86-61e2b2d64ee6-credential-socket\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld\" (UID: \"4df5d536-0c2d-43ae-9e86-61e2b2d64ee6\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld"
Nov 01 14:37:46.103864 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:46.103559    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fefc13c7-831b-45aa-b73c-69814cc7e5b4-credential-socket\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc\" (UID: \"fefc13c7-831b-45aa-b73c-69814cc7e5b4\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc"
Nov 01 14:37:46.103864 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:46.103598    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fefc13c7-831b-45aa-b73c-69814cc7e5b4-istiod-ca-cert\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc\" (UID: \"fefc13c7-831b-45aa-b73c-69814cc7e5b4\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc"
Nov 01 14:37:46.103864 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:46.103637    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4df5d536-0c2d-43ae-9e86-61e2b2d64ee6-istio-podinfo\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld\" (UID: \"4df5d536-0c2d-43ae-9e86-61e2b2d64ee6\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld"
Nov 01 14:37:46.151909 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:46.151759    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc"]
Nov 01 14:37:46.204692 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:46.204654    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4df5d536-0c2d-43ae-9e86-61e2b2d64ee6-istiod-ca-cert\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld\" (UID: \"4df5d536-0c2d-43ae-9e86-61e2b2d64ee6\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld"
Nov 01 14:37:46.204851 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:46.204708    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fefc13c7-831b-45aa-b73c-69814cc7e5b4-istio-data\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc\" (UID: \"fefc13c7-831b-45aa-b73c-69814cc7e5b4\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc"
Nov 01 14:37:46.204851 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:46.204753    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fefc13c7-831b-45aa-b73c-69814cc7e5b4-istio-token\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc\" (UID: \"fefc13c7-831b-45aa-b73c-69814cc7e5b4\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc"
Nov 01 14:37:46.204851 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:46.204780    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4df5d536-0c2d-43ae-9e86-61e2b2d64ee6-credential-socket\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld\" (UID: \"4df5d536-0c2d-43ae-9e86-61e2b2d64ee6\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld"
Nov 01 14:37:46.204958 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:46.204851    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fefc13c7-831b-45aa-b73c-69814cc7e5b4-credential-socket\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc\" (UID: \"fefc13c7-831b-45aa-b73c-69814cc7e5b4\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc"
Nov 01 14:37:46.204958 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:46.204883    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fefc13c7-831b-45aa-b73c-69814cc7e5b4-istiod-ca-cert\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc\" (UID: \"fefc13c7-831b-45aa-b73c-69814cc7e5b4\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc"
Nov 01 14:37:46.204958 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:46.204919    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4df5d536-0c2d-43ae-9e86-61e2b2d64ee6-istio-podinfo\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld\" (UID: \"4df5d536-0c2d-43ae-9e86-61e2b2d64ee6\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld"
Nov 01 14:37:46.205046 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:46.204976    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4df5d536-0c2d-43ae-9e86-61e2b2d64ee6-workload-certs\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld\" (UID: \"4df5d536-0c2d-43ae-9e86-61e2b2d64ee6\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld"
Nov 01 14:37:46.205046 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:46.205022    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fefc13c7-831b-45aa-b73c-69814cc7e5b4-istio-envoy\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc\" (UID: \"fefc13c7-831b-45aa-b73c-69814cc7e5b4\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc"
Nov 01 14:37:46.205117 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:46.205051    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fefc13c7-831b-45aa-b73c-69814cc7e5b4-workload-socket\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc\" (UID: \"fefc13c7-831b-45aa-b73c-69814cc7e5b4\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc"
Nov 01 14:37:46.205117 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:46.205075    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fefc13c7-831b-45aa-b73c-69814cc7e5b4-istio-podinfo\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc\" (UID: \"fefc13c7-831b-45aa-b73c-69814cc7e5b4\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc"
Nov 01 14:37:46.205117 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:46.205097    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4df5d536-0c2d-43ae-9e86-61e2b2d64ee6-istio-data\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld\" (UID: \"4df5d536-0c2d-43ae-9e86-61e2b2d64ee6\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld"
Nov 01 14:37:46.205239 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:46.205128    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fefc13c7-831b-45aa-b73c-69814cc7e5b4-workload-certs\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc\" (UID: \"fefc13c7-831b-45aa-b73c-69814cc7e5b4\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc"
Nov 01 14:37:46.205239 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:46.205155    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4df5d536-0c2d-43ae-9e86-61e2b2d64ee6-istio-envoy\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld\" (UID: \"4df5d536-0c2d-43ae-9e86-61e2b2d64ee6\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld"
Nov 01 14:37:46.205239 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:46.205209    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4df5d536-0c2d-43ae-9e86-61e2b2d64ee6-istio-token\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld\" (UID: \"4df5d536-0c2d-43ae-9e86-61e2b2d64ee6\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld"
Nov 01 14:37:46.205332 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:46.205260    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4df5d536-0c2d-43ae-9e86-61e2b2d64ee6-workload-socket\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld\" (UID: \"4df5d536-0c2d-43ae-9e86-61e2b2d64ee6\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld"
Nov 01 14:37:46.205332 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:46.205283    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqljm\" (UniqueName: \"kubernetes.io/projected/fefc13c7-831b-45aa-b73c-69814cc7e5b4-kube-api-access-kqljm\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc\" (UID: \"fefc13c7-831b-45aa-b73c-69814cc7e5b4\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc"
Nov 01 14:37:46.205332 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:46.205317    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mjs8\" (UniqueName: \"kubernetes.io/projected/4df5d536-0c2d-43ae-9e86-61e2b2d64ee6-kube-api-access-7mjs8\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld\" (UID: \"4df5d536-0c2d-43ae-9e86-61e2b2d64ee6\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld"
Nov 01 14:37:46.206112 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:46.206088    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4df5d536-0c2d-43ae-9e86-61e2b2d64ee6-istiod-ca-cert\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld\" (UID: \"4df5d536-0c2d-43ae-9e86-61e2b2d64ee6\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld"
Nov 01 14:37:46.206753 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:46.206724    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fefc13c7-831b-45aa-b73c-69814cc7e5b4-istiod-ca-cert\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc\" (UID: \"fefc13c7-831b-45aa-b73c-69814cc7e5b4\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc"
Nov 01 14:37:46.207214 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:46.207162    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4df5d536-0c2d-43ae-9e86-61e2b2d64ee6-credential-socket\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld\" (UID: \"4df5d536-0c2d-43ae-9e86-61e2b2d64ee6\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld"
Nov 01 14:37:46.207914 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:46.207882    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4df5d536-0c2d-43ae-9e86-61e2b2d64ee6-workload-socket\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld\" (UID: \"4df5d536-0c2d-43ae-9e86-61e2b2d64ee6\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld"
Nov 01 14:37:46.208361 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:46.208340    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4df5d536-0c2d-43ae-9e86-61e2b2d64ee6-istio-data\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld\" (UID: \"4df5d536-0c2d-43ae-9e86-61e2b2d64ee6\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld"
Nov 01 14:37:46.208527 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:46.208507    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4df5d536-0c2d-43ae-9e86-61e2b2d64ee6-workload-certs\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld\" (UID: \"4df5d536-0c2d-43ae-9e86-61e2b2d64ee6\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld"
Nov 01 14:37:46.208676 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:46.208660    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fefc13c7-831b-45aa-b73c-69814cc7e5b4-istio-data\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc\" (UID: \"fefc13c7-831b-45aa-b73c-69814cc7e5b4\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc"
Nov 01 14:37:46.208994 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:46.208974    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fefc13c7-831b-45aa-b73c-69814cc7e5b4-workload-socket\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc\" (UID: \"fefc13c7-831b-45aa-b73c-69814cc7e5b4\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc"
Nov 01 14:37:46.209141 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:46.209122    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fefc13c7-831b-45aa-b73c-69814cc7e5b4-workload-certs\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc\" (UID: \"fefc13c7-831b-45aa-b73c-69814cc7e5b4\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc"
Nov 01 14:37:46.209547 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:46.209496    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fefc13c7-831b-45aa-b73c-69814cc7e5b4-credential-socket\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc\" (UID: \"fefc13c7-831b-45aa-b73c-69814cc7e5b4\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc"
Nov 01 14:37:46.211027 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:46.211006    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fefc13c7-831b-45aa-b73c-69814cc7e5b4-istio-envoy\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc\" (UID: \"fefc13c7-831b-45aa-b73c-69814cc7e5b4\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc"
Nov 01 14:37:46.213163 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:46.213134    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4df5d536-0c2d-43ae-9e86-61e2b2d64ee6-istio-podinfo\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld\" (UID: \"4df5d536-0c2d-43ae-9e86-61e2b2d64ee6\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld"
Nov 01 14:37:46.213306 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:46.213287    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4df5d536-0c2d-43ae-9e86-61e2b2d64ee6-istio-envoy\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld\" (UID: \"4df5d536-0c2d-43ae-9e86-61e2b2d64ee6\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld"
Nov 01 14:37:46.213536 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:46.213518    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fefc13c7-831b-45aa-b73c-69814cc7e5b4-istio-podinfo\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc\" (UID: \"fefc13c7-831b-45aa-b73c-69814cc7e5b4\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc"
Nov 01 14:37:46.259904 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:46.259833    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mjs8\" (UniqueName: \"kubernetes.io/projected/4df5d536-0c2d-43ae-9e86-61e2b2d64ee6-kube-api-access-7mjs8\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld\" (UID: \"4df5d536-0c2d-43ae-9e86-61e2b2d64ee6\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld"
Nov 01 14:37:46.260943 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:46.260916    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fefc13c7-831b-45aa-b73c-69814cc7e5b4-istio-token\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc\" (UID: \"fefc13c7-831b-45aa-b73c-69814cc7e5b4\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc"
Nov 01 14:37:46.261828 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:46.261802    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqljm\" (UniqueName: \"kubernetes.io/projected/fefc13c7-831b-45aa-b73c-69814cc7e5b4-kube-api-access-kqljm\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc\" (UID: \"fefc13c7-831b-45aa-b73c-69814cc7e5b4\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc"
Nov 01 14:37:46.262654 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:46.262632    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4df5d536-0c2d-43ae-9e86-61e2b2d64ee6-istio-token\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld\" (UID: \"4df5d536-0c2d-43ae-9e86-61e2b2d64ee6\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld"
Nov 01 14:37:46.306256 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:46.306223    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld"
Nov 01 14:37:46.367404 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:46.366725    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc"
Nov 01 14:37:46.735456 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:46.735422    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld"]
Nov 01 14:37:46.740402 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:37:46.740373    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4df5d536_0c2d_43ae_9e86_61e2b2d64ee6.slice/crio-e8b467444eb570460d5e60be72f29208528b3c465d3c40d880e8972d4197534c WatchSource:0}: Error finding container e8b467444eb570460d5e60be72f29208528b3c465d3c40d880e8972d4197534c: Status 404 returned error can't find the container with id e8b467444eb570460d5e60be72f29208528b3c465d3c40d880e8972d4197534c
Nov 01 14:37:46.742814 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:46.742774    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 14:37:46.742913 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:46.742850    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 14:37:46.742913 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:46.742879    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 14:37:46.794761 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:37:46.794726    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfefc13c7_831b_45aa_b73c_69814cc7e5b4.slice/crio-c2dc3942611b1c63caa413ab6ad3b89f60c6079cd3e5dd3224f4531dae0a3f06 WatchSource:0}: Error finding container c2dc3942611b1c63caa413ab6ad3b89f60c6079cd3e5dd3224f4531dae0a3f06: Status 404 returned error can't find the container with id c2dc3942611b1c63caa413ab6ad3b89f60c6079cd3e5dd3224f4531dae0a3f06
Nov 01 14:37:46.798027 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:46.797979    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 14:37:46.798124 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:46.798075    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 14:37:46.798226 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:46.798121    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 14:37:46.799083 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:46.798974    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc"]
Nov 01 14:37:47.067247 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:47.066501    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc" event={"ID":"fefc13c7-831b-45aa-b73c-69814cc7e5b4","Type":"ContainerStarted","Data":"938a947427a8543f64ed0ffe58b39891827dec8ed66a6a5babb6fbc166a373bc"}
Nov 01 14:37:47.067247 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:47.066716    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc" event={"ID":"fefc13c7-831b-45aa-b73c-69814cc7e5b4","Type":"ContainerStarted","Data":"c2dc3942611b1c63caa413ab6ad3b89f60c6079cd3e5dd3224f4531dae0a3f06"}
Nov 01 14:37:47.074213 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:47.071595    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld" event={"ID":"4df5d536-0c2d-43ae-9e86-61e2b2d64ee6","Type":"ContainerStarted","Data":"041dce0d8ce366c943416cbceee166bd690b19d12c584e2146c8742d5d62083d"}
Nov 01 14:37:47.074213 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:47.071642    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld" event={"ID":"4df5d536-0c2d-43ae-9e86-61e2b2d64ee6","Type":"ContainerStarted","Data":"e8b467444eb570460d5e60be72f29208528b3c465d3c40d880e8972d4197534c"}
Nov 01 14:37:47.190352 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:47.189675    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc" podStartSLOduration=2.189656534 podStartE2EDuration="2.189656534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:37:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:37:47.174974392 +0000 UTC m=+1254.504837131" watchObservedRunningTime="2026-11-01 14:37:47.189656534 +0000 UTC m=+1254.519519273"
Nov 01 14:37:47.288787 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:47.288733    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld" podStartSLOduration=2.28871962 podStartE2EDuration="2.28871962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:37:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:37:47.285558178 +0000 UTC m=+1254.615420916" watchObservedRunningTime="2026-11-01 14:37:47.28871962 +0000 UTC m=+1254.618582357"
Nov 01 14:37:47.306972 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:47.306937    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld"
Nov 01 14:37:47.308951 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:47.308450    4993 patch_prober.go:29] interesting pod/e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Nov 01 14:37:47.308951 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:47.308501    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld" podUID="4df5d536-0c2d-43ae-9e86-61e2b2d64ee6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Nov 01 14:37:47.367345 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:47.367244    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc"
Nov 01 14:37:47.369157 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:47.369128    4993 patch_prober.go:29] interesting pod/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 01 14:37:47.369324 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:47.369190    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc" podUID="fefc13c7-831b-45aa-b73c-69814cc7e5b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 01 14:37:48.314194 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:48.313714    4993 patch_prober.go:29] interesting pod/e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Nov 01 14:37:48.314194 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:48.313785    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld" podUID="4df5d536-0c2d-43ae-9e86-61e2b2d64ee6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Nov 01 14:37:48.369159 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:48.369122    4993 patch_prober.go:29] interesting pod/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 01 14:37:48.369489 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:48.369458    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc" podUID="fefc13c7-831b-45aa-b73c-69814cc7e5b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 01 14:37:49.315192 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:49.314742    4993 patch_prober.go:29] interesting pod/e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Nov 01 14:37:49.315192 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:49.314818    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld" podUID="4df5d536-0c2d-43ae-9e86-61e2b2d64ee6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Nov 01 14:37:49.425645 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:49.425461    4993 patch_prober.go:29] interesting pod/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 01 14:37:49.425934 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:49.425901    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc" podUID="fefc13c7-831b-45aa-b73c-69814cc7e5b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 01 14:37:50.310642 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:50.310592    4993 patch_prober.go:29] interesting pod/e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Nov 01 14:37:50.310941 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:50.310903    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld" podUID="4df5d536-0c2d-43ae-9e86-61e2b2d64ee6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Nov 01 14:37:50.370814 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:50.370778    4993 patch_prober.go:29] interesting pod/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 01 14:37:50.371470 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:50.371420    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc" podUID="fefc13c7-831b-45aa-b73c-69814cc7e5b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 01 14:37:51.308613 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:51.308575    4993 patch_prober.go:29] interesting pod/e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Nov 01 14:37:51.308901 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:51.308864    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld" podUID="4df5d536-0c2d-43ae-9e86-61e2b2d64ee6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Nov 01 14:37:51.369704 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:51.369581    4993 patch_prober.go:29] interesting pod/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 01 14:37:51.369704 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:51.369643    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc" podUID="fefc13c7-831b-45aa-b73c-69814cc7e5b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 01 14:37:52.317205 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:52.307522    4993 patch_prober.go:29] interesting pod/e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Nov 01 14:37:52.317205 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:52.307583    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld" podUID="4df5d536-0c2d-43ae-9e86-61e2b2d64ee6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Nov 01 14:37:52.391192 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:52.390969    4993 patch_prober.go:29] interesting pod/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 01 14:37:52.391192 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:52.391032    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc" podUID="fefc13c7-831b-45aa-b73c-69814cc7e5b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 01 14:37:52.806708 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:52.806659    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-00a787/distinct-2-w1-00a787-kserve-56f5f9dc7d-bhxxr" podUID="c0180369-b893-47c4-a683-ad9b6da82101" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 14:37:52.807225 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:52.806837    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-007270/simulator-w2-007270-kserve-5bf8844f79-c7v9v" podUID="2ae81a81-e763-40ac-8f5e-4bd38f9a3e68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 14:37:53.003435 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:53.003383    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f8e2c/simulator-w0-1f8e2c-kserve-755bf645d-zf8t4" podUID="bbc61e12-d326-4d31-be81-88fb641337c7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Nov 01 14:37:53.203306 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:53.203203    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-00a787/simulator-w1-00a787-kserve-5bdcf4b567-4xsn7" podUID="6b8fc4b9-5b38-4d33-8f56-c8913e7e403f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Nov 01 14:37:53.207110 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:53.207072    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-w3-5b0dcc-kserve-57cb5cb5cf-msjth" podUID="e3ac5585-dc23-445d-89fc-b68f90dac95a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Nov 01 14:37:53.323196 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:53.321914    4993 patch_prober.go:29] interesting pod/e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Nov 01 14:37:53.323196 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:53.321990    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld" podUID="4df5d536-0c2d-43ae-9e86-61e2b2d64ee6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Nov 01 14:37:53.373673 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:53.373634    4993 patch_prober.go:29] interesting pod/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 01 14:37:53.373964 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:53.373932    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc" podUID="fefc13c7-831b-45aa-b73c-69814cc7e5b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 01 14:37:53.378306 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:53.378266    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-00a787/distinct-w1-00a787-kserve-dbd65bf4f-lbcwn" podUID="e1c97a57-0e89-4370-bc30-c75368d340f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 14:37:53.432325 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:53.432267    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536a87/premium-w4-536a87-kserve-74c6fdbc8-sf89f" podUID="8a720a76-488a-4a65-98ad-31ca84fa7842" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 14:37:53.700194 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:53.696804    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f8e2c/distinct-w0-1f8e2c-kserve-6657447954-rpml6" podUID="39e0e8e2-87d8-4332-bb5d-86f3849861be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Nov 01 14:37:53.817189 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:53.814382    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-007270/simulator-w2-007270-kserve-5bf8844f79-c7v9v" podUID="2ae81a81-e763-40ac-8f5e-4bd38f9a3e68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 14:37:53.873202 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:53.841154    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-00a787/distinct-2-w1-00a787-kserve-56f5f9dc7d-bhxxr" podUID="c0180369-b893-47c4-a683-ad9b6da82101" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Nov 01 14:37:53.873202 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:53.858138    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-007270/simulator-w2-007270-kserve-5bf8844f79-c7v9v"
Nov 01 14:37:53.873202 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:53.858235    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-00a787/distinct-2-w1-00a787-kserve-56f5f9dc7d-bhxxr"
Nov 01 14:37:54.144863 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:54.144815    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f8e2c/simulator-w0-1f8e2c-kserve-755bf645d-zf8t4" podUID="bbc61e12-d326-4d31-be81-88fb641337c7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Nov 01 14:37:54.146652 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:54.146617    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f8e2c/simulator-w0-1f8e2c-kserve-755bf645d-zf8t4"
Nov 01 14:37:54.223778 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:54.223723    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-w3-5b0dcc-kserve-57cb5cb5cf-msjth" podUID="e3ac5585-dc23-445d-89fc-b68f90dac95a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 14:37:54.225217 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:54.225162    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-00a787/simulator-w1-00a787-kserve-5bdcf4b567-4xsn7" podUID="6b8fc4b9-5b38-4d33-8f56-c8913e7e403f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 14:37:54.226421 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:54.226401    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-w3-5b0dcc-kserve-57cb5cb5cf-msjth"
Nov 01 14:37:54.253386 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:54.253360    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-00a787/simulator-w1-00a787-kserve-5bdcf4b567-4xsn7"
Nov 01 14:37:54.311668 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:54.311631    4993 patch_prober.go:29] interesting pod/e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Nov 01 14:37:54.311991 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:54.311937    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld" podUID="4df5d536-0c2d-43ae-9e86-61e2b2d64ee6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Nov 01 14:37:54.392194 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:54.389253    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-00a787/distinct-w1-00a787-kserve-dbd65bf4f-lbcwn" podUID="e1c97a57-0e89-4370-bc30-c75368d340f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Nov 01 14:37:54.392194 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:54.391460    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-00a787/distinct-w1-00a787-kserve-dbd65bf4f-lbcwn"
Nov 01 14:37:54.415275 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:54.402769    4993 patch_prober.go:29] interesting pod/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 01 14:37:54.415563 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:54.415527    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc" podUID="fefc13c7-831b-45aa-b73c-69814cc7e5b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 01 14:37:54.439192 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:54.436264    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536a87/premium-w4-536a87-kserve-74c6fdbc8-sf89f" podUID="8a720a76-488a-4a65-98ad-31ca84fa7842" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 14:37:54.439192 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:54.438063    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-536a87/premium-w4-536a87-kserve-74c6fdbc8-sf89f"
Nov 01 14:37:54.829631 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:54.829584    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f8e2c/distinct-w0-1f8e2c-kserve-6657447954-rpml6" podUID="39e0e8e2-87d8-4332-bb5d-86f3849861be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 14:37:54.849195 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:54.829877    4993 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-00a787/distinct-w1-00a787-kserve-dbd65bf4f-lbcwn" podUID="e1c97a57-0e89-4370-bc30-c75368d340f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 14:37:55.140991 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:54.889368    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-00a787/distinct-2-w1-00a787-kserve-56f5f9dc7d-bhxxr" podUID="c0180369-b893-47c4-a683-ad9b6da82101" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 14:37:55.141600 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:54.914455    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f8e2c/distinct-w0-1f8e2c-kserve-6657447954-rpml6"
Nov 01 14:37:55.141937 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:55.141916    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-00a787/simulator-w1-00a787-kserve-5bdcf4b567-4xsn7"
Nov 01 14:37:55.142157 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:54.944405    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-007270/simulator-w2-007270-kserve-5bf8844f79-c7v9v" podUID="2ae81a81-e763-40ac-8f5e-4bd38f9a3e68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 14:37:55.144368 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:55.144347    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-w3-5b0dcc-kserve-57cb5cb5cf-msjth"
Nov 01 14:37:55.162028 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:55.155301    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f8e2c/simulator-w0-1f8e2c-kserve-755bf645d-zf8t4" podUID="bbc61e12-d326-4d31-be81-88fb641337c7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 14:37:55.190093 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:55.190060    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-007270/simulator-w2-007270-kserve-5bf8844f79-c7v9v"
Nov 01 14:37:55.286705 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:55.286670    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f8e2c/simulator-w0-1f8e2c-kserve-755bf645d-zf8t4"
Nov 01 14:37:55.312705 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:55.305704    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f8e2c/distinct-w0-1f8e2c-kserve-6657447954-rpml6"
Nov 01 14:37:55.315261 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:55.315223    4993 patch_prober.go:29] interesting pod/e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Nov 01 14:37:55.315705 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:55.315675    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld" podUID="4df5d536-0c2d-43ae-9e86-61e2b2d64ee6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Nov 01 14:37:55.316306 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:55.316238    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-536a87/premium-w4-536a87-kserve-74c6fdbc8-sf89f"
Nov 01 14:37:55.316432 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:55.316337    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-00a787/distinct-2-w1-00a787-kserve-56f5f9dc7d-bhxxr"
Nov 01 14:37:55.323539 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:55.323365    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-00a787/distinct-w1-00a787-kserve-dbd65bf4f-lbcwn"
Nov 01 14:37:55.368224 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:55.368188    4993 patch_prober.go:29] interesting pod/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 01 14:37:55.368386 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:55.368250    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc" podUID="fefc13c7-831b-45aa-b73c-69814cc7e5b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 01 14:37:56.461236 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:56.451683    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc"
Nov 01 14:37:56.461236 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:56.453430    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc"
Nov 01 14:37:56.461236 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:56.455713    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc"
Nov 01 14:37:56.925791 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:56.925756    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld"
Nov 01 14:37:56.926663 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:56.926638    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld"
Nov 01 14:37:56.927403 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:37:56.927384    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld"
Nov 01 14:37:57.500613 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:37:57.500574    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:38:01.109751 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:01.109716    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9"]
Nov 01 14:38:01.115128 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:01.115105    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9"
Nov 01 14:38:01.153351 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:01.153315    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzx67\" (UniqueName: \"kubernetes.io/projected/9e600803-b75d-48b9-847b-d236b79cf527-kube-api-access-hzx67\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9\" (UID: \"9e600803-b75d-48b9-847b-d236b79cf527\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9"
Nov 01 14:38:01.153555 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:01.153380    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e600803-b75d-48b9-847b-d236b79cf527-istio-envoy\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9\" (UID: \"9e600803-b75d-48b9-847b-d236b79cf527\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9"
Nov 01 14:38:01.153555 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:01.153482    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e600803-b75d-48b9-847b-d236b79cf527-workload-socket\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9\" (UID: \"9e600803-b75d-48b9-847b-d236b79cf527\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9"
Nov 01 14:38:01.153555 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:01.153545    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e600803-b75d-48b9-847b-d236b79cf527-credential-socket\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9\" (UID: \"9e600803-b75d-48b9-847b-d236b79cf527\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9"
Nov 01 14:38:01.153750 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:01.153617    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e600803-b75d-48b9-847b-d236b79cf527-istio-data\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9\" (UID: \"9e600803-b75d-48b9-847b-d236b79cf527\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9"
Nov 01 14:38:01.153750 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:01.153664    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e600803-b75d-48b9-847b-d236b79cf527-istio-podinfo\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9\" (UID: \"9e600803-b75d-48b9-847b-d236b79cf527\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9"
Nov 01 14:38:01.153832 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:01.153764    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e600803-b75d-48b9-847b-d236b79cf527-istio-token\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9\" (UID: \"9e600803-b75d-48b9-847b-d236b79cf527\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9"
Nov 01 14:38:01.153832 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:01.153783    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e600803-b75d-48b9-847b-d236b79cf527-istiod-ca-cert\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9\" (UID: \"9e600803-b75d-48b9-847b-d236b79cf527\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9"
Nov 01 14:38:01.153928 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:01.153900    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e600803-b75d-48b9-847b-d236b79cf527-workload-certs\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9\" (UID: \"9e600803-b75d-48b9-847b-d236b79cf527\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9"
Nov 01 14:38:01.214455 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:01.214415    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9"]
Nov 01 14:38:01.254934 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:01.254897    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e600803-b75d-48b9-847b-d236b79cf527-istio-token\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9\" (UID: \"9e600803-b75d-48b9-847b-d236b79cf527\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9"
Nov 01 14:38:01.254934 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:01.254936    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e600803-b75d-48b9-847b-d236b79cf527-istiod-ca-cert\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9\" (UID: \"9e600803-b75d-48b9-847b-d236b79cf527\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9"
Nov 01 14:38:01.255147 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:01.255013    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e600803-b75d-48b9-847b-d236b79cf527-workload-certs\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9\" (UID: \"9e600803-b75d-48b9-847b-d236b79cf527\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9"
Nov 01 14:38:01.255147 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:01.255072    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzx67\" (UniqueName: \"kubernetes.io/projected/9e600803-b75d-48b9-847b-d236b79cf527-kube-api-access-hzx67\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9\" (UID: \"9e600803-b75d-48b9-847b-d236b79cf527\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9"
Nov 01 14:38:01.255147 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:01.255108    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e600803-b75d-48b9-847b-d236b79cf527-istio-envoy\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9\" (UID: \"9e600803-b75d-48b9-847b-d236b79cf527\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9"
Nov 01 14:38:01.255147 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:01.255137    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e600803-b75d-48b9-847b-d236b79cf527-workload-socket\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9\" (UID: \"9e600803-b75d-48b9-847b-d236b79cf527\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9"
Nov 01 14:38:01.255367 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:01.255205    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e600803-b75d-48b9-847b-d236b79cf527-credential-socket\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9\" (UID: \"9e600803-b75d-48b9-847b-d236b79cf527\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9"
Nov 01 14:38:01.255367 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:01.255255    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e600803-b75d-48b9-847b-d236b79cf527-istio-data\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9\" (UID: \"9e600803-b75d-48b9-847b-d236b79cf527\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9"
Nov 01 14:38:01.255367 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:01.255302    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e600803-b75d-48b9-847b-d236b79cf527-istio-podinfo\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9\" (UID: \"9e600803-b75d-48b9-847b-d236b79cf527\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9"
Nov 01 14:38:01.256023 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:01.255988    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e600803-b75d-48b9-847b-d236b79cf527-workload-socket\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9\" (UID: \"9e600803-b75d-48b9-847b-d236b79cf527\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9"
Nov 01 14:38:01.256142 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:01.255992    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e600803-b75d-48b9-847b-d236b79cf527-workload-certs\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9\" (UID: \"9e600803-b75d-48b9-847b-d236b79cf527\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9"
Nov 01 14:38:01.256142 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:01.256057    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e600803-b75d-48b9-847b-d236b79cf527-credential-socket\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9\" (UID: \"9e600803-b75d-48b9-847b-d236b79cf527\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9"
Nov 01 14:38:01.256142 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:01.256091    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e600803-b75d-48b9-847b-d236b79cf527-istio-data\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9\" (UID: \"9e600803-b75d-48b9-847b-d236b79cf527\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9"
Nov 01 14:38:01.256284 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:01.256214    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e600803-b75d-48b9-847b-d236b79cf527-istiod-ca-cert\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9\" (UID: \"9e600803-b75d-48b9-847b-d236b79cf527\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9"
Nov 01 14:38:01.257629 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:01.257601    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e600803-b75d-48b9-847b-d236b79cf527-istio-envoy\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9\" (UID: \"9e600803-b75d-48b9-847b-d236b79cf527\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9"
Nov 01 14:38:01.257915 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:01.257898    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e600803-b75d-48b9-847b-d236b79cf527-istio-podinfo\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9\" (UID: \"9e600803-b75d-48b9-847b-d236b79cf527\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9"
Nov 01 14:38:01.308232 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:01.308195    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e600803-b75d-48b9-847b-d236b79cf527-istio-token\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9\" (UID: \"9e600803-b75d-48b9-847b-d236b79cf527\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9"
Nov 01 14:38:01.308369 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:01.308273    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzx67\" (UniqueName: \"kubernetes.io/projected/9e600803-b75d-48b9-847b-d236b79cf527-kube-api-access-hzx67\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9\" (UID: \"9e600803-b75d-48b9-847b-d236b79cf527\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9"
Nov 01 14:38:01.428762 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:01.428680    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9"
Nov 01 14:38:01.860227 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:01.860191    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9"]
Nov 01 14:38:01.864002 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:38:01.863978    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e600803_b75d_48b9_847b_d236b79cf527.slice/crio-1faf051a50f213d65a87df54bef1764c2d61188197b9cc81b4ad9541c9d6ec60 WatchSource:0}: Error finding container 1faf051a50f213d65a87df54bef1764c2d61188197b9cc81b4ad9541c9d6ec60: Status 404 returned error can't find the container with id 1faf051a50f213d65a87df54bef1764c2d61188197b9cc81b4ad9541c9d6ec60
Nov 01 14:38:01.867012 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:01.866971    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 14:38:01.867099 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:01.867059    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 14:38:01.867138 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:01.867107    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 14:38:02.398527 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:02.398481    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9" event={"ID":"9e600803-b75d-48b9-847b-d236b79cf527","Type":"ContainerStarted","Data":"e9bb8ae4bd6b6ae513fb8f6b73b3ec284e4067a30a4aef63fb246cd0ad7f1702"}
Nov 01 14:38:02.398978 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:02.398544    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9" event={"ID":"9e600803-b75d-48b9-847b-d236b79cf527","Type":"ContainerStarted","Data":"1faf051a50f213d65a87df54bef1764c2d61188197b9cc81b4ad9541c9d6ec60"}
Nov 01 14:38:02.428966 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:02.428931    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9"
Nov 01 14:38:02.430726 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:02.430701    4993 patch_prober.go:29] interesting pod/e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Nov 01 14:38:02.430844 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:02.430749    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9" podUID="9e600803-b75d-48b9-847b-d236b79cf527" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Nov 01 14:38:02.573193 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:02.554458    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9" podStartSLOduration=1.55443853 podStartE2EDuration="1.55443853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:38:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:38:02.55364991 +0000 UTC m=+1269.883512647" watchObservedRunningTime="2026-11-01 14:38:02.55443853 +0000 UTC m=+1269.884301285"
Nov 01 14:38:03.429920 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:03.429879    4993 patch_prober.go:29] interesting pod/e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Nov 01 14:38:03.430686 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:03.429940    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9" podUID="9e600803-b75d-48b9-847b-d236b79cf527" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Nov 01 14:38:04.429919 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:04.429885    4993 patch_prober.go:29] interesting pod/e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Nov 01 14:38:04.430367 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:04.429942    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9" podUID="9e600803-b75d-48b9-847b-d236b79cf527" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Nov 01 14:38:05.429196 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:05.429137    4993 patch_prober.go:29] interesting pod/e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Nov 01 14:38:05.429397 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:05.429226    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9" podUID="9e600803-b75d-48b9-847b-d236b79cf527" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Nov 01 14:38:06.429718 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:06.429674    4993 patch_prober.go:29] interesting pod/e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Nov 01 14:38:06.430241 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:06.429733    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9" podUID="9e600803-b75d-48b9-847b-d236b79cf527" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Nov 01 14:38:07.429885 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:07.429850    4993 patch_prober.go:29] interesting pod/e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Nov 01 14:38:07.430340 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:07.429900    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9" podUID="9e600803-b75d-48b9-847b-d236b79cf527" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Nov 01 14:38:08.430034 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:08.429993    4993 patch_prober.go:29] interesting pod/e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Nov 01 14:38:08.430509 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:08.430078    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9" podUID="9e600803-b75d-48b9-847b-d236b79cf527" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Nov 01 14:38:10.488446 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:10.488386    4993 patch_prober.go:29] interesting pod/e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 14:38:10.489230 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:10.489160    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9" podUID="9e600803-b75d-48b9-847b-d236b79cf527" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 14:38:11.158319 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:11.158257    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-w3-5b0dcc-kserve-57cb5cb5cf-msjth" podUID="e3ac5585-dc23-445d-89fc-b68f90dac95a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 14:38:11.158984 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:11.158949    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-00a787/distinct-2-w1-00a787-kserve-56f5f9dc7d-bhxxr" podUID="c0180369-b893-47c4-a683-ad9b6da82101" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Nov 01 14:38:11.160630 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:11.160604    4993 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-00a787/simulator-w1-00a787-kserve-5bdcf4b567-4xsn7" podUID="6b8fc4b9-5b38-4d33-8f56-c8913e7e403f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 14:38:11.199314 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:11.199260    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-007270/simulator-w2-007270-kserve-5bf8844f79-c7v9v" podUID="2ae81a81-e763-40ac-8f5e-4bd38f9a3e68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 14:38:11.298120 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:11.298077    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f8e2c/simulator-w0-1f8e2c-kserve-755bf645d-zf8t4" podUID="bbc61e12-d326-4d31-be81-88fb641337c7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 14:38:11.311301 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:11.311254    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f8e2c/distinct-w0-1f8e2c-kserve-6657447954-rpml6" podUID="39e0e8e2-87d8-4332-bb5d-86f3849861be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 14:38:11.336452 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:11.336400    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536a87/premium-w4-536a87-kserve-74c6fdbc8-sf89f" podUID="8a720a76-488a-4a65-98ad-31ca84fa7842" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 14:38:11.365266 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:11.365211    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-00a787/distinct-w1-00a787-kserve-dbd65bf4f-lbcwn" podUID="e1c97a57-0e89-4370-bc30-c75368d340f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Nov 01 14:38:11.540336 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:11.540290    4993 patch_prober.go:29] interesting pod/e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 14:38:11.540957 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:11.540924    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9" podUID="9e600803-b75d-48b9-847b-d236b79cf527" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 14:38:11.922242 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:11.920270    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-00a787/simulator-w1-00a787-kserve-5bdcf4b567-4xsn7" podUID="6b8fc4b9-5b38-4d33-8f56-c8913e7e403f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 14:38:11.922592 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:11.922503    4993 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1f8e2c/distinct-w0-1f8e2c-kserve-6657447954-rpml6" podUID="39e0e8e2-87d8-4332-bb5d-86f3849861be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 14:38:12.165153 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:12.163738    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9"
Nov 01 14:38:12.165153 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:12.164954    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9"
Nov 01 14:38:12.173579 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:12.173360    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9"
Nov 01 14:38:27.508383 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:38:27.508339    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:38:30.252378 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:30.252336    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c9b0827c-65bb9b948d-6bbgc"]
Nov 01 14:38:30.257948 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:30.257923    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9b0827c-65bb9b948d-6bbgc"
Nov 01 14:38:30.317151 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:30.317120    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-c9b0827c-serving-cert\""
Nov 01 14:38:30.359130 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:30.359094    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c9b0827c-65bb9b948d-6bbgc"]
Nov 01 14:38:30.388473 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:30.388432    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d3109df-047b-4a1f-a6d7-668ce0e4a43a-openshift-service-ca\") pod \"maas-api-e2e-mt-c9b0827c-65bb9b948d-6bbgc\" (UID: \"5d3109df-047b-4a1f-a6d7-668ce0e4a43a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9b0827c-65bb9b948d-6bbgc"
Nov 01 14:38:30.388764 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:30.388737    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d3109df-047b-4a1f-a6d7-668ce0e4a43a-metrics-certs\") pod \"maas-api-e2e-mt-c9b0827c-65bb9b948d-6bbgc\" (UID: \"5d3109df-047b-4a1f-a6d7-668ce0e4a43a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9b0827c-65bb9b948d-6bbgc"
Nov 01 14:38:30.388912 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:30.388895    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mxjx\" (UniqueName: \"kubernetes.io/projected/5d3109df-047b-4a1f-a6d7-668ce0e4a43a-kube-api-access-9mxjx\") pod \"maas-api-e2e-mt-c9b0827c-65bb9b948d-6bbgc\" (UID: \"5d3109df-047b-4a1f-a6d7-668ce0e4a43a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9b0827c-65bb9b948d-6bbgc"
Nov 01 14:38:30.389309 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:30.389280    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d3109df-047b-4a1f-a6d7-668ce0e4a43a-maas-api-tls\") pod \"maas-api-e2e-mt-c9b0827c-65bb9b948d-6bbgc\" (UID: \"5d3109df-047b-4a1f-a6d7-668ce0e4a43a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9b0827c-65bb9b948d-6bbgc"
Nov 01 14:38:30.490576 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:30.490524    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d3109df-047b-4a1f-a6d7-668ce0e4a43a-metrics-certs\") pod \"maas-api-e2e-mt-c9b0827c-65bb9b948d-6bbgc\" (UID: \"5d3109df-047b-4a1f-a6d7-668ce0e4a43a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9b0827c-65bb9b948d-6bbgc"
Nov 01 14:38:30.490794 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:30.490585    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mxjx\" (UniqueName: \"kubernetes.io/projected/5d3109df-047b-4a1f-a6d7-668ce0e4a43a-kube-api-access-9mxjx\") pod \"maas-api-e2e-mt-c9b0827c-65bb9b948d-6bbgc\" (UID: \"5d3109df-047b-4a1f-a6d7-668ce0e4a43a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9b0827c-65bb9b948d-6bbgc"
Nov 01 14:38:30.490917 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:30.490895    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d3109df-047b-4a1f-a6d7-668ce0e4a43a-maas-api-tls\") pod \"maas-api-e2e-mt-c9b0827c-65bb9b948d-6bbgc\" (UID: \"5d3109df-047b-4a1f-a6d7-668ce0e4a43a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9b0827c-65bb9b948d-6bbgc"
Nov 01 14:38:30.491141 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:30.491111    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d3109df-047b-4a1f-a6d7-668ce0e4a43a-openshift-service-ca\") pod \"maas-api-e2e-mt-c9b0827c-65bb9b948d-6bbgc\" (UID: \"5d3109df-047b-4a1f-a6d7-668ce0e4a43a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9b0827c-65bb9b948d-6bbgc"
Nov 01 14:38:30.491240 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:38:30.491002    4993 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-c9b0827c-serving-cert: secret "maas-api-e2e-mt-c9b0827c-serving-cert" not found
Nov 01 14:38:30.491287 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:38:30.491239    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d3109df-047b-4a1f-a6d7-668ce0e4a43a-maas-api-tls podName:5d3109df-047b-4a1f-a6d7-668ce0e4a43a nodeName:}" failed. No retries permitted until 2026-11-01 14:38:30.991215786 +0000 UTC m=+1298.321078515 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5d3109df-047b-4a1f-a6d7-668ce0e4a43a-maas-api-tls") pod "maas-api-e2e-mt-c9b0827c-65bb9b948d-6bbgc" (UID: "5d3109df-047b-4a1f-a6d7-668ce0e4a43a") : secret "maas-api-e2e-mt-c9b0827c-serving-cert" not found
Nov 01 14:38:30.491996 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:30.491971    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d3109df-047b-4a1f-a6d7-668ce0e4a43a-openshift-service-ca\") pod \"maas-api-e2e-mt-c9b0827c-65bb9b948d-6bbgc\" (UID: \"5d3109df-047b-4a1f-a6d7-668ce0e4a43a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9b0827c-65bb9b948d-6bbgc"
Nov 01 14:38:30.493310 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:30.493282    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d3109df-047b-4a1f-a6d7-668ce0e4a43a-metrics-certs\") pod \"maas-api-e2e-mt-c9b0827c-65bb9b948d-6bbgc\" (UID: \"5d3109df-047b-4a1f-a6d7-668ce0e4a43a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9b0827c-65bb9b948d-6bbgc"
Nov 01 14:38:30.546742 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:30.546657    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mxjx\" (UniqueName: \"kubernetes.io/projected/5d3109df-047b-4a1f-a6d7-668ce0e4a43a-kube-api-access-9mxjx\") pod \"maas-api-e2e-mt-c9b0827c-65bb9b948d-6bbgc\" (UID: \"5d3109df-047b-4a1f-a6d7-668ce0e4a43a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9b0827c-65bb9b948d-6bbgc"
Nov 01 14:38:31.003289 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:31.003249    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d3109df-047b-4a1f-a6d7-668ce0e4a43a-maas-api-tls\") pod \"maas-api-e2e-mt-c9b0827c-65bb9b948d-6bbgc\" (UID: \"5d3109df-047b-4a1f-a6d7-668ce0e4a43a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9b0827c-65bb9b948d-6bbgc"
Nov 01 14:38:31.005736 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:31.005712    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d3109df-047b-4a1f-a6d7-668ce0e4a43a-maas-api-tls\") pod \"maas-api-e2e-mt-c9b0827c-65bb9b948d-6bbgc\" (UID: \"5d3109df-047b-4a1f-a6d7-668ce0e4a43a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9b0827c-65bb9b948d-6bbgc"
Nov 01 14:38:31.176579 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:31.176542    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9b0827c-65bb9b948d-6bbgc"
Nov 01 14:38:31.357271 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:31.356826    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7"]
Nov 01 14:38:31.374983 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:31.374947    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7"
Nov 01 14:38:31.407116 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:31.407078    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7"]
Nov 01 14:38:31.514698 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:31.514656    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30998c96-4380-4e46-89db-c2f328351f71-istio-envoy\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7\" (UID: \"30998c96-4380-4e46-89db-c2f328351f71\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7"
Nov 01 14:38:31.514904 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:31.514716    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr4gj\" (UniqueName: \"kubernetes.io/projected/30998c96-4380-4e46-89db-c2f328351f71-kube-api-access-wr4gj\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7\" (UID: \"30998c96-4380-4e46-89db-c2f328351f71\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7"
Nov 01 14:38:31.514904 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:31.514846    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30998c96-4380-4e46-89db-c2f328351f71-workload-certs\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7\" (UID: \"30998c96-4380-4e46-89db-c2f328351f71\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7"
Nov 01 14:38:31.514998 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:31.514912    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30998c96-4380-4e46-89db-c2f328351f71-istio-data\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7\" (UID: \"30998c96-4380-4e46-89db-c2f328351f71\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7"
Nov 01 14:38:31.514998 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:31.514952    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30998c96-4380-4e46-89db-c2f328351f71-istio-token\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7\" (UID: \"30998c96-4380-4e46-89db-c2f328351f71\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7"
Nov 01 14:38:31.515094 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:31.515060    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30998c96-4380-4e46-89db-c2f328351f71-credential-socket\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7\" (UID: \"30998c96-4380-4e46-89db-c2f328351f71\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7"
Nov 01 14:38:31.515268 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:31.515119    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30998c96-4380-4e46-89db-c2f328351f71-istiod-ca-cert\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7\" (UID: \"30998c96-4380-4e46-89db-c2f328351f71\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7"
Nov 01 14:38:31.515372 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:31.515356    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30998c96-4380-4e46-89db-c2f328351f71-workload-socket\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7\" (UID: \"30998c96-4380-4e46-89db-c2f328351f71\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7"
Nov 01 14:38:31.515437 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:31.515422    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30998c96-4380-4e46-89db-c2f328351f71-istio-podinfo\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7\" (UID: \"30998c96-4380-4e46-89db-c2f328351f71\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7"
Nov 01 14:38:31.572823 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:31.570973    4993 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 14:38:31.581626 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:31.581588    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c9b0827c-65bb9b948d-6bbgc"]
Nov 01 14:38:31.616927 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:31.616822    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30998c96-4380-4e46-89db-c2f328351f71-workload-socket\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7\" (UID: \"30998c96-4380-4e46-89db-c2f328351f71\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7"
Nov 01 14:38:31.616927 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:31.616897    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30998c96-4380-4e46-89db-c2f328351f71-istio-podinfo\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7\" (UID: \"30998c96-4380-4e46-89db-c2f328351f71\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7"
Nov 01 14:38:31.617210 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:31.616955    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30998c96-4380-4e46-89db-c2f328351f71-istio-envoy\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7\" (UID: \"30998c96-4380-4e46-89db-c2f328351f71\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7"
Nov 01 14:38:31.617210 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:31.616990    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr4gj\" (UniqueName: \"kubernetes.io/projected/30998c96-4380-4e46-89db-c2f328351f71-kube-api-access-wr4gj\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7\" (UID: \"30998c96-4380-4e46-89db-c2f328351f71\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7"
Nov 01 14:38:31.617210 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:31.617032    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30998c96-4380-4e46-89db-c2f328351f71-workload-certs\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7\" (UID: \"30998c96-4380-4e46-89db-c2f328351f71\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7"
Nov 01 14:38:31.617210 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:31.617055    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30998c96-4380-4e46-89db-c2f328351f71-istio-data\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7\" (UID: \"30998c96-4380-4e46-89db-c2f328351f71\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7"
Nov 01 14:38:31.617210 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:31.617100    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30998c96-4380-4e46-89db-c2f328351f71-istio-token\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7\" (UID: \"30998c96-4380-4e46-89db-c2f328351f71\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7"
Nov 01 14:38:31.617210 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:31.617162    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30998c96-4380-4e46-89db-c2f328351f71-credential-socket\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7\" (UID: \"30998c96-4380-4e46-89db-c2f328351f71\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7"
Nov 01 14:38:31.617507 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:31.617217    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30998c96-4380-4e46-89db-c2f328351f71-istiod-ca-cert\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7\" (UID: \"30998c96-4380-4e46-89db-c2f328351f71\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7"
Nov 01 14:38:31.618591 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:31.617873    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30998c96-4380-4e46-89db-c2f328351f71-istio-data\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7\" (UID: \"30998c96-4380-4e46-89db-c2f328351f71\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7"
Nov 01 14:38:31.618591 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:31.618131    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30998c96-4380-4e46-89db-c2f328351f71-workload-certs\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7\" (UID: \"30998c96-4380-4e46-89db-c2f328351f71\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7"
Nov 01 14:38:31.618591 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:31.618323    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30998c96-4380-4e46-89db-c2f328351f71-credential-socket\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7\" (UID: \"30998c96-4380-4e46-89db-c2f328351f71\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7"
Nov 01 14:38:31.618913 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:31.618875    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30998c96-4380-4e46-89db-c2f328351f71-workload-socket\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7\" (UID: \"30998c96-4380-4e46-89db-c2f328351f71\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7"
Nov 01 14:38:31.619396 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:31.619366    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30998c96-4380-4e46-89db-c2f328351f71-istiod-ca-cert\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7\" (UID: \"30998c96-4380-4e46-89db-c2f328351f71\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7"
Nov 01 14:38:31.621794 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:31.621589    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30998c96-4380-4e46-89db-c2f328351f71-istio-podinfo\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7\" (UID: \"30998c96-4380-4e46-89db-c2f328351f71\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7"
Nov 01 14:38:31.621794 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:31.621779    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30998c96-4380-4e46-89db-c2f328351f71-istio-envoy\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7\" (UID: \"30998c96-4380-4e46-89db-c2f328351f71\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7"
Nov 01 14:38:31.666214 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:31.661554    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9b0827c-65bb9b948d-6bbgc" event={"ID":"5d3109df-047b-4a1f-a6d7-668ce0e4a43a","Type":"ContainerStarted","Data":"1e7f66af5305f7e12f5e9e59ed5306e07f98f9d5243b7eee4385f5cfcb0ca121"}
Nov 01 14:38:31.672684 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:31.672652    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30998c96-4380-4e46-89db-c2f328351f71-istio-token\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7\" (UID: \"30998c96-4380-4e46-89db-c2f328351f71\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7"
Nov 01 14:38:31.679267 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:31.679218    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr4gj\" (UniqueName: \"kubernetes.io/projected/30998c96-4380-4e46-89db-c2f328351f71-kube-api-access-wr4gj\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7\" (UID: \"30998c96-4380-4e46-89db-c2f328351f71\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7"
Nov 01 14:38:31.691907 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:31.691858    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7"
Nov 01 14:38:32.514542 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:32.514502    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7"]
Nov 01 14:38:32.540478 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:32.540438    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 14:38:32.540705 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:32.540682    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 14:38:32.540861 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:32.540839    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 14:38:32.701805 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:32.701708    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7" event={"ID":"30998c96-4380-4e46-89db-c2f328351f71","Type":"ContainerStarted","Data":"6f6ae2fe410a41d3ee0a79b8da82b65b609860615cb70986d636ac153f3e2cc5"}
Nov 01 14:38:33.724192 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:33.723294    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9b0827c-65bb9b948d-6bbgc" event={"ID":"5d3109df-047b-4a1f-a6d7-668ce0e4a43a","Type":"ContainerStarted","Data":"41329d4428abef02ab6340e7f6998f0cdecd2701c55df0875a49ae94a775f687"}
Nov 01 14:38:33.724666 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:33.724637    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9b0827c-65bb9b948d-6bbgc"
Nov 01 14:38:33.737241 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:33.727040    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7" event={"ID":"30998c96-4380-4e46-89db-c2f328351f71","Type":"ContainerStarted","Data":"3395dd973dc9b24731aa32f565c12de9b3aaafdee4aa92189970da6cfa8d1d18"}
Nov 01 14:38:34.082194 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:34.070226    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9b0827c-65bb9b948d-6bbgc" podStartSLOduration=2.945622603 podStartE2EDuration="4.070206986s" totalImagesPullingTime="1.124584383s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:38:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:38:33.927987444 +0000 UTC m=+1301.257850181" watchObservedRunningTime="2026-11-01 14:38:34.070206986 +0000 UTC m=+1301.400069734"
Nov 01 14:38:34.106197 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:34.105315    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7" podStartSLOduration=3.105292556 podStartE2EDuration="3.105292556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:38:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:38:34.041661407 +0000 UTC m=+1301.371524171" watchObservedRunningTime="2026-11-01 14:38:34.105292556 +0000 UTC m=+1301.435155294"
Nov 01 14:38:34.697194 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:34.696516    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7"
Nov 01 14:38:34.714193 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:34.697912    4993 patch_prober.go:29] interesting pod/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Nov 01 14:38:34.714193 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:34.698001    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7" podUID="30998c96-4380-4e46-89db-c2f328351f71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Nov 01 14:38:35.355193 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:35.344296    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-w3-5b0dcc-kserve-57cb5cb5cf-msjth" podUID="e3ac5585-dc23-445d-89fc-b68f90dac95a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 14:38:35.694627 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:35.694517    4993 patch_prober.go:29] interesting pod/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Nov 01 14:38:35.694912 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:35.694866    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7" podUID="30998c96-4380-4e46-89db-c2f328351f71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Nov 01 14:38:36.052194 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:36.033295    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-00a787/simulator-w1-00a787-kserve-5bdcf4b567-4xsn7" podUID="6b8fc4b9-5b38-4d33-8f56-c8913e7e403f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 14:38:36.166195 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:36.165518    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-00a787/distinct-2-w1-00a787-kserve-56f5f9dc7d-bhxxr" podUID="c0180369-b893-47c4-a683-ad9b6da82101" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 14:38:36.192340 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:36.192290    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-007270/simulator-w2-007270-kserve-5bf8844f79-c7v9v" podUID="2ae81a81-e763-40ac-8f5e-4bd38f9a3e68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 14:38:36.287346 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:36.287295    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f8e2c/simulator-w0-1f8e2c-kserve-755bf645d-zf8t4" podUID="bbc61e12-d326-4d31-be81-88fb641337c7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 14:38:36.316397 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:36.316291    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f8e2c/distinct-w0-1f8e2c-kserve-6657447954-rpml6" podUID="39e0e8e2-87d8-4332-bb5d-86f3849861be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Nov 01 14:38:36.352146 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:36.348976    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-00a787/distinct-w1-00a787-kserve-dbd65bf4f-lbcwn" podUID="e1c97a57-0e89-4370-bc30-c75368d340f9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 14:38:36.352146 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:36.350096    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-w3-5b0dcc-kserve-57cb5cb5cf-msjth" podUID="e3ac5585-dc23-445d-89fc-b68f90dac95a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 14:38:36.352146 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:36.350538    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-w3-5b0dcc-kserve-57cb5cb5cf-msjth"
Nov 01 14:38:36.361439 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:36.361401    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536a87/premium-w4-536a87-kserve-74c6fdbc8-sf89f" podUID="8a720a76-488a-4a65-98ad-31ca84fa7842" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Nov 01 14:38:36.546329 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:36.546275    4993 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1f8e2c/simulator-w0-1f8e2c-kserve-755bf645d-zf8t4" podUID="bbc61e12-d326-4d31-be81-88fb641337c7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 14:38:36.645616 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:36.645528    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-w3-5b0dcc-kserve-57cb5cb5cf-msjth"
Nov 01 14:38:36.712388 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:36.712350    4993 patch_prober.go:29] interesting pod/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Nov 01 14:38:36.712580 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:36.712420    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7" podUID="30998c96-4380-4e46-89db-c2f328351f71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Nov 01 14:38:37.693592 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:37.693553    4993 patch_prober.go:29] interesting pod/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Nov 01 14:38:37.693970 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:37.693633    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7" podUID="30998c96-4380-4e46-89db-c2f328351f71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Nov 01 14:38:38.693549 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:38.693515    4993 patch_prober.go:29] interesting pod/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Nov 01 14:38:38.693747 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:38.693587    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7" podUID="30998c96-4380-4e46-89db-c2f328351f71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Nov 01 14:38:39.693412 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:39.693375    4993 patch_prober.go:29] interesting pod/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Nov 01 14:38:39.693603 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:39.693441    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7" podUID="30998c96-4380-4e46-89db-c2f328351f71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Nov 01 14:38:40.693488 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:40.693441    4993 patch_prober.go:29] interesting pod/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Nov 01 14:38:40.694020 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:40.693503    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7" podUID="30998c96-4380-4e46-89db-c2f328351f71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Nov 01 14:38:41.170343 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:41.170311    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9b0827c-65bb9b948d-6bbgc"
Nov 01 14:38:41.743002 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:41.742948    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7"
Nov 01 14:38:41.743597 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:41.743406    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7"
Nov 01 14:38:41.744567 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:41.744545    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7"
Nov 01 14:38:54.294016 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:54.293973    4993 generic.go:356] "Generic (PLEG): container finished" podID="e6e13755-98b2-495b-9bd0-a1dd39b880df" containerID="6289ebdc14f802c76d2475c3cb7d75bf28eeb140b2bed8c5eec02969bf759bc6" exitCode=0
Nov 01 14:38:54.294706 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:54.294596    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-5h4tl-w5zwd" event={"ID":"e6e13755-98b2-495b-9bd0-a1dd39b880df","Type":"ContainerDied","Data":"6289ebdc14f802c76d2475c3cb7d75bf28eeb140b2bed8c5eec02969bf759bc6"}
Nov 01 14:38:55.458040 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:55.458011    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-5h4tl-w5zwd"
Nov 01 14:38:55.549699 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:55.549612    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6e13755-98b2-495b-9bd0-a1dd39b880df-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6e13755-98b2-495b-9bd0-a1dd39b880df-trusted-ca-bundle\") pod \"e6e13755-98b2-495b-9bd0-a1dd39b880df\" (UID: \"e6e13755-98b2-495b-9bd0-a1dd39b880df\") "
Nov 01 14:38:55.549699 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:55.549703    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6e13755-98b2-495b-9bd0-a1dd39b880df-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e6e13755-98b2-495b-9bd0-a1dd39b880df-archives-path\") pod \"e6e13755-98b2-495b-9bd0-a1dd39b880df\" (UID: \"e6e13755-98b2-495b-9bd0-a1dd39b880df\") "
Nov 01 14:38:55.549992 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:55.549880    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6e13755-98b2-495b-9bd0-a1dd39b880df-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6e13755-98b2-495b-9bd0-a1dd39b880df-service-ca-bundle\") pod \"e6e13755-98b2-495b-9bd0-a1dd39b880df\" (UID: \"e6e13755-98b2-495b-9bd0-a1dd39b880df\") "
Nov 01 14:38:55.549992 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:55.549954    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6e13755-98b2-495b-9bd0-a1dd39b880df-kube-api-access-256jv\" (UniqueName: \"kubernetes.io/projected/e6e13755-98b2-495b-9bd0-a1dd39b880df-kube-api-access-256jv\") pod \"e6e13755-98b2-495b-9bd0-a1dd39b880df\" (UID: \"e6e13755-98b2-495b-9bd0-a1dd39b880df\") "
Nov 01 14:38:55.551072 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:55.551042    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6e13755-98b2-495b-9bd0-a1dd39b880df-archives-path" pod "e6e13755-98b2-495b-9bd0-a1dd39b880df" (UID: "e6e13755-98b2-495b-9bd0-a1dd39b880df"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:38:55.551442 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:55.551409    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6e13755-98b2-495b-9bd0-a1dd39b880df-service-ca-bundle" pod "e6e13755-98b2-495b-9bd0-a1dd39b880df" (UID: "e6e13755-98b2-495b-9bd0-a1dd39b880df"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:38:55.551616 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:55.551419    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6e13755-98b2-495b-9bd0-a1dd39b880df-trusted-ca-bundle" pod "e6e13755-98b2-495b-9bd0-a1dd39b880df" (UID: "e6e13755-98b2-495b-9bd0-a1dd39b880df"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:38:55.552771 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:55.552751    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6e13755-98b2-495b-9bd0-a1dd39b880df-kube-api-access-256jv" pod "e6e13755-98b2-495b-9bd0-a1dd39b880df" (UID: "e6e13755-98b2-495b-9bd0-a1dd39b880df"). InnerVolumeSpecName "kube-api-access-256jv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:38:55.652391 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:55.652353    4993 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6e13755-98b2-495b-9bd0-a1dd39b880df-trusted-ca-bundle\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:38:55.652391 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:55.652391    4993 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e6e13755-98b2-495b-9bd0-a1dd39b880df-archives-path\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:38:55.652597 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:55.652409    4993 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6e13755-98b2-495b-9bd0-a1dd39b880df-service-ca-bundle\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:38:55.652597 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:55.652431    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-256jv\" (UniqueName: \"kubernetes.io/projected/e6e13755-98b2-495b-9bd0-a1dd39b880df-kube-api-access-256jv\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:38:56.312261 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:56.312230    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-5h4tl-w5zwd"
Nov 01 14:38:56.312640 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:56.312289    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-5h4tl-w5zwd" event={"ID":"e6e13755-98b2-495b-9bd0-a1dd39b880df","Type":"ContainerDied","Data":"9ee13766b67d31e4b22299e771783004ca9dc5626034349a8ba2ecb3740bc1ec"}
Nov 01 14:38:56.312773 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:38:56.312697    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ee13766b67d31e4b22299e771783004ca9dc5626034349a8ba2ecb3740bc1ec"
Nov 01 14:38:57.500848 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:38:57.500810    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:39:27.502580 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:39:27.502530    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:39:38.522896 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:38.522859    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-c9b0827c/e2e-lifecycle-model-c9b0827c-kserve-78c9dd7bf-9p2bh"]
Nov 01 14:39:38.523427 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:38.523413    4993 cpu_manager.go:414] "containerMap: removing container" podUID="e6e13755-98b2-495b-9bd0-a1dd39b880df" containerName="insights-gathering"
Nov 01 14:39:38.523480 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:38.523428    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6e13755-98b2-495b-9bd0-a1dd39b880df" containerName="insights-gathering"
Nov 01 14:39:38.523554 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:38.523544    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6e13755-98b2-495b-9bd0-a1dd39b880df" containerName="insights-gathering"
Nov 01 14:39:38.527254 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:38.527228    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c9b0827c/e2e-lifecycle-model-c9b0827c-kserve-78c9dd7bf-9p2bh"
Nov 01 14:39:38.576021 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:38.575989    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-c9b0827c\"/\"kube-root-ca.crt\""
Nov 01 14:39:38.622041 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:38.621997    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-c9b0827c\"/\"openshift-service-ca.crt\""
Nov 01 14:39:38.622041 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:38.622039    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-c9b0827c\"/\"default-dockercfg-8l88l\""
Nov 01 14:39:38.622328 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:38.622104    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-c9b0827c\"/\"e2e-lifecycle-model-c9b0827c-kserve-self-signed-certs\""
Nov 01 14:39:38.625491 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:38.625463    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-c9b0827c/e2e-lifecycle-model-c9b0827c-kserve-78c9dd7bf-9p2bh"]
Nov 01 14:39:38.666965 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:38.666857    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57e7bcd7-adc6-49f3-a2b2-529ea5e5603a-dshm\") pod \"e2e-lifecycle-model-c9b0827c-kserve-78c9dd7bf-9p2bh\" (UID: \"57e7bcd7-adc6-49f3-a2b2-529ea5e5603a\") " pod="ai-tenant-e2e-mt-c9b0827c/e2e-lifecycle-model-c9b0827c-kserve-78c9dd7bf-9p2bh"
Nov 01 14:39:38.666965 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:38.666917    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57e7bcd7-adc6-49f3-a2b2-529ea5e5603a-model-cache\") pod \"e2e-lifecycle-model-c9b0827c-kserve-78c9dd7bf-9p2bh\" (UID: \"57e7bcd7-adc6-49f3-a2b2-529ea5e5603a\") " pod="ai-tenant-e2e-mt-c9b0827c/e2e-lifecycle-model-c9b0827c-kserve-78c9dd7bf-9p2bh"
Nov 01 14:39:38.667398 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:38.667372    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57e7bcd7-adc6-49f3-a2b2-529ea5e5603a-tls-certs\") pod \"e2e-lifecycle-model-c9b0827c-kserve-78c9dd7bf-9p2bh\" (UID: \"57e7bcd7-adc6-49f3-a2b2-529ea5e5603a\") " pod="ai-tenant-e2e-mt-c9b0827c/e2e-lifecycle-model-c9b0827c-kserve-78c9dd7bf-9p2bh"
Nov 01 14:39:38.667480 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:38.667435    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57e7bcd7-adc6-49f3-a2b2-529ea5e5603a-tmp-dir\") pod \"e2e-lifecycle-model-c9b0827c-kserve-78c9dd7bf-9p2bh\" (UID: \"57e7bcd7-adc6-49f3-a2b2-529ea5e5603a\") " pod="ai-tenant-e2e-mt-c9b0827c/e2e-lifecycle-model-c9b0827c-kserve-78c9dd7bf-9p2bh"
Nov 01 14:39:38.667480 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:38.667474    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57e7bcd7-adc6-49f3-a2b2-529ea5e5603a-home\") pod \"e2e-lifecycle-model-c9b0827c-kserve-78c9dd7bf-9p2bh\" (UID: \"57e7bcd7-adc6-49f3-a2b2-529ea5e5603a\") " pod="ai-tenant-e2e-mt-c9b0827c/e2e-lifecycle-model-c9b0827c-kserve-78c9dd7bf-9p2bh"
Nov 01 14:39:38.667585 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:38.667531    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2bf2\" (UniqueName: \"kubernetes.io/projected/57e7bcd7-adc6-49f3-a2b2-529ea5e5603a-kube-api-access-m2bf2\") pod \"e2e-lifecycle-model-c9b0827c-kserve-78c9dd7bf-9p2bh\" (UID: \"57e7bcd7-adc6-49f3-a2b2-529ea5e5603a\") " pod="ai-tenant-e2e-mt-c9b0827c/e2e-lifecycle-model-c9b0827c-kserve-78c9dd7bf-9p2bh"
Nov 01 14:39:38.768956 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:38.768916    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57e7bcd7-adc6-49f3-a2b2-529ea5e5603a-dshm\") pod \"e2e-lifecycle-model-c9b0827c-kserve-78c9dd7bf-9p2bh\" (UID: \"57e7bcd7-adc6-49f3-a2b2-529ea5e5603a\") " pod="ai-tenant-e2e-mt-c9b0827c/e2e-lifecycle-model-c9b0827c-kserve-78c9dd7bf-9p2bh"
Nov 01 14:39:38.768956 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:38.768960    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57e7bcd7-adc6-49f3-a2b2-529ea5e5603a-model-cache\") pod \"e2e-lifecycle-model-c9b0827c-kserve-78c9dd7bf-9p2bh\" (UID: \"57e7bcd7-adc6-49f3-a2b2-529ea5e5603a\") " pod="ai-tenant-e2e-mt-c9b0827c/e2e-lifecycle-model-c9b0827c-kserve-78c9dd7bf-9p2bh"
Nov 01 14:39:38.769224 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:38.768978    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57e7bcd7-adc6-49f3-a2b2-529ea5e5603a-tls-certs\") pod \"e2e-lifecycle-model-c9b0827c-kserve-78c9dd7bf-9p2bh\" (UID: \"57e7bcd7-adc6-49f3-a2b2-529ea5e5603a\") " pod="ai-tenant-e2e-mt-c9b0827c/e2e-lifecycle-model-c9b0827c-kserve-78c9dd7bf-9p2bh"
Nov 01 14:39:38.769224 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:38.769097    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57e7bcd7-adc6-49f3-a2b2-529ea5e5603a-tmp-dir\") pod \"e2e-lifecycle-model-c9b0827c-kserve-78c9dd7bf-9p2bh\" (UID: \"57e7bcd7-adc6-49f3-a2b2-529ea5e5603a\") " pod="ai-tenant-e2e-mt-c9b0827c/e2e-lifecycle-model-c9b0827c-kserve-78c9dd7bf-9p2bh"
Nov 01 14:39:38.769224 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:38.769139    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57e7bcd7-adc6-49f3-a2b2-529ea5e5603a-home\") pod \"e2e-lifecycle-model-c9b0827c-kserve-78c9dd7bf-9p2bh\" (UID: \"57e7bcd7-adc6-49f3-a2b2-529ea5e5603a\") " pod="ai-tenant-e2e-mt-c9b0827c/e2e-lifecycle-model-c9b0827c-kserve-78c9dd7bf-9p2bh"
Nov 01 14:39:38.769386 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:38.769227    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2bf2\" (UniqueName: \"kubernetes.io/projected/57e7bcd7-adc6-49f3-a2b2-529ea5e5603a-kube-api-access-m2bf2\") pod \"e2e-lifecycle-model-c9b0827c-kserve-78c9dd7bf-9p2bh\" (UID: \"57e7bcd7-adc6-49f3-a2b2-529ea5e5603a\") " pod="ai-tenant-e2e-mt-c9b0827c/e2e-lifecycle-model-c9b0827c-kserve-78c9dd7bf-9p2bh"
Nov 01 14:39:38.769479 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:38.769452    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57e7bcd7-adc6-49f3-a2b2-529ea5e5603a-model-cache\") pod \"e2e-lifecycle-model-c9b0827c-kserve-78c9dd7bf-9p2bh\" (UID: \"57e7bcd7-adc6-49f3-a2b2-529ea5e5603a\") " pod="ai-tenant-e2e-mt-c9b0827c/e2e-lifecycle-model-c9b0827c-kserve-78c9dd7bf-9p2bh"
Nov 01 14:39:38.769890 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:38.769864    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57e7bcd7-adc6-49f3-a2b2-529ea5e5603a-tmp-dir\") pod \"e2e-lifecycle-model-c9b0827c-kserve-78c9dd7bf-9p2bh\" (UID: \"57e7bcd7-adc6-49f3-a2b2-529ea5e5603a\") " pod="ai-tenant-e2e-mt-c9b0827c/e2e-lifecycle-model-c9b0827c-kserve-78c9dd7bf-9p2bh"
Nov 01 14:39:38.770015 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:38.769895    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57e7bcd7-adc6-49f3-a2b2-529ea5e5603a-home\") pod \"e2e-lifecycle-model-c9b0827c-kserve-78c9dd7bf-9p2bh\" (UID: \"57e7bcd7-adc6-49f3-a2b2-529ea5e5603a\") " pod="ai-tenant-e2e-mt-c9b0827c/e2e-lifecycle-model-c9b0827c-kserve-78c9dd7bf-9p2bh"
Nov 01 14:39:38.771534 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:38.771510    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57e7bcd7-adc6-49f3-a2b2-529ea5e5603a-dshm\") pod \"e2e-lifecycle-model-c9b0827c-kserve-78c9dd7bf-9p2bh\" (UID: \"57e7bcd7-adc6-49f3-a2b2-529ea5e5603a\") " pod="ai-tenant-e2e-mt-c9b0827c/e2e-lifecycle-model-c9b0827c-kserve-78c9dd7bf-9p2bh"
Nov 01 14:39:38.772084 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:38.772063    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57e7bcd7-adc6-49f3-a2b2-529ea5e5603a-tls-certs\") pod \"e2e-lifecycle-model-c9b0827c-kserve-78c9dd7bf-9p2bh\" (UID: \"57e7bcd7-adc6-49f3-a2b2-529ea5e5603a\") " pod="ai-tenant-e2e-mt-c9b0827c/e2e-lifecycle-model-c9b0827c-kserve-78c9dd7bf-9p2bh"
Nov 01 14:39:38.822373 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:38.822294    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2bf2\" (UniqueName: \"kubernetes.io/projected/57e7bcd7-adc6-49f3-a2b2-529ea5e5603a-kube-api-access-m2bf2\") pod \"e2e-lifecycle-model-c9b0827c-kserve-78c9dd7bf-9p2bh\" (UID: \"57e7bcd7-adc6-49f3-a2b2-529ea5e5603a\") " pod="ai-tenant-e2e-mt-c9b0827c/e2e-lifecycle-model-c9b0827c-kserve-78c9dd7bf-9p2bh"
Nov 01 14:39:38.838703 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:38.838673    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c9b0827c/e2e-lifecycle-model-c9b0827c-kserve-78c9dd7bf-9p2bh"
Nov 01 14:39:39.159005 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:39.158915    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-466a6e-7b74b87c87-f9qvk"]
Nov 01 14:39:39.164197 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:39.164159    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466a6e-7b74b87c87-f9qvk"
Nov 01 14:39:39.212756 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:39.212722    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-466a6e-serving-cert\""
Nov 01 14:39:39.222405 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:39.222375    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-466a6e-7b74b87c87-f9qvk"]
Nov 01 14:39:39.271063 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:39.269847    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-c9b0827c/e2e-lifecycle-model-c9b0827c-kserve-78c9dd7bf-9p2bh"]
Nov 01 14:39:39.276890 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:39:39.276859    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod57e7bcd7_adc6_49f3_a2b2_529ea5e5603a.slice/crio-01081914ca10d2122727d26732072fe94ec0f8b438da54e8e518493ba00b5c5b WatchSource:0}: Error finding container 01081914ca10d2122727d26732072fe94ec0f8b438da54e8e518493ba00b5c5b: Status 404 returned error can't find the container with id 01081914ca10d2122727d26732072fe94ec0f8b438da54e8e518493ba00b5c5b
Nov 01 14:39:39.295355 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:39.295318    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a56cf6f-3dce-46b9-8b71-757373386d7f-maas-api-tls\") pod \"maas-api-e2e-api-a-466a6e-7b74b87c87-f9qvk\" (UID: \"0a56cf6f-3dce-46b9-8b71-757373386d7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466a6e-7b74b87c87-f9qvk"
Nov 01 14:39:39.295512 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:39.295404    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f92l4\" (UniqueName: \"kubernetes.io/projected/0a56cf6f-3dce-46b9-8b71-757373386d7f-kube-api-access-f92l4\") pod \"maas-api-e2e-api-a-466a6e-7b74b87c87-f9qvk\" (UID: \"0a56cf6f-3dce-46b9-8b71-757373386d7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466a6e-7b74b87c87-f9qvk"
Nov 01 14:39:39.295512 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:39.295465    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a56cf6f-3dce-46b9-8b71-757373386d7f-metrics-certs\") pod \"maas-api-e2e-api-a-466a6e-7b74b87c87-f9qvk\" (UID: \"0a56cf6f-3dce-46b9-8b71-757373386d7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466a6e-7b74b87c87-f9qvk"
Nov 01 14:39:39.295650 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:39.295549    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a56cf6f-3dce-46b9-8b71-757373386d7f-openshift-service-ca\") pod \"maas-api-e2e-api-a-466a6e-7b74b87c87-f9qvk\" (UID: \"0a56cf6f-3dce-46b9-8b71-757373386d7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466a6e-7b74b87c87-f9qvk"
Nov 01 14:39:39.396675 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:39.396636    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a56cf6f-3dce-46b9-8b71-757373386d7f-maas-api-tls\") pod \"maas-api-e2e-api-a-466a6e-7b74b87c87-f9qvk\" (UID: \"0a56cf6f-3dce-46b9-8b71-757373386d7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466a6e-7b74b87c87-f9qvk"
Nov 01 14:39:39.396951 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:39.396932    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f92l4\" (UniqueName: \"kubernetes.io/projected/0a56cf6f-3dce-46b9-8b71-757373386d7f-kube-api-access-f92l4\") pod \"maas-api-e2e-api-a-466a6e-7b74b87c87-f9qvk\" (UID: \"0a56cf6f-3dce-46b9-8b71-757373386d7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466a6e-7b74b87c87-f9qvk"
Nov 01 14:39:39.397104 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:39.397084    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a56cf6f-3dce-46b9-8b71-757373386d7f-metrics-certs\") pod \"maas-api-e2e-api-a-466a6e-7b74b87c87-f9qvk\" (UID: \"0a56cf6f-3dce-46b9-8b71-757373386d7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466a6e-7b74b87c87-f9qvk"
Nov 01 14:39:39.397377 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:39.397360    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a56cf6f-3dce-46b9-8b71-757373386d7f-openshift-service-ca\") pod \"maas-api-e2e-api-a-466a6e-7b74b87c87-f9qvk\" (UID: \"0a56cf6f-3dce-46b9-8b71-757373386d7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466a6e-7b74b87c87-f9qvk"
Nov 01 14:39:39.398460 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:39.398432    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a56cf6f-3dce-46b9-8b71-757373386d7f-openshift-service-ca\") pod \"maas-api-e2e-api-a-466a6e-7b74b87c87-f9qvk\" (UID: \"0a56cf6f-3dce-46b9-8b71-757373386d7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466a6e-7b74b87c87-f9qvk"
Nov 01 14:39:39.398753 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:39:39.398730    4993 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-466a6e-serving-cert: secret "maas-api-e2e-api-a-466a6e-serving-cert" not found
Nov 01 14:39:39.399896 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:39:39.399880    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a56cf6f-3dce-46b9-8b71-757373386d7f-maas-api-tls podName:0a56cf6f-3dce-46b9-8b71-757373386d7f nodeName:}" failed. No retries permitted until 2026-11-01 14:39:39.899858301 +0000 UTC m=+1367.229721033 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0a56cf6f-3dce-46b9-8b71-757373386d7f-maas-api-tls") pod "maas-api-e2e-api-a-466a6e-7b74b87c87-f9qvk" (UID: "0a56cf6f-3dce-46b9-8b71-757373386d7f") : secret "maas-api-e2e-api-a-466a6e-serving-cert" not found
Nov 01 14:39:39.403364 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:39.403293    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a56cf6f-3dce-46b9-8b71-757373386d7f-metrics-certs\") pod \"maas-api-e2e-api-a-466a6e-7b74b87c87-f9qvk\" (UID: \"0a56cf6f-3dce-46b9-8b71-757373386d7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466a6e-7b74b87c87-f9qvk"
Nov 01 14:39:39.452736 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:39.452660    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f92l4\" (UniqueName: \"kubernetes.io/projected/0a56cf6f-3dce-46b9-8b71-757373386d7f-kube-api-access-f92l4\") pod \"maas-api-e2e-api-a-466a6e-7b74b87c87-f9qvk\" (UID: \"0a56cf6f-3dce-46b9-8b71-757373386d7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466a6e-7b74b87c87-f9qvk"
Nov 01 14:39:39.675537 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:39.675497    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c9b0827c/e2e-lifecycle-model-c9b0827c-kserve-78c9dd7bf-9p2bh" event={"ID":"57e7bcd7-adc6-49f3-a2b2-529ea5e5603a","Type":"ContainerStarted","Data":"2005cf74e5d21957527e37649d7ad4fc7e6ea8cb247e7e4d077c25eb2edb8141"}
Nov 01 14:39:39.675990 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:39.675548    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c9b0827c/e2e-lifecycle-model-c9b0827c-kserve-78c9dd7bf-9p2bh" event={"ID":"57e7bcd7-adc6-49f3-a2b2-529ea5e5603a","Type":"ContainerStarted","Data":"01081914ca10d2122727d26732072fe94ec0f8b438da54e8e518493ba00b5c5b"}
Nov 01 14:39:39.782847 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:39.782791    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-c9b0827c/e2e-lifecycle-model-c9b0827c-kserve-78c9dd7bf-9p2bh" podStartSLOduration=1.782776901 podStartE2EDuration="1.782776901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:39:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:39:39.780852503 +0000 UTC m=+1367.110715235" watchObservedRunningTime="2026-11-01 14:39:39.782776901 +0000 UTC m=+1367.112639639"
Nov 01 14:39:39.839209 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:39.839148    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-c9b0827c/e2e-lifecycle-model-c9b0827c-kserve-78c9dd7bf-9p2bh"
Nov 01 14:39:39.903845 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:39.903798    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a56cf6f-3dce-46b9-8b71-757373386d7f-maas-api-tls\") pod \"maas-api-e2e-api-a-466a6e-7b74b87c87-f9qvk\" (UID: \"0a56cf6f-3dce-46b9-8b71-757373386d7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466a6e-7b74b87c87-f9qvk"
Nov 01 14:39:39.906125 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:39.906100    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a56cf6f-3dce-46b9-8b71-757373386d7f-maas-api-tls\") pod \"maas-api-e2e-api-a-466a6e-7b74b87c87-f9qvk\" (UID: \"0a56cf6f-3dce-46b9-8b71-757373386d7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466a6e-7b74b87c87-f9qvk"
Nov 01 14:39:40.082520 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:40.082431    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466a6e-7b74b87c87-f9qvk"
Nov 01 14:39:40.337538 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:40.337457    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wcdb6"]
Nov 01 14:39:40.341705 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:40.341683    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wcdb6"
Nov 01 14:39:40.440500 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:40.440470    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wcdb6"]
Nov 01 14:39:40.481018 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:40.480578    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-84ded4-openshift-default-dockercfg-td8dn\""
Nov 01 14:39:40.485974 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:40.485683    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-466a6e-7b74b87c87-f9qvk"]
Nov 01 14:39:40.487743 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:39:40.487713    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a56cf6f_3dce_46b9_8b71_757373386d7f.slice/crio-3fa3d1bd5877d87ed3ab7183f44dfc36d6cd589a6e6728b1b64a8e5409f4f658 WatchSource:0}: Error finding container 3fa3d1bd5877d87ed3ab7183f44dfc36d6cd589a6e6728b1b64a8e5409f4f658: Status 404 returned error can't find the container with id 3fa3d1bd5877d87ed3ab7183f44dfc36d6cd589a6e6728b1b64a8e5409f4f658
Nov 01 14:39:40.511730 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:40.511697    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65c82e9c-1c36-4fce-9ba2-fa15d5704683-workload-socket\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-wcdb6\" (UID: \"65c82e9c-1c36-4fce-9ba2-fa15d5704683\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wcdb6"
Nov 01 14:39:40.511847 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:40.511764    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65c82e9c-1c36-4fce-9ba2-fa15d5704683-istio-envoy\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-wcdb6\" (UID: \"65c82e9c-1c36-4fce-9ba2-fa15d5704683\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wcdb6"
Nov 01 14:39:40.511924 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:40.511900    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65c82e9c-1c36-4fce-9ba2-fa15d5704683-credential-socket\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-wcdb6\" (UID: \"65c82e9c-1c36-4fce-9ba2-fa15d5704683\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wcdb6"
Nov 01 14:39:40.511961 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:40.511933    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65c82e9c-1c36-4fce-9ba2-fa15d5704683-istio-podinfo\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-wcdb6\" (UID: \"65c82e9c-1c36-4fce-9ba2-fa15d5704683\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wcdb6"
Nov 01 14:39:40.511994 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:40.511963    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65c82e9c-1c36-4fce-9ba2-fa15d5704683-istio-data\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-wcdb6\" (UID: \"65c82e9c-1c36-4fce-9ba2-fa15d5704683\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wcdb6"
Nov 01 14:39:40.512035 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:40.512021    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65c82e9c-1c36-4fce-9ba2-fa15d5704683-istio-token\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-wcdb6\" (UID: \"65c82e9c-1c36-4fce-9ba2-fa15d5704683\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wcdb6"
Nov 01 14:39:40.512139 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:40.512124    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88qlk\" (UniqueName: \"kubernetes.io/projected/65c82e9c-1c36-4fce-9ba2-fa15d5704683-kube-api-access-88qlk\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-wcdb6\" (UID: \"65c82e9c-1c36-4fce-9ba2-fa15d5704683\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wcdb6"
Nov 01 14:39:40.512219 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:40.512198    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65c82e9c-1c36-4fce-9ba2-fa15d5704683-istiod-ca-cert\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-wcdb6\" (UID: \"65c82e9c-1c36-4fce-9ba2-fa15d5704683\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wcdb6"
Nov 01 14:39:40.512290 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:40.512275    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65c82e9c-1c36-4fce-9ba2-fa15d5704683-workload-certs\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-wcdb6\" (UID: \"65c82e9c-1c36-4fce-9ba2-fa15d5704683\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wcdb6"
Nov 01 14:39:40.614253 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:40.614135    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65c82e9c-1c36-4fce-9ba2-fa15d5704683-workload-certs\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-wcdb6\" (UID: \"65c82e9c-1c36-4fce-9ba2-fa15d5704683\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wcdb6"
Nov 01 14:39:40.614253 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:40.614222    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65c82e9c-1c36-4fce-9ba2-fa15d5704683-workload-certs\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-wcdb6\" (UID: \"65c82e9c-1c36-4fce-9ba2-fa15d5704683\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wcdb6"
Nov 01 14:39:40.614476 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:40.614347    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65c82e9c-1c36-4fce-9ba2-fa15d5704683-workload-socket\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-wcdb6\" (UID: \"65c82e9c-1c36-4fce-9ba2-fa15d5704683\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wcdb6"
Nov 01 14:39:40.614476 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:40.614434    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65c82e9c-1c36-4fce-9ba2-fa15d5704683-istio-envoy\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-wcdb6\" (UID: \"65c82e9c-1c36-4fce-9ba2-fa15d5704683\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wcdb6"
Nov 01 14:39:40.614583 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:40.614560    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65c82e9c-1c36-4fce-9ba2-fa15d5704683-credential-socket\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-wcdb6\" (UID: \"65c82e9c-1c36-4fce-9ba2-fa15d5704683\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wcdb6"
Nov 01 14:39:40.614647 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:40.614587    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65c82e9c-1c36-4fce-9ba2-fa15d5704683-istio-podinfo\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-wcdb6\" (UID: \"65c82e9c-1c36-4fce-9ba2-fa15d5704683\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wcdb6"
Nov 01 14:39:40.614647 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:40.614627    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65c82e9c-1c36-4fce-9ba2-fa15d5704683-istio-data\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-wcdb6\" (UID: \"65c82e9c-1c36-4fce-9ba2-fa15d5704683\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wcdb6"
Nov 01 14:39:40.614758 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:40.614651    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65c82e9c-1c36-4fce-9ba2-fa15d5704683-istio-token\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-wcdb6\" (UID: \"65c82e9c-1c36-4fce-9ba2-fa15d5704683\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wcdb6"
Nov 01 14:39:40.614758 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:40.614687    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88qlk\" (UniqueName: \"kubernetes.io/projected/65c82e9c-1c36-4fce-9ba2-fa15d5704683-kube-api-access-88qlk\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-wcdb6\" (UID: \"65c82e9c-1c36-4fce-9ba2-fa15d5704683\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wcdb6"
Nov 01 14:39:40.614758 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:40.614719    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65c82e9c-1c36-4fce-9ba2-fa15d5704683-istiod-ca-cert\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-wcdb6\" (UID: \"65c82e9c-1c36-4fce-9ba2-fa15d5704683\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wcdb6"
Nov 01 14:39:40.615528 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:40.615494    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65c82e9c-1c36-4fce-9ba2-fa15d5704683-istiod-ca-cert\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-wcdb6\" (UID: \"65c82e9c-1c36-4fce-9ba2-fa15d5704683\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wcdb6"
Nov 01 14:39:40.615788 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:40.615764    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65c82e9c-1c36-4fce-9ba2-fa15d5704683-workload-socket\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-wcdb6\" (UID: \"65c82e9c-1c36-4fce-9ba2-fa15d5704683\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wcdb6"
Nov 01 14:39:40.616383 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:40.616336    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65c82e9c-1c36-4fce-9ba2-fa15d5704683-istio-data\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-wcdb6\" (UID: \"65c82e9c-1c36-4fce-9ba2-fa15d5704683\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wcdb6"
Nov 01 14:39:40.616486 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:40.616406    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65c82e9c-1c36-4fce-9ba2-fa15d5704683-credential-socket\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-wcdb6\" (UID: \"65c82e9c-1c36-4fce-9ba2-fa15d5704683\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wcdb6"
Nov 01 14:39:40.620058 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:40.619085    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65c82e9c-1c36-4fce-9ba2-fa15d5704683-istio-envoy\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-wcdb6\" (UID: \"65c82e9c-1c36-4fce-9ba2-fa15d5704683\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wcdb6"
Nov 01 14:39:40.620783 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:40.620737    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65c82e9c-1c36-4fce-9ba2-fa15d5704683-istio-podinfo\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-wcdb6\" (UID: \"65c82e9c-1c36-4fce-9ba2-fa15d5704683\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wcdb6"
Nov 01 14:39:40.669969 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:40.669933    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88qlk\" (UniqueName: \"kubernetes.io/projected/65c82e9c-1c36-4fce-9ba2-fa15d5704683-kube-api-access-88qlk\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-wcdb6\" (UID: \"65c82e9c-1c36-4fce-9ba2-fa15d5704683\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wcdb6"
Nov 01 14:39:40.670151 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:40.670081    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65c82e9c-1c36-4fce-9ba2-fa15d5704683-istio-token\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-wcdb6\" (UID: \"65c82e9c-1c36-4fce-9ba2-fa15d5704683\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wcdb6"
Nov 01 14:39:40.685131 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:40.685097    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466a6e-7b74b87c87-f9qvk" event={"ID":"0a56cf6f-3dce-46b9-8b71-757373386d7f","Type":"ContainerStarted","Data":"3fa3d1bd5877d87ed3ab7183f44dfc36d6cd589a6e6728b1b64a8e5409f4f658"}
Nov 01 14:39:40.962866 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:40.962770    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wcdb6"
Nov 01 14:39:41.394560 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:41.394216    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wcdb6"]
Nov 01 14:39:41.397654 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:39:41.397594    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65c82e9c_1c36_4fce_9ba2_fa15d5704683.slice/crio-1ffdead22ca2dff99cd7a18402345c148de4cc2cf19d2e1ac75db50de5ef0101 WatchSource:0}: Error finding container 1ffdead22ca2dff99cd7a18402345c148de4cc2cf19d2e1ac75db50de5ef0101: Status 404 returned error can't find the container with id 1ffdead22ca2dff99cd7a18402345c148de4cc2cf19d2e1ac75db50de5ef0101
Nov 01 14:39:41.401115 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:41.401069    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 14:39:41.401223 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:41.401159    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 14:39:41.401223 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:41.401218    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 14:39:41.694548 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:41.694445    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466a6e-7b74b87c87-f9qvk" event={"ID":"0a56cf6f-3dce-46b9-8b71-757373386d7f","Type":"ContainerStarted","Data":"e8872ab7a79e5fb74267da19c37f18f2e3371b4afdea31844563c9d1da7e9b7d"}
Nov 01 14:39:41.694548 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:41.694519    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466a6e-7b74b87c87-f9qvk"
Nov 01 14:39:41.696425 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:41.696391    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wcdb6" event={"ID":"65c82e9c-1c36-4fce-9ba2-fa15d5704683","Type":"ContainerStarted","Data":"4685497fd885fcae63e46696bdabc3b0ac25ad06cab4e55c78cf1c7a8bd4e7d9"}
Nov 01 14:39:41.696537 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:41.696452    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wcdb6" event={"ID":"65c82e9c-1c36-4fce-9ba2-fa15d5704683","Type":"ContainerStarted","Data":"1ffdead22ca2dff99cd7a18402345c148de4cc2cf19d2e1ac75db50de5ef0101"}
Nov 01 14:39:41.802664 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:41.802605    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466a6e-7b74b87c87-f9qvk" podStartSLOduration=1.869551039 podStartE2EDuration="2.802585672s" totalImagesPullingTime="933.034633ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:39:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:39:41.79970068 +0000 UTC m=+1369.129563418" watchObservedRunningTime="2026-11-01 14:39:41.802585672 +0000 UTC m=+1369.132448409"
Nov 01 14:39:41.907891 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:41.907741    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wcdb6" podStartSLOduration=1.907724186 podStartE2EDuration="1.907724186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:39:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:39:41.907473106 +0000 UTC m=+1369.237335843" watchObservedRunningTime="2026-11-01 14:39:41.907724186 +0000 UTC m=+1369.237586922"
Nov 01 14:39:41.967995 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:41.965606    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wcdb6"
Nov 01 14:39:42.977462 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:42.977399    4993 patch_prober.go:29] interesting pod/e2e-api-a-84ded4-openshift-default-d9cbb4847-wcdb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Nov 01 14:39:42.978025 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:42.977540    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wcdb6" podUID="65c82e9c-1c36-4fce-9ba2-fa15d5704683" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Nov 01 14:39:43.973193 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:43.963661    4993 patch_prober.go:29] interesting pod/e2e-api-a-84ded4-openshift-default-d9cbb4847-wcdb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Nov 01 14:39:43.973193 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:43.963727    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wcdb6" podUID="65c82e9c-1c36-4fce-9ba2-fa15d5704683" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Nov 01 14:39:44.976195 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:44.975583    4993 patch_prober.go:29] interesting pod/e2e-api-a-84ded4-openshift-default-d9cbb4847-wcdb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Nov 01 14:39:44.976195 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:44.975655    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wcdb6" podUID="65c82e9c-1c36-4fce-9ba2-fa15d5704683" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Nov 01 14:39:45.965593 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:45.965555    4993 patch_prober.go:29] interesting pod/e2e-api-a-84ded4-openshift-default-d9cbb4847-wcdb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Nov 01 14:39:45.966611 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:45.966569    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wcdb6" podUID="65c82e9c-1c36-4fce-9ba2-fa15d5704683" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Nov 01 14:39:46.968194 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:46.965618    4993 patch_prober.go:29] interesting pod/e2e-api-a-84ded4-openshift-default-d9cbb4847-wcdb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Nov 01 14:39:46.968194 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:46.965680    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wcdb6" podUID="65c82e9c-1c36-4fce-9ba2-fa15d5704683" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Nov 01 14:39:47.708973 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:47.708941    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466a6e-7b74b87c87-f9qvk"
Nov 01 14:39:47.968306 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:47.968229    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wcdb6"
Nov 01 14:39:47.968732 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:47.968712    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wcdb6"
Nov 01 14:39:47.969459 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:47.969437    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wcdb6"
Nov 01 14:39:48.839042 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:48.839005    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-c9b0827c/e2e-lifecycle-model-c9b0827c-kserve-78c9dd7bf-9p2bh"
Nov 01 14:39:48.853092 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:48.853065    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-c9b0827c/e2e-lifecycle-model-c9b0827c-kserve-78c9dd7bf-9p2bh"
Nov 01 14:39:49.756655 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:39:49.756619    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-c9b0827c/e2e-lifecycle-model-c9b0827c-kserve-78c9dd7bf-9p2bh"
Nov 01 14:39:57.502735 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:39:57.502675    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:40:13.949083 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:13.949050    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-536a87/distinct-w4-536a87-kserve-6bdb5c8897-lmcls"]
Nov 01 14:40:13.952965 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:13.952945    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-536a87/distinct-w4-536a87-kserve-6bdb5c8897-lmcls"
Nov 01 14:40:14.047629 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:14.047589    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-536a87\"/\"distinct-w4-536a87-kserve-self-signed-certs\""
Nov 01 14:40:14.053184 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:14.053142    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-536a87/distinct-w4-536a87-kserve-6bdb5c8897-lmcls"]
Nov 01 14:40:14.131980 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:14.131930    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27dvp\" (UniqueName: \"kubernetes.io/projected/3cdf53f6-df5a-4cdc-b751-e26c71484700-kube-api-access-27dvp\") pod \"distinct-w4-536a87-kserve-6bdb5c8897-lmcls\" (UID: \"3cdf53f6-df5a-4cdc-b751-e26c71484700\") " pod="e2e-models-e2e-worker-w4-536a87/distinct-w4-536a87-kserve-6bdb5c8897-lmcls"
Nov 01 14:40:14.132308 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:14.132276    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cdf53f6-df5a-4cdc-b751-e26c71484700-tmp-dir\") pod \"distinct-w4-536a87-kserve-6bdb5c8897-lmcls\" (UID: \"3cdf53f6-df5a-4cdc-b751-e26c71484700\") " pod="e2e-models-e2e-worker-w4-536a87/distinct-w4-536a87-kserve-6bdb5c8897-lmcls"
Nov 01 14:40:14.132419 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:14.132356    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cdf53f6-df5a-4cdc-b751-e26c71484700-model-cache\") pod \"distinct-w4-536a87-kserve-6bdb5c8897-lmcls\" (UID: \"3cdf53f6-df5a-4cdc-b751-e26c71484700\") " pod="e2e-models-e2e-worker-w4-536a87/distinct-w4-536a87-kserve-6bdb5c8897-lmcls"
Nov 01 14:40:14.132467 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:14.132459    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cdf53f6-df5a-4cdc-b751-e26c71484700-dshm\") pod \"distinct-w4-536a87-kserve-6bdb5c8897-lmcls\" (UID: \"3cdf53f6-df5a-4cdc-b751-e26c71484700\") " pod="e2e-models-e2e-worker-w4-536a87/distinct-w4-536a87-kserve-6bdb5c8897-lmcls"
Nov 01 14:40:14.132521 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:14.132508    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3cdf53f6-df5a-4cdc-b751-e26c71484700-home\") pod \"distinct-w4-536a87-kserve-6bdb5c8897-lmcls\" (UID: \"3cdf53f6-df5a-4cdc-b751-e26c71484700\") " pod="e2e-models-e2e-worker-w4-536a87/distinct-w4-536a87-kserve-6bdb5c8897-lmcls"
Nov 01 14:40:14.132572 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:14.132535    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cdf53f6-df5a-4cdc-b751-e26c71484700-tls-certs\") pod \"distinct-w4-536a87-kserve-6bdb5c8897-lmcls\" (UID: \"3cdf53f6-df5a-4cdc-b751-e26c71484700\") " pod="e2e-models-e2e-worker-w4-536a87/distinct-w4-536a87-kserve-6bdb5c8897-lmcls"
Nov 01 14:40:14.233714 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:14.233682    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cdf53f6-df5a-4cdc-b751-e26c71484700-model-cache\") pod \"distinct-w4-536a87-kserve-6bdb5c8897-lmcls\" (UID: \"3cdf53f6-df5a-4cdc-b751-e26c71484700\") " pod="e2e-models-e2e-worker-w4-536a87/distinct-w4-536a87-kserve-6bdb5c8897-lmcls"
Nov 01 14:40:14.233890 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:14.233764    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cdf53f6-df5a-4cdc-b751-e26c71484700-dshm\") pod \"distinct-w4-536a87-kserve-6bdb5c8897-lmcls\" (UID: \"3cdf53f6-df5a-4cdc-b751-e26c71484700\") " pod="e2e-models-e2e-worker-w4-536a87/distinct-w4-536a87-kserve-6bdb5c8897-lmcls"
Nov 01 14:40:14.233890 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:14.233817    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3cdf53f6-df5a-4cdc-b751-e26c71484700-home\") pod \"distinct-w4-536a87-kserve-6bdb5c8897-lmcls\" (UID: \"3cdf53f6-df5a-4cdc-b751-e26c71484700\") " pod="e2e-models-e2e-worker-w4-536a87/distinct-w4-536a87-kserve-6bdb5c8897-lmcls"
Nov 01 14:40:14.233998 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:14.233961    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cdf53f6-df5a-4cdc-b751-e26c71484700-tls-certs\") pod \"distinct-w4-536a87-kserve-6bdb5c8897-lmcls\" (UID: \"3cdf53f6-df5a-4cdc-b751-e26c71484700\") " pod="e2e-models-e2e-worker-w4-536a87/distinct-w4-536a87-kserve-6bdb5c8897-lmcls"
Nov 01 14:40:14.234218 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:14.234154    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cdf53f6-df5a-4cdc-b751-e26c71484700-model-cache\") pod \"distinct-w4-536a87-kserve-6bdb5c8897-lmcls\" (UID: \"3cdf53f6-df5a-4cdc-b751-e26c71484700\") " pod="e2e-models-e2e-worker-w4-536a87/distinct-w4-536a87-kserve-6bdb5c8897-lmcls"
Nov 01 14:40:14.234218 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:14.234194    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3cdf53f6-df5a-4cdc-b751-e26c71484700-home\") pod \"distinct-w4-536a87-kserve-6bdb5c8897-lmcls\" (UID: \"3cdf53f6-df5a-4cdc-b751-e26c71484700\") " pod="e2e-models-e2e-worker-w4-536a87/distinct-w4-536a87-kserve-6bdb5c8897-lmcls"
Nov 01 14:40:14.234438 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:14.234280    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27dvp\" (UniqueName: \"kubernetes.io/projected/3cdf53f6-df5a-4cdc-b751-e26c71484700-kube-api-access-27dvp\") pod \"distinct-w4-536a87-kserve-6bdb5c8897-lmcls\" (UID: \"3cdf53f6-df5a-4cdc-b751-e26c71484700\") " pod="e2e-models-e2e-worker-w4-536a87/distinct-w4-536a87-kserve-6bdb5c8897-lmcls"
Nov 01 14:40:14.234438 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:14.234329    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cdf53f6-df5a-4cdc-b751-e26c71484700-tmp-dir\") pod \"distinct-w4-536a87-kserve-6bdb5c8897-lmcls\" (UID: \"3cdf53f6-df5a-4cdc-b751-e26c71484700\") " pod="e2e-models-e2e-worker-w4-536a87/distinct-w4-536a87-kserve-6bdb5c8897-lmcls"
Nov 01 14:40:14.234704 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:14.234679    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cdf53f6-df5a-4cdc-b751-e26c71484700-tmp-dir\") pod \"distinct-w4-536a87-kserve-6bdb5c8897-lmcls\" (UID: \"3cdf53f6-df5a-4cdc-b751-e26c71484700\") " pod="e2e-models-e2e-worker-w4-536a87/distinct-w4-536a87-kserve-6bdb5c8897-lmcls"
Nov 01 14:40:14.236477 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:14.236450    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cdf53f6-df5a-4cdc-b751-e26c71484700-dshm\") pod \"distinct-w4-536a87-kserve-6bdb5c8897-lmcls\" (UID: \"3cdf53f6-df5a-4cdc-b751-e26c71484700\") " pod="e2e-models-e2e-worker-w4-536a87/distinct-w4-536a87-kserve-6bdb5c8897-lmcls"
Nov 01 14:40:14.236676 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:14.236656    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cdf53f6-df5a-4cdc-b751-e26c71484700-tls-certs\") pod \"distinct-w4-536a87-kserve-6bdb5c8897-lmcls\" (UID: \"3cdf53f6-df5a-4cdc-b751-e26c71484700\") " pod="e2e-models-e2e-worker-w4-536a87/distinct-w4-536a87-kserve-6bdb5c8897-lmcls"
Nov 01 14:40:14.287039 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:14.287001    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27dvp\" (UniqueName: \"kubernetes.io/projected/3cdf53f6-df5a-4cdc-b751-e26c71484700-kube-api-access-27dvp\") pod \"distinct-w4-536a87-kserve-6bdb5c8897-lmcls\" (UID: \"3cdf53f6-df5a-4cdc-b751-e26c71484700\") " pod="e2e-models-e2e-worker-w4-536a87/distinct-w4-536a87-kserve-6bdb5c8897-lmcls"
Nov 01 14:40:14.566726 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:14.566638    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-536a87/distinct-w4-536a87-kserve-6bdb5c8897-lmcls"
Nov 01 14:40:14.939598 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:14.939508    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-536a87/distinct-w4-536a87-kserve-6bdb5c8897-lmcls"]
Nov 01 14:40:14.947672 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:40:14.947638    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3cdf53f6_df5a_4cdc_b751_e26c71484700.slice/crio-36396d5e1673024628bff726e102c63c50fb859a1a918fab907ce843a22e2034 WatchSource:0}: Error finding container 36396d5e1673024628bff726e102c63c50fb859a1a918fab907ce843a22e2034: Status 404 returned error can't find the container with id 36396d5e1673024628bff726e102c63c50fb859a1a918fab907ce843a22e2034
Nov 01 14:40:15.941560 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:15.941517    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-536a87/distinct-w4-536a87-kserve-6bdb5c8897-lmcls" event={"ID":"3cdf53f6-df5a-4cdc-b751-e26c71484700","Type":"ContainerStarted","Data":"34c3e8d99f425379366e0d6ea908a239eae0c724bf8d28b0e49305a464d85806"}
Nov 01 14:40:15.942142 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:15.941569    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-536a87/distinct-w4-536a87-kserve-6bdb5c8897-lmcls" event={"ID":"3cdf53f6-df5a-4cdc-b751-e26c71484700","Type":"ContainerStarted","Data":"36396d5e1673024628bff726e102c63c50fb859a1a918fab907ce843a22e2034"}
Nov 01 14:40:16.049326 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:16.049260    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-536a87/distinct-w4-536a87-kserve-6bdb5c8897-lmcls" podStartSLOduration=3.049245525 podStartE2EDuration="3.049245525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:40:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:40:16.047082984 +0000 UTC m=+1403.376945734" watchObservedRunningTime="2026-11-01 14:40:16.049245525 +0000 UTC m=+1403.379108262"
Nov 01 14:40:16.567008 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:16.566966    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-536a87/distinct-w4-536a87-kserve-6bdb5c8897-lmcls"
Nov 01 14:40:24.567495 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:24.567453    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-536a87/distinct-w4-536a87-kserve-6bdb5c8897-lmcls"
Nov 01 14:40:24.580644 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:24.580616    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-536a87/distinct-w4-536a87-kserve-6bdb5c8897-lmcls"
Nov 01 14:40:25.019677 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:25.019645    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-536a87/distinct-w4-536a87-kserve-6bdb5c8897-lmcls"
Nov 01 14:40:27.501903 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:40:27.501868    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:40:32.761043 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:32.761001    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-536a87/distinct-2-w4-536a87-kserve-6ff7c9fb94-scgv7"]
Nov 01 14:40:32.765401 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:32.765377    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-536a87/distinct-2-w4-536a87-kserve-6ff7c9fb94-scgv7"
Nov 01 14:40:32.824696 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:32.824658    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56b01618-1df3-4ba6-8766-46fa85c27773-tmp-dir\") pod \"distinct-2-w4-536a87-kserve-6ff7c9fb94-scgv7\" (UID: \"56b01618-1df3-4ba6-8766-46fa85c27773\") " pod="e2e-models-e2e-worker-w4-536a87/distinct-2-w4-536a87-kserve-6ff7c9fb94-scgv7"
Nov 01 14:40:32.824865 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:32.824725    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56b01618-1df3-4ba6-8766-46fa85c27773-home\") pod \"distinct-2-w4-536a87-kserve-6ff7c9fb94-scgv7\" (UID: \"56b01618-1df3-4ba6-8766-46fa85c27773\") " pod="e2e-models-e2e-worker-w4-536a87/distinct-2-w4-536a87-kserve-6ff7c9fb94-scgv7"
Nov 01 14:40:32.824865 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:32.824834    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56b01618-1df3-4ba6-8766-46fa85c27773-dshm\") pod \"distinct-2-w4-536a87-kserve-6ff7c9fb94-scgv7\" (UID: \"56b01618-1df3-4ba6-8766-46fa85c27773\") " pod="e2e-models-e2e-worker-w4-536a87/distinct-2-w4-536a87-kserve-6ff7c9fb94-scgv7"
Nov 01 14:40:32.824981 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:32.824954    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvj9t\" (UniqueName: \"kubernetes.io/projected/56b01618-1df3-4ba6-8766-46fa85c27773-kube-api-access-rvj9t\") pod \"distinct-2-w4-536a87-kserve-6ff7c9fb94-scgv7\" (UID: \"56b01618-1df3-4ba6-8766-46fa85c27773\") " pod="e2e-models-e2e-worker-w4-536a87/distinct-2-w4-536a87-kserve-6ff7c9fb94-scgv7"
Nov 01 14:40:32.825129 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:32.825112    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56b01618-1df3-4ba6-8766-46fa85c27773-tls-certs\") pod \"distinct-2-w4-536a87-kserve-6ff7c9fb94-scgv7\" (UID: \"56b01618-1df3-4ba6-8766-46fa85c27773\") " pod="e2e-models-e2e-worker-w4-536a87/distinct-2-w4-536a87-kserve-6ff7c9fb94-scgv7"
Nov 01 14:40:32.825266 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:32.825241    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56b01618-1df3-4ba6-8766-46fa85c27773-model-cache\") pod \"distinct-2-w4-536a87-kserve-6ff7c9fb94-scgv7\" (UID: \"56b01618-1df3-4ba6-8766-46fa85c27773\") " pod="e2e-models-e2e-worker-w4-536a87/distinct-2-w4-536a87-kserve-6ff7c9fb94-scgv7"
Nov 01 14:40:32.860245 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:32.860200    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-536a87\"/\"distinct-2-w4-536a87-kserve-self-signed-certs\""
Nov 01 14:40:32.865727 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:32.865694    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-536a87/distinct-2-w4-536a87-kserve-6ff7c9fb94-scgv7"]
Nov 01 14:40:32.926316 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:32.926271    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56b01618-1df3-4ba6-8766-46fa85c27773-model-cache\") pod \"distinct-2-w4-536a87-kserve-6ff7c9fb94-scgv7\" (UID: \"56b01618-1df3-4ba6-8766-46fa85c27773\") " pod="e2e-models-e2e-worker-w4-536a87/distinct-2-w4-536a87-kserve-6ff7c9fb94-scgv7"
Nov 01 14:40:32.926510 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:32.926413    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56b01618-1df3-4ba6-8766-46fa85c27773-tmp-dir\") pod \"distinct-2-w4-536a87-kserve-6ff7c9fb94-scgv7\" (UID: \"56b01618-1df3-4ba6-8766-46fa85c27773\") " pod="e2e-models-e2e-worker-w4-536a87/distinct-2-w4-536a87-kserve-6ff7c9fb94-scgv7"
Nov 01 14:40:32.926510 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:32.926460    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56b01618-1df3-4ba6-8766-46fa85c27773-home\") pod \"distinct-2-w4-536a87-kserve-6ff7c9fb94-scgv7\" (UID: \"56b01618-1df3-4ba6-8766-46fa85c27773\") " pod="e2e-models-e2e-worker-w4-536a87/distinct-2-w4-536a87-kserve-6ff7c9fb94-scgv7"
Nov 01 14:40:32.926510 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:32.926497    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56b01618-1df3-4ba6-8766-46fa85c27773-dshm\") pod \"distinct-2-w4-536a87-kserve-6ff7c9fb94-scgv7\" (UID: \"56b01618-1df3-4ba6-8766-46fa85c27773\") " pod="e2e-models-e2e-worker-w4-536a87/distinct-2-w4-536a87-kserve-6ff7c9fb94-scgv7"
Nov 01 14:40:32.926688 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:32.926556    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvj9t\" (UniqueName: \"kubernetes.io/projected/56b01618-1df3-4ba6-8766-46fa85c27773-kube-api-access-rvj9t\") pod \"distinct-2-w4-536a87-kserve-6ff7c9fb94-scgv7\" (UID: \"56b01618-1df3-4ba6-8766-46fa85c27773\") " pod="e2e-models-e2e-worker-w4-536a87/distinct-2-w4-536a87-kserve-6ff7c9fb94-scgv7"
Nov 01 14:40:32.926747 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:32.926687    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56b01618-1df3-4ba6-8766-46fa85c27773-tls-certs\") pod \"distinct-2-w4-536a87-kserve-6ff7c9fb94-scgv7\" (UID: \"56b01618-1df3-4ba6-8766-46fa85c27773\") " pod="e2e-models-e2e-worker-w4-536a87/distinct-2-w4-536a87-kserve-6ff7c9fb94-scgv7"
Nov 01 14:40:32.927295 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:32.927266    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56b01618-1df3-4ba6-8766-46fa85c27773-home\") pod \"distinct-2-w4-536a87-kserve-6ff7c9fb94-scgv7\" (UID: \"56b01618-1df3-4ba6-8766-46fa85c27773\") " pod="e2e-models-e2e-worker-w4-536a87/distinct-2-w4-536a87-kserve-6ff7c9fb94-scgv7"
Nov 01 14:40:32.927618 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:32.927591    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56b01618-1df3-4ba6-8766-46fa85c27773-tmp-dir\") pod \"distinct-2-w4-536a87-kserve-6ff7c9fb94-scgv7\" (UID: \"56b01618-1df3-4ba6-8766-46fa85c27773\") " pod="e2e-models-e2e-worker-w4-536a87/distinct-2-w4-536a87-kserve-6ff7c9fb94-scgv7"
Nov 01 14:40:32.927725 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:32.927655    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56b01618-1df3-4ba6-8766-46fa85c27773-model-cache\") pod \"distinct-2-w4-536a87-kserve-6ff7c9fb94-scgv7\" (UID: \"56b01618-1df3-4ba6-8766-46fa85c27773\") " pod="e2e-models-e2e-worker-w4-536a87/distinct-2-w4-536a87-kserve-6ff7c9fb94-scgv7"
Nov 01 14:40:32.929842 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:32.929808    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56b01618-1df3-4ba6-8766-46fa85c27773-dshm\") pod \"distinct-2-w4-536a87-kserve-6ff7c9fb94-scgv7\" (UID: \"56b01618-1df3-4ba6-8766-46fa85c27773\") " pod="e2e-models-e2e-worker-w4-536a87/distinct-2-w4-536a87-kserve-6ff7c9fb94-scgv7"
Nov 01 14:40:32.930441 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:32.930414    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56b01618-1df3-4ba6-8766-46fa85c27773-tls-certs\") pod \"distinct-2-w4-536a87-kserve-6ff7c9fb94-scgv7\" (UID: \"56b01618-1df3-4ba6-8766-46fa85c27773\") " pod="e2e-models-e2e-worker-w4-536a87/distinct-2-w4-536a87-kserve-6ff7c9fb94-scgv7"
Nov 01 14:40:32.984195 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:32.981927    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvj9t\" (UniqueName: \"kubernetes.io/projected/56b01618-1df3-4ba6-8766-46fa85c27773-kube-api-access-rvj9t\") pod \"distinct-2-w4-536a87-kserve-6ff7c9fb94-scgv7\" (UID: \"56b01618-1df3-4ba6-8766-46fa85c27773\") " pod="e2e-models-e2e-worker-w4-536a87/distinct-2-w4-536a87-kserve-6ff7c9fb94-scgv7"
Nov 01 14:40:33.079737 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:33.079642    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-536a87/distinct-2-w4-536a87-kserve-6ff7c9fb94-scgv7"
Nov 01 14:40:33.496131 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:33.496087    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-536a87/distinct-2-w4-536a87-kserve-6ff7c9fb94-scgv7"]
Nov 01 14:40:33.500934 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:40:33.500906    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod56b01618_1df3_4ba6_8766_46fa85c27773.slice/crio-baac5501299abe77b8031a135aed18f58528dceb6c547cf04efd1011fc730869 WatchSource:0}: Error finding container baac5501299abe77b8031a135aed18f58528dceb6c547cf04efd1011fc730869: Status 404 returned error can't find the container with id baac5501299abe77b8031a135aed18f58528dceb6c547cf04efd1011fc730869
Nov 01 14:40:34.074255 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:34.074216    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-536a87/distinct-2-w4-536a87-kserve-6ff7c9fb94-scgv7" event={"ID":"56b01618-1df3-4ba6-8766-46fa85c27773","Type":"ContainerStarted","Data":"96631854d96fbc1fb56b5d7015a6d7cc20a6b6b6e1e4d36508f3af75dcddf9b1"}
Nov 01 14:40:34.074731 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:34.074272    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-536a87/distinct-2-w4-536a87-kserve-6ff7c9fb94-scgv7" event={"ID":"56b01618-1df3-4ba6-8766-46fa85c27773","Type":"ContainerStarted","Data":"baac5501299abe77b8031a135aed18f58528dceb6c547cf04efd1011fc730869"}
Nov 01 14:40:34.080626 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:34.080587    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-536a87/distinct-2-w4-536a87-kserve-6ff7c9fb94-scgv7"
Nov 01 14:40:34.184447 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:34.184390    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-536a87/distinct-2-w4-536a87-kserve-6ff7c9fb94-scgv7" podStartSLOduration=2.184376427 podStartE2EDuration="2.184376427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:40:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:40:34.18142142 +0000 UTC m=+1421.511284157" watchObservedRunningTime="2026-11-01 14:40:34.184376427 +0000 UTC m=+1421.514239163"
Nov 01 14:40:43.080085 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:43.080046    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-536a87/distinct-2-w4-536a87-kserve-6ff7c9fb94-scgv7"
Nov 01 14:40:43.093291 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:43.093261    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-536a87/distinct-2-w4-536a87-kserve-6ff7c9fb94-scgv7"
Nov 01 14:40:43.142305 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:43.142274    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-536a87/distinct-2-w4-536a87-kserve-6ff7c9fb94-scgv7"
Nov 01 14:40:44.049443 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:44.049406    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a32dd3-76b9567cbf-vjls6"]
Nov 01 14:40:44.053934 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:44.053910    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a32dd3-76b9567cbf-vjls6"
Nov 01 14:40:44.103096 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:44.103052    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-a32dd3-serving-cert\""
Nov 01 14:40:44.154188 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:44.153809    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8cbg\" (UniqueName: \"kubernetes.io/projected/871064b8-bc60-43e3-9267-6d42fb86afcf-kube-api-access-j8cbg\") pod \"maas-api-e2e-api-b-a32dd3-76b9567cbf-vjls6\" (UID: \"871064b8-bc60-43e3-9267-6d42fb86afcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a32dd3-76b9567cbf-vjls6"
Nov 01 14:40:44.154188 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:44.153882    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/871064b8-bc60-43e3-9267-6d42fb86afcf-maas-api-tls\") pod \"maas-api-e2e-api-b-a32dd3-76b9567cbf-vjls6\" (UID: \"871064b8-bc60-43e3-9267-6d42fb86afcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a32dd3-76b9567cbf-vjls6"
Nov 01 14:40:44.154188 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:44.154026    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/871064b8-bc60-43e3-9267-6d42fb86afcf-openshift-service-ca\") pod \"maas-api-e2e-api-b-a32dd3-76b9567cbf-vjls6\" (UID: \"871064b8-bc60-43e3-9267-6d42fb86afcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a32dd3-76b9567cbf-vjls6"
Nov 01 14:40:44.154188 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:44.154049    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/871064b8-bc60-43e3-9267-6d42fb86afcf-metrics-certs\") pod \"maas-api-e2e-api-b-a32dd3-76b9567cbf-vjls6\" (UID: \"871064b8-bc60-43e3-9267-6d42fb86afcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a32dd3-76b9567cbf-vjls6"
Nov 01 14:40:44.154533 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:44.154408    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a32dd3-76b9567cbf-vjls6"]
Nov 01 14:40:44.255132 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:44.255085    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8cbg\" (UniqueName: \"kubernetes.io/projected/871064b8-bc60-43e3-9267-6d42fb86afcf-kube-api-access-j8cbg\") pod \"maas-api-e2e-api-b-a32dd3-76b9567cbf-vjls6\" (UID: \"871064b8-bc60-43e3-9267-6d42fb86afcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a32dd3-76b9567cbf-vjls6"
Nov 01 14:40:44.255343 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:44.255160    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/871064b8-bc60-43e3-9267-6d42fb86afcf-maas-api-tls\") pod \"maas-api-e2e-api-b-a32dd3-76b9567cbf-vjls6\" (UID: \"871064b8-bc60-43e3-9267-6d42fb86afcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a32dd3-76b9567cbf-vjls6"
Nov 01 14:40:44.255343 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:40:44.255286    4993 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-a32dd3-serving-cert: secret "maas-api-e2e-api-b-a32dd3-serving-cert" not found
Nov 01 14:40:44.255469 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:44.255345    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/871064b8-bc60-43e3-9267-6d42fb86afcf-openshift-service-ca\") pod \"maas-api-e2e-api-b-a32dd3-76b9567cbf-vjls6\" (UID: \"871064b8-bc60-43e3-9267-6d42fb86afcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a32dd3-76b9567cbf-vjls6"
Nov 01 14:40:44.255469 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:40:44.255369    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/871064b8-bc60-43e3-9267-6d42fb86afcf-maas-api-tls podName:871064b8-bc60-43e3-9267-6d42fb86afcf nodeName:}" failed. No retries permitted until 2026-11-01 14:40:44.755346727 +0000 UTC m=+1432.085209447 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/871064b8-bc60-43e3-9267-6d42fb86afcf-maas-api-tls") pod "maas-api-e2e-api-b-a32dd3-76b9567cbf-vjls6" (UID: "871064b8-bc60-43e3-9267-6d42fb86afcf") : secret "maas-api-e2e-api-b-a32dd3-serving-cert" not found
Nov 01 14:40:44.255469 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:44.255402    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/871064b8-bc60-43e3-9267-6d42fb86afcf-metrics-certs\") pod \"maas-api-e2e-api-b-a32dd3-76b9567cbf-vjls6\" (UID: \"871064b8-bc60-43e3-9267-6d42fb86afcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a32dd3-76b9567cbf-vjls6"
Nov 01 14:40:44.256139 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:44.256105    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/871064b8-bc60-43e3-9267-6d42fb86afcf-openshift-service-ca\") pod \"maas-api-e2e-api-b-a32dd3-76b9567cbf-vjls6\" (UID: \"871064b8-bc60-43e3-9267-6d42fb86afcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a32dd3-76b9567cbf-vjls6"
Nov 01 14:40:44.258506 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:44.258480    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/871064b8-bc60-43e3-9267-6d42fb86afcf-metrics-certs\") pod \"maas-api-e2e-api-b-a32dd3-76b9567cbf-vjls6\" (UID: \"871064b8-bc60-43e3-9267-6d42fb86afcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a32dd3-76b9567cbf-vjls6"
Nov 01 14:40:44.333097 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:44.333014    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8cbg\" (UniqueName: \"kubernetes.io/projected/871064b8-bc60-43e3-9267-6d42fb86afcf-kube-api-access-j8cbg\") pod \"maas-api-e2e-api-b-a32dd3-76b9567cbf-vjls6\" (UID: \"871064b8-bc60-43e3-9267-6d42fb86afcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a32dd3-76b9567cbf-vjls6"
Nov 01 14:40:44.772593 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:44.772550    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/871064b8-bc60-43e3-9267-6d42fb86afcf-maas-api-tls\") pod \"maas-api-e2e-api-b-a32dd3-76b9567cbf-vjls6\" (UID: \"871064b8-bc60-43e3-9267-6d42fb86afcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a32dd3-76b9567cbf-vjls6"
Nov 01 14:40:44.775293 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:44.775265    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/871064b8-bc60-43e3-9267-6d42fb86afcf-maas-api-tls\") pod \"maas-api-e2e-api-b-a32dd3-76b9567cbf-vjls6\" (UID: \"871064b8-bc60-43e3-9267-6d42fb86afcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a32dd3-76b9567cbf-vjls6"
Nov 01 14:40:44.973424 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:44.973386    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a32dd3-76b9567cbf-vjls6"
Nov 01 14:40:45.351034 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:45.350957    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a32dd3-76b9567cbf-vjls6"]
Nov 01 14:40:45.356859 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:40:45.356830    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod871064b8_bc60_43e3_9267_6d42fb86afcf.slice/crio-1b9856951d74e0e563c83d4e1b399bd0fd8b0bef74eabb707b6d297952a6e318 WatchSource:0}: Error finding container 1b9856951d74e0e563c83d4e1b399bd0fd8b0bef74eabb707b6d297952a6e318: Status 404 returned error can't find the container with id 1b9856951d74e0e563c83d4e1b399bd0fd8b0bef74eabb707b6d297952a6e318
Nov 01 14:40:46.160975 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:46.160930    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a32dd3-76b9567cbf-vjls6" event={"ID":"871064b8-bc60-43e3-9267-6d42fb86afcf","Type":"ContainerStarted","Data":"1b9856951d74e0e563c83d4e1b399bd0fd8b0bef74eabb707b6d297952a6e318"}
Nov 01 14:40:47.170078 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:47.170039    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a32dd3-76b9567cbf-vjls6" event={"ID":"871064b8-bc60-43e3-9267-6d42fb86afcf","Type":"ContainerStarted","Data":"8268a15f401578d785d0c8d22ef611faf4f2dcb6fc50f6a55cfed35bafe7e80f"}
Nov 01 14:40:47.170575 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:47.170138    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a32dd3-76b9567cbf-vjls6"
Nov 01 14:40:47.285820 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:47.285751    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a32dd3-76b9567cbf-vjls6" podStartSLOduration=2.285156205 podStartE2EDuration="3.285737654s" totalImagesPullingTime="1.000581449s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:40:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:40:47.283449371 +0000 UTC m=+1434.613312126" watchObservedRunningTime="2026-11-01 14:40:47.285737654 +0000 UTC m=+1434.615600390"
Nov 01 14:40:53.182760 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:40:53.182726    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a32dd3-76b9567cbf-vjls6"
Nov 01 14:40:57.503715 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:40:57.503674    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:41:14.585069 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:41:14.585015    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j"]
Nov 01 14:41:14.592617 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:41:14.592587    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j"
Nov 01 14:41:14.642286 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:41:14.642252    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j"]
Nov 01 14:41:14.685035 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:41:14.685004    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-536a87-openshift-default-dockercfg-588rv\""
Nov 01 14:41:14.740114 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:41:14.740078    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/174f6587-8229-4907-892f-27df74d3f997-workload-certs\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j\" (UID: \"174f6587-8229-4907-892f-27df74d3f997\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j"
Nov 01 14:41:14.740114 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:41:14.740119    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/174f6587-8229-4907-892f-27df74d3f997-workload-socket\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j\" (UID: \"174f6587-8229-4907-892f-27df74d3f997\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j"
Nov 01 14:41:14.740411 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:41:14.740300    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/174f6587-8229-4907-892f-27df74d3f997-istio-data\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j\" (UID: \"174f6587-8229-4907-892f-27df74d3f997\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j"
Nov 01 14:41:14.740411 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:41:14.740363    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/174f6587-8229-4907-892f-27df74d3f997-istio-token\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j\" (UID: \"174f6587-8229-4907-892f-27df74d3f997\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j"
Nov 01 14:41:14.740501 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:41:14.740414    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/174f6587-8229-4907-892f-27df74d3f997-credential-socket\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j\" (UID: \"174f6587-8229-4907-892f-27df74d3f997\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j"
Nov 01 14:41:14.740553 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:41:14.740504    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/174f6587-8229-4907-892f-27df74d3f997-istio-envoy\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j\" (UID: \"174f6587-8229-4907-892f-27df74d3f997\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j"
Nov 01 14:41:14.740597 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:41:14.740570    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgrp6\" (UniqueName: \"kubernetes.io/projected/174f6587-8229-4907-892f-27df74d3f997-kube-api-access-sgrp6\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j\" (UID: \"174f6587-8229-4907-892f-27df74d3f997\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j"
Nov 01 14:41:14.740639 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:41:14.740625    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/174f6587-8229-4907-892f-27df74d3f997-istio-podinfo\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j\" (UID: \"174f6587-8229-4907-892f-27df74d3f997\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j"
Nov 01 14:41:14.740714 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:41:14.740697    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/174f6587-8229-4907-892f-27df74d3f997-istiod-ca-cert\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j\" (UID: \"174f6587-8229-4907-892f-27df74d3f997\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j"
Nov 01 14:41:14.841779 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:41:14.841678    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgrp6\" (UniqueName: \"kubernetes.io/projected/174f6587-8229-4907-892f-27df74d3f997-kube-api-access-sgrp6\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j\" (UID: \"174f6587-8229-4907-892f-27df74d3f997\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j"
Nov 01 14:41:14.841779 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:41:14.841772    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/174f6587-8229-4907-892f-27df74d3f997-istio-podinfo\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j\" (UID: \"174f6587-8229-4907-892f-27df74d3f997\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j"
Nov 01 14:41:14.842022 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:41:14.841812    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/174f6587-8229-4907-892f-27df74d3f997-istiod-ca-cert\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j\" (UID: \"174f6587-8229-4907-892f-27df74d3f997\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j"
Nov 01 14:41:14.842022 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:41:14.841880    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/174f6587-8229-4907-892f-27df74d3f997-workload-certs\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j\" (UID: \"174f6587-8229-4907-892f-27df74d3f997\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j"
Nov 01 14:41:14.842022 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:41:14.841920    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/174f6587-8229-4907-892f-27df74d3f997-workload-socket\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j\" (UID: \"174f6587-8229-4907-892f-27df74d3f997\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j"
Nov 01 14:41:14.842022 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:41:14.842002    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/174f6587-8229-4907-892f-27df74d3f997-istio-data\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j\" (UID: \"174f6587-8229-4907-892f-27df74d3f997\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j"
Nov 01 14:41:14.842312 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:41:14.842053    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/174f6587-8229-4907-892f-27df74d3f997-istio-token\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j\" (UID: \"174f6587-8229-4907-892f-27df74d3f997\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j"
Nov 01 14:41:14.842312 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:41:14.842104    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/174f6587-8229-4907-892f-27df74d3f997-credential-socket\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j\" (UID: \"174f6587-8229-4907-892f-27df74d3f997\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j"
Nov 01 14:41:14.842312 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:41:14.842195    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/174f6587-8229-4907-892f-27df74d3f997-istio-envoy\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j\" (UID: \"174f6587-8229-4907-892f-27df74d3f997\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j"
Nov 01 14:41:14.842485 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:41:14.842413    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/174f6587-8229-4907-892f-27df74d3f997-workload-certs\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j\" (UID: \"174f6587-8229-4907-892f-27df74d3f997\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j"
Nov 01 14:41:14.842602 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:41:14.842564    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/174f6587-8229-4907-892f-27df74d3f997-workload-socket\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j\" (UID: \"174f6587-8229-4907-892f-27df74d3f997\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j"
Nov 01 14:41:14.842742 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:41:14.842660    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/174f6587-8229-4907-892f-27df74d3f997-istio-data\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j\" (UID: \"174f6587-8229-4907-892f-27df74d3f997\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j"
Nov 01 14:41:14.842807 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:41:14.842786    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/174f6587-8229-4907-892f-27df74d3f997-credential-socket\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j\" (UID: \"174f6587-8229-4907-892f-27df74d3f997\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j"
Nov 01 14:41:14.842948 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:41:14.842929    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/174f6587-8229-4907-892f-27df74d3f997-istiod-ca-cert\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j\" (UID: \"174f6587-8229-4907-892f-27df74d3f997\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j"
Nov 01 14:41:14.844510 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:41:14.844468    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/174f6587-8229-4907-892f-27df74d3f997-istio-podinfo\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j\" (UID: \"174f6587-8229-4907-892f-27df74d3f997\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j"
Nov 01 14:41:14.844616 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:41:14.844557    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/174f6587-8229-4907-892f-27df74d3f997-istio-envoy\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j\" (UID: \"174f6587-8229-4907-892f-27df74d3f997\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j"
Nov 01 14:41:14.894920 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:41:14.894876    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/174f6587-8229-4907-892f-27df74d3f997-istio-token\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j\" (UID: \"174f6587-8229-4907-892f-27df74d3f997\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j"
Nov 01 14:41:14.895095 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:41:14.894982    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgrp6\" (UniqueName: \"kubernetes.io/projected/174f6587-8229-4907-892f-27df74d3f997-kube-api-access-sgrp6\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j\" (UID: \"174f6587-8229-4907-892f-27df74d3f997\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j"
Nov 01 14:41:14.906789 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:41:14.906761    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j"
Nov 01 14:41:15.340088 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:41:15.340051    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j"]
Nov 01 14:41:15.345693 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:41:15.345668    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod174f6587_8229_4907_892f_27df74d3f997.slice/crio-3eaea0f8bea3c2bf9c45b06d976d17e3a1399b860bdb6c0ad7b0b61dd5fa96e8 WatchSource:0}: Error finding container 3eaea0f8bea3c2bf9c45b06d976d17e3a1399b860bdb6c0ad7b0b61dd5fa96e8: Status 404 returned error can't find the container with id 3eaea0f8bea3c2bf9c45b06d976d17e3a1399b860bdb6c0ad7b0b61dd5fa96e8
Nov 01 14:41:15.348255 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:41:15.348222    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 14:41:15.348338 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:41:15.348313    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 14:41:15.348392 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:41:15.348351    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 14:41:15.371920 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:41:15.371891    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j" event={"ID":"174f6587-8229-4907-892f-27df74d3f997","Type":"ContainerStarted","Data":"3eaea0f8bea3c2bf9c45b06d976d17e3a1399b860bdb6c0ad7b0b61dd5fa96e8"}
Nov 01 14:41:16.391100 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:41:16.391055    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j" event={"ID":"174f6587-8229-4907-892f-27df74d3f997","Type":"ContainerStarted","Data":"91c96200be516bfec1975a3eef77eb5710b9a3d0fc03cb884ce399e24c19ff33"}
Nov 01 14:41:16.545926 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:41:16.545870    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j" podStartSLOduration=2.5458534779999997 podStartE2EDuration="2.545853478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:41:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:41:16.545333977 +0000 UTC m=+1463.875196745" watchObservedRunningTime="2026-11-01 14:41:16.545853478 +0000 UTC m=+1463.875716216"
Nov 01 14:41:16.907027 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:41:16.906993    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j"
Nov 01 14:41:16.908792 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:41:16.908763    4993 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 01 14:41:16.908926 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:41:16.908822    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j" podUID="174f6587-8229-4907-892f-27df74d3f997" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 01 14:41:17.907714 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:41:17.907676    4993 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 01 14:41:17.908128 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:41:17.907729    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j" podUID="174f6587-8229-4907-892f-27df74d3f997" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 01 14:41:18.907434 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:41:18.907398    4993 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 01 14:41:18.907601 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:41:18.907453    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j" podUID="174f6587-8229-4907-892f-27df74d3f997" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 01 14:41:19.907536 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:41:19.907501    4993 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 01 14:41:19.907954 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:41:19.907560    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j" podUID="174f6587-8229-4907-892f-27df74d3f997" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 01 14:41:20.907536 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:41:20.907497    4993 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 01 14:41:20.907796 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:41:20.907558    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j" podUID="174f6587-8229-4907-892f-27df74d3f997" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 01 14:41:20.945365 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:41:20.945332    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-116e85-65676f99fd-b7vxz"]
Nov 01 14:41:20.950874 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:41:20.950843    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-116e85-65676f99fd-b7vxz"
Nov 01 14:41:21.008402 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:41:21.008373    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-116e85-serving-cert\""
Nov 01 14:41:21.009129 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:41:21.009099    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-116e85-65676f99fd-b7vxz"]
Nov 01 14:41:21.043605 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:41:21.043564    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgjcr\" (UniqueName: \"kubernetes.io/projected/cefdf46f-afbf-44e8-8d54-a3d5b0e7c348-kube-api-access-mgjcr\") pod \"maas-api-e2e-api-b-116e85-65676f99fd-b7vxz\" (UID: \"cefdf46f-afbf-44e8-8d54-a3d5b0e7c348\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-116e85-65676f99fd-b7vxz"
Nov 01 14:41:21.043896 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:41:21.043871    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cefdf46f-afbf-44e8-8d54-a3d5b0e7c348-openshift-service-ca\") pod \"maas-api-e2e-api-b-116e85-65676f99fd-b7vxz\" (UID: \"cefdf46f-afbf-44e8-8d54-a3d5b0e7c348\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-116e85-65676f99fd-b7vxz"
Nov 01 14:41:21.044244 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:41:21.044221    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cefdf46f-afbf-44e8-8d54-a3d5b0e7c348-maas-api-tls\") pod \"maas-api-e2e-api-b-116e85-65676f99fd-b7vxz\" (UID: \"cefdf46f-afbf-44e8-8d54-a3d5b0e7c348\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-116e85-65676f99fd-b7vxz"
Nov 01 14:41:21.044460 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:41:21.044371    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cefdf46f-afbf-44e8-8d54-a3d5b0e7c348-metrics-certs\") pod \"maas-api-e2e-api-b-116e85-65676f99fd-b7vxz\" (UID: \"cefdf46f-afbf-44e8-8d54-a3d5b0e7c348\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-116e85-65676f99fd-b7vxz"
Nov 01 14:41:21.146344 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:41:21.146304    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgjcr\" (UniqueName: \"kubernetes.io/projected/cefdf46f-afbf-44e8-8d54-a3d5b0e7c348-kube-api-access-mgjcr\") pod \"maas-api-e2e-api-b-116e85-65676f99fd-b7vxz\" (UID: \"cefdf46f-afbf-44e8-8d54-a3d5b0e7c348\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-116e85-65676f99fd-b7vxz"
Nov 01 14:41:21.146537 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:41:21.146398    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cefdf46f-afbf-44e8-8d54-a3d5b0e7c348-openshift-service-ca\") pod \"maas-api-e2e-api-b-116e85-65676f99fd-b7vxz\" (UID: \"cefdf46f-afbf-44e8-8d54-a3d5b0e7c348\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-116e85-65676f99fd-b7vxz"
Nov 01 14:41:21.146638 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:41:21.146614    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cefdf46f-afbf-44e8-8d54-a3d5b0e7c348-maas-api-tls\") pod \"maas-api-e2e-api-b-116e85-65676f99fd-b7vxz\" (UID: \"cefdf46f-afbf-44e8-8d54-a3d5b0e7c348\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-116e85-65676f99fd-b7vxz"
Nov 01 14:41:21.146679 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:41:21.146651    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cefdf46f-afbf-44e8-8d54-a3d5b0e7c348-metrics-certs\") pod \"maas-api-e2e-api-b-116e85-65676f99fd-b7vxz\" (UID: \"cefdf46f-afbf-44e8-8d54-a3d5b0e7c348\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-116e85-65676f99fd-b7vxz"
Nov 01 14:41:21.146764 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:41:21.146749    4993 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-116e85-serving-cert: secret "maas-api-e2e-api-b-116e85-serving-cert" not found
Nov 01 14:41:21.146825 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:41:21.146808    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cefdf46f-afbf-44e8-8d54-a3d5b0e7c348-maas-api-tls podName:cefdf46f-afbf-44e8-8d54-a3d5b0e7c348 nodeName:}" failed. No retries permitted until 2026-11-01 14:41:21.646793467 +0000 UTC m=+1468.976656187 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cefdf46f-afbf-44e8-8d54-a3d5b0e7c348-maas-api-tls") pod "maas-api-e2e-api-b-116e85-65676f99fd-b7vxz" (UID: "cefdf46f-afbf-44e8-8d54-a3d5b0e7c348") : secret "maas-api-e2e-api-b-116e85-serving-cert" not found
Nov 01 14:41:21.147243 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:41:21.147220    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cefdf46f-afbf-44e8-8d54-a3d5b0e7c348-openshift-service-ca\") pod \"maas-api-e2e-api-b-116e85-65676f99fd-b7vxz\" (UID: \"cefdf46f-afbf-44e8-8d54-a3d5b0e7c348\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-116e85-65676f99fd-b7vxz"
Nov 01 14:41:21.149603 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:41:21.149579    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cefdf46f-afbf-44e8-8d54-a3d5b0e7c348-metrics-certs\") pod \"maas-api-e2e-api-b-116e85-65676f99fd-b7vxz\" (UID: \"cefdf46f-afbf-44e8-8d54-a3d5b0e7c348\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-116e85-65676f99fd-b7vxz"
Nov 01 14:41:21.212675 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:41:21.212260    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgjcr\" (UniqueName: \"kubernetes.io/projected/cefdf46f-afbf-44e8-8d54-a3d5b0e7c348-kube-api-access-mgjcr\") pod \"maas-api-e2e-api-b-116e85-65676f99fd-b7vxz\" (UID: \"cefdf46f-afbf-44e8-8d54-a3d5b0e7c348\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-116e85-65676f99fd-b7vxz"
Nov 01 14:41:21.657568 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:41:21.657532    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cefdf46f-afbf-44e8-8d54-a3d5b0e7c348-maas-api-tls\") pod \"maas-api-e2e-api-b-116e85-65676f99fd-b7vxz\" (UID: \"cefdf46f-afbf-44e8-8d54-a3d5b0e7c348\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-116e85-65676f99fd-b7vxz"
Nov 01 14:41:21.660096 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:41:21.660069    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cefdf46f-afbf-44e8-8d54-a3d5b0e7c348-maas-api-tls\") pod \"maas-api-e2e-api-b-116e85-65676f99fd-b7vxz\" (UID: \"cefdf46f-afbf-44e8-8d54-a3d5b0e7c348\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-116e85-65676f99fd-b7vxz"
Nov 01 14:41:21.892464 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:41:21.892432    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-116e85-65676f99fd-b7vxz"
Nov 01 14:41:21.907773 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:41:21.907719    4993 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 01 14:41:21.908566 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:41:21.908535    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j" podUID="174f6587-8229-4907-892f-27df74d3f997" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 01 14:41:22.289497 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:41:22.289449    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-116e85-65676f99fd-b7vxz"]
Nov 01 14:41:22.293895 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:41:22.293668    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcefdf46f_afbf_44e8_8d54_a3d5b0e7c348.slice/crio-3e802e2450a5846b1e7392510d6104f0f568d0b3958aa00ef43c1e32c81a59b8 WatchSource:0}: Error finding container 3e802e2450a5846b1e7392510d6104f0f568d0b3958aa00ef43c1e32c81a59b8: Status 404 returned error can't find the container with id 3e802e2450a5846b1e7392510d6104f0f568d0b3958aa00ef43c1e32c81a59b8
Nov 01 14:41:22.437746 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:41:22.437704    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-116e85-65676f99fd-b7vxz" event={"ID":"cefdf46f-afbf-44e8-8d54-a3d5b0e7c348","Type":"ContainerStarted","Data":"3e802e2450a5846b1e7392510d6104f0f568d0b3958aa00ef43c1e32c81a59b8"}
Nov 01 14:41:23.006779 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:41:23.006738    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j"
Nov 01 14:41:23.007732 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:41:23.007706    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j"
Nov 01 14:41:23.009404 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:41:23.008839    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j"
Nov 01 14:41:23.445630 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:41:23.445494    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-116e85-65676f99fd-b7vxz" event={"ID":"cefdf46f-afbf-44e8-8d54-a3d5b0e7c348","Type":"ContainerStarted","Data":"289a6e314f4b585abe36fa9325bc78662de2527d5f2a6860051865028cc3fd0a"}
Nov 01 14:41:23.445630 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:41:23.445584    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-116e85-65676f99fd-b7vxz"
Nov 01 14:41:23.553987 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:41:23.553929    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-116e85-65676f99fd-b7vxz" podStartSLOduration=2.718819965 podStartE2EDuration="3.553914484s" totalImagesPullingTime="835.094519ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:41:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:41:23.551895943 +0000 UTC m=+1470.881758681" watchObservedRunningTime="2026-11-01 14:41:23.553914484 +0000 UTC m=+1470.883777221"
Nov 01 14:41:27.504979 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:41:27.504916    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:41:29.458452 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:41:29.458417    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-116e85-65676f99fd-b7vxz"
Nov 01 14:41:52.964489 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:41:52.964455    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xmwtp_bb59051d-c2df-4362-a157-00f14f915e79/console-operator/3.log"
Nov 01 14:41:52.965030 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:41:52.964571    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xmwtp_bb59051d-c2df-4362-a157-00f14f915e79/console-operator/3.log"
Nov 01 14:41:57.506095 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:41:57.506045    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:42:19.084696 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:19.084622    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wcdb6"]
Nov 01 14:42:19.085242 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:19.085013    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wcdb6" podUID="65c82e9c-1c36-4fce-9ba2-fa15d5704683" containerName="istio-proxy" containerID="cri-o://4685497fd885fcae63e46696bdabc3b0ac25ad06cab4e55c78cf1c7a8bd4e7d9" gracePeriod=30
Nov 01 14:42:24.386712 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:24.386686    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wcdb6"
Nov 01 14:42:24.563856 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:24.563808    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65c82e9c-1c36-4fce-9ba2-fa15d5704683-istio-token\" (UniqueName: \"kubernetes.io/projected/65c82e9c-1c36-4fce-9ba2-fa15d5704683-istio-token\") pod \"65c82e9c-1c36-4fce-9ba2-fa15d5704683\" (UID: \"65c82e9c-1c36-4fce-9ba2-fa15d5704683\") "
Nov 01 14:42:24.564058 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:24.563910    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65c82e9c-1c36-4fce-9ba2-fa15d5704683-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65c82e9c-1c36-4fce-9ba2-fa15d5704683-istio-envoy\") pod \"65c82e9c-1c36-4fce-9ba2-fa15d5704683\" (UID: \"65c82e9c-1c36-4fce-9ba2-fa15d5704683\") "
Nov 01 14:42:24.564058 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:24.563987    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/65c82e9c-1c36-4fce-9ba2-fa15d5704683-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65c82e9c-1c36-4fce-9ba2-fa15d5704683-istio-podinfo\") pod \"65c82e9c-1c36-4fce-9ba2-fa15d5704683\" (UID: \"65c82e9c-1c36-4fce-9ba2-fa15d5704683\") "
Nov 01 14:42:24.564238 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:24.564120    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65c82e9c-1c36-4fce-9ba2-fa15d5704683-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65c82e9c-1c36-4fce-9ba2-fa15d5704683-istio-data\") pod \"65c82e9c-1c36-4fce-9ba2-fa15d5704683\" (UID: \"65c82e9c-1c36-4fce-9ba2-fa15d5704683\") "
Nov 01 14:42:24.564621 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:24.564588    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65c82e9c-1c36-4fce-9ba2-fa15d5704683-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65c82e9c-1c36-4fce-9ba2-fa15d5704683-istiod-ca-cert\") pod \"65c82e9c-1c36-4fce-9ba2-fa15d5704683\" (UID: \"65c82e9c-1c36-4fce-9ba2-fa15d5704683\") "
Nov 01 14:42:24.564734 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:24.564649    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65c82e9c-1c36-4fce-9ba2-fa15d5704683-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65c82e9c-1c36-4fce-9ba2-fa15d5704683-workload-socket\") pod \"65c82e9c-1c36-4fce-9ba2-fa15d5704683\" (UID: \"65c82e9c-1c36-4fce-9ba2-fa15d5704683\") "
Nov 01 14:42:24.564734 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:24.564689    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65c82e9c-1c36-4fce-9ba2-fa15d5704683-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65c82e9c-1c36-4fce-9ba2-fa15d5704683-credential-socket\") pod \"65c82e9c-1c36-4fce-9ba2-fa15d5704683\" (UID: \"65c82e9c-1c36-4fce-9ba2-fa15d5704683\") "
Nov 01 14:42:24.564734 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:24.564718    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65c82e9c-1c36-4fce-9ba2-fa15d5704683-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65c82e9c-1c36-4fce-9ba2-fa15d5704683-workload-certs\") pod \"65c82e9c-1c36-4fce-9ba2-fa15d5704683\" (UID: \"65c82e9c-1c36-4fce-9ba2-fa15d5704683\") "
Nov 01 14:42:24.564905 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:24.564789    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65c82e9c-1c36-4fce-9ba2-fa15d5704683-kube-api-access-88qlk\" (UniqueName: \"kubernetes.io/projected/65c82e9c-1c36-4fce-9ba2-fa15d5704683-kube-api-access-88qlk\") pod \"65c82e9c-1c36-4fce-9ba2-fa15d5704683\" (UID: \"65c82e9c-1c36-4fce-9ba2-fa15d5704683\") "
Nov 01 14:42:24.566112 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:24.566080    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65c82e9c-1c36-4fce-9ba2-fa15d5704683-credential-socket" pod "65c82e9c-1c36-4fce-9ba2-fa15d5704683" (UID: "65c82e9c-1c36-4fce-9ba2-fa15d5704683"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:42:24.566476 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:24.566452    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65c82e9c-1c36-4fce-9ba2-fa15d5704683-istiod-ca-cert" pod "65c82e9c-1c36-4fce-9ba2-fa15d5704683" (UID: "65c82e9c-1c36-4fce-9ba2-fa15d5704683"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:42:24.566563 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:24.566532    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65c82e9c-1c36-4fce-9ba2-fa15d5704683-workload-certs" pod "65c82e9c-1c36-4fce-9ba2-fa15d5704683" (UID: "65c82e9c-1c36-4fce-9ba2-fa15d5704683"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:42:24.566766 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:24.566746    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65c82e9c-1c36-4fce-9ba2-fa15d5704683-workload-socket" pod "65c82e9c-1c36-4fce-9ba2-fa15d5704683" (UID: "65c82e9c-1c36-4fce-9ba2-fa15d5704683"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:42:24.566949 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:24.566929    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65c82e9c-1c36-4fce-9ba2-fa15d5704683-istio-data" pod "65c82e9c-1c36-4fce-9ba2-fa15d5704683" (UID: "65c82e9c-1c36-4fce-9ba2-fa15d5704683"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:42:24.573505 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:24.573462    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65c82e9c-1c36-4fce-9ba2-fa15d5704683-istio-token" pod "65c82e9c-1c36-4fce-9ba2-fa15d5704683" (UID: "65c82e9c-1c36-4fce-9ba2-fa15d5704683"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:42:24.575611 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:24.575579    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65c82e9c-1c36-4fce-9ba2-fa15d5704683-istio-envoy" pod "65c82e9c-1c36-4fce-9ba2-fa15d5704683" (UID: "65c82e9c-1c36-4fce-9ba2-fa15d5704683"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:42:24.575611 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:24.575590    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/65c82e9c-1c36-4fce-9ba2-fa15d5704683-istio-podinfo" pod "65c82e9c-1c36-4fce-9ba2-fa15d5704683" (UID: "65c82e9c-1c36-4fce-9ba2-fa15d5704683"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 14:42:24.575841 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:24.575822    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65c82e9c-1c36-4fce-9ba2-fa15d5704683-kube-api-access-88qlk" pod "65c82e9c-1c36-4fce-9ba2-fa15d5704683" (UID: "65c82e9c-1c36-4fce-9ba2-fa15d5704683"). InnerVolumeSpecName "kube-api-access-88qlk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:42:24.667339 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:24.667253    4993 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65c82e9c-1c36-4fce-9ba2-fa15d5704683-istio-data\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:42:24.667339 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:24.667289    4993 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65c82e9c-1c36-4fce-9ba2-fa15d5704683-istiod-ca-cert\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:42:24.667339 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:24.667305    4993 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65c82e9c-1c36-4fce-9ba2-fa15d5704683-workload-socket\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:42:24.667339 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:24.667318    4993 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65c82e9c-1c36-4fce-9ba2-fa15d5704683-credential-socket\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:42:24.667339 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:24.667333    4993 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65c82e9c-1c36-4fce-9ba2-fa15d5704683-workload-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:42:24.667602 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:24.667347    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88qlk\" (UniqueName: \"kubernetes.io/projected/65c82e9c-1c36-4fce-9ba2-fa15d5704683-kube-api-access-88qlk\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:42:24.667602 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:24.667362    4993 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65c82e9c-1c36-4fce-9ba2-fa15d5704683-istio-token\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:42:24.667602 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:24.667377    4993 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65c82e9c-1c36-4fce-9ba2-fa15d5704683-istio-envoy\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:42:24.667602 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:24.667391    4993 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65c82e9c-1c36-4fce-9ba2-fa15d5704683-istio-podinfo\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:42:24.975489 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:24.975396    4993 generic.go:356] "Generic (PLEG): container finished" podID="65c82e9c-1c36-4fce-9ba2-fa15d5704683" containerID="4685497fd885fcae63e46696bdabc3b0ac25ad06cab4e55c78cf1c7a8bd4e7d9" exitCode=0
Nov 01 14:42:24.975489 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:24.975453    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wcdb6" event={"ID":"65c82e9c-1c36-4fce-9ba2-fa15d5704683","Type":"ContainerDied","Data":"4685497fd885fcae63e46696bdabc3b0ac25ad06cab4e55c78cf1c7a8bd4e7d9"}
Nov 01 14:42:24.975489 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:24.975491    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wcdb6" event={"ID":"65c82e9c-1c36-4fce-9ba2-fa15d5704683","Type":"ContainerDied","Data":"1ffdead22ca2dff99cd7a18402345c148de4cc2cf19d2e1ac75db50de5ef0101"}
Nov 01 14:42:24.975750 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:24.975512    4993 scope.go:122] "RemoveContainer" containerID="4685497fd885fcae63e46696bdabc3b0ac25ad06cab4e55c78cf1c7a8bd4e7d9"
Nov 01 14:42:24.975750 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:24.975621    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wcdb6"
Nov 01 14:42:24.989304 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:24.989265    4993 scope.go:122] "RemoveContainer" containerID="4685497fd885fcae63e46696bdabc3b0ac25ad06cab4e55c78cf1c7a8bd4e7d9"
Nov 01 14:42:24.989645 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:42:24.989615    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4685497fd885fcae63e46696bdabc3b0ac25ad06cab4e55c78cf1c7a8bd4e7d9\": container with ID starting with 4685497fd885fcae63e46696bdabc3b0ac25ad06cab4e55c78cf1c7a8bd4e7d9 not found: ID does not exist" containerID="4685497fd885fcae63e46696bdabc3b0ac25ad06cab4e55c78cf1c7a8bd4e7d9"
Nov 01 14:42:24.989744 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:24.989656    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4685497fd885fcae63e46696bdabc3b0ac25ad06cab4e55c78cf1c7a8bd4e7d9"} err="failed to get container status \"4685497fd885fcae63e46696bdabc3b0ac25ad06cab4e55c78cf1c7a8bd4e7d9\": rpc error: code = NotFound desc = could not find container \"4685497fd885fcae63e46696bdabc3b0ac25ad06cab4e55c78cf1c7a8bd4e7d9\": container with ID starting with 4685497fd885fcae63e46696bdabc3b0ac25ad06cab4e55c78cf1c7a8bd4e7d9 not found: ID does not exist"
Nov 01 14:42:25.133393 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:25.133362    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wcdb6"]
Nov 01 14:42:25.139014 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:25.138986    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-wcdb6"]
Nov 01 14:42:26.797733 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:26.797690    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65c82e9c-1c36-4fce-9ba2-fa15d5704683" path="/var/lib/kubelet/pods/65c82e9c-1c36-4fce-9ba2-fa15d5704683/volumes"
Nov 01 14:42:27.503304 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:42:27.503256    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:42:29.145394 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:29.145358    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a32dd3-76b9567cbf-vjls6"]
Nov 01 14:42:29.145950 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:29.145591    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a32dd3-76b9567cbf-vjls6" podUID="871064b8-bc60-43e3-9267-6d42fb86afcf" containerName="maas-api" containerID="cri-o://8268a15f401578d785d0c8d22ef611faf4f2dcb6fc50f6a55cfed35bafe7e80f" gracePeriod=30
Nov 01 14:42:29.710809 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:29.708023    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a32dd3-76b9567cbf-vjls6"
Nov 01 14:42:29.748574 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:29.748529    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/871064b8-bc60-43e3-9267-6d42fb86afcf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/871064b8-bc60-43e3-9267-6d42fb86afcf-maas-api-tls\") pod \"871064b8-bc60-43e3-9267-6d42fb86afcf\" (UID: \"871064b8-bc60-43e3-9267-6d42fb86afcf\") "
Nov 01 14:42:29.748769 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:29.748658    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/871064b8-bc60-43e3-9267-6d42fb86afcf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/871064b8-bc60-43e3-9267-6d42fb86afcf-metrics-certs\") pod \"871064b8-bc60-43e3-9267-6d42fb86afcf\" (UID: \"871064b8-bc60-43e3-9267-6d42fb86afcf\") "
Nov 01 14:42:29.748769 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:29.748689    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/871064b8-bc60-43e3-9267-6d42fb86afcf-kube-api-access-j8cbg\" (UniqueName: \"kubernetes.io/projected/871064b8-bc60-43e3-9267-6d42fb86afcf-kube-api-access-j8cbg\") pod \"871064b8-bc60-43e3-9267-6d42fb86afcf\" (UID: \"871064b8-bc60-43e3-9267-6d42fb86afcf\") "
Nov 01 14:42:29.748769 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:29.748751    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/871064b8-bc60-43e3-9267-6d42fb86afcf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/871064b8-bc60-43e3-9267-6d42fb86afcf-openshift-service-ca\") pod \"871064b8-bc60-43e3-9267-6d42fb86afcf\" (UID: \"871064b8-bc60-43e3-9267-6d42fb86afcf\") "
Nov 01 14:42:29.749608 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:29.749581    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/871064b8-bc60-43e3-9267-6d42fb86afcf-openshift-service-ca" pod "871064b8-bc60-43e3-9267-6d42fb86afcf" (UID: "871064b8-bc60-43e3-9267-6d42fb86afcf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:42:29.758258 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:29.756613    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/871064b8-bc60-43e3-9267-6d42fb86afcf-maas-api-tls" pod "871064b8-bc60-43e3-9267-6d42fb86afcf" (UID: "871064b8-bc60-43e3-9267-6d42fb86afcf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:42:29.760641 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:29.760608    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/871064b8-bc60-43e3-9267-6d42fb86afcf-metrics-certs" pod "871064b8-bc60-43e3-9267-6d42fb86afcf" (UID: "871064b8-bc60-43e3-9267-6d42fb86afcf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:42:29.760750 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:29.760639    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/871064b8-bc60-43e3-9267-6d42fb86afcf-kube-api-access-j8cbg" pod "871064b8-bc60-43e3-9267-6d42fb86afcf" (UID: "871064b8-bc60-43e3-9267-6d42fb86afcf"). InnerVolumeSpecName "kube-api-access-j8cbg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:42:29.855777 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:29.855734    4993 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/871064b8-bc60-43e3-9267-6d42fb86afcf-metrics-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:42:29.855777 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:29.855772    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8cbg\" (UniqueName: \"kubernetes.io/projected/871064b8-bc60-43e3-9267-6d42fb86afcf-kube-api-access-j8cbg\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:42:29.856142 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:29.855792    4993 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/871064b8-bc60-43e3-9267-6d42fb86afcf-openshift-service-ca\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:42:29.856142 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:29.855806    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/871064b8-bc60-43e3-9267-6d42fb86afcf-maas-api-tls\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:42:30.014233 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:30.014109    4993 generic.go:356] "Generic (PLEG): container finished" podID="871064b8-bc60-43e3-9267-6d42fb86afcf" containerID="8268a15f401578d785d0c8d22ef611faf4f2dcb6fc50f6a55cfed35bafe7e80f" exitCode=0
Nov 01 14:42:30.014233 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:30.014159    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a32dd3-76b9567cbf-vjls6"
Nov 01 14:42:30.014233 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:30.014193    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a32dd3-76b9567cbf-vjls6" event={"ID":"871064b8-bc60-43e3-9267-6d42fb86afcf","Type":"ContainerDied","Data":"8268a15f401578d785d0c8d22ef611faf4f2dcb6fc50f6a55cfed35bafe7e80f"}
Nov 01 14:42:30.014508 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:30.014257    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a32dd3-76b9567cbf-vjls6" event={"ID":"871064b8-bc60-43e3-9267-6d42fb86afcf","Type":"ContainerDied","Data":"1b9856951d74e0e563c83d4e1b399bd0fd8b0bef74eabb707b6d297952a6e318"}
Nov 01 14:42:30.014508 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:30.014288    4993 scope.go:122] "RemoveContainer" containerID="8268a15f401578d785d0c8d22ef611faf4f2dcb6fc50f6a55cfed35bafe7e80f"
Nov 01 14:42:30.025878 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:30.025852    4993 scope.go:122] "RemoveContainer" containerID="8268a15f401578d785d0c8d22ef611faf4f2dcb6fc50f6a55cfed35bafe7e80f"
Nov 01 14:42:30.026261 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:42:30.026235    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8268a15f401578d785d0c8d22ef611faf4f2dcb6fc50f6a55cfed35bafe7e80f\": container with ID starting with 8268a15f401578d785d0c8d22ef611faf4f2dcb6fc50f6a55cfed35bafe7e80f not found: ID does not exist" containerID="8268a15f401578d785d0c8d22ef611faf4f2dcb6fc50f6a55cfed35bafe7e80f"
Nov 01 14:42:30.026349 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:30.026273    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8268a15f401578d785d0c8d22ef611faf4f2dcb6fc50f6a55cfed35bafe7e80f"} err="failed to get container status \"8268a15f401578d785d0c8d22ef611faf4f2dcb6fc50f6a55cfed35bafe7e80f\": rpc error: code = NotFound desc = could not find container \"8268a15f401578d785d0c8d22ef611faf4f2dcb6fc50f6a55cfed35bafe7e80f\": container with ID starting with 8268a15f401578d785d0c8d22ef611faf4f2dcb6fc50f6a55cfed35bafe7e80f not found: ID does not exist"
Nov 01 14:42:30.168150 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:30.168118    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a32dd3-76b9567cbf-vjls6"]
Nov 01 14:42:30.172090 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:30.172065    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a32dd3-76b9567cbf-vjls6"]
Nov 01 14:42:30.797091 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:30.797055    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="871064b8-bc60-43e3-9267-6d42fb86afcf" path="/var/lib/kubelet/pods/871064b8-bc60-43e3-9267-6d42fb86afcf/volumes"
Nov 01 14:42:49.834560 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:49.834525    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-466a6e-7b74b87c87-f9qvk"]
Nov 01 14:42:49.835039 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:49.834785    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466a6e-7b74b87c87-f9qvk" podUID="0a56cf6f-3dce-46b9-8b71-757373386d7f" containerName="maas-api" containerID="cri-o://e8872ab7a79e5fb74267da19c37f18f2e3371b4afdea31844563c9d1da7e9b7d" gracePeriod=30
Nov 01 14:42:50.214620 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:50.214575    4993 generic.go:356] "Generic (PLEG): container finished" podID="0a56cf6f-3dce-46b9-8b71-757373386d7f" containerID="e8872ab7a79e5fb74267da19c37f18f2e3371b4afdea31844563c9d1da7e9b7d" exitCode=0
Nov 01 14:42:50.214762 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:50.214628    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466a6e-7b74b87c87-f9qvk" event={"ID":"0a56cf6f-3dce-46b9-8b71-757373386d7f","Type":"ContainerDied","Data":"e8872ab7a79e5fb74267da19c37f18f2e3371b4afdea31844563c9d1da7e9b7d"}
Nov 01 14:42:50.214762 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:50.214694    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466a6e-7b74b87c87-f9qvk" event={"ID":"0a56cf6f-3dce-46b9-8b71-757373386d7f","Type":"ContainerDied","Data":"3fa3d1bd5877d87ed3ab7183f44dfc36d6cd589a6e6728b1b64a8e5409f4f658"}
Nov 01 14:42:50.214762 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:50.214698    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466a6e-7b74b87c87-f9qvk"
Nov 01 14:42:50.214762 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:50.214719    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3fa3d1bd5877d87ed3ab7183f44dfc36d6cd589a6e6728b1b64a8e5409f4f658"
Nov 01 14:42:50.381201 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:50.379895    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a56cf6f-3dce-46b9-8b71-757373386d7f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a56cf6f-3dce-46b9-8b71-757373386d7f-openshift-service-ca\") pod \"0a56cf6f-3dce-46b9-8b71-757373386d7f\" (UID: \"0a56cf6f-3dce-46b9-8b71-757373386d7f\") "
Nov 01 14:42:50.381201 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:50.379954    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a56cf6f-3dce-46b9-8b71-757373386d7f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a56cf6f-3dce-46b9-8b71-757373386d7f-maas-api-tls\") pod \"0a56cf6f-3dce-46b9-8b71-757373386d7f\" (UID: \"0a56cf6f-3dce-46b9-8b71-757373386d7f\") "
Nov 01 14:42:50.381201 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:50.380088    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a56cf6f-3dce-46b9-8b71-757373386d7f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a56cf6f-3dce-46b9-8b71-757373386d7f-metrics-certs\") pod \"0a56cf6f-3dce-46b9-8b71-757373386d7f\" (UID: \"0a56cf6f-3dce-46b9-8b71-757373386d7f\") "
Nov 01 14:42:50.381201 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:50.380228    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a56cf6f-3dce-46b9-8b71-757373386d7f-kube-api-access-f92l4\" (UniqueName: \"kubernetes.io/projected/0a56cf6f-3dce-46b9-8b71-757373386d7f-kube-api-access-f92l4\") pod \"0a56cf6f-3dce-46b9-8b71-757373386d7f\" (UID: \"0a56cf6f-3dce-46b9-8b71-757373386d7f\") "
Nov 01 14:42:50.381201 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:50.380394    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a56cf6f-3dce-46b9-8b71-757373386d7f-openshift-service-ca" pod "0a56cf6f-3dce-46b9-8b71-757373386d7f" (UID: "0a56cf6f-3dce-46b9-8b71-757373386d7f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:42:50.381201 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:50.380853    4993 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a56cf6f-3dce-46b9-8b71-757373386d7f-openshift-service-ca\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:42:50.398242 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:50.383649    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a56cf6f-3dce-46b9-8b71-757373386d7f-kube-api-access-f92l4" pod "0a56cf6f-3dce-46b9-8b71-757373386d7f" (UID: "0a56cf6f-3dce-46b9-8b71-757373386d7f"). InnerVolumeSpecName "kube-api-access-f92l4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:42:50.416628 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:50.416592    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a56cf6f-3dce-46b9-8b71-757373386d7f-maas-api-tls" pod "0a56cf6f-3dce-46b9-8b71-757373386d7f" (UID: "0a56cf6f-3dce-46b9-8b71-757373386d7f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:42:50.416822 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:50.416796    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a56cf6f-3dce-46b9-8b71-757373386d7f-metrics-certs" pod "0a56cf6f-3dce-46b9-8b71-757373386d7f" (UID: "0a56cf6f-3dce-46b9-8b71-757373386d7f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:42:50.482749 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:50.482717    4993 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a56cf6f-3dce-46b9-8b71-757373386d7f-metrics-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:42:50.482749 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:50.482747    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f92l4\" (UniqueName: \"kubernetes.io/projected/0a56cf6f-3dce-46b9-8b71-757373386d7f-kube-api-access-f92l4\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:42:50.482921 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:50.482760    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a56cf6f-3dce-46b9-8b71-757373386d7f-maas-api-tls\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:42:51.220951 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:51.220873    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-466a6e-7b74b87c87-f9qvk"
Nov 01 14:42:51.375259 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:51.375221    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-466a6e-7b74b87c87-f9qvk"]
Nov 01 14:42:51.421299 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:51.421263    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-466a6e-7b74b87c87-f9qvk"]
Nov 01 14:42:52.799647 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:42:52.799607    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a56cf6f-3dce-46b9-8b71-757373386d7f" path="/var/lib/kubelet/pods/0a56cf6f-3dce-46b9-8b71-757373386d7f/volumes"
Nov 01 14:42:57.501624 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:42:57.501557    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:43:05.403441 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:05.403353    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c9b0827c-65bb9b948d-6bbgc"]
Nov 01 14:43:05.403895 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:05.403583    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9b0827c-65bb9b948d-6bbgc" podUID="5d3109df-047b-4a1f-a6d7-668ce0e4a43a" containerName="maas-api" containerID="cri-o://41329d4428abef02ab6340e7f6998f0cdecd2701c55df0875a49ae94a775f687" gracePeriod=30
Nov 01 14:43:05.728019 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:05.727991    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9b0827c-65bb9b948d-6bbgc"
Nov 01 14:43:05.820712 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:05.819916    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d3109df-047b-4a1f-a6d7-668ce0e4a43a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d3109df-047b-4a1f-a6d7-668ce0e4a43a-openshift-service-ca\") pod \"5d3109df-047b-4a1f-a6d7-668ce0e4a43a\" (UID: \"5d3109df-047b-4a1f-a6d7-668ce0e4a43a\") "
Nov 01 14:43:05.820712 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:05.820038    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d3109df-047b-4a1f-a6d7-668ce0e4a43a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d3109df-047b-4a1f-a6d7-668ce0e4a43a-metrics-certs\") pod \"5d3109df-047b-4a1f-a6d7-668ce0e4a43a\" (UID: \"5d3109df-047b-4a1f-a6d7-668ce0e4a43a\") "
Nov 01 14:43:05.820712 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:05.820087    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d3109df-047b-4a1f-a6d7-668ce0e4a43a-kube-api-access-9mxjx\" (UniqueName: \"kubernetes.io/projected/5d3109df-047b-4a1f-a6d7-668ce0e4a43a-kube-api-access-9mxjx\") pod \"5d3109df-047b-4a1f-a6d7-668ce0e4a43a\" (UID: \"5d3109df-047b-4a1f-a6d7-668ce0e4a43a\") "
Nov 01 14:43:05.820712 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:05.820195    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d3109df-047b-4a1f-a6d7-668ce0e4a43a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d3109df-047b-4a1f-a6d7-668ce0e4a43a-maas-api-tls\") pod \"5d3109df-047b-4a1f-a6d7-668ce0e4a43a\" (UID: \"5d3109df-047b-4a1f-a6d7-668ce0e4a43a\") "
Nov 01 14:43:05.821912 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:05.821879    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d3109df-047b-4a1f-a6d7-668ce0e4a43a-openshift-service-ca" pod "5d3109df-047b-4a1f-a6d7-668ce0e4a43a" (UID: "5d3109df-047b-4a1f-a6d7-668ce0e4a43a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:43:05.826339 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:05.825660    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d3109df-047b-4a1f-a6d7-668ce0e4a43a-kube-api-access-9mxjx" pod "5d3109df-047b-4a1f-a6d7-668ce0e4a43a" (UID: "5d3109df-047b-4a1f-a6d7-668ce0e4a43a"). InnerVolumeSpecName "kube-api-access-9mxjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:43:05.826339 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:05.825837    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d3109df-047b-4a1f-a6d7-668ce0e4a43a-metrics-certs" pod "5d3109df-047b-4a1f-a6d7-668ce0e4a43a" (UID: "5d3109df-047b-4a1f-a6d7-668ce0e4a43a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:43:05.826339 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:05.826155    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d3109df-047b-4a1f-a6d7-668ce0e4a43a-maas-api-tls" pod "5d3109df-047b-4a1f-a6d7-668ce0e4a43a" (UID: "5d3109df-047b-4a1f-a6d7-668ce0e4a43a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:43:05.922290 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:05.922260    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d3109df-047b-4a1f-a6d7-668ce0e4a43a-maas-api-tls\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:43:05.922290 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:05.922287    4993 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d3109df-047b-4a1f-a6d7-668ce0e4a43a-openshift-service-ca\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:43:05.922290 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:05.922296    4993 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d3109df-047b-4a1f-a6d7-668ce0e4a43a-metrics-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:43:05.922513 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:05.922306    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mxjx\" (UniqueName: \"kubernetes.io/projected/5d3109df-047b-4a1f-a6d7-668ce0e4a43a-kube-api-access-9mxjx\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:43:06.324402 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:06.324370    4993 generic.go:356] "Generic (PLEG): container finished" podID="5d3109df-047b-4a1f-a6d7-668ce0e4a43a" containerID="41329d4428abef02ab6340e7f6998f0cdecd2701c55df0875a49ae94a775f687" exitCode=0
Nov 01 14:43:06.324587 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:06.324415    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9b0827c-65bb9b948d-6bbgc"
Nov 01 14:43:06.324587 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:06.324456    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9b0827c-65bb9b948d-6bbgc" event={"ID":"5d3109df-047b-4a1f-a6d7-668ce0e4a43a","Type":"ContainerDied","Data":"41329d4428abef02ab6340e7f6998f0cdecd2701c55df0875a49ae94a775f687"}
Nov 01 14:43:06.324587 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:06.324518    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9b0827c-65bb9b948d-6bbgc" event={"ID":"5d3109df-047b-4a1f-a6d7-668ce0e4a43a","Type":"ContainerDied","Data":"1e7f66af5305f7e12f5e9e59ed5306e07f98f9d5243b7eee4385f5cfcb0ca121"}
Nov 01 14:43:06.324587 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:06.324547    4993 scope.go:122] "RemoveContainer" containerID="41329d4428abef02ab6340e7f6998f0cdecd2701c55df0875a49ae94a775f687"
Nov 01 14:43:06.336079 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:06.336059    4993 scope.go:122] "RemoveContainer" containerID="41329d4428abef02ab6340e7f6998f0cdecd2701c55df0875a49ae94a775f687"
Nov 01 14:43:06.336396 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:43:06.336377    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41329d4428abef02ab6340e7f6998f0cdecd2701c55df0875a49ae94a775f687\": container with ID starting with 41329d4428abef02ab6340e7f6998f0cdecd2701c55df0875a49ae94a775f687 not found: ID does not exist" containerID="41329d4428abef02ab6340e7f6998f0cdecd2701c55df0875a49ae94a775f687"
Nov 01 14:43:06.336470 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:06.336404    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41329d4428abef02ab6340e7f6998f0cdecd2701c55df0875a49ae94a775f687"} err="failed to get container status \"41329d4428abef02ab6340e7f6998f0cdecd2701c55df0875a49ae94a775f687\": rpc error: code = NotFound desc = could not find container \"41329d4428abef02ab6340e7f6998f0cdecd2701c55df0875a49ae94a775f687\": container with ID starting with 41329d4428abef02ab6340e7f6998f0cdecd2701c55df0875a49ae94a775f687 not found: ID does not exist"
Nov 01 14:43:06.482359 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:06.482319    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c9b0827c-65bb9b948d-6bbgc"]
Nov 01 14:43:06.487554 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:06.487526    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c9b0827c-65bb9b948d-6bbgc"]
Nov 01 14:43:06.796273 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:06.796239    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d3109df-047b-4a1f-a6d7-668ce0e4a43a" path="/var/lib/kubelet/pods/5d3109df-047b-4a1f-a6d7-668ce0e4a43a/volumes"
Nov 01 14:43:24.156391 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:24.156354    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr"]
Nov 01 14:43:24.156896 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:24.156578    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr" podUID="0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8" containerName="istio-proxy" containerID="cri-o://91e56a8579f5bb611a79d73a592f92672f8a575458be3ff73967704425a366b4" gracePeriod=30
Nov 01 14:43:27.502152 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:43:27.502115    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:43:28.427373 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:28.427336    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-26a0c1-769bfd4c5f-9jwpp"]
Nov 01 14:43:28.427931 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:28.427917    4993 cpu_manager.go:414] "containerMap: removing container" podUID="0a56cf6f-3dce-46b9-8b71-757373386d7f" containerName="maas-api"
Nov 01 14:43:28.427982 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:28.427933    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a56cf6f-3dce-46b9-8b71-757373386d7f" containerName="maas-api"
Nov 01 14:43:28.427982 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:28.427948    4993 cpu_manager.go:414] "containerMap: removing container" podUID="871064b8-bc60-43e3-9267-6d42fb86afcf" containerName="maas-api"
Nov 01 14:43:28.427982 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:28.427953    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="871064b8-bc60-43e3-9267-6d42fb86afcf" containerName="maas-api"
Nov 01 14:43:28.427982 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:28.427969    4993 cpu_manager.go:414] "containerMap: removing container" podUID="5d3109df-047b-4a1f-a6d7-668ce0e4a43a" containerName="maas-api"
Nov 01 14:43:28.427982 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:28.427974    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d3109df-047b-4a1f-a6d7-668ce0e4a43a" containerName="maas-api"
Nov 01 14:43:28.428133 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:28.427984    4993 cpu_manager.go:414] "containerMap: removing container" podUID="65c82e9c-1c36-4fce-9ba2-fa15d5704683" containerName="istio-proxy"
Nov 01 14:43:28.428133 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:28.427989    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65c82e9c-1c36-4fce-9ba2-fa15d5704683" containerName="istio-proxy"
Nov 01 14:43:28.428133 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:28.428071    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d3109df-047b-4a1f-a6d7-668ce0e4a43a" containerName="maas-api"
Nov 01 14:43:28.428133 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:28.428082    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="65c82e9c-1c36-4fce-9ba2-fa15d5704683" containerName="istio-proxy"
Nov 01 14:43:28.428133 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:28.428093    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a56cf6f-3dce-46b9-8b71-757373386d7f" containerName="maas-api"
Nov 01 14:43:28.428133 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:28.428101    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="871064b8-bc60-43e3-9267-6d42fb86afcf" containerName="maas-api"
Nov 01 14:43:28.431663 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:28.431644    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26a0c1-769bfd4c5f-9jwpp"
Nov 01 14:43:28.452752 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:28.452726    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-26a0c1-769bfd4c5f-9jwpp"]
Nov 01 14:43:28.474954 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:28.474912    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d460221-d6fe-4574-84eb-c589a367eebe-maas-api-tls\") pod \"maas-api-e2e-api-b-26a0c1-769bfd4c5f-9jwpp\" (UID: \"4d460221-d6fe-4574-84eb-c589a367eebe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26a0c1-769bfd4c5f-9jwpp"
Nov 01 14:43:28.475108 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:28.475000    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5xcw\" (UniqueName: \"kubernetes.io/projected/4d460221-d6fe-4574-84eb-c589a367eebe-kube-api-access-n5xcw\") pod \"maas-api-e2e-api-b-26a0c1-769bfd4c5f-9jwpp\" (UID: \"4d460221-d6fe-4574-84eb-c589a367eebe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26a0c1-769bfd4c5f-9jwpp"
Nov 01 14:43:28.475192 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:28.475155    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d460221-d6fe-4574-84eb-c589a367eebe-metrics-certs\") pod \"maas-api-e2e-api-b-26a0c1-769bfd4c5f-9jwpp\" (UID: \"4d460221-d6fe-4574-84eb-c589a367eebe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26a0c1-769bfd4c5f-9jwpp"
Nov 01 14:43:28.475419 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:28.475392    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d460221-d6fe-4574-84eb-c589a367eebe-openshift-service-ca\") pod \"maas-api-e2e-api-b-26a0c1-769bfd4c5f-9jwpp\" (UID: \"4d460221-d6fe-4574-84eb-c589a367eebe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26a0c1-769bfd4c5f-9jwpp"
Nov 01 14:43:28.480370 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:28.480351    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-26a0c1-serving-cert\""
Nov 01 14:43:28.576647 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:28.576597    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d460221-d6fe-4574-84eb-c589a367eebe-metrics-certs\") pod \"maas-api-e2e-api-b-26a0c1-769bfd4c5f-9jwpp\" (UID: \"4d460221-d6fe-4574-84eb-c589a367eebe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26a0c1-769bfd4c5f-9jwpp"
Nov 01 14:43:28.577257 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:28.576697    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d460221-d6fe-4574-84eb-c589a367eebe-openshift-service-ca\") pod \"maas-api-e2e-api-b-26a0c1-769bfd4c5f-9jwpp\" (UID: \"4d460221-d6fe-4574-84eb-c589a367eebe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26a0c1-769bfd4c5f-9jwpp"
Nov 01 14:43:28.577257 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:28.576828    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d460221-d6fe-4574-84eb-c589a367eebe-maas-api-tls\") pod \"maas-api-e2e-api-b-26a0c1-769bfd4c5f-9jwpp\" (UID: \"4d460221-d6fe-4574-84eb-c589a367eebe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26a0c1-769bfd4c5f-9jwpp"
Nov 01 14:43:28.577257 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:28.576894    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5xcw\" (UniqueName: \"kubernetes.io/projected/4d460221-d6fe-4574-84eb-c589a367eebe-kube-api-access-n5xcw\") pod \"maas-api-e2e-api-b-26a0c1-769bfd4c5f-9jwpp\" (UID: \"4d460221-d6fe-4574-84eb-c589a367eebe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26a0c1-769bfd4c5f-9jwpp"
Nov 01 14:43:28.577592 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:43:28.577570    4993 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-26a0c1-serving-cert: secret "maas-api-e2e-api-b-26a0c1-serving-cert" not found
Nov 01 14:43:28.577802 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:43:28.577791    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d460221-d6fe-4574-84eb-c589a367eebe-maas-api-tls podName:4d460221-d6fe-4574-84eb-c589a367eebe nodeName:}" failed. No retries permitted until 2026-11-01 14:43:29.077768522 +0000 UTC m=+1596.407631256 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4d460221-d6fe-4574-84eb-c589a367eebe-maas-api-tls") pod "maas-api-e2e-api-b-26a0c1-769bfd4c5f-9jwpp" (UID: "4d460221-d6fe-4574-84eb-c589a367eebe") : secret "maas-api-e2e-api-b-26a0c1-serving-cert" not found
Nov 01 14:43:28.578121 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:28.578001    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d460221-d6fe-4574-84eb-c589a367eebe-openshift-service-ca\") pod \"maas-api-e2e-api-b-26a0c1-769bfd4c5f-9jwpp\" (UID: \"4d460221-d6fe-4574-84eb-c589a367eebe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26a0c1-769bfd4c5f-9jwpp"
Nov 01 14:43:28.580791 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:28.580756    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d460221-d6fe-4574-84eb-c589a367eebe-metrics-certs\") pod \"maas-api-e2e-api-b-26a0c1-769bfd4c5f-9jwpp\" (UID: \"4d460221-d6fe-4574-84eb-c589a367eebe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26a0c1-769bfd4c5f-9jwpp"
Nov 01 14:43:28.636917 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:28.636888    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5xcw\" (UniqueName: \"kubernetes.io/projected/4d460221-d6fe-4574-84eb-c589a367eebe-kube-api-access-n5xcw\") pod \"maas-api-e2e-api-b-26a0c1-769bfd4c5f-9jwpp\" (UID: \"4d460221-d6fe-4574-84eb-c589a367eebe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26a0c1-769bfd4c5f-9jwpp"
Nov 01 14:43:29.087231 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:29.087086    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d460221-d6fe-4574-84eb-c589a367eebe-maas-api-tls\") pod \"maas-api-e2e-api-b-26a0c1-769bfd4c5f-9jwpp\" (UID: \"4d460221-d6fe-4574-84eb-c589a367eebe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26a0c1-769bfd4c5f-9jwpp"
Nov 01 14:43:29.089529 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:29.089499    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d460221-d6fe-4574-84eb-c589a367eebe-maas-api-tls\") pod \"maas-api-e2e-api-b-26a0c1-769bfd4c5f-9jwpp\" (UID: \"4d460221-d6fe-4574-84eb-c589a367eebe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26a0c1-769bfd4c5f-9jwpp"
Nov 01 14:43:29.347491 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:29.344775    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26a0c1-769bfd4c5f-9jwpp"
Nov 01 14:43:29.473228 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:29.472911    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr"
Nov 01 14:43:29.530520 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:29.530478    4993 generic.go:356] "Generic (PLEG): container finished" podID="0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8" containerID="91e56a8579f5bb611a79d73a592f92672f8a575458be3ff73967704425a366b4" exitCode=0
Nov 01 14:43:29.530738 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:29.530544    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr" event={"ID":"0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8","Type":"ContainerDied","Data":"91e56a8579f5bb611a79d73a592f92672f8a575458be3ff73967704425a366b4"}
Nov 01 14:43:29.530738 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:29.530636    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr" event={"ID":"0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8","Type":"ContainerDied","Data":"7be62e690055f257532b935e5c16414e4f1e0eb90e6705fc17d14b6cc516a6d5"}
Nov 01 14:43:29.530738 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:29.530666    4993 scope.go:122] "RemoveContainer" containerID="91e56a8579f5bb611a79d73a592f92672f8a575458be3ff73967704425a366b4"
Nov 01 14:43:29.530919 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:29.530823    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr"
Nov 01 14:43:29.545454 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:29.545433    4993 scope.go:122] "RemoveContainer" containerID="91e56a8579f5bb611a79d73a592f92672f8a575458be3ff73967704425a366b4"
Nov 01 14:43:29.545836 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:43:29.545811    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91e56a8579f5bb611a79d73a592f92672f8a575458be3ff73967704425a366b4\": container with ID starting with 91e56a8579f5bb611a79d73a592f92672f8a575458be3ff73967704425a366b4 not found: ID does not exist" containerID="91e56a8579f5bb611a79d73a592f92672f8a575458be3ff73967704425a366b4"
Nov 01 14:43:29.545900 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:29.545844    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91e56a8579f5bb611a79d73a592f92672f8a575458be3ff73967704425a366b4"} err="failed to get container status \"91e56a8579f5bb611a79d73a592f92672f8a575458be3ff73967704425a366b4\": rpc error: code = NotFound desc = could not find container \"91e56a8579f5bb611a79d73a592f92672f8a575458be3ff73967704425a366b4\": container with ID starting with 91e56a8579f5bb611a79d73a592f92672f8a575458be3ff73967704425a366b4 not found: ID does not exist"
Nov 01 14:43:29.597939 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:29.597906    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8-workload-socket\") pod \"0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8\" (UID: \"0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8\") "
Nov 01 14:43:29.598399 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:29.597960    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8-kube-api-access-zdzrz\" (UniqueName: \"kubernetes.io/projected/0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8-kube-api-access-zdzrz\") pod \"0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8\" (UID: \"0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8\") "
Nov 01 14:43:29.598399 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:29.598004    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8-istio-token\" (UniqueName: \"kubernetes.io/projected/0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8-istio-token\") pod \"0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8\" (UID: \"0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8\") "
Nov 01 14:43:29.598399 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:29.598217    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8-istio-data\") pod \"0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8\" (UID: \"0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8\") "
Nov 01 14:43:29.598399 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:29.598278    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8-istio-podinfo\") pod \"0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8\" (UID: \"0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8\") "
Nov 01 14:43:29.598399 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:29.598378    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8-credential-socket\") pod \"0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8\" (UID: \"0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8\") "
Nov 01 14:43:29.598688 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:29.598636    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8-workload-socket" pod "0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8" (UID: "0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:43:29.598688 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:29.598630    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8-credential-socket" pod "0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8" (UID: "0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:43:29.598688 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:29.598665    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8-istio-envoy\") pod \"0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8\" (UID: \"0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8\") "
Nov 01 14:43:29.598846 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:29.598789    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8-workload-certs\") pod \"0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8\" (UID: \"0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8\") "
Nov 01 14:43:29.598846 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:29.598795    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8-istio-data" pod "0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8" (UID: "0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:43:29.598948 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:29.598871    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8-istiod-ca-cert\") pod \"0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8\" (UID: \"0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8\") "
Nov 01 14:43:29.599223 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:29.599198    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8-workload-certs" pod "0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8" (UID: "0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:43:29.599418 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:29.599393    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8-istiod-ca-cert" pod "0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8" (UID: "0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:43:29.599550 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:29.599534    4993 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8-istiod-ca-cert\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:43:29.599598 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:29.599554    4993 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8-workload-socket\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:43:29.599598 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:29.599568    4993 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8-istio-data\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:43:29.599598 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:29.599582    4993 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8-credential-socket\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:43:29.599598 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:29.599596    4993 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8-workload-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:43:29.602231 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:29.602205    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8-istio-token" pod "0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8" (UID: "0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:43:29.602331 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:29.602240    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8-kube-api-access-zdzrz" pod "0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8" (UID: "0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8"). InnerVolumeSpecName "kube-api-access-zdzrz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:43:29.602331 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:29.602301    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8-istio-podinfo" pod "0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8" (UID: "0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 14:43:29.603811 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:29.603790    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8-istio-envoy" pod "0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8" (UID: "0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:43:29.700477 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:29.700441    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdzrz\" (UniqueName: \"kubernetes.io/projected/0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8-kube-api-access-zdzrz\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:43:29.700477 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:29.700471    4993 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8-istio-token\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:43:29.700477 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:29.700482    4993 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8-istio-podinfo\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:43:29.700714 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:29.700491    4993 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8-istio-envoy\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:43:29.787826 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:29.787774    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-26a0c1-769bfd4c5f-9jwpp"]
Nov 01 14:43:29.793074 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:43:29.793042    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d460221_d6fe_4574_84eb_c589a367eebe.slice/crio-a600d85d84db0b3c61a495d23ee1f2045816685ea38e419bc6db942388796819 WatchSource:0}: Error finding container a600d85d84db0b3c61a495d23ee1f2045816685ea38e419bc6db942388796819: Status 404 returned error can't find the container with id a600d85d84db0b3c61a495d23ee1f2045816685ea38e419bc6db942388796819
Nov 01 14:43:29.949590 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:29.949494    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr"]
Nov 01 14:43:29.953095 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:29.953065    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr"]
Nov 01 14:43:30.388523 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:30.388483    4993 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 14:43:30.388724 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:30.388551    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfbxr" podUID="0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 14:43:30.538707 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:30.538664    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26a0c1-769bfd4c5f-9jwpp" event={"ID":"4d460221-d6fe-4574-84eb-c589a367eebe","Type":"ContainerStarted","Data":"a600d85d84db0b3c61a495d23ee1f2045816685ea38e419bc6db942388796819"}
Nov 01 14:43:30.802027 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:30.801990    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8" path="/var/lib/kubelet/pods/0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8/volumes"
Nov 01 14:43:31.557953 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:31.557898    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26a0c1-769bfd4c5f-9jwpp" event={"ID":"4d460221-d6fe-4574-84eb-c589a367eebe","Type":"ContainerStarted","Data":"13db682a608e184fea4f724bbf773e6c133debd22c3356d3abc655a53992b103"}
Nov 01 14:43:31.558863 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:31.558827    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26a0c1-769bfd4c5f-9jwpp"
Nov 01 14:43:31.668106 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:31.668050    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26a0c1-769bfd4c5f-9jwpp" podStartSLOduration=2.836941849 podStartE2EDuration="3.66803509s" totalImagesPullingTime="831.093241ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:43:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:43:31.665637208 +0000 UTC m=+1598.995499946" watchObservedRunningTime="2026-11-01 14:43:31.66803509 +0000 UTC m=+1598.997897826"
Nov 01 14:43:33.647481 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:33.647447    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld"]
Nov 01 14:43:33.647998 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:33.647712    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld" podUID="4df5d536-0c2d-43ae-9e86-61e2b2d64ee6" containerName="istio-proxy" containerID="cri-o://041dce0d8ce366c943416cbceee166bd690b19d12c584e2146c8742d5d62083d" gracePeriod=30
Nov 01 14:43:37.571857 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:37.571829    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26a0c1-769bfd4c5f-9jwpp"
Nov 01 14:43:38.927564 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:38.927502    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld"
Nov 01 14:43:39.042627 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:39.042581    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4df5d536-0c2d-43ae-9e86-61e2b2d64ee6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4df5d536-0c2d-43ae-9e86-61e2b2d64ee6-istiod-ca-cert\") pod \"4df5d536-0c2d-43ae-9e86-61e2b2d64ee6\" (UID: \"4df5d536-0c2d-43ae-9e86-61e2b2d64ee6\") "
Nov 01 14:43:39.042785 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:39.042645    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4df5d536-0c2d-43ae-9e86-61e2b2d64ee6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4df5d536-0c2d-43ae-9e86-61e2b2d64ee6-istio-podinfo\") pod \"4df5d536-0c2d-43ae-9e86-61e2b2d64ee6\" (UID: \"4df5d536-0c2d-43ae-9e86-61e2b2d64ee6\") "
Nov 01 14:43:39.042785 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:39.042688    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4df5d536-0c2d-43ae-9e86-61e2b2d64ee6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4df5d536-0c2d-43ae-9e86-61e2b2d64ee6-workload-certs\") pod \"4df5d536-0c2d-43ae-9e86-61e2b2d64ee6\" (UID: \"4df5d536-0c2d-43ae-9e86-61e2b2d64ee6\") "
Nov 01 14:43:39.042910 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:39.042785    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4df5d536-0c2d-43ae-9e86-61e2b2d64ee6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4df5d536-0c2d-43ae-9e86-61e2b2d64ee6-workload-socket\") pod \"4df5d536-0c2d-43ae-9e86-61e2b2d64ee6\" (UID: \"4df5d536-0c2d-43ae-9e86-61e2b2d64ee6\") "
Nov 01 14:43:39.042910 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:39.042828    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4df5d536-0c2d-43ae-9e86-61e2b2d64ee6-kube-api-access-7mjs8\" (UniqueName: \"kubernetes.io/projected/4df5d536-0c2d-43ae-9e86-61e2b2d64ee6-kube-api-access-7mjs8\") pod \"4df5d536-0c2d-43ae-9e86-61e2b2d64ee6\" (UID: \"4df5d536-0c2d-43ae-9e86-61e2b2d64ee6\") "
Nov 01 14:43:39.043028 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:39.042952    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4df5d536-0c2d-43ae-9e86-61e2b2d64ee6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4df5d536-0c2d-43ae-9e86-61e2b2d64ee6-istio-envoy\") pod \"4df5d536-0c2d-43ae-9e86-61e2b2d64ee6\" (UID: \"4df5d536-0c2d-43ae-9e86-61e2b2d64ee6\") "
Nov 01 14:43:39.043028 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:39.042981    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4df5d536-0c2d-43ae-9e86-61e2b2d64ee6-istio-token\" (UniqueName: \"kubernetes.io/projected/4df5d536-0c2d-43ae-9e86-61e2b2d64ee6-istio-token\") pod \"4df5d536-0c2d-43ae-9e86-61e2b2d64ee6\" (UID: \"4df5d536-0c2d-43ae-9e86-61e2b2d64ee6\") "
Nov 01 14:43:39.043028 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:39.043010    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4df5d536-0c2d-43ae-9e86-61e2b2d64ee6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4df5d536-0c2d-43ae-9e86-61e2b2d64ee6-credential-socket\") pod \"4df5d536-0c2d-43ae-9e86-61e2b2d64ee6\" (UID: \"4df5d536-0c2d-43ae-9e86-61e2b2d64ee6\") "
Nov 01 14:43:39.043203 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:39.043020    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4df5d536-0c2d-43ae-9e86-61e2b2d64ee6-istiod-ca-cert" pod "4df5d536-0c2d-43ae-9e86-61e2b2d64ee6" (UID: "4df5d536-0c2d-43ae-9e86-61e2b2d64ee6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:43:39.043203 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:39.043024    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4df5d536-0c2d-43ae-9e86-61e2b2d64ee6-workload-certs" pod "4df5d536-0c2d-43ae-9e86-61e2b2d64ee6" (UID: "4df5d536-0c2d-43ae-9e86-61e2b2d64ee6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:43:39.043203 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:39.043103    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4df5d536-0c2d-43ae-9e86-61e2b2d64ee6-workload-socket" pod "4df5d536-0c2d-43ae-9e86-61e2b2d64ee6" (UID: "4df5d536-0c2d-43ae-9e86-61e2b2d64ee6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:43:39.043203 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:39.043109    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4df5d536-0c2d-43ae-9e86-61e2b2d64ee6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4df5d536-0c2d-43ae-9e86-61e2b2d64ee6-istio-data\") pod \"4df5d536-0c2d-43ae-9e86-61e2b2d64ee6\" (UID: \"4df5d536-0c2d-43ae-9e86-61e2b2d64ee6\") "
Nov 01 14:43:39.043409 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:39.043340    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4df5d536-0c2d-43ae-9e86-61e2b2d64ee6-credential-socket" pod "4df5d536-0c2d-43ae-9e86-61e2b2d64ee6" (UID: "4df5d536-0c2d-43ae-9e86-61e2b2d64ee6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:43:39.043409 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:39.043366    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4df5d536-0c2d-43ae-9e86-61e2b2d64ee6-istio-data" pod "4df5d536-0c2d-43ae-9e86-61e2b2d64ee6" (UID: "4df5d536-0c2d-43ae-9e86-61e2b2d64ee6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:43:39.043788 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:39.043764    4993 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4df5d536-0c2d-43ae-9e86-61e2b2d64ee6-istiod-ca-cert\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:43:39.043878 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:39.043792    4993 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4df5d536-0c2d-43ae-9e86-61e2b2d64ee6-workload-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:43:39.043878 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:39.043807    4993 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4df5d536-0c2d-43ae-9e86-61e2b2d64ee6-workload-socket\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:43:39.043878 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:39.043823    4993 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4df5d536-0c2d-43ae-9e86-61e2b2d64ee6-credential-socket\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:43:39.043878 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:39.043836    4993 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4df5d536-0c2d-43ae-9e86-61e2b2d64ee6-istio-data\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:43:39.045268 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:39.045237    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4df5d536-0c2d-43ae-9e86-61e2b2d64ee6-istio-envoy" pod "4df5d536-0c2d-43ae-9e86-61e2b2d64ee6" (UID: "4df5d536-0c2d-43ae-9e86-61e2b2d64ee6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:43:39.045268 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:39.045248    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4df5d536-0c2d-43ae-9e86-61e2b2d64ee6-kube-api-access-7mjs8" pod "4df5d536-0c2d-43ae-9e86-61e2b2d64ee6" (UID: "4df5d536-0c2d-43ae-9e86-61e2b2d64ee6"). InnerVolumeSpecName "kube-api-access-7mjs8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:43:39.045542 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:39.045521    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4df5d536-0c2d-43ae-9e86-61e2b2d64ee6-istio-podinfo" pod "4df5d536-0c2d-43ae-9e86-61e2b2d64ee6" (UID: "4df5d536-0c2d-43ae-9e86-61e2b2d64ee6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 14:43:39.045587 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:39.045552    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4df5d536-0c2d-43ae-9e86-61e2b2d64ee6-istio-token" pod "4df5d536-0c2d-43ae-9e86-61e2b2d64ee6" (UID: "4df5d536-0c2d-43ae-9e86-61e2b2d64ee6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:43:39.145959 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:39.145832    4993 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4df5d536-0c2d-43ae-9e86-61e2b2d64ee6-istio-podinfo\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:43:39.145959 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:39.145872    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mjs8\" (UniqueName: \"kubernetes.io/projected/4df5d536-0c2d-43ae-9e86-61e2b2d64ee6-kube-api-access-7mjs8\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:43:39.145959 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:39.145891    4993 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4df5d536-0c2d-43ae-9e86-61e2b2d64ee6-istio-envoy\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:43:39.145959 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:39.145907    4993 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4df5d536-0c2d-43ae-9e86-61e2b2d64ee6-istio-token\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:43:39.619236 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:39.619191    4993 generic.go:356] "Generic (PLEG): container finished" podID="4df5d536-0c2d-43ae-9e86-61e2b2d64ee6" containerID="041dce0d8ce366c943416cbceee166bd690b19d12c584e2146c8742d5d62083d" exitCode=0
Nov 01 14:43:39.619451 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:39.619253    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld"
Nov 01 14:43:39.619451 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:39.619280    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld" event={"ID":"4df5d536-0c2d-43ae-9e86-61e2b2d64ee6","Type":"ContainerDied","Data":"041dce0d8ce366c943416cbceee166bd690b19d12c584e2146c8742d5d62083d"}
Nov 01 14:43:39.619451 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:39.619438    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld" event={"ID":"4df5d536-0c2d-43ae-9e86-61e2b2d64ee6","Type":"ContainerDied","Data":"e8b467444eb570460d5e60be72f29208528b3c465d3c40d880e8972d4197534c"}
Nov 01 14:43:39.619637 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:39.619466    4993 scope.go:122] "RemoveContainer" containerID="041dce0d8ce366c943416cbceee166bd690b19d12c584e2146c8742d5d62083d"
Nov 01 14:43:39.634376 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:39.633803    4993 scope.go:122] "RemoveContainer" containerID="041dce0d8ce366c943416cbceee166bd690b19d12c584e2146c8742d5d62083d"
Nov 01 14:43:39.634376 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:43:39.634205    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"041dce0d8ce366c943416cbceee166bd690b19d12c584e2146c8742d5d62083d\": container with ID starting with 041dce0d8ce366c943416cbceee166bd690b19d12c584e2146c8742d5d62083d not found: ID does not exist" containerID="041dce0d8ce366c943416cbceee166bd690b19d12c584e2146c8742d5d62083d"
Nov 01 14:43:39.634376 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:39.634244    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"041dce0d8ce366c943416cbceee166bd690b19d12c584e2146c8742d5d62083d"} err="failed to get container status \"041dce0d8ce366c943416cbceee166bd690b19d12c584e2146c8742d5d62083d\": rpc error: code = NotFound desc = could not find container \"041dce0d8ce366c943416cbceee166bd690b19d12c584e2146c8742d5d62083d\": container with ID starting with 041dce0d8ce366c943416cbceee166bd690b19d12c584e2146c8742d5d62083d not found: ID does not exist"
Nov 01 14:43:39.784726 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:39.784694    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld"]
Nov 01 14:43:39.789660 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:39.789614    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-9xlld"]
Nov 01 14:43:40.796630 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:40.796589    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4df5d536-0c2d-43ae-9e86-61e2b2d64ee6" path="/var/lib/kubelet/pods/4df5d536-0c2d-43ae-9e86-61e2b2d64ee6/volumes"
Nov 01 14:43:49.992995 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:49.992955    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-ks7q6"]
Nov 01 14:43:49.993567 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:49.993550    4993 cpu_manager.go:414] "containerMap: removing container" podUID="4df5d536-0c2d-43ae-9e86-61e2b2d64ee6" containerName="istio-proxy"
Nov 01 14:43:49.993612 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:49.993568    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4df5d536-0c2d-43ae-9e86-61e2b2d64ee6" containerName="istio-proxy"
Nov 01 14:43:49.993612 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:49.993588    4993 cpu_manager.go:414] "containerMap: removing container" podUID="0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8" containerName="istio-proxy"
Nov 01 14:43:49.993612 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:49.993595    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8" containerName="istio-proxy"
Nov 01 14:43:49.993723 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:49.993713    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="4df5d536-0c2d-43ae-9e86-61e2b2d64ee6" containerName="istio-proxy"
Nov 01 14:43:49.993757 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:49.993725    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c2f4202-6c87-4721-bc5e-5fb7eb6fa6a8" containerName="istio-proxy"
Nov 01 14:43:49.997154 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:49.997134    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-ks7q6"
Nov 01 14:43:50.096516 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:50.096482    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-ks7q6"]
Nov 01 14:43:50.123799 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:50.123761    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x64wf\" (UniqueName: \"kubernetes.io/projected/040af721-9da6-4df4-a334-a5d95c482f3b-kube-api-access-x64wf\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-ks7q6\" (UID: \"040af721-9da6-4df4-a334-a5d95c482f3b\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-ks7q6"
Nov 01 14:43:50.123950 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:50.123811    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/040af721-9da6-4df4-a334-a5d95c482f3b-credential-socket\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-ks7q6\" (UID: \"040af721-9da6-4df4-a334-a5d95c482f3b\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-ks7q6"
Nov 01 14:43:50.123950 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:50.123919    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/040af721-9da6-4df4-a334-a5d95c482f3b-istio-envoy\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-ks7q6\" (UID: \"040af721-9da6-4df4-a334-a5d95c482f3b\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-ks7q6"
Nov 01 14:43:50.124064 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:50.123992    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/040af721-9da6-4df4-a334-a5d95c482f3b-istio-data\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-ks7q6\" (UID: \"040af721-9da6-4df4-a334-a5d95c482f3b\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-ks7q6"
Nov 01 14:43:50.124207 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:50.124162    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/040af721-9da6-4df4-a334-a5d95c482f3b-workload-certs\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-ks7q6\" (UID: \"040af721-9da6-4df4-a334-a5d95c482f3b\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-ks7q6"
Nov 01 14:43:50.124259 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:50.124234    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/040af721-9da6-4df4-a334-a5d95c482f3b-istiod-ca-cert\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-ks7q6\" (UID: \"040af721-9da6-4df4-a334-a5d95c482f3b\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-ks7q6"
Nov 01 14:43:50.124312 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:50.124297    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/040af721-9da6-4df4-a334-a5d95c482f3b-istio-podinfo\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-ks7q6\" (UID: \"040af721-9da6-4df4-a334-a5d95c482f3b\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-ks7q6"
Nov 01 14:43:50.124410 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:50.124393    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/040af721-9da6-4df4-a334-a5d95c482f3b-workload-socket\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-ks7q6\" (UID: \"040af721-9da6-4df4-a334-a5d95c482f3b\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-ks7q6"
Nov 01 14:43:50.124477 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:50.124461    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/040af721-9da6-4df4-a334-a5d95c482f3b-istio-token\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-ks7q6\" (UID: \"040af721-9da6-4df4-a334-a5d95c482f3b\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-ks7q6"
Nov 01 14:43:50.139228 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:50.139201    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-26a0c1-openshift-default-dockercfg-xrpzs\""
Nov 01 14:43:50.225752 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:50.225719    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/040af721-9da6-4df4-a334-a5d95c482f3b-workload-certs\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-ks7q6\" (UID: \"040af721-9da6-4df4-a334-a5d95c482f3b\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-ks7q6"
Nov 01 14:43:50.225752 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:50.225758    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/040af721-9da6-4df4-a334-a5d95c482f3b-istiod-ca-cert\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-ks7q6\" (UID: \"040af721-9da6-4df4-a334-a5d95c482f3b\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-ks7q6"
Nov 01 14:43:50.225959 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:50.225806    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/040af721-9da6-4df4-a334-a5d95c482f3b-istio-podinfo\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-ks7q6\" (UID: \"040af721-9da6-4df4-a334-a5d95c482f3b\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-ks7q6"
Nov 01 14:43:50.225959 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:50.225837    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/040af721-9da6-4df4-a334-a5d95c482f3b-workload-socket\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-ks7q6\" (UID: \"040af721-9da6-4df4-a334-a5d95c482f3b\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-ks7q6"
Nov 01 14:43:50.225959 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:50.225901    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/040af721-9da6-4df4-a334-a5d95c482f3b-istio-token\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-ks7q6\" (UID: \"040af721-9da6-4df4-a334-a5d95c482f3b\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-ks7q6"
Nov 01 14:43:50.226102 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:50.225963    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x64wf\" (UniqueName: \"kubernetes.io/projected/040af721-9da6-4df4-a334-a5d95c482f3b-kube-api-access-x64wf\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-ks7q6\" (UID: \"040af721-9da6-4df4-a334-a5d95c482f3b\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-ks7q6"
Nov 01 14:43:50.226102 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:50.225990    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/040af721-9da6-4df4-a334-a5d95c482f3b-credential-socket\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-ks7q6\" (UID: \"040af721-9da6-4df4-a334-a5d95c482f3b\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-ks7q6"
Nov 01 14:43:50.226102 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:50.226028    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/040af721-9da6-4df4-a334-a5d95c482f3b-istio-envoy\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-ks7q6\" (UID: \"040af721-9da6-4df4-a334-a5d95c482f3b\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-ks7q6"
Nov 01 14:43:50.226102 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:50.226077    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/040af721-9da6-4df4-a334-a5d95c482f3b-istio-data\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-ks7q6\" (UID: \"040af721-9da6-4df4-a334-a5d95c482f3b\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-ks7q6"
Nov 01 14:43:50.226393 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:50.226149    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/040af721-9da6-4df4-a334-a5d95c482f3b-workload-certs\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-ks7q6\" (UID: \"040af721-9da6-4df4-a334-a5d95c482f3b\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-ks7q6"
Nov 01 14:43:50.226393 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:50.226309    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/040af721-9da6-4df4-a334-a5d95c482f3b-workload-socket\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-ks7q6\" (UID: \"040af721-9da6-4df4-a334-a5d95c482f3b\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-ks7q6"
Nov 01 14:43:50.226496 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:50.226422    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/040af721-9da6-4df4-a334-a5d95c482f3b-credential-socket\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-ks7q6\" (UID: \"040af721-9da6-4df4-a334-a5d95c482f3b\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-ks7q6"
Nov 01 14:43:50.226594 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:50.226573    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/040af721-9da6-4df4-a334-a5d95c482f3b-istio-data\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-ks7q6\" (UID: \"040af721-9da6-4df4-a334-a5d95c482f3b\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-ks7q6"
Nov 01 14:43:50.226717 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:50.226698    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/040af721-9da6-4df4-a334-a5d95c482f3b-istiod-ca-cert\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-ks7q6\" (UID: \"040af721-9da6-4df4-a334-a5d95c482f3b\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-ks7q6"
Nov 01 14:43:50.228315 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:50.228292    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/040af721-9da6-4df4-a334-a5d95c482f3b-istio-envoy\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-ks7q6\" (UID: \"040af721-9da6-4df4-a334-a5d95c482f3b\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-ks7q6"
Nov 01 14:43:50.228553 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:50.228536    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/040af721-9da6-4df4-a334-a5d95c482f3b-istio-podinfo\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-ks7q6\" (UID: \"040af721-9da6-4df4-a334-a5d95c482f3b\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-ks7q6"
Nov 01 14:43:50.280510 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:50.280444    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/040af721-9da6-4df4-a334-a5d95c482f3b-istio-token\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-ks7q6\" (UID: \"040af721-9da6-4df4-a334-a5d95c482f3b\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-ks7q6"
Nov 01 14:43:50.280765 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:50.280742    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x64wf\" (UniqueName: \"kubernetes.io/projected/040af721-9da6-4df4-a334-a5d95c482f3b-kube-api-access-x64wf\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-ks7q6\" (UID: \"040af721-9da6-4df4-a334-a5d95c482f3b\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-ks7q6"
Nov 01 14:43:50.311687 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:50.311660    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-ks7q6"
Nov 01 14:43:50.745204 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:50.745138    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-ks7q6"]
Nov 01 14:43:50.750053 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:43:50.750025    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod040af721_9da6_4df4_a334_a5d95c482f3b.slice/crio-c2c421cc10766da39bc12d6dc17641065944c4ae22d2d61be351a891e2c56ea0 WatchSource:0}: Error finding container c2c421cc10766da39bc12d6dc17641065944c4ae22d2d61be351a891e2c56ea0: Status 404 returned error can't find the container with id c2c421cc10766da39bc12d6dc17641065944c4ae22d2d61be351a891e2c56ea0
Nov 01 14:43:50.753518 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:50.753475    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 14:43:50.753591 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:50.753569    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 14:43:50.753626 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:50.753612    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 14:43:51.709274 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:51.709231    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-ks7q6" event={"ID":"040af721-9da6-4df4-a334-a5d95c482f3b","Type":"ContainerStarted","Data":"8733d4402c69b4f52328853a6a7de8ec05d632d71f82b5fd3d906931df1f08ef"}
Nov 01 14:43:51.709759 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:51.709331    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-ks7q6" event={"ID":"040af721-9da6-4df4-a334-a5d95c482f3b","Type":"ContainerStarted","Data":"c2c421cc10766da39bc12d6dc17641065944c4ae22d2d61be351a891e2c56ea0"}
Nov 01 14:43:51.922228 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:51.922142    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-ks7q6" podStartSLOduration=2.9221236839999998 podStartE2EDuration="2.922123684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:43:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:43:51.872746351 +0000 UTC m=+1619.202609086" watchObservedRunningTime="2026-11-01 14:43:51.922123684 +0000 UTC m=+1619.251986421"
Nov 01 14:43:51.930861 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:51.930827    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-116e85-65676f99fd-b7vxz"]
Nov 01 14:43:51.931097 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:51.931071    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-116e85-65676f99fd-b7vxz" podUID="cefdf46f-afbf-44e8-8d54-a3d5b0e7c348" containerName="maas-api" containerID="cri-o://289a6e314f4b585abe36fa9325bc78662de2527d5f2a6860051865028cc3fd0a" gracePeriod=30
Nov 01 14:43:52.328711 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:52.322889    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-ks7q6"
Nov 01 14:43:52.329770 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:52.329407    4993 patch_prober.go:29] interesting pod/e2e-api-b-26a0c1-openshift-default-76c87f685-ks7q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Nov 01 14:43:52.329770 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:52.329465    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-ks7q6" podUID="040af721-9da6-4df4-a334-a5d95c482f3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Nov 01 14:43:52.702017 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:52.701987    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-116e85-65676f99fd-b7vxz"
Nov 01 14:43:52.721678 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:52.721641    4993 generic.go:356] "Generic (PLEG): container finished" podID="cefdf46f-afbf-44e8-8d54-a3d5b0e7c348" containerID="289a6e314f4b585abe36fa9325bc78662de2527d5f2a6860051865028cc3fd0a" exitCode=0
Nov 01 14:43:52.722183 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:52.721685    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-116e85-65676f99fd-b7vxz"
Nov 01 14:43:52.722183 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:52.721723    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-116e85-65676f99fd-b7vxz" event={"ID":"cefdf46f-afbf-44e8-8d54-a3d5b0e7c348","Type":"ContainerDied","Data":"289a6e314f4b585abe36fa9325bc78662de2527d5f2a6860051865028cc3fd0a"}
Nov 01 14:43:52.722183 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:52.721788    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-116e85-65676f99fd-b7vxz" event={"ID":"cefdf46f-afbf-44e8-8d54-a3d5b0e7c348","Type":"ContainerDied","Data":"3e802e2450a5846b1e7392510d6104f0f568d0b3958aa00ef43c1e32c81a59b8"}
Nov 01 14:43:52.722183 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:52.721825    4993 scope.go:122] "RemoveContainer" containerID="289a6e314f4b585abe36fa9325bc78662de2527d5f2a6860051865028cc3fd0a"
Nov 01 14:43:52.731656 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:52.731640    4993 scope.go:122] "RemoveContainer" containerID="289a6e314f4b585abe36fa9325bc78662de2527d5f2a6860051865028cc3fd0a"
Nov 01 14:43:52.731954 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:43:52.731931    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"289a6e314f4b585abe36fa9325bc78662de2527d5f2a6860051865028cc3fd0a\": container with ID starting with 289a6e314f4b585abe36fa9325bc78662de2527d5f2a6860051865028cc3fd0a not found: ID does not exist" containerID="289a6e314f4b585abe36fa9325bc78662de2527d5f2a6860051865028cc3fd0a"
Nov 01 14:43:52.732053 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:52.731967    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"289a6e314f4b585abe36fa9325bc78662de2527d5f2a6860051865028cc3fd0a"} err="failed to get container status \"289a6e314f4b585abe36fa9325bc78662de2527d5f2a6860051865028cc3fd0a\": rpc error: code = NotFound desc = could not find container \"289a6e314f4b585abe36fa9325bc78662de2527d5f2a6860051865028cc3fd0a\": container with ID starting with 289a6e314f4b585abe36fa9325bc78662de2527d5f2a6860051865028cc3fd0a not found: ID does not exist"
Nov 01 14:43:52.818482 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:52.818448    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cefdf46f-afbf-44e8-8d54-a3d5b0e7c348-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cefdf46f-afbf-44e8-8d54-a3d5b0e7c348-openshift-service-ca\") pod \"cefdf46f-afbf-44e8-8d54-a3d5b0e7c348\" (UID: \"cefdf46f-afbf-44e8-8d54-a3d5b0e7c348\") "
Nov 01 14:43:52.818675 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:52.818619    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cefdf46f-afbf-44e8-8d54-a3d5b0e7c348-kube-api-access-mgjcr\" (UniqueName: \"kubernetes.io/projected/cefdf46f-afbf-44e8-8d54-a3d5b0e7c348-kube-api-access-mgjcr\") pod \"cefdf46f-afbf-44e8-8d54-a3d5b0e7c348\" (UID: \"cefdf46f-afbf-44e8-8d54-a3d5b0e7c348\") "
Nov 01 14:43:52.818756 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:52.818689    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cefdf46f-afbf-44e8-8d54-a3d5b0e7c348-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cefdf46f-afbf-44e8-8d54-a3d5b0e7c348-maas-api-tls\") pod \"cefdf46f-afbf-44e8-8d54-a3d5b0e7c348\" (UID: \"cefdf46f-afbf-44e8-8d54-a3d5b0e7c348\") "
Nov 01 14:43:52.818812 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:52.818760    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cefdf46f-afbf-44e8-8d54-a3d5b0e7c348-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cefdf46f-afbf-44e8-8d54-a3d5b0e7c348-metrics-certs\") pod \"cefdf46f-afbf-44e8-8d54-a3d5b0e7c348\" (UID: \"cefdf46f-afbf-44e8-8d54-a3d5b0e7c348\") "
Nov 01 14:43:52.818812 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:52.818789    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cefdf46f-afbf-44e8-8d54-a3d5b0e7c348-openshift-service-ca" pod "cefdf46f-afbf-44e8-8d54-a3d5b0e7c348" (UID: "cefdf46f-afbf-44e8-8d54-a3d5b0e7c348"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:43:52.821698 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:52.819383    4993 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cefdf46f-afbf-44e8-8d54-a3d5b0e7c348-openshift-service-ca\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:43:52.821698 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:52.821245    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cefdf46f-afbf-44e8-8d54-a3d5b0e7c348-maas-api-tls" pod "cefdf46f-afbf-44e8-8d54-a3d5b0e7c348" (UID: "cefdf46f-afbf-44e8-8d54-a3d5b0e7c348"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:43:52.827316 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:52.822103    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cefdf46f-afbf-44e8-8d54-a3d5b0e7c348-kube-api-access-mgjcr" pod "cefdf46f-afbf-44e8-8d54-a3d5b0e7c348" (UID: "cefdf46f-afbf-44e8-8d54-a3d5b0e7c348"). InnerVolumeSpecName "kube-api-access-mgjcr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:43:52.827316 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:52.822215    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cefdf46f-afbf-44e8-8d54-a3d5b0e7c348-metrics-certs" pod "cefdf46f-afbf-44e8-8d54-a3d5b0e7c348" (UID: "cefdf46f-afbf-44e8-8d54-a3d5b0e7c348"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:43:52.920598 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:52.920521    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgjcr\" (UniqueName: \"kubernetes.io/projected/cefdf46f-afbf-44e8-8d54-a3d5b0e7c348-kube-api-access-mgjcr\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:43:52.920598 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:52.920556    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cefdf46f-afbf-44e8-8d54-a3d5b0e7c348-maas-api-tls\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:43:52.920598 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:52.920570    4993 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cefdf46f-afbf-44e8-8d54-a3d5b0e7c348-metrics-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:43:53.154192 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:53.143622    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-116e85-65676f99fd-b7vxz"]
Nov 01 14:43:53.177247 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:53.166549    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-116e85-65676f99fd-b7vxz"]
Nov 01 14:43:53.312127 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:53.312088    4993 patch_prober.go:29] interesting pod/e2e-api-b-26a0c1-openshift-default-76c87f685-ks7q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Nov 01 14:43:53.312301 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:53.312197    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-ks7q6" podUID="040af721-9da6-4df4-a334-a5d95c482f3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Nov 01 14:43:53.741974 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:53.741938    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c9b0827c-65bb9b948d-6kxb4"]
Nov 01 14:43:53.742495 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:53.742481    4993 cpu_manager.go:414] "containerMap: removing container" podUID="cefdf46f-afbf-44e8-8d54-a3d5b0e7c348" containerName="maas-api"
Nov 01 14:43:53.742558 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:53.742496    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cefdf46f-afbf-44e8-8d54-a3d5b0e7c348" containerName="maas-api"
Nov 01 14:43:53.742649 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:53.742638    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="cefdf46f-afbf-44e8-8d54-a3d5b0e7c348" containerName="maas-api"
Nov 01 14:43:53.747567 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:53.747545    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9b0827c-65bb9b948d-6kxb4"
Nov 01 14:43:53.795819 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:53.795785    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-c9b0827c-serving-cert\""
Nov 01 14:43:53.806880 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:53.806849    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c9b0827c-65bb9b948d-6kxb4"]
Nov 01 14:43:53.832529 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:53.832491    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7549aec7-a94c-47a2-945d-1a2a4d31fc69-openshift-service-ca\") pod \"maas-api-e2e-mt-c9b0827c-65bb9b948d-6kxb4\" (UID: \"7549aec7-a94c-47a2-945d-1a2a4d31fc69\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9b0827c-65bb9b948d-6kxb4"
Nov 01 14:43:53.832699 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:53.832600    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7549aec7-a94c-47a2-945d-1a2a4d31fc69-metrics-certs\") pod \"maas-api-e2e-mt-c9b0827c-65bb9b948d-6kxb4\" (UID: \"7549aec7-a94c-47a2-945d-1a2a4d31fc69\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9b0827c-65bb9b948d-6kxb4"
Nov 01 14:43:53.832763 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:53.832717    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7549aec7-a94c-47a2-945d-1a2a4d31fc69-maas-api-tls\") pod \"maas-api-e2e-mt-c9b0827c-65bb9b948d-6kxb4\" (UID: \"7549aec7-a94c-47a2-945d-1a2a4d31fc69\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9b0827c-65bb9b948d-6kxb4"
Nov 01 14:43:53.833076 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:53.832892    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2qgh\" (UniqueName: \"kubernetes.io/projected/7549aec7-a94c-47a2-945d-1a2a4d31fc69-kube-api-access-w2qgh\") pod \"maas-api-e2e-mt-c9b0827c-65bb9b948d-6kxb4\" (UID: \"7549aec7-a94c-47a2-945d-1a2a4d31fc69\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9b0827c-65bb9b948d-6kxb4"
Nov 01 14:43:53.934545 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:53.934512    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7549aec7-a94c-47a2-945d-1a2a4d31fc69-openshift-service-ca\") pod \"maas-api-e2e-mt-c9b0827c-65bb9b948d-6kxb4\" (UID: \"7549aec7-a94c-47a2-945d-1a2a4d31fc69\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9b0827c-65bb9b948d-6kxb4"
Nov 01 14:43:53.934697 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:53.934552    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7549aec7-a94c-47a2-945d-1a2a4d31fc69-metrics-certs\") pod \"maas-api-e2e-mt-c9b0827c-65bb9b948d-6kxb4\" (UID: \"7549aec7-a94c-47a2-945d-1a2a4d31fc69\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9b0827c-65bb9b948d-6kxb4"
Nov 01 14:43:53.934697 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:53.934598    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7549aec7-a94c-47a2-945d-1a2a4d31fc69-maas-api-tls\") pod \"maas-api-e2e-mt-c9b0827c-65bb9b948d-6kxb4\" (UID: \"7549aec7-a94c-47a2-945d-1a2a4d31fc69\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9b0827c-65bb9b948d-6kxb4"
Nov 01 14:43:53.934697 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:53.934684    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2qgh\" (UniqueName: \"kubernetes.io/projected/7549aec7-a94c-47a2-945d-1a2a4d31fc69-kube-api-access-w2qgh\") pod \"maas-api-e2e-mt-c9b0827c-65bb9b948d-6kxb4\" (UID: \"7549aec7-a94c-47a2-945d-1a2a4d31fc69\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9b0827c-65bb9b948d-6kxb4"
Nov 01 14:43:53.935108 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:43:53.935082    4993 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-c9b0827c-serving-cert: secret "maas-api-e2e-mt-c9b0827c-serving-cert" not found
Nov 01 14:43:53.935329 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:53.935133    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7549aec7-a94c-47a2-945d-1a2a4d31fc69-openshift-service-ca\") pod \"maas-api-e2e-mt-c9b0827c-65bb9b948d-6kxb4\" (UID: \"7549aec7-a94c-47a2-945d-1a2a4d31fc69\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9b0827c-65bb9b948d-6kxb4"
Nov 01 14:43:53.935426 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:43:53.935412    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7549aec7-a94c-47a2-945d-1a2a4d31fc69-maas-api-tls podName:7549aec7-a94c-47a2-945d-1a2a4d31fc69 nodeName:}" failed. No retries permitted until 2026-11-01 14:43:54.435310605 +0000 UTC m=+1621.765173324 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7549aec7-a94c-47a2-945d-1a2a4d31fc69-maas-api-tls") pod "maas-api-e2e-mt-c9b0827c-65bb9b948d-6kxb4" (UID: "7549aec7-a94c-47a2-945d-1a2a4d31fc69") : secret "maas-api-e2e-mt-c9b0827c-serving-cert" not found
Nov 01 14:43:53.937059 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:53.937040    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7549aec7-a94c-47a2-945d-1a2a4d31fc69-metrics-certs\") pod \"maas-api-e2e-mt-c9b0827c-65bb9b948d-6kxb4\" (UID: \"7549aec7-a94c-47a2-945d-1a2a4d31fc69\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9b0827c-65bb9b948d-6kxb4"
Nov 01 14:43:53.991478 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:53.991409    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2qgh\" (UniqueName: \"kubernetes.io/projected/7549aec7-a94c-47a2-945d-1a2a4d31fc69-kube-api-access-w2qgh\") pod \"maas-api-e2e-mt-c9b0827c-65bb9b948d-6kxb4\" (UID: \"7549aec7-a94c-47a2-945d-1a2a4d31fc69\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9b0827c-65bb9b948d-6kxb4"
Nov 01 14:43:54.313189 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:54.312639    4993 patch_prober.go:29] interesting pod/e2e-api-b-26a0c1-openshift-default-76c87f685-ks7q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Nov 01 14:43:54.313189 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:54.312701    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-ks7q6" podUID="040af721-9da6-4df4-a334-a5d95c482f3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Nov 01 14:43:54.441845 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:54.441798    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7549aec7-a94c-47a2-945d-1a2a4d31fc69-maas-api-tls\") pod \"maas-api-e2e-mt-c9b0827c-65bb9b948d-6kxb4\" (UID: \"7549aec7-a94c-47a2-945d-1a2a4d31fc69\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9b0827c-65bb9b948d-6kxb4"
Nov 01 14:43:54.444353 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:54.444321    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7549aec7-a94c-47a2-945d-1a2a4d31fc69-maas-api-tls\") pod \"maas-api-e2e-mt-c9b0827c-65bb9b948d-6kxb4\" (UID: \"7549aec7-a94c-47a2-945d-1a2a4d31fc69\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9b0827c-65bb9b948d-6kxb4"
Nov 01 14:43:54.689986 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:54.689900    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9b0827c-65bb9b948d-6kxb4"
Nov 01 14:43:54.805963 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:54.805419    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cefdf46f-afbf-44e8-8d54-a3d5b0e7c348" path="/var/lib/kubelet/pods/cefdf46f-afbf-44e8-8d54-a3d5b0e7c348/volumes"
Nov 01 14:43:55.083143 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:55.083098    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c9b0827c-65bb9b948d-6kxb4"]
Nov 01 14:43:55.091931 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:43:55.091899    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7549aec7_a94c_47a2_945d_1a2a4d31fc69.slice/crio-683a16187f5ad9148c8452ffeea9a2971605fd4bf23ab8ac95ec797991ae7deb WatchSource:0}: Error finding container 683a16187f5ad9148c8452ffeea9a2971605fd4bf23ab8ac95ec797991ae7deb: Status 404 returned error can't find the container with id 683a16187f5ad9148c8452ffeea9a2971605fd4bf23ab8ac95ec797991ae7deb
Nov 01 14:43:55.093409 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:55.093391    4993 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 14:43:55.312526 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:55.312492    4993 patch_prober.go:29] interesting pod/e2e-api-b-26a0c1-openshift-default-76c87f685-ks7q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Nov 01 14:43:55.312699 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:55.312551    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-ks7q6" podUID="040af721-9da6-4df4-a334-a5d95c482f3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Nov 01 14:43:55.742990 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:55.742949    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9b0827c-65bb9b948d-6kxb4" event={"ID":"7549aec7-a94c-47a2-945d-1a2a4d31fc69","Type":"ContainerStarted","Data":"683a16187f5ad9148c8452ffeea9a2971605fd4bf23ab8ac95ec797991ae7deb"}
Nov 01 14:43:56.315711 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:56.315619    4993 patch_prober.go:29] interesting pod/e2e-api-b-26a0c1-openshift-default-76c87f685-ks7q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Nov 01 14:43:56.316301 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:56.316263    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-ks7q6" podUID="040af721-9da6-4df4-a334-a5d95c482f3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Nov 01 14:43:56.751061 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:56.751025    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9b0827c-65bb9b948d-6kxb4" event={"ID":"7549aec7-a94c-47a2-945d-1a2a4d31fc69","Type":"ContainerStarted","Data":"a1c8ac22071d401cc4c09040bde2c53c68ac2191eecf5b09f312f0ad485ece78"}
Nov 01 14:43:56.751244 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:56.751147    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9b0827c-65bb9b948d-6kxb4"
Nov 01 14:43:56.869810 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:56.869757    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9b0827c-65bb9b948d-6kxb4" podStartSLOduration=3.020791905 podStartE2EDuration="3.869741428s" totalImagesPullingTime="848.949523ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:43:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:43:56.866671171 +0000 UTC m=+1624.196533905" watchObservedRunningTime="2026-11-01 14:43:56.869741428 +0000 UTC m=+1624.199604212"
Nov 01 14:43:57.316951 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:57.316917    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-ks7q6"
Nov 01 14:43:57.317787 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:57.317746    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-ks7q6"
Nov 01 14:43:57.318431 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:43:57.318410    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-ks7q6"
Nov 01 14:43:57.513103 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:43:57.513065    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:44:00.260954 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:44:00.260071    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx"]
Nov 01 14:44:00.260954 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:44:00.260313    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx" podUID="20e06dcd-6062-4dfa-9294-388c801fa8f7" containerName="istio-proxy" containerID="cri-o://21dd3ee3e960a36e2c49c599548586c522b594d17eb4e857d272c7144291c3d1" gracePeriod=30
Nov 01 14:44:02.767498 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:44:02.767463    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9b0827c-65bb9b948d-6kxb4"
Nov 01 14:44:04.990576 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:44:04.990537    4993 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 14:44:04.990969 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:44:04.990607    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx" podUID="20e06dcd-6062-4dfa-9294-388c801fa8f7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 14:44:05.548794 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:44:05.548729    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx"
Nov 01 14:44:05.658064 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:44:05.658027    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/20e06dcd-6062-4dfa-9294-388c801fa8f7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20e06dcd-6062-4dfa-9294-388c801fa8f7-istio-podinfo\") pod \"20e06dcd-6062-4dfa-9294-388c801fa8f7\" (UID: \"20e06dcd-6062-4dfa-9294-388c801fa8f7\") "
Nov 01 14:44:05.658257 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:44:05.658076    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20e06dcd-6062-4dfa-9294-388c801fa8f7-istio-token\" (UniqueName: \"kubernetes.io/projected/20e06dcd-6062-4dfa-9294-388c801fa8f7-istio-token\") pod \"20e06dcd-6062-4dfa-9294-388c801fa8f7\" (UID: \"20e06dcd-6062-4dfa-9294-388c801fa8f7\") "
Nov 01 14:44:05.658257 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:44:05.658126    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20e06dcd-6062-4dfa-9294-388c801fa8f7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20e06dcd-6062-4dfa-9294-388c801fa8f7-istio-envoy\") pod \"20e06dcd-6062-4dfa-9294-388c801fa8f7\" (UID: \"20e06dcd-6062-4dfa-9294-388c801fa8f7\") "
Nov 01 14:44:05.658257 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:44:05.658199    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20e06dcd-6062-4dfa-9294-388c801fa8f7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20e06dcd-6062-4dfa-9294-388c801fa8f7-istiod-ca-cert\") pod \"20e06dcd-6062-4dfa-9294-388c801fa8f7\" (UID: \"20e06dcd-6062-4dfa-9294-388c801fa8f7\") "
Nov 01 14:44:05.658257 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:44:05.658253    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20e06dcd-6062-4dfa-9294-388c801fa8f7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20e06dcd-6062-4dfa-9294-388c801fa8f7-istio-data\") pod \"20e06dcd-6062-4dfa-9294-388c801fa8f7\" (UID: \"20e06dcd-6062-4dfa-9294-388c801fa8f7\") "
Nov 01 14:44:05.658493 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:44:05.658302    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20e06dcd-6062-4dfa-9294-388c801fa8f7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20e06dcd-6062-4dfa-9294-388c801fa8f7-credential-socket\") pod \"20e06dcd-6062-4dfa-9294-388c801fa8f7\" (UID: \"20e06dcd-6062-4dfa-9294-388c801fa8f7\") "
Nov 01 14:44:05.658493 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:44:05.658442    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20e06dcd-6062-4dfa-9294-388c801fa8f7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20e06dcd-6062-4dfa-9294-388c801fa8f7-workload-socket\") pod \"20e06dcd-6062-4dfa-9294-388c801fa8f7\" (UID: \"20e06dcd-6062-4dfa-9294-388c801fa8f7\") "
Nov 01 14:44:05.658493 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:44:05.658479    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20e06dcd-6062-4dfa-9294-388c801fa8f7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20e06dcd-6062-4dfa-9294-388c801fa8f7-workload-certs\") pod \"20e06dcd-6062-4dfa-9294-388c801fa8f7\" (UID: \"20e06dcd-6062-4dfa-9294-388c801fa8f7\") "
Nov 01 14:44:05.658674 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:44:05.658518    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20e06dcd-6062-4dfa-9294-388c801fa8f7-kube-api-access-6p56r\" (UniqueName: \"kubernetes.io/projected/20e06dcd-6062-4dfa-9294-388c801fa8f7-kube-api-access-6p56r\") pod \"20e06dcd-6062-4dfa-9294-388c801fa8f7\" (UID: \"20e06dcd-6062-4dfa-9294-388c801fa8f7\") "
Nov 01 14:44:05.658674 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:44:05.658517    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20e06dcd-6062-4dfa-9294-388c801fa8f7-istiod-ca-cert" pod "20e06dcd-6062-4dfa-9294-388c801fa8f7" (UID: "20e06dcd-6062-4dfa-9294-388c801fa8f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:44:05.658772 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:44:05.658701    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20e06dcd-6062-4dfa-9294-388c801fa8f7-workload-socket" pod "20e06dcd-6062-4dfa-9294-388c801fa8f7" (UID: "20e06dcd-6062-4dfa-9294-388c801fa8f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:44:05.658772 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:44:05.658712    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20e06dcd-6062-4dfa-9294-388c801fa8f7-workload-certs" pod "20e06dcd-6062-4dfa-9294-388c801fa8f7" (UID: "20e06dcd-6062-4dfa-9294-388c801fa8f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:44:05.659049 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:44:05.658685    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20e06dcd-6062-4dfa-9294-388c801fa8f7-credential-socket" pod "20e06dcd-6062-4dfa-9294-388c801fa8f7" (UID: "20e06dcd-6062-4dfa-9294-388c801fa8f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:44:05.659049 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:44:05.659020    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20e06dcd-6062-4dfa-9294-388c801fa8f7-istio-data" pod "20e06dcd-6062-4dfa-9294-388c801fa8f7" (UID: "20e06dcd-6062-4dfa-9294-388c801fa8f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:44:05.659310 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:44:05.659292    4993 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20e06dcd-6062-4dfa-9294-388c801fa8f7-workload-socket\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:05.659390 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:44:05.659316    4993 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20e06dcd-6062-4dfa-9294-388c801fa8f7-workload-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:05.659390 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:44:05.659331    4993 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20e06dcd-6062-4dfa-9294-388c801fa8f7-istiod-ca-cert\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:05.659390 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:44:05.659345    4993 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20e06dcd-6062-4dfa-9294-388c801fa8f7-istio-data\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:05.659390 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:44:05.659360    4993 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20e06dcd-6062-4dfa-9294-388c801fa8f7-credential-socket\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:05.660711 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:44:05.660679    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20e06dcd-6062-4dfa-9294-388c801fa8f7-istio-token" pod "20e06dcd-6062-4dfa-9294-388c801fa8f7" (UID: "20e06dcd-6062-4dfa-9294-388c801fa8f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:44:05.660822 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:44:05.660802    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/20e06dcd-6062-4dfa-9294-388c801fa8f7-istio-podinfo" pod "20e06dcd-6062-4dfa-9294-388c801fa8f7" (UID: "20e06dcd-6062-4dfa-9294-388c801fa8f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 14:44:05.660968 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:44:05.660938    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20e06dcd-6062-4dfa-9294-388c801fa8f7-istio-envoy" pod "20e06dcd-6062-4dfa-9294-388c801fa8f7" (UID: "20e06dcd-6062-4dfa-9294-388c801fa8f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:44:05.661077 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:44:05.661005    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20e06dcd-6062-4dfa-9294-388c801fa8f7-kube-api-access-6p56r" pod "20e06dcd-6062-4dfa-9294-388c801fa8f7" (UID: "20e06dcd-6062-4dfa-9294-388c801fa8f7"). InnerVolumeSpecName "kube-api-access-6p56r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:44:05.760877 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:44:05.760831    4993 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20e06dcd-6062-4dfa-9294-388c801fa8f7-istio-podinfo\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:05.760877 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:44:05.760873    4993 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20e06dcd-6062-4dfa-9294-388c801fa8f7-istio-token\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:05.761139 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:44:05.760889    4993 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20e06dcd-6062-4dfa-9294-388c801fa8f7-istio-envoy\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:05.761139 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:44:05.760903    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6p56r\" (UniqueName: \"kubernetes.io/projected/20e06dcd-6062-4dfa-9294-388c801fa8f7-kube-api-access-6p56r\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:44:05.912067 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:44:05.911947    4993 generic.go:356] "Generic (PLEG): container finished" podID="20e06dcd-6062-4dfa-9294-388c801fa8f7" containerID="21dd3ee3e960a36e2c49c599548586c522b594d17eb4e857d272c7144291c3d1" exitCode=0
Nov 01 14:44:05.912344 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:44:05.912014    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx"
Nov 01 14:44:05.912490 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:44:05.912037    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx" event={"ID":"20e06dcd-6062-4dfa-9294-388c801fa8f7","Type":"ContainerDied","Data":"21dd3ee3e960a36e2c49c599548586c522b594d17eb4e857d272c7144291c3d1"}
Nov 01 14:44:05.912561 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:44:05.912488    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx" event={"ID":"20e06dcd-6062-4dfa-9294-388c801fa8f7","Type":"ContainerDied","Data":"04a9ec64ad8e48a40b22e036cf484f7c6651dca9fabb8f01edd2a578d0b4235b"}
Nov 01 14:44:05.912561 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:44:05.912518    4993 scope.go:122] "RemoveContainer" containerID="21dd3ee3e960a36e2c49c599548586c522b594d17eb4e857d272c7144291c3d1"
Nov 01 14:44:05.924532 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:44:05.924510    4993 scope.go:122] "RemoveContainer" containerID="21dd3ee3e960a36e2c49c599548586c522b594d17eb4e857d272c7144291c3d1"
Nov 01 14:44:05.924963 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:44:05.924899    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21dd3ee3e960a36e2c49c599548586c522b594d17eb4e857d272c7144291c3d1\": container with ID starting with 21dd3ee3e960a36e2c49c599548586c522b594d17eb4e857d272c7144291c3d1 not found: ID does not exist" containerID="21dd3ee3e960a36e2c49c599548586c522b594d17eb4e857d272c7144291c3d1"
Nov 01 14:44:05.924963 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:44:05.924939    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21dd3ee3e960a36e2c49c599548586c522b594d17eb4e857d272c7144291c3d1"} err="failed to get container status \"21dd3ee3e960a36e2c49c599548586c522b594d17eb4e857d272c7144291c3d1\": rpc error: code = NotFound desc = could not find container \"21dd3ee3e960a36e2c49c599548586c522b594d17eb4e857d272c7144291c3d1\": container with ID starting with 21dd3ee3e960a36e2c49c599548586c522b594d17eb4e857d272c7144291c3d1 not found: ID does not exist"
Nov 01 14:44:06.066766 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:44:06.066725    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx"]
Nov 01 14:44:06.114959 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:44:06.114901    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-4lhhx"]
Nov 01 14:44:06.796532 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:44:06.796500    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20e06dcd-6062-4dfa-9294-388c801fa8f7" path="/var/lib/kubelet/pods/20e06dcd-6062-4dfa-9294-388c801fa8f7/volumes"
Nov 01 14:44:27.501405 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:44:27.501367    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:44:33.016498 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:44:33.016410    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-71e39a-7bd7c6f974-mcvvx"]
Nov 01 14:44:33.018825 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:44:33.017147    4993 cpu_manager.go:414] "containerMap: removing container" podUID="20e06dcd-6062-4dfa-9294-388c801fa8f7" containerName="istio-proxy"
Nov 01 14:44:33.018825 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:44:33.017178    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20e06dcd-6062-4dfa-9294-388c801fa8f7" containerName="istio-proxy"
Nov 01 14:44:33.018825 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:44:33.017303    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="20e06dcd-6062-4dfa-9294-388c801fa8f7" containerName="istio-proxy"
Nov 01 14:44:33.020361 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:44:33.020335    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-71e39a-7bd7c6f974-mcvvx"
Nov 01 14:44:33.071130 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:44:33.071084    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-71e39a-serving-cert\""
Nov 01 14:44:33.120952 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:44:33.120918    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-71e39a-7bd7c6f974-mcvvx"]
Nov 01 14:44:33.196871 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:44:33.196839    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d99efc6-0d6e-4cb5-a65e-6ac9eae07ea9-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-71e39a-7bd7c6f974-mcvvx\" (UID: \"1d99efc6-0d6e-4cb5-a65e-6ac9eae07ea9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-71e39a-7bd7c6f974-mcvvx"
Nov 01 14:44:33.197072 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:44:33.196969    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d99efc6-0d6e-4cb5-a65e-6ac9eae07ea9-maas-api-tls\") pod \"maas-api-e2e-ipp-a-71e39a-7bd7c6f974-mcvvx\" (UID: \"1d99efc6-0d6e-4cb5-a65e-6ac9eae07ea9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-71e39a-7bd7c6f974-mcvvx"
Nov 01 14:44:33.197072 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:44:33.197011    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d99efc6-0d6e-4cb5-a65e-6ac9eae07ea9-metrics-certs\") pod \"maas-api-e2e-ipp-a-71e39a-7bd7c6f974-mcvvx\" (UID: \"1d99efc6-0d6e-4cb5-a65e-6ac9eae07ea9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-71e39a-7bd7c6f974-mcvvx"
Nov 01 14:44:33.197072 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:44:33.197031    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr9kk\" (UniqueName: \"kubernetes.io/projected/1d99efc6-0d6e-4cb5-a65e-6ac9eae07ea9-kube-api-access-xr9kk\") pod \"maas-api-e2e-ipp-a-71e39a-7bd7c6f974-mcvvx\" (UID: \"1d99efc6-0d6e-4cb5-a65e-6ac9eae07ea9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-71e39a-7bd7c6f974-mcvvx"
Nov 01 14:44:33.297876 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:44:33.297781    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d99efc6-0d6e-4cb5-a65e-6ac9eae07ea9-metrics-certs\") pod \"maas-api-e2e-ipp-a-71e39a-7bd7c6f974-mcvvx\" (UID: \"1d99efc6-0d6e-4cb5-a65e-6ac9eae07ea9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-71e39a-7bd7c6f974-mcvvx"
Nov 01 14:44:33.297876 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:44:33.297827    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr9kk\" (UniqueName: \"kubernetes.io/projected/1d99efc6-0d6e-4cb5-a65e-6ac9eae07ea9-kube-api-access-xr9kk\") pod \"maas-api-e2e-ipp-a-71e39a-7bd7c6f974-mcvvx\" (UID: \"1d99efc6-0d6e-4cb5-a65e-6ac9eae07ea9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-71e39a-7bd7c6f974-mcvvx"
Nov 01 14:44:33.298115 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:44:33.297889    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d99efc6-0d6e-4cb5-a65e-6ac9eae07ea9-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-71e39a-7bd7c6f974-mcvvx\" (UID: \"1d99efc6-0d6e-4cb5-a65e-6ac9eae07ea9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-71e39a-7bd7c6f974-mcvvx"
Nov 01 14:44:33.298115 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:44:33.297992    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d99efc6-0d6e-4cb5-a65e-6ac9eae07ea9-maas-api-tls\") pod \"maas-api-e2e-ipp-a-71e39a-7bd7c6f974-mcvvx\" (UID: \"1d99efc6-0d6e-4cb5-a65e-6ac9eae07ea9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-71e39a-7bd7c6f974-mcvvx"
Nov 01 14:44:33.299065 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:44:33.299039    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d99efc6-0d6e-4cb5-a65e-6ac9eae07ea9-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-71e39a-7bd7c6f974-mcvvx\" (UID: \"1d99efc6-0d6e-4cb5-a65e-6ac9eae07ea9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-71e39a-7bd7c6f974-mcvvx"
Nov 01 14:44:33.300911 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:44:33.300882    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d99efc6-0d6e-4cb5-a65e-6ac9eae07ea9-metrics-certs\") pod \"maas-api-e2e-ipp-a-71e39a-7bd7c6f974-mcvvx\" (UID: \"1d99efc6-0d6e-4cb5-a65e-6ac9eae07ea9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-71e39a-7bd7c6f974-mcvvx"
Nov 01 14:44:33.301080 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:44:33.300989    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d99efc6-0d6e-4cb5-a65e-6ac9eae07ea9-maas-api-tls\") pod \"maas-api-e2e-ipp-a-71e39a-7bd7c6f974-mcvvx\" (UID: \"1d99efc6-0d6e-4cb5-a65e-6ac9eae07ea9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-71e39a-7bd7c6f974-mcvvx"
Nov 01 14:44:33.352113 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:44:33.352077    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr9kk\" (UniqueName: \"kubernetes.io/projected/1d99efc6-0d6e-4cb5-a65e-6ac9eae07ea9-kube-api-access-xr9kk\") pod \"maas-api-e2e-ipp-a-71e39a-7bd7c6f974-mcvvx\" (UID: \"1d99efc6-0d6e-4cb5-a65e-6ac9eae07ea9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-71e39a-7bd7c6f974-mcvvx"
Nov 01 14:44:33.633761 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:44:33.633667    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-71e39a-7bd7c6f974-mcvvx"
Nov 01 14:44:34.036235 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:44:34.035988    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-71e39a-7bd7c6f974-mcvvx"]
Nov 01 14:44:34.199315 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:44:34.199271    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-71e39a-7bd7c6f974-mcvvx" event={"ID":"1d99efc6-0d6e-4cb5-a65e-6ac9eae07ea9","Type":"ContainerStarted","Data":"eed5bfd2cc4607733bc1ccb6a023a4de060d5a334ec4ddf7683c9a9d032aae14"}
Nov 01 14:44:35.213620 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:44:35.213522    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-71e39a-7bd7c6f974-mcvvx" event={"ID":"1d99efc6-0d6e-4cb5-a65e-6ac9eae07ea9","Type":"ContainerStarted","Data":"609d4270e614008808a3b5239e98efd49dd09eb6bdda629d7ea77c18abbc855e"}
Nov 01 14:44:35.214072 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:44:35.213827    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-71e39a-7bd7c6f974-mcvvx"
Nov 01 14:44:35.322263 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:44:35.322200    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-71e39a-7bd7c6f974-mcvvx" podStartSLOduration=2.470599363 podStartE2EDuration="3.322181233s" totalImagesPullingTime="851.58187ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:44:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:44:35.31992712 +0000 UTC m=+1662.649789857" watchObservedRunningTime="2026-11-01 14:44:35.322181233 +0000 UTC m=+1662.652043967"
Nov 01 14:44:41.226280 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:44:41.226246    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-71e39a-7bd7c6f974-mcvvx"
Nov 01 14:44:57.504545 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:44:57.504507    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:45:00.269372 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:00.269338    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892405-t8xr7"]
Nov 01 14:45:00.276298 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:00.276255    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892405-mm2mf"]
Nov 01 14:45:00.278562 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:00.278535    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892405-t8xr7"
Nov 01 14:45:00.281130 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:00.281107    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892405-mm2mf"
Nov 01 14:45:00.320425 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:00.320383    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892405-t8xr7"]
Nov 01 14:45:00.330576 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:00.330539    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\""
Nov 01 14:45:00.368906 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:00.368876    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892405-mm2mf"]
Nov 01 14:45:00.381374 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:00.381335    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9ffca0d-15ec-4324-b8c9-dd23d23898e7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-536a87-29892405-mm2mf\" (UID: \"c9ffca0d-15ec-4324-b8c9-dd23d23898e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892405-mm2mf"
Nov 01 14:45:00.381794 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:00.381688    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/903f1b4f-57e1-4fe4-95ba-14b49e728b9a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892405-t8xr7\" (UID: \"903f1b4f-57e1-4fe4-95ba-14b49e728b9a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892405-t8xr7"
Nov 01 14:45:00.383259 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:00.381908    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql95l\" (UniqueName: \"kubernetes.io/projected/c9ffca0d-15ec-4324-b8c9-dd23d23898e7-kube-api-access-ql95l\") pod \"maas-api-key-cleanup-e2e-worker-w4-536a87-29892405-mm2mf\" (UID: \"c9ffca0d-15ec-4324-b8c9-dd23d23898e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892405-mm2mf"
Nov 01 14:45:00.383259 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:00.381940    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwxps\" (UniqueName: \"kubernetes.io/projected/903f1b4f-57e1-4fe4-95ba-14b49e728b9a-kube-api-access-rwxps\") pod \"maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892405-t8xr7\" (UID: \"903f1b4f-57e1-4fe4-95ba-14b49e728b9a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892405-t8xr7"
Nov 01 14:45:00.482818 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:00.482780    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql95l\" (UniqueName: \"kubernetes.io/projected/c9ffca0d-15ec-4324-b8c9-dd23d23898e7-kube-api-access-ql95l\") pod \"maas-api-key-cleanup-e2e-worker-w4-536a87-29892405-mm2mf\" (UID: \"c9ffca0d-15ec-4324-b8c9-dd23d23898e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892405-mm2mf"
Nov 01 14:45:00.482818 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:00.482823    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwxps\" (UniqueName: \"kubernetes.io/projected/903f1b4f-57e1-4fe4-95ba-14b49e728b9a-kube-api-access-rwxps\") pod \"maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892405-t8xr7\" (UID: \"903f1b4f-57e1-4fe4-95ba-14b49e728b9a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892405-t8xr7"
Nov 01 14:45:00.483090 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:00.482917    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9ffca0d-15ec-4324-b8c9-dd23d23898e7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-536a87-29892405-mm2mf\" (UID: \"c9ffca0d-15ec-4324-b8c9-dd23d23898e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892405-mm2mf"
Nov 01 14:45:00.483090 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:00.483017    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/903f1b4f-57e1-4fe4-95ba-14b49e728b9a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892405-t8xr7\" (UID: \"903f1b4f-57e1-4fe4-95ba-14b49e728b9a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892405-t8xr7"
Nov 01 14:45:00.483727 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:00.483706    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/903f1b4f-57e1-4fe4-95ba-14b49e728b9a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892405-t8xr7\" (UID: \"903f1b4f-57e1-4fe4-95ba-14b49e728b9a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892405-t8xr7"
Nov 01 14:45:00.483727 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:00.483720    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9ffca0d-15ec-4324-b8c9-dd23d23898e7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-536a87-29892405-mm2mf\" (UID: \"c9ffca0d-15ec-4324-b8c9-dd23d23898e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892405-mm2mf"
Nov 01 14:45:00.537833 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:00.537766    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwxps\" (UniqueName: \"kubernetes.io/projected/903f1b4f-57e1-4fe4-95ba-14b49e728b9a-kube-api-access-rwxps\") pod \"maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892405-t8xr7\" (UID: \"903f1b4f-57e1-4fe4-95ba-14b49e728b9a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892405-t8xr7"
Nov 01 14:45:00.537833 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:00.537793    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql95l\" (UniqueName: \"kubernetes.io/projected/c9ffca0d-15ec-4324-b8c9-dd23d23898e7-kube-api-access-ql95l\") pod \"maas-api-key-cleanup-e2e-worker-w4-536a87-29892405-mm2mf\" (UID: \"c9ffca0d-15ec-4324-b8c9-dd23d23898e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892405-mm2mf"
Nov 01 14:45:00.607928 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:00.607890    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892405-t8xr7"
Nov 01 14:45:00.617216 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:00.617154    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892405-mm2mf"
Nov 01 14:45:00.998552 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:00.998512    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892405-t8xr7"]
Nov 01 14:45:01.003780 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:45:01.003745    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod903f1b4f_57e1_4fe4_95ba_14b49e728b9a.slice/crio-0b68faaa481fab399b8c1ffe46a933bd5b7c5bbc3512700f95d6fb631cb91a6f WatchSource:0}: Error finding container 0b68faaa481fab399b8c1ffe46a933bd5b7c5bbc3512700f95d6fb631cb91a6f: Status 404 returned error can't find the container with id 0b68faaa481fab399b8c1ffe46a933bd5b7c5bbc3512700f95d6fb631cb91a6f
Nov 01 14:45:01.049103 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:01.049073    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892405-mm2mf"]
Nov 01 14:45:01.054628 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:45:01.054591    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9ffca0d_15ec_4324_b8c9_dd23d23898e7.slice/crio-3e12f903790a613577d54da3e6c2ed72a211695a35315ade0a1dc39f2ec04c6f WatchSource:0}: Error finding container 3e12f903790a613577d54da3e6c2ed72a211695a35315ade0a1dc39f2ec04c6f: Status 404 returned error can't find the container with id 3e12f903790a613577d54da3e6c2ed72a211695a35315ade0a1dc39f2ec04c6f
Nov 01 14:45:01.402408 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:01.402369    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892405-mm2mf" event={"ID":"c9ffca0d-15ec-4324-b8c9-dd23d23898e7","Type":"ContainerStarted","Data":"208eddbf7f39a81cd0a0761bb0ddd959c1a99acc52af58fba9ee2f5c8b27880d"}
Nov 01 14:45:01.402905 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:01.402426    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892405-mm2mf" event={"ID":"c9ffca0d-15ec-4324-b8c9-dd23d23898e7","Type":"ContainerStarted","Data":"3e12f903790a613577d54da3e6c2ed72a211695a35315ade0a1dc39f2ec04c6f"}
Nov 01 14:45:01.403912 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:01.403887    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892405-t8xr7" event={"ID":"903f1b4f-57e1-4fe4-95ba-14b49e728b9a","Type":"ContainerStarted","Data":"0b68faaa481fab399b8c1ffe46a933bd5b7c5bbc3512700f95d6fb631cb91a6f"}
Nov 01 14:45:01.511939 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:01.511885    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892405-mm2mf" podStartSLOduration=1.5118710210000001 podStartE2EDuration="1.511871021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:45:01.510020279 +0000 UTC m=+1688.839883012" watchObservedRunningTime="2026-11-01 14:45:01.511871021 +0000 UTC m=+1688.841733757"
Nov 01 14:45:02.412162 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:02.412126    4993 generic.go:356] "Generic (PLEG): container finished" podID="903f1b4f-57e1-4fe4-95ba-14b49e728b9a" containerID="cee47b34f5d68ae6e9bad592f4b195c2df613249188e3783a6f57f06da1ebc47" exitCode=60
Nov 01 14:45:02.412713 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:02.412214    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892405-t8xr7" event={"ID":"903f1b4f-57e1-4fe4-95ba-14b49e728b9a","Type":"ContainerDied","Data":"cee47b34f5d68ae6e9bad592f4b195c2df613249188e3783a6f57f06da1ebc47"}
Nov 01 14:45:02.412713 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:02.412591    4993 scope.go:122] "RemoveContainer" containerID="cee47b34f5d68ae6e9bad592f4b195c2df613249188e3783a6f57f06da1ebc47"
Nov 01 14:45:02.414669 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:02.414637    4993 generic.go:356] "Generic (PLEG): container finished" podID="c9ffca0d-15ec-4324-b8c9-dd23d23898e7" containerID="208eddbf7f39a81cd0a0761bb0ddd959c1a99acc52af58fba9ee2f5c8b27880d" exitCode=60
Nov 01 14:45:02.414767 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:02.414749    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892405-mm2mf" event={"ID":"c9ffca0d-15ec-4324-b8c9-dd23d23898e7","Type":"ContainerDied","Data":"208eddbf7f39a81cd0a0761bb0ddd959c1a99acc52af58fba9ee2f5c8b27880d"}
Nov 01 14:45:02.414983 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:02.414967    4993 scope.go:122] "RemoveContainer" containerID="208eddbf7f39a81cd0a0761bb0ddd959c1a99acc52af58fba9ee2f5c8b27880d"
Nov 01 14:45:03.422827 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:03.422784    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892405-t8xr7" event={"ID":"903f1b4f-57e1-4fe4-95ba-14b49e728b9a","Type":"ContainerStarted","Data":"6639d605405ec643be8b88773f48f49c8add8ef4ee7d0f2f93b0e84e51662f67"}
Nov 01 14:45:03.424925 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:03.424897    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892405-mm2mf" event={"ID":"c9ffca0d-15ec-4324-b8c9-dd23d23898e7","Type":"ContainerStarted","Data":"e9b64ee554f7afb47eacf40a21e16107b8ab38560923820c1e8303323a34431a"}
Nov 01 14:45:03.535680 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:03.535634    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892405-t8xr7" podStartSLOduration=3.535618977 podStartE2EDuration="3.535618977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:45:03.532636843 +0000 UTC m=+1690.862499579" watchObservedRunningTime="2026-11-01 14:45:03.535618977 +0000 UTC m=+1690.865481713"
Nov 01 14:45:04.432729 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:04.432689    4993 generic.go:356] "Generic (PLEG): container finished" podID="c9ffca0d-15ec-4324-b8c9-dd23d23898e7" containerID="e9b64ee554f7afb47eacf40a21e16107b8ab38560923820c1e8303323a34431a" exitCode=60
Nov 01 14:45:04.433348 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:04.432793    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892405-mm2mf" event={"ID":"c9ffca0d-15ec-4324-b8c9-dd23d23898e7","Type":"ContainerDied","Data":"e9b64ee554f7afb47eacf40a21e16107b8ab38560923820c1e8303323a34431a"}
Nov 01 14:45:04.433348 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:04.432881    4993 scope.go:122] "RemoveContainer" containerID="208eddbf7f39a81cd0a0761bb0ddd959c1a99acc52af58fba9ee2f5c8b27880d"
Nov 01 14:45:04.433348 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:04.433087    4993 scope.go:122] "RemoveContainer" containerID="e9b64ee554f7afb47eacf40a21e16107b8ab38560923820c1e8303323a34431a"
Nov 01 14:45:04.433497 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:45:04.433442    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-536a87-29892405-mm2mf_odh-ai-gateway-infra(c9ffca0d-15ec-4324-b8c9-dd23d23898e7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892405-mm2mf" podUID="c9ffca0d-15ec-4324-b8c9-dd23d23898e7"
Nov 01 14:45:04.435547 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:04.435526    4993 generic.go:356] "Generic (PLEG): container finished" podID="903f1b4f-57e1-4fe4-95ba-14b49e728b9a" containerID="6639d605405ec643be8b88773f48f49c8add8ef4ee7d0f2f93b0e84e51662f67" exitCode=60
Nov 01 14:45:04.435686 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:04.435573    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892405-t8xr7" event={"ID":"903f1b4f-57e1-4fe4-95ba-14b49e728b9a","Type":"ContainerDied","Data":"6639d605405ec643be8b88773f48f49c8add8ef4ee7d0f2f93b0e84e51662f67"}
Nov 01 14:45:04.435905 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:04.435890    4993 scope.go:122] "RemoveContainer" containerID="6639d605405ec643be8b88773f48f49c8add8ef4ee7d0f2f93b0e84e51662f67"
Nov 01 14:45:04.436180 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:45:04.436147    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892405-t8xr7_odh-ai-gateway-infra(903f1b4f-57e1-4fe4-95ba-14b49e728b9a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892405-t8xr7" podUID="903f1b4f-57e1-4fe4-95ba-14b49e728b9a"
Nov 01 14:45:04.447552 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:04.447527    4993 scope.go:122] "RemoveContainer" containerID="cee47b34f5d68ae6e9bad592f4b195c2df613249188e3783a6f57f06da1ebc47"
Nov 01 14:45:05.444087 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:05.444050    4993 scope.go:122] "RemoveContainer" containerID="6639d605405ec643be8b88773f48f49c8add8ef4ee7d0f2f93b0e84e51662f67"
Nov 01 14:45:05.444602 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:45:05.444461    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892405-t8xr7_odh-ai-gateway-infra(903f1b4f-57e1-4fe4-95ba-14b49e728b9a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892405-t8xr7" podUID="903f1b4f-57e1-4fe4-95ba-14b49e728b9a"
Nov 01 14:45:05.445745 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:05.445723    4993 scope.go:122] "RemoveContainer" containerID="e9b64ee554f7afb47eacf40a21e16107b8ab38560923820c1e8303323a34431a"
Nov 01 14:45:05.445973 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:45:05.445954    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-536a87-29892405-mm2mf_odh-ai-gateway-infra(c9ffca0d-15ec-4324-b8c9-dd23d23898e7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892405-mm2mf" podUID="c9ffca0d-15ec-4324-b8c9-dd23d23898e7"
Nov 01 14:45:07.804216 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:07.803049    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-c9b0827c/e2e-lifecycle-model-c9b0827c-kserve-78c9dd7bf-9p2bh"]
Nov 01 14:45:14.787978 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:14.787943    4993 scope.go:122] "RemoveContainer" containerID="e9b64ee554f7afb47eacf40a21e16107b8ab38560923820c1e8303323a34431a"
Nov 01 14:45:14.789150 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:14.788825    4993 scope.go:122] "RemoveContainer" containerID="6639d605405ec643be8b88773f48f49c8add8ef4ee7d0f2f93b0e84e51662f67"
Nov 01 14:45:15.516731 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:15.516687    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892405-t8xr7" event={"ID":"903f1b4f-57e1-4fe4-95ba-14b49e728b9a","Type":"ContainerStarted","Data":"d3869549708b04be7aa74970abfb1fbb37be49ff82e6fd708a19135fea64170e"}
Nov 01 14:45:15.518740 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:15.518712    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892405-mm2mf" event={"ID":"c9ffca0d-15ec-4324-b8c9-dd23d23898e7","Type":"ContainerStarted","Data":"b09116e839d356d8544a2e8261dc26eb2240c1fce2a8ea0507d1bd0894f119a4"}
Nov 01 14:45:16.529895 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:16.529862    4993 generic.go:356] "Generic (PLEG): container finished" podID="903f1b4f-57e1-4fe4-95ba-14b49e728b9a" containerID="d3869549708b04be7aa74970abfb1fbb37be49ff82e6fd708a19135fea64170e" exitCode=60
Nov 01 14:45:16.530407 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:16.529940    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892405-t8xr7" event={"ID":"903f1b4f-57e1-4fe4-95ba-14b49e728b9a","Type":"ContainerDied","Data":"d3869549708b04be7aa74970abfb1fbb37be49ff82e6fd708a19135fea64170e"}
Nov 01 14:45:16.530407 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:16.530015    4993 scope.go:122] "RemoveContainer" containerID="6639d605405ec643be8b88773f48f49c8add8ef4ee7d0f2f93b0e84e51662f67"
Nov 01 14:45:16.530545 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:16.530421    4993 scope.go:122] "RemoveContainer" containerID="d3869549708b04be7aa74970abfb1fbb37be49ff82e6fd708a19135fea64170e"
Nov 01 14:45:16.530803 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:45:16.530777    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892405-t8xr7_odh-ai-gateway-infra(903f1b4f-57e1-4fe4-95ba-14b49e728b9a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892405-t8xr7" podUID="903f1b4f-57e1-4fe4-95ba-14b49e728b9a"
Nov 01 14:45:16.532050 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:16.532030    4993 generic.go:356] "Generic (PLEG): container finished" podID="c9ffca0d-15ec-4324-b8c9-dd23d23898e7" containerID="b09116e839d356d8544a2e8261dc26eb2240c1fce2a8ea0507d1bd0894f119a4" exitCode=60
Nov 01 14:45:16.532122 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:16.532080    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892405-mm2mf" event={"ID":"c9ffca0d-15ec-4324-b8c9-dd23d23898e7","Type":"ContainerDied","Data":"b09116e839d356d8544a2e8261dc26eb2240c1fce2a8ea0507d1bd0894f119a4"}
Nov 01 14:45:16.532439 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:16.532426    4993 scope.go:122] "RemoveContainer" containerID="b09116e839d356d8544a2e8261dc26eb2240c1fce2a8ea0507d1bd0894f119a4"
Nov 01 14:45:16.532678 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:45:16.532664    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-536a87-29892405-mm2mf_odh-ai-gateway-infra(c9ffca0d-15ec-4324-b8c9-dd23d23898e7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892405-mm2mf" podUID="c9ffca0d-15ec-4324-b8c9-dd23d23898e7"
Nov 01 14:45:16.542280 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:16.542251    4993 scope.go:122] "RemoveContainer" containerID="e9b64ee554f7afb47eacf40a21e16107b8ab38560923820c1e8303323a34431a"
Nov 01 14:45:16.636583 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:16.636545    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892405-t8xr7"]
Nov 01 14:45:16.744021 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:16.743985    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892405-mm2mf"]
Nov 01 14:45:17.748719 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:17.748691    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892405-t8xr7"
Nov 01 14:45:17.753349 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:17.753322    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892405-mm2mf"
Nov 01 14:45:17.805612 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:17.805533    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9ffca0d-15ec-4324-b8c9-dd23d23898e7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9ffca0d-15ec-4324-b8c9-dd23d23898e7-maas-api-service-ca\") pod \"c9ffca0d-15ec-4324-b8c9-dd23d23898e7\" (UID: \"c9ffca0d-15ec-4324-b8c9-dd23d23898e7\") "
Nov 01 14:45:17.805762 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:17.805730    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/903f1b4f-57e1-4fe4-95ba-14b49e728b9a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/903f1b4f-57e1-4fe4-95ba-14b49e728b9a-maas-api-service-ca\") pod \"903f1b4f-57e1-4fe4-95ba-14b49e728b9a\" (UID: \"903f1b4f-57e1-4fe4-95ba-14b49e728b9a\") "
Nov 01 14:45:17.805805 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:17.805773    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9ffca0d-15ec-4324-b8c9-dd23d23898e7-kube-api-access-ql95l\" (UniqueName: \"kubernetes.io/projected/c9ffca0d-15ec-4324-b8c9-dd23d23898e7-kube-api-access-ql95l\") pod \"c9ffca0d-15ec-4324-b8c9-dd23d23898e7\" (UID: \"c9ffca0d-15ec-4324-b8c9-dd23d23898e7\") "
Nov 01 14:45:17.805943 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:17.805924    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9ffca0d-15ec-4324-b8c9-dd23d23898e7-maas-api-service-ca" pod "c9ffca0d-15ec-4324-b8c9-dd23d23898e7" (UID: "c9ffca0d-15ec-4324-b8c9-dd23d23898e7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:45:17.806070 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:17.806004    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/903f1b4f-57e1-4fe4-95ba-14b49e728b9a-kube-api-access-rwxps\" (UniqueName: \"kubernetes.io/projected/903f1b4f-57e1-4fe4-95ba-14b49e728b9a-kube-api-access-rwxps\") pod \"903f1b4f-57e1-4fe4-95ba-14b49e728b9a\" (UID: \"903f1b4f-57e1-4fe4-95ba-14b49e728b9a\") "
Nov 01 14:45:17.806192 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:17.806044    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/903f1b4f-57e1-4fe4-95ba-14b49e728b9a-maas-api-service-ca" pod "903f1b4f-57e1-4fe4-95ba-14b49e728b9a" (UID: "903f1b4f-57e1-4fe4-95ba-14b49e728b9a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:45:17.806591 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:17.806568    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9ffca0d-15ec-4324-b8c9-dd23d23898e7-maas-api-service-ca\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:45:17.806591 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:17.806590    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/903f1b4f-57e1-4fe4-95ba-14b49e728b9a-maas-api-service-ca\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:45:17.808007 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:17.807983    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/903f1b4f-57e1-4fe4-95ba-14b49e728b9a-kube-api-access-rwxps" pod "903f1b4f-57e1-4fe4-95ba-14b49e728b9a" (UID: "903f1b4f-57e1-4fe4-95ba-14b49e728b9a"). InnerVolumeSpecName "kube-api-access-rwxps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:45:17.808098 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:17.808010    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9ffca0d-15ec-4324-b8c9-dd23d23898e7-kube-api-access-ql95l" pod "c9ffca0d-15ec-4324-b8c9-dd23d23898e7" (UID: "c9ffca0d-15ec-4324-b8c9-dd23d23898e7"). InnerVolumeSpecName "kube-api-access-ql95l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:45:17.911209 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:17.910025    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwxps\" (UniqueName: \"kubernetes.io/projected/903f1b4f-57e1-4fe4-95ba-14b49e728b9a-kube-api-access-rwxps\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:45:17.911209 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:17.910062    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ql95l\" (UniqueName: \"kubernetes.io/projected/c9ffca0d-15ec-4324-b8c9-dd23d23898e7-kube-api-access-ql95l\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:45:18.556149 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:18.554906    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892405-t8xr7" event={"ID":"903f1b4f-57e1-4fe4-95ba-14b49e728b9a","Type":"ContainerDied","Data":"0b68faaa481fab399b8c1ffe46a933bd5b7c5bbc3512700f95d6fb631cb91a6f"}
Nov 01 14:45:18.556149 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:18.554982    4993 scope.go:122] "RemoveContainer" containerID="d3869549708b04be7aa74970abfb1fbb37be49ff82e6fd708a19135fea64170e"
Nov 01 14:45:18.556149 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:18.555075    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892405-t8xr7"
Nov 01 14:45:18.621740 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:18.613227    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892405-mm2mf" event={"ID":"c9ffca0d-15ec-4324-b8c9-dd23d23898e7","Type":"ContainerDied","Data":"3e12f903790a613577d54da3e6c2ed72a211695a35315ade0a1dc39f2ec04c6f"}
Nov 01 14:45:18.621740 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:18.613271    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892405-mm2mf"
Nov 01 14:45:18.688026 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:18.688002    4993 scope.go:122] "RemoveContainer" containerID="b09116e839d356d8544a2e8261dc26eb2240c1fce2a8ea0507d1bd0894f119a4"
Nov 01 14:45:18.786263 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:18.786221    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892405-mm2mf"]
Nov 01 14:45:18.787058 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:18.787022    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892405-mm2mf"]
Nov 01 14:45:18.795861 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:18.795833    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9ffca0d-15ec-4324-b8c9-dd23d23898e7" path="/var/lib/kubelet/pods/c9ffca0d-15ec-4324-b8c9-dd23d23898e7/volumes"
Nov 01 14:45:18.884590 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:18.884506    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892405-t8xr7"]
Nov 01 14:45:18.894511 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:18.894438    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892405-t8xr7"]
Nov 01 14:45:20.796582 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:20.796550    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="903f1b4f-57e1-4fe4-95ba-14b49e728b9a" path="/var/lib/kubelet/pods/903f1b4f-57e1-4fe4-95ba-14b49e728b9a/volumes"
Nov 01 14:45:22.886542 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:22.886493    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-c9b0827c/e2e-lifecycle-model-c9b0827c-kserve-78c9dd7bf-9p2bh" podUID="57e7bcd7-adc6-49f3-a2b2-529ea5e5603a" containerName="main" containerID="cri-o://2005cf74e5d21957527e37649d7ad4fc7e6ea8cb247e7e4d077c25eb2edb8141" gracePeriod=45
Nov 01 14:45:23.177397 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:23.177368    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c9b0827c/e2e-lifecycle-model-c9b0827c-kserve-78c9dd7bf-9p2bh"
Nov 01 14:45:23.337580 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:23.337541    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57e7bcd7-adc6-49f3-a2b2-529ea5e5603a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57e7bcd7-adc6-49f3-a2b2-529ea5e5603a-model-cache\") pod \"57e7bcd7-adc6-49f3-a2b2-529ea5e5603a\" (UID: \"57e7bcd7-adc6-49f3-a2b2-529ea5e5603a\") "
Nov 01 14:45:23.337754 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:23.337686    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57e7bcd7-adc6-49f3-a2b2-529ea5e5603a-tls-certs\" (UniqueName: \"kubernetes.io/secret/57e7bcd7-adc6-49f3-a2b2-529ea5e5603a-tls-certs\") pod \"57e7bcd7-adc6-49f3-a2b2-529ea5e5603a\" (UID: \"57e7bcd7-adc6-49f3-a2b2-529ea5e5603a\") "
Nov 01 14:45:23.337754 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:23.337721    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57e7bcd7-adc6-49f3-a2b2-529ea5e5603a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57e7bcd7-adc6-49f3-a2b2-529ea5e5603a-tmp-dir\") pod \"57e7bcd7-adc6-49f3-a2b2-529ea5e5603a\" (UID: \"57e7bcd7-adc6-49f3-a2b2-529ea5e5603a\") "
Nov 01 14:45:23.337939 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:23.337815    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57e7bcd7-adc6-49f3-a2b2-529ea5e5603a-model-cache" pod "57e7bcd7-adc6-49f3-a2b2-529ea5e5603a" (UID: "57e7bcd7-adc6-49f3-a2b2-529ea5e5603a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:45:23.337939 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:23.337824    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57e7bcd7-adc6-49f3-a2b2-529ea5e5603a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/57e7bcd7-adc6-49f3-a2b2-529ea5e5603a-dshm\") pod \"57e7bcd7-adc6-49f3-a2b2-529ea5e5603a\" (UID: \"57e7bcd7-adc6-49f3-a2b2-529ea5e5603a\") "
Nov 01 14:45:23.338060 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:23.337995    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57e7bcd7-adc6-49f3-a2b2-529ea5e5603a-kube-api-access-m2bf2\" (UniqueName: \"kubernetes.io/projected/57e7bcd7-adc6-49f3-a2b2-529ea5e5603a-kube-api-access-m2bf2\") pod \"57e7bcd7-adc6-49f3-a2b2-529ea5e5603a\" (UID: \"57e7bcd7-adc6-49f3-a2b2-529ea5e5603a\") "
Nov 01 14:45:23.338117 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:23.338075    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57e7bcd7-adc6-49f3-a2b2-529ea5e5603a-home\" (UniqueName: \"kubernetes.io/empty-dir/57e7bcd7-adc6-49f3-a2b2-529ea5e5603a-home\") pod \"57e7bcd7-adc6-49f3-a2b2-529ea5e5603a\" (UID: \"57e7bcd7-adc6-49f3-a2b2-529ea5e5603a\") "
Nov 01 14:45:23.338454 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:23.338427    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57e7bcd7-adc6-49f3-a2b2-529ea5e5603a-tmp-dir" pod "57e7bcd7-adc6-49f3-a2b2-529ea5e5603a" (UID: "57e7bcd7-adc6-49f3-a2b2-529ea5e5603a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:45:23.338631 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:23.338443    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57e7bcd7-adc6-49f3-a2b2-529ea5e5603a-home" pod "57e7bcd7-adc6-49f3-a2b2-529ea5e5603a" (UID: "57e7bcd7-adc6-49f3-a2b2-529ea5e5603a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:45:23.338849 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:23.338820    4993 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57e7bcd7-adc6-49f3-a2b2-529ea5e5603a-model-cache\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:45:23.338937 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:23.338864    4993 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57e7bcd7-adc6-49f3-a2b2-529ea5e5603a-tmp-dir\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:45:23.340482 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:23.340445    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57e7bcd7-adc6-49f3-a2b2-529ea5e5603a-tls-certs" pod "57e7bcd7-adc6-49f3-a2b2-529ea5e5603a" (UID: "57e7bcd7-adc6-49f3-a2b2-529ea5e5603a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:45:23.340920 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:23.340891    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57e7bcd7-adc6-49f3-a2b2-529ea5e5603a-kube-api-access-m2bf2" pod "57e7bcd7-adc6-49f3-a2b2-529ea5e5603a" (UID: "57e7bcd7-adc6-49f3-a2b2-529ea5e5603a"). InnerVolumeSpecName "kube-api-access-m2bf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:45:23.341379 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:23.341359    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57e7bcd7-adc6-49f3-a2b2-529ea5e5603a-dshm" pod "57e7bcd7-adc6-49f3-a2b2-529ea5e5603a" (UID: "57e7bcd7-adc6-49f3-a2b2-529ea5e5603a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:45:23.439972 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:23.439885    4993 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57e7bcd7-adc6-49f3-a2b2-529ea5e5603a-home\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:45:23.439972 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:23.439920    4993 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57e7bcd7-adc6-49f3-a2b2-529ea5e5603a-tls-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:45:23.439972 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:23.439929    4993 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57e7bcd7-adc6-49f3-a2b2-529ea5e5603a-dshm\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:45:23.439972 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:23.439938    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2bf2\" (UniqueName: \"kubernetes.io/projected/57e7bcd7-adc6-49f3-a2b2-529ea5e5603a-kube-api-access-m2bf2\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:45:23.652324 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:23.652291    4993 generic.go:356] "Generic (PLEG): container finished" podID="57e7bcd7-adc6-49f3-a2b2-529ea5e5603a" containerID="2005cf74e5d21957527e37649d7ad4fc7e6ea8cb247e7e4d077c25eb2edb8141" exitCode=0
Nov 01 14:45:23.652527 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:23.652461    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c9b0827c/e2e-lifecycle-model-c9b0827c-kserve-78c9dd7bf-9p2bh" event={"ID":"57e7bcd7-adc6-49f3-a2b2-529ea5e5603a","Type":"ContainerDied","Data":"2005cf74e5d21957527e37649d7ad4fc7e6ea8cb247e7e4d077c25eb2edb8141"}
Nov 01 14:45:23.652527 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:23.652491    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c9b0827c/e2e-lifecycle-model-c9b0827c-kserve-78c9dd7bf-9p2bh" event={"ID":"57e7bcd7-adc6-49f3-a2b2-529ea5e5603a","Type":"ContainerDied","Data":"01081914ca10d2122727d26732072fe94ec0f8b438da54e8e518493ba00b5c5b"}
Nov 01 14:45:23.652527 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:23.652507    4993 scope.go:122] "RemoveContainer" containerID="2005cf74e5d21957527e37649d7ad4fc7e6ea8cb247e7e4d077c25eb2edb8141"
Nov 01 14:45:23.652675 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:23.652616    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c9b0827c/e2e-lifecycle-model-c9b0827c-kserve-78c9dd7bf-9p2bh"
Nov 01 14:45:23.668406 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:23.668378    4993 scope.go:122] "RemoveContainer" containerID="2005cf74e5d21957527e37649d7ad4fc7e6ea8cb247e7e4d077c25eb2edb8141"
Nov 01 14:45:23.668746 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:45:23.668725    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2005cf74e5d21957527e37649d7ad4fc7e6ea8cb247e7e4d077c25eb2edb8141\": container with ID starting with 2005cf74e5d21957527e37649d7ad4fc7e6ea8cb247e7e4d077c25eb2edb8141 not found: ID does not exist" containerID="2005cf74e5d21957527e37649d7ad4fc7e6ea8cb247e7e4d077c25eb2edb8141"
Nov 01 14:45:23.668816 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:23.668758    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2005cf74e5d21957527e37649d7ad4fc7e6ea8cb247e7e4d077c25eb2edb8141"} err="failed to get container status \"2005cf74e5d21957527e37649d7ad4fc7e6ea8cb247e7e4d077c25eb2edb8141\": rpc error: code = NotFound desc = could not find container \"2005cf74e5d21957527e37649d7ad4fc7e6ea8cb247e7e4d077c25eb2edb8141\": container with ID starting with 2005cf74e5d21957527e37649d7ad4fc7e6ea8cb247e7e4d077c25eb2edb8141 not found: ID does not exist"
Nov 01 14:45:23.808516 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:23.808480    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-c9b0827c/e2e-lifecycle-model-c9b0827c-kserve-78c9dd7bf-9p2bh"]
Nov 01 14:45:23.856453 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:23.856411    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-c9b0827c/e2e-lifecycle-model-c9b0827c-kserve-78c9dd7bf-9p2bh"]
Nov 01 14:45:24.796943 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:24.796911    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57e7bcd7-adc6-49f3-a2b2-529ea5e5603a" path="/var/lib/kubelet/pods/57e7bcd7-adc6-49f3-a2b2-529ea5e5603a/volumes"
Nov 01 14:45:27.522435 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:45:27.522376    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:45:47.320211 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:47.304392    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c9b0827c-65bb9b948d-6kxb4"]
Nov 01 14:45:47.320211 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:47.304652    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9b0827c-65bb9b948d-6kxb4" podUID="7549aec7-a94c-47a2-945d-1a2a4d31fc69" containerName="maas-api" containerID="cri-o://a1c8ac22071d401cc4c09040bde2c53c68ac2191eecf5b09f312f0ad485ece78" gracePeriod=30
Nov 01 14:45:47.653819 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:47.653787    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9b0827c-65bb9b948d-6kxb4"
Nov 01 14:45:47.744487 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:47.744427    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7549aec7-a94c-47a2-945d-1a2a4d31fc69-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7549aec7-a94c-47a2-945d-1a2a4d31fc69-maas-api-tls\") pod \"7549aec7-a94c-47a2-945d-1a2a4d31fc69\" (UID: \"7549aec7-a94c-47a2-945d-1a2a4d31fc69\") "
Nov 01 14:45:47.744852 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:47.744824    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7549aec7-a94c-47a2-945d-1a2a4d31fc69-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7549aec7-a94c-47a2-945d-1a2a4d31fc69-openshift-service-ca\") pod \"7549aec7-a94c-47a2-945d-1a2a4d31fc69\" (UID: \"7549aec7-a94c-47a2-945d-1a2a4d31fc69\") "
Nov 01 14:45:47.745279 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:47.745253    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7549aec7-a94c-47a2-945d-1a2a4d31fc69-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7549aec7-a94c-47a2-945d-1a2a4d31fc69-metrics-certs\") pod \"7549aec7-a94c-47a2-945d-1a2a4d31fc69\" (UID: \"7549aec7-a94c-47a2-945d-1a2a4d31fc69\") "
Nov 01 14:45:47.745440 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:47.745422    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7549aec7-a94c-47a2-945d-1a2a4d31fc69-kube-api-access-w2qgh\" (UniqueName: \"kubernetes.io/projected/7549aec7-a94c-47a2-945d-1a2a4d31fc69-kube-api-access-w2qgh\") pod \"7549aec7-a94c-47a2-945d-1a2a4d31fc69\" (UID: \"7549aec7-a94c-47a2-945d-1a2a4d31fc69\") "
Nov 01 14:45:47.748580 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:47.748550    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7549aec7-a94c-47a2-945d-1a2a4d31fc69-openshift-service-ca" pod "7549aec7-a94c-47a2-945d-1a2a4d31fc69" (UID: "7549aec7-a94c-47a2-945d-1a2a4d31fc69"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:45:47.753895 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:47.753851    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7549aec7-a94c-47a2-945d-1a2a4d31fc69-metrics-certs" pod "7549aec7-a94c-47a2-945d-1a2a4d31fc69" (UID: "7549aec7-a94c-47a2-945d-1a2a4d31fc69"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:45:47.754140 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:47.754120    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7549aec7-a94c-47a2-945d-1a2a4d31fc69-kube-api-access-w2qgh" pod "7549aec7-a94c-47a2-945d-1a2a4d31fc69" (UID: "7549aec7-a94c-47a2-945d-1a2a4d31fc69"). InnerVolumeSpecName "kube-api-access-w2qgh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:45:47.755500 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:47.755476    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7549aec7-a94c-47a2-945d-1a2a4d31fc69-maas-api-tls" pod "7549aec7-a94c-47a2-945d-1a2a4d31fc69" (UID: "7549aec7-a94c-47a2-945d-1a2a4d31fc69"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:45:47.834541 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:47.834501    4993 generic.go:356] "Generic (PLEG): container finished" podID="7549aec7-a94c-47a2-945d-1a2a4d31fc69" containerID="a1c8ac22071d401cc4c09040bde2c53c68ac2191eecf5b09f312f0ad485ece78" exitCode=0
Nov 01 14:45:47.834738 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:47.834558    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9b0827c-65bb9b948d-6kxb4" event={"ID":"7549aec7-a94c-47a2-945d-1a2a4d31fc69","Type":"ContainerDied","Data":"a1c8ac22071d401cc4c09040bde2c53c68ac2191eecf5b09f312f0ad485ece78"}
Nov 01 14:45:47.834738 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:47.834606    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9b0827c-65bb9b948d-6kxb4" event={"ID":"7549aec7-a94c-47a2-945d-1a2a4d31fc69","Type":"ContainerDied","Data":"683a16187f5ad9148c8452ffeea9a2971605fd4bf23ab8ac95ec797991ae7deb"}
Nov 01 14:45:47.834738 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:47.834636    4993 scope.go:122] "RemoveContainer" containerID="a1c8ac22071d401cc4c09040bde2c53c68ac2191eecf5b09f312f0ad485ece78"
Nov 01 14:45:47.834926 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:47.834800    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9b0827c-65bb9b948d-6kxb4"
Nov 01 14:45:47.847739 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:47.847712    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7549aec7-a94c-47a2-945d-1a2a4d31fc69-maas-api-tls\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:45:47.847739 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:47.847739    4993 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7549aec7-a94c-47a2-945d-1a2a4d31fc69-openshift-service-ca\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:45:47.847928 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:47.847748    4993 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7549aec7-a94c-47a2-945d-1a2a4d31fc69-metrics-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:45:47.847928 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:47.847758    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2qgh\" (UniqueName: \"kubernetes.io/projected/7549aec7-a94c-47a2-945d-1a2a4d31fc69-kube-api-access-w2qgh\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:45:47.849557 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:47.849534    4993 scope.go:122] "RemoveContainer" containerID="a1c8ac22071d401cc4c09040bde2c53c68ac2191eecf5b09f312f0ad485ece78"
Nov 01 14:45:47.849854 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:45:47.849833    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1c8ac22071d401cc4c09040bde2c53c68ac2191eecf5b09f312f0ad485ece78\": container with ID starting with a1c8ac22071d401cc4c09040bde2c53c68ac2191eecf5b09f312f0ad485ece78 not found: ID does not exist" containerID="a1c8ac22071d401cc4c09040bde2c53c68ac2191eecf5b09f312f0ad485ece78"
Nov 01 14:45:47.849930 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:47.849863    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1c8ac22071d401cc4c09040bde2c53c68ac2191eecf5b09f312f0ad485ece78"} err="failed to get container status \"a1c8ac22071d401cc4c09040bde2c53c68ac2191eecf5b09f312f0ad485ece78\": rpc error: code = NotFound desc = could not find container \"a1c8ac22071d401cc4c09040bde2c53c68ac2191eecf5b09f312f0ad485ece78\": container with ID starting with a1c8ac22071d401cc4c09040bde2c53c68ac2191eecf5b09f312f0ad485ece78 not found: ID does not exist"
Nov 01 14:45:47.991145 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:47.991113    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c9b0827c-65bb9b948d-6kxb4"]
Nov 01 14:45:47.994278 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:47.994252    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c9b0827c-65bb9b948d-6kxb4"]
Nov 01 14:45:48.798930 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:48.798887    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7549aec7-a94c-47a2-945d-1a2a4d31fc69" path="/var/lib/kubelet/pods/7549aec7-a94c-47a2-945d-1a2a4d31fc69/volumes"
Nov 01 14:45:53.288080 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:45:53.288044    4993 scope.go:122] "RemoveContainer" containerID="e8872ab7a79e5fb74267da19c37f18f2e3371b4afdea31844563c9d1da7e9b7d"
Nov 01 14:45:57.502271 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:45:57.502228    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:46:13.006138 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:13.006097    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-26a0c1-769bfd4c5f-9jwpp"]
Nov 01 14:46:13.006627 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:13.006354    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26a0c1-769bfd4c5f-9jwpp" podUID="4d460221-d6fe-4574-84eb-c589a367eebe" containerName="maas-api" containerID="cri-o://13db682a608e184fea4f724bbf773e6c133debd22c3356d3abc655a53992b103" gracePeriod=30
Nov 01 14:46:13.323203 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:13.323135    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26a0c1-769bfd4c5f-9jwpp"
Nov 01 14:46:13.420235 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:13.420198    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d460221-d6fe-4574-84eb-c589a367eebe-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d460221-d6fe-4574-84eb-c589a367eebe-metrics-certs\") pod \"4d460221-d6fe-4574-84eb-c589a367eebe\" (UID: \"4d460221-d6fe-4574-84eb-c589a367eebe\") "
Nov 01 14:46:13.420418 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:13.420353    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d460221-d6fe-4574-84eb-c589a367eebe-kube-api-access-n5xcw\" (UniqueName: \"kubernetes.io/projected/4d460221-d6fe-4574-84eb-c589a367eebe-kube-api-access-n5xcw\") pod \"4d460221-d6fe-4574-84eb-c589a367eebe\" (UID: \"4d460221-d6fe-4574-84eb-c589a367eebe\") "
Nov 01 14:46:13.420478 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:13.420415    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d460221-d6fe-4574-84eb-c589a367eebe-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d460221-d6fe-4574-84eb-c589a367eebe-maas-api-tls\") pod \"4d460221-d6fe-4574-84eb-c589a367eebe\" (UID: \"4d460221-d6fe-4574-84eb-c589a367eebe\") "
Nov 01 14:46:13.420583 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:13.420565    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d460221-d6fe-4574-84eb-c589a367eebe-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d460221-d6fe-4574-84eb-c589a367eebe-openshift-service-ca\") pod \"4d460221-d6fe-4574-84eb-c589a367eebe\" (UID: \"4d460221-d6fe-4574-84eb-c589a367eebe\") "
Nov 01 14:46:13.421508 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:13.421475    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d460221-d6fe-4574-84eb-c589a367eebe-openshift-service-ca" pod "4d460221-d6fe-4574-84eb-c589a367eebe" (UID: "4d460221-d6fe-4574-84eb-c589a367eebe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:46:13.422724 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:13.422700    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d460221-d6fe-4574-84eb-c589a367eebe-kube-api-access-n5xcw" pod "4d460221-d6fe-4574-84eb-c589a367eebe" (UID: "4d460221-d6fe-4574-84eb-c589a367eebe"). InnerVolumeSpecName "kube-api-access-n5xcw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:46:13.423131 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:13.423108    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d460221-d6fe-4574-84eb-c589a367eebe-maas-api-tls" pod "4d460221-d6fe-4574-84eb-c589a367eebe" (UID: "4d460221-d6fe-4574-84eb-c589a367eebe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:46:13.423600 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:13.423579    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d460221-d6fe-4574-84eb-c589a367eebe-metrics-certs" pod "4d460221-d6fe-4574-84eb-c589a367eebe" (UID: "4d460221-d6fe-4574-84eb-c589a367eebe"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:46:13.522008 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:13.521972    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5xcw\" (UniqueName: \"kubernetes.io/projected/4d460221-d6fe-4574-84eb-c589a367eebe-kube-api-access-n5xcw\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:46:13.522008 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:13.522006    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d460221-d6fe-4574-84eb-c589a367eebe-maas-api-tls\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:46:13.522261 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:13.522022    4993 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d460221-d6fe-4574-84eb-c589a367eebe-openshift-service-ca\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:46:13.522261 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:13.522036    4993 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d460221-d6fe-4574-84eb-c589a367eebe-metrics-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:46:14.099139 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:14.099100    4993 generic.go:356] "Generic (PLEG): container finished" podID="4d460221-d6fe-4574-84eb-c589a367eebe" containerID="13db682a608e184fea4f724bbf773e6c133debd22c3356d3abc655a53992b103" exitCode=0
Nov 01 14:46:14.099881 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:14.099845    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26a0c1-769bfd4c5f-9jwpp" event={"ID":"4d460221-d6fe-4574-84eb-c589a367eebe","Type":"ContainerDied","Data":"13db682a608e184fea4f724bbf773e6c133debd22c3356d3abc655a53992b103"}
Nov 01 14:46:14.100039 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:14.099919    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26a0c1-769bfd4c5f-9jwpp" event={"ID":"4d460221-d6fe-4574-84eb-c589a367eebe","Type":"ContainerDied","Data":"a600d85d84db0b3c61a495d23ee1f2045816685ea38e419bc6db942388796819"}
Nov 01 14:46:14.100039 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:14.099951    4993 scope.go:122] "RemoveContainer" containerID="13db682a608e184fea4f724bbf773e6c133debd22c3356d3abc655a53992b103"
Nov 01 14:46:14.100212 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:14.100108    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-26a0c1-769bfd4c5f-9jwpp"
Nov 01 14:46:14.116985 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:14.116956    4993 scope.go:122] "RemoveContainer" containerID="13db682a608e184fea4f724bbf773e6c133debd22c3356d3abc655a53992b103"
Nov 01 14:46:14.117573 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:46:14.117547    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13db682a608e184fea4f724bbf773e6c133debd22c3356d3abc655a53992b103\": container with ID starting with 13db682a608e184fea4f724bbf773e6c133debd22c3356d3abc655a53992b103 not found: ID does not exist" containerID="13db682a608e184fea4f724bbf773e6c133debd22c3356d3abc655a53992b103"
Nov 01 14:46:14.117681 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:14.117580    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13db682a608e184fea4f724bbf773e6c133debd22c3356d3abc655a53992b103"} err="failed to get container status \"13db682a608e184fea4f724bbf773e6c133debd22c3356d3abc655a53992b103\": rpc error: code = NotFound desc = could not find container \"13db682a608e184fea4f724bbf773e6c133debd22c3356d3abc655a53992b103\": container with ID starting with 13db682a608e184fea4f724bbf773e6c133debd22c3356d3abc655a53992b103 not found: ID does not exist"
Nov 01 14:46:14.257434 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:14.257398    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-26a0c1-769bfd4c5f-9jwpp"]
Nov 01 14:46:14.262515 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:14.262461    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-26a0c1-769bfd4c5f-9jwpp"]
Nov 01 14:46:14.796474 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:14.796441    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d460221-d6fe-4574-84eb-c589a367eebe" path="/var/lib/kubelet/pods/4d460221-d6fe-4574-84eb-c589a367eebe/volumes"
Nov 01 14:46:16.233892 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:16.233855    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j"]
Nov 01 14:46:16.234551 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:16.234121    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j" podUID="174f6587-8229-4907-892f-27df74d3f997" containerName="istio-proxy" containerID="cri-o://91c96200be516bfec1975a3eef77eb5710b9a3d0fc03cb884ce399e24c19ff33" gracePeriod=30
Nov 01 14:46:19.022804 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:19.022768    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-ks7q6"]
Nov 01 14:46:19.023590 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:19.023553    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-ks7q6" podUID="040af721-9da6-4df4-a334-a5d95c482f3b" containerName="istio-proxy" containerID="cri-o://8733d4402c69b4f52328853a6a7de8ec05d632d71f82b5fd3d906931df1f08ef" gracePeriod=30
Nov 01 14:46:21.533768 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:21.533735    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j"
Nov 01 14:46:21.638503 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:21.638471    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/174f6587-8229-4907-892f-27df74d3f997-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/174f6587-8229-4907-892f-27df74d3f997-istio-podinfo\") pod \"174f6587-8229-4907-892f-27df74d3f997\" (UID: \"174f6587-8229-4907-892f-27df74d3f997\") "
Nov 01 14:46:21.638698 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:21.638612    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/174f6587-8229-4907-892f-27df74d3f997-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/174f6587-8229-4907-892f-27df74d3f997-istio-data\") pod \"174f6587-8229-4907-892f-27df74d3f997\" (UID: \"174f6587-8229-4907-892f-27df74d3f997\") "
Nov 01 14:46:21.638698 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:21.638641    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/174f6587-8229-4907-892f-27df74d3f997-kube-api-access-sgrp6\" (UniqueName: \"kubernetes.io/projected/174f6587-8229-4907-892f-27df74d3f997-kube-api-access-sgrp6\") pod \"174f6587-8229-4907-892f-27df74d3f997\" (UID: \"174f6587-8229-4907-892f-27df74d3f997\") "
Nov 01 14:46:21.638698 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:21.638671    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/174f6587-8229-4907-892f-27df74d3f997-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/174f6587-8229-4907-892f-27df74d3f997-workload-socket\") pod \"174f6587-8229-4907-892f-27df74d3f997\" (UID: \"174f6587-8229-4907-892f-27df74d3f997\") "
Nov 01 14:46:21.638698 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:21.638691    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/174f6587-8229-4907-892f-27df74d3f997-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/174f6587-8229-4907-892f-27df74d3f997-credential-socket\") pod \"174f6587-8229-4907-892f-27df74d3f997\" (UID: \"174f6587-8229-4907-892f-27df74d3f997\") "
Nov 01 14:46:21.638914 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:21.638709    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/174f6587-8229-4907-892f-27df74d3f997-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/174f6587-8229-4907-892f-27df74d3f997-istiod-ca-cert\") pod \"174f6587-8229-4907-892f-27df74d3f997\" (UID: \"174f6587-8229-4907-892f-27df74d3f997\") "
Nov 01 14:46:21.638998 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:21.638967    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/174f6587-8229-4907-892f-27df74d3f997-credential-socket" pod "174f6587-8229-4907-892f-27df74d3f997" (UID: "174f6587-8229-4907-892f-27df74d3f997"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:46:21.639060 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:21.639003    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/174f6587-8229-4907-892f-27df74d3f997-istio-data" pod "174f6587-8229-4907-892f-27df74d3f997" (UID: "174f6587-8229-4907-892f-27df74d3f997"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:46:21.639060 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:21.639014    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/174f6587-8229-4907-892f-27df74d3f997-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/174f6587-8229-4907-892f-27df74d3f997-istio-envoy\") pod \"174f6587-8229-4907-892f-27df74d3f997\" (UID: \"174f6587-8229-4907-892f-27df74d3f997\") "
Nov 01 14:46:21.639060 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:21.638985    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/174f6587-8229-4907-892f-27df74d3f997-workload-socket" pod "174f6587-8229-4907-892f-27df74d3f997" (UID: "174f6587-8229-4907-892f-27df74d3f997"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:46:21.639254 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:21.639115    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/174f6587-8229-4907-892f-27df74d3f997-istiod-ca-cert" pod "174f6587-8229-4907-892f-27df74d3f997" (UID: "174f6587-8229-4907-892f-27df74d3f997"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:46:21.639254 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:21.639211    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/174f6587-8229-4907-892f-27df74d3f997-istio-token\" (UniqueName: \"kubernetes.io/projected/174f6587-8229-4907-892f-27df74d3f997-istio-token\") pod \"174f6587-8229-4907-892f-27df74d3f997\" (UID: \"174f6587-8229-4907-892f-27df74d3f997\") "
Nov 01 14:46:21.639359 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:21.639293    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/174f6587-8229-4907-892f-27df74d3f997-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/174f6587-8229-4907-892f-27df74d3f997-workload-certs\") pod \"174f6587-8229-4907-892f-27df74d3f997\" (UID: \"174f6587-8229-4907-892f-27df74d3f997\") "
Nov 01 14:46:21.639623 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:21.639603    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/174f6587-8229-4907-892f-27df74d3f997-workload-certs" pod "174f6587-8229-4907-892f-27df74d3f997" (UID: "174f6587-8229-4907-892f-27df74d3f997"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:46:21.639923 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:21.639897    4993 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/174f6587-8229-4907-892f-27df74d3f997-istio-data\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:46:21.640036 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:21.639925    4993 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/174f6587-8229-4907-892f-27df74d3f997-workload-socket\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:46:21.640036 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:21.639941    4993 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/174f6587-8229-4907-892f-27df74d3f997-credential-socket\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:46:21.640036 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:21.639954    4993 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/174f6587-8229-4907-892f-27df74d3f997-istiod-ca-cert\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:46:21.640036 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:21.639968    4993 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/174f6587-8229-4907-892f-27df74d3f997-workload-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:46:21.641040 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:21.641018    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/174f6587-8229-4907-892f-27df74d3f997-kube-api-access-sgrp6" pod "174f6587-8229-4907-892f-27df74d3f997" (UID: "174f6587-8229-4907-892f-27df74d3f997"). InnerVolumeSpecName "kube-api-access-sgrp6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:46:21.641281 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:21.641249    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/174f6587-8229-4907-892f-27df74d3f997-istio-podinfo" pod "174f6587-8229-4907-892f-27df74d3f997" (UID: "174f6587-8229-4907-892f-27df74d3f997"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 14:46:21.641382 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:21.641354    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/174f6587-8229-4907-892f-27df74d3f997-istio-envoy" pod "174f6587-8229-4907-892f-27df74d3f997" (UID: "174f6587-8229-4907-892f-27df74d3f997"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:46:21.641575 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:21.641550    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/174f6587-8229-4907-892f-27df74d3f997-istio-token" pod "174f6587-8229-4907-892f-27df74d3f997" (UID: "174f6587-8229-4907-892f-27df74d3f997"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:46:21.741103 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:21.741067    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgrp6\" (UniqueName: \"kubernetes.io/projected/174f6587-8229-4907-892f-27df74d3f997-kube-api-access-sgrp6\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:46:21.741103 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:21.741096    4993 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/174f6587-8229-4907-892f-27df74d3f997-istio-envoy\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:46:21.741103 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:21.741105    4993 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/174f6587-8229-4907-892f-27df74d3f997-istio-token\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:46:21.741103 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:21.741114    4993 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/174f6587-8229-4907-892f-27df74d3f997-istio-podinfo\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:46:22.169102 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:22.169064    4993 generic.go:356] "Generic (PLEG): container finished" podID="174f6587-8229-4907-892f-27df74d3f997" containerID="91c96200be516bfec1975a3eef77eb5710b9a3d0fc03cb884ce399e24c19ff33" exitCode=0
Nov 01 14:46:22.169330 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:22.169118    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j"
Nov 01 14:46:22.169330 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:22.169160    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j" event={"ID":"174f6587-8229-4907-892f-27df74d3f997","Type":"ContainerDied","Data":"91c96200be516bfec1975a3eef77eb5710b9a3d0fc03cb884ce399e24c19ff33"}
Nov 01 14:46:22.169330 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:22.169253    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j" event={"ID":"174f6587-8229-4907-892f-27df74d3f997","Type":"ContainerDied","Data":"3eaea0f8bea3c2bf9c45b06d976d17e3a1399b860bdb6c0ad7b0b61dd5fa96e8"}
Nov 01 14:46:22.169330 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:22.169283    4993 scope.go:122] "RemoveContainer" containerID="91c96200be516bfec1975a3eef77eb5710b9a3d0fc03cb884ce399e24c19ff33"
Nov 01 14:46:22.180887 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:22.180859    4993 scope.go:122] "RemoveContainer" containerID="91c96200be516bfec1975a3eef77eb5710b9a3d0fc03cb884ce399e24c19ff33"
Nov 01 14:46:22.181334 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:46:22.181303    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91c96200be516bfec1975a3eef77eb5710b9a3d0fc03cb884ce399e24c19ff33\": container with ID starting with 91c96200be516bfec1975a3eef77eb5710b9a3d0fc03cb884ce399e24c19ff33 not found: ID does not exist" containerID="91c96200be516bfec1975a3eef77eb5710b9a3d0fc03cb884ce399e24c19ff33"
Nov 01 14:46:22.181418 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:22.181348    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91c96200be516bfec1975a3eef77eb5710b9a3d0fc03cb884ce399e24c19ff33"} err="failed to get container status \"91c96200be516bfec1975a3eef77eb5710b9a3d0fc03cb884ce399e24c19ff33\": rpc error: code = NotFound desc = could not find container \"91c96200be516bfec1975a3eef77eb5710b9a3d0fc03cb884ce399e24c19ff33\": container with ID starting with 91c96200be516bfec1975a3eef77eb5710b9a3d0fc03cb884ce399e24c19ff33 not found: ID does not exist"
Nov 01 14:46:22.323974 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:22.323938    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j"]
Nov 01 14:46:22.369423 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:22.369386    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-ps68j"]
Nov 01 14:46:22.795953 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:22.795918    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="174f6587-8229-4907-892f-27df74d3f997" path="/var/lib/kubelet/pods/174f6587-8229-4907-892f-27df74d3f997/volumes"
Nov 01 14:46:24.185926 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:24.185892    4993 generic.go:356] "Generic (PLEG): container finished" podID="040af721-9da6-4df4-a334-a5d95c482f3b" containerID="8733d4402c69b4f52328853a6a7de8ec05d632d71f82b5fd3d906931df1f08ef" exitCode=0
Nov 01 14:46:24.186321 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:24.185950    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-ks7q6" event={"ID":"040af721-9da6-4df4-a334-a5d95c482f3b","Type":"ContainerDied","Data":"8733d4402c69b4f52328853a6a7de8ec05d632d71f82b5fd3d906931df1f08ef"}
Nov 01 14:46:24.320416 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:24.320382    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-ks7q6"
Nov 01 14:46:24.371970 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:24.371930    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/040af721-9da6-4df4-a334-a5d95c482f3b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/040af721-9da6-4df4-a334-a5d95c482f3b-credential-socket\") pod \"040af721-9da6-4df4-a334-a5d95c482f3b\" (UID: \"040af721-9da6-4df4-a334-a5d95c482f3b\") "
Nov 01 14:46:24.372128 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:24.371991    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/040af721-9da6-4df4-a334-a5d95c482f3b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/040af721-9da6-4df4-a334-a5d95c482f3b-workload-certs\") pod \"040af721-9da6-4df4-a334-a5d95c482f3b\" (UID: \"040af721-9da6-4df4-a334-a5d95c482f3b\") "
Nov 01 14:46:24.372128 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:24.372040    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/040af721-9da6-4df4-a334-a5d95c482f3b-istio-token\" (UniqueName: \"kubernetes.io/projected/040af721-9da6-4df4-a334-a5d95c482f3b-istio-token\") pod \"040af721-9da6-4df4-a334-a5d95c482f3b\" (UID: \"040af721-9da6-4df4-a334-a5d95c482f3b\") "
Nov 01 14:46:24.372128 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:24.372109    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/040af721-9da6-4df4-a334-a5d95c482f3b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/040af721-9da6-4df4-a334-a5d95c482f3b-workload-socket\") pod \"040af721-9da6-4df4-a334-a5d95c482f3b\" (UID: \"040af721-9da6-4df4-a334-a5d95c482f3b\") "
Nov 01 14:46:24.372272 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:24.372157    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/040af721-9da6-4df4-a334-a5d95c482f3b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/040af721-9da6-4df4-a334-a5d95c482f3b-istio-envoy\") pod \"040af721-9da6-4df4-a334-a5d95c482f3b\" (UID: \"040af721-9da6-4df4-a334-a5d95c482f3b\") "
Nov 01 14:46:24.372272 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:24.372211    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/040af721-9da6-4df4-a334-a5d95c482f3b-credential-socket" pod "040af721-9da6-4df4-a334-a5d95c482f3b" (UID: "040af721-9da6-4df4-a334-a5d95c482f3b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:46:24.372272 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:24.372232    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/040af721-9da6-4df4-a334-a5d95c482f3b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/040af721-9da6-4df4-a334-a5d95c482f3b-istiod-ca-cert\") pod \"040af721-9da6-4df4-a334-a5d95c482f3b\" (UID: \"040af721-9da6-4df4-a334-a5d95c482f3b\") "
Nov 01 14:46:24.372272 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:24.372263    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/040af721-9da6-4df4-a334-a5d95c482f3b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/040af721-9da6-4df4-a334-a5d95c482f3b-istio-podinfo\") pod \"040af721-9da6-4df4-a334-a5d95c482f3b\" (UID: \"040af721-9da6-4df4-a334-a5d95c482f3b\") "
Nov 01 14:46:24.372426 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:24.372319    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/040af721-9da6-4df4-a334-a5d95c482f3b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/040af721-9da6-4df4-a334-a5d95c482f3b-istio-data\") pod \"040af721-9da6-4df4-a334-a5d95c482f3b\" (UID: \"040af721-9da6-4df4-a334-a5d95c482f3b\") "
Nov 01 14:46:24.372426 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:24.372321    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/040af721-9da6-4df4-a334-a5d95c482f3b-workload-certs" pod "040af721-9da6-4df4-a334-a5d95c482f3b" (UID: "040af721-9da6-4df4-a334-a5d95c482f3b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:46:24.372426 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:24.372394    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/040af721-9da6-4df4-a334-a5d95c482f3b-workload-socket" pod "040af721-9da6-4df4-a334-a5d95c482f3b" (UID: "040af721-9da6-4df4-a334-a5d95c482f3b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:46:24.372577 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:24.372448    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/040af721-9da6-4df4-a334-a5d95c482f3b-kube-api-access-x64wf\" (UniqueName: \"kubernetes.io/projected/040af721-9da6-4df4-a334-a5d95c482f3b-kube-api-access-x64wf\") pod \"040af721-9da6-4df4-a334-a5d95c482f3b\" (UID: \"040af721-9da6-4df4-a334-a5d95c482f3b\") "
Nov 01 14:46:24.372751 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:24.372701    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/040af721-9da6-4df4-a334-a5d95c482f3b-istio-data" pod "040af721-9da6-4df4-a334-a5d95c482f3b" (UID: "040af721-9da6-4df4-a334-a5d95c482f3b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:46:24.372751 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:24.372725    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/040af721-9da6-4df4-a334-a5d95c482f3b-istiod-ca-cert" pod "040af721-9da6-4df4-a334-a5d95c482f3b" (UID: "040af721-9da6-4df4-a334-a5d95c482f3b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:46:24.373129 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:24.373103    4993 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/040af721-9da6-4df4-a334-a5d95c482f3b-workload-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:46:24.373237 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:24.373134    4993 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/040af721-9da6-4df4-a334-a5d95c482f3b-workload-socket\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:46:24.373237 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:24.373150    4993 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/040af721-9da6-4df4-a334-a5d95c482f3b-istiod-ca-cert\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:46:24.373237 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:24.373194    4993 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/040af721-9da6-4df4-a334-a5d95c482f3b-istio-data\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:46:24.373237 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:24.373210    4993 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/040af721-9da6-4df4-a334-a5d95c482f3b-credential-socket\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:46:24.374898 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:24.374868    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/040af721-9da6-4df4-a334-a5d95c482f3b-istio-envoy" pod "040af721-9da6-4df4-a334-a5d95c482f3b" (UID: "040af721-9da6-4df4-a334-a5d95c482f3b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:46:24.374898 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:24.374894    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/040af721-9da6-4df4-a334-a5d95c482f3b-istio-token" pod "040af721-9da6-4df4-a334-a5d95c482f3b" (UID: "040af721-9da6-4df4-a334-a5d95c482f3b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:46:24.375112 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:24.374952    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/040af721-9da6-4df4-a334-a5d95c482f3b-kube-api-access-x64wf" pod "040af721-9da6-4df4-a334-a5d95c482f3b" (UID: "040af721-9da6-4df4-a334-a5d95c482f3b"). InnerVolumeSpecName "kube-api-access-x64wf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:46:24.375112 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:24.375095    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/040af721-9da6-4df4-a334-a5d95c482f3b-istio-podinfo" pod "040af721-9da6-4df4-a334-a5d95c482f3b" (UID: "040af721-9da6-4df4-a334-a5d95c482f3b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 14:46:24.474933 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:24.474893    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x64wf\" (UniqueName: \"kubernetes.io/projected/040af721-9da6-4df4-a334-a5d95c482f3b-kube-api-access-x64wf\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:46:24.474933 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:24.474931    4993 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/040af721-9da6-4df4-a334-a5d95c482f3b-istio-token\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:46:24.475160 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:24.474944    4993 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/040af721-9da6-4df4-a334-a5d95c482f3b-istio-envoy\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:46:24.475160 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:24.474958    4993 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/040af721-9da6-4df4-a334-a5d95c482f3b-istio-podinfo\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:46:25.193310 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:25.193219    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-ks7q6"
Nov 01 14:46:25.193310 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:25.193227    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-ks7q6" event={"ID":"040af721-9da6-4df4-a334-a5d95c482f3b","Type":"ContainerDied","Data":"c2c421cc10766da39bc12d6dc17641065944c4ae22d2d61be351a891e2c56ea0"}
Nov 01 14:46:25.193310 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:25.193302    4993 scope.go:122] "RemoveContainer" containerID="8733d4402c69b4f52328853a6a7de8ec05d632d71f82b5fd3d906931df1f08ef"
Nov 01 14:46:25.392661 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:25.392628    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-ks7q6"]
Nov 01 14:46:25.438798 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:25.438766    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-ks7q6"]
Nov 01 14:46:26.798293 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:26.798256    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="040af721-9da6-4df4-a334-a5d95c482f3b" path="/var/lib/kubelet/pods/040af721-9da6-4df4-a334-a5d95c482f3b/volumes"
Nov 01 14:46:27.499412 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:46:27.499373    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:46:53.042223 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:53.042191    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xmwtp_bb59051d-c2df-4362-a157-00f14f915e79/console-operator/3.log"
Nov 01 14:46:53.043691 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:46:53.043667    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xmwtp_bb59051d-c2df-4362-a157-00f14f915e79/console-operator/3.log"
Nov 01 14:46:57.502373 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:46:57.502334    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:47:00.428832 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:00.428791    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cbac83-76b7b8d447-5chmg"]
Nov 01 14:47:00.429487 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:00.429466    4993 cpu_manager.go:414] "containerMap: removing container" podUID="c9ffca0d-15ec-4324-b8c9-dd23d23898e7" containerName="cleanup"
Nov 01 14:47:00.429545 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:00.429488    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9ffca0d-15ec-4324-b8c9-dd23d23898e7" containerName="cleanup"
Nov 01 14:47:00.429545 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:00.429499    4993 cpu_manager.go:414] "containerMap: removing container" podUID="040af721-9da6-4df4-a334-a5d95c482f3b" containerName="istio-proxy"
Nov 01 14:47:00.429545 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:00.429504    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="040af721-9da6-4df4-a334-a5d95c482f3b" containerName="istio-proxy"
Nov 01 14:47:00.429545 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:00.429516    4993 cpu_manager.go:414] "containerMap: removing container" podUID="c9ffca0d-15ec-4324-b8c9-dd23d23898e7" containerName="cleanup"
Nov 01 14:47:00.429545 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:00.429521    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9ffca0d-15ec-4324-b8c9-dd23d23898e7" containerName="cleanup"
Nov 01 14:47:00.429545 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:00.429541    4993 cpu_manager.go:414] "containerMap: removing container" podUID="903f1b4f-57e1-4fe4-95ba-14b49e728b9a" containerName="cleanup"
Nov 01 14:47:00.429545 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:00.429546    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="903f1b4f-57e1-4fe4-95ba-14b49e728b9a" containerName="cleanup"
Nov 01 14:47:00.429763 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:00.429554    4993 cpu_manager.go:414] "containerMap: removing container" podUID="57e7bcd7-adc6-49f3-a2b2-529ea5e5603a" containerName="main"
Nov 01 14:47:00.429763 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:00.429559    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57e7bcd7-adc6-49f3-a2b2-529ea5e5603a" containerName="main"
Nov 01 14:47:00.429763 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:00.429568    4993 cpu_manager.go:414] "containerMap: removing container" podUID="174f6587-8229-4907-892f-27df74d3f997" containerName="istio-proxy"
Nov 01 14:47:00.429763 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:00.429572    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="174f6587-8229-4907-892f-27df74d3f997" containerName="istio-proxy"
Nov 01 14:47:00.429763 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:00.429585    4993 cpu_manager.go:414] "containerMap: removing container" podUID="c9ffca0d-15ec-4324-b8c9-dd23d23898e7" containerName="cleanup"
Nov 01 14:47:00.429763 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:00.429589    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9ffca0d-15ec-4324-b8c9-dd23d23898e7" containerName="cleanup"
Nov 01 14:47:00.429763 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:00.429594    4993 cpu_manager.go:414] "containerMap: removing container" podUID="7549aec7-a94c-47a2-945d-1a2a4d31fc69" containerName="maas-api"
Nov 01 14:47:00.429763 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:00.429599    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7549aec7-a94c-47a2-945d-1a2a4d31fc69" containerName="maas-api"
Nov 01 14:47:00.429763 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:00.429604    4993 cpu_manager.go:414] "containerMap: removing container" podUID="903f1b4f-57e1-4fe4-95ba-14b49e728b9a" containerName="cleanup"
Nov 01 14:47:00.429763 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:00.429608    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="903f1b4f-57e1-4fe4-95ba-14b49e728b9a" containerName="cleanup"
Nov 01 14:47:00.429763 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:00.429616    4993 cpu_manager.go:414] "containerMap: removing container" podUID="4d460221-d6fe-4574-84eb-c589a367eebe" containerName="maas-api"
Nov 01 14:47:00.429763 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:00.429621    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d460221-d6fe-4574-84eb-c589a367eebe" containerName="maas-api"
Nov 01 14:47:00.429763 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:00.429694    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9ffca0d-15ec-4324-b8c9-dd23d23898e7" containerName="cleanup"
Nov 01 14:47:00.429763 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:00.429704    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="903f1b4f-57e1-4fe4-95ba-14b49e728b9a" containerName="cleanup"
Nov 01 14:47:00.429763 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:00.429709    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="57e7bcd7-adc6-49f3-a2b2-529ea5e5603a" containerName="main"
Nov 01 14:47:00.429763 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:00.429718    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="174f6587-8229-4907-892f-27df74d3f997" containerName="istio-proxy"
Nov 01 14:47:00.429763 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:00.429725    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="903f1b4f-57e1-4fe4-95ba-14b49e728b9a" containerName="cleanup"
Nov 01 14:47:00.429763 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:00.429732    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="7549aec7-a94c-47a2-945d-1a2a4d31fc69" containerName="maas-api"
Nov 01 14:47:00.429763 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:00.429739    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d460221-d6fe-4574-84eb-c589a367eebe" containerName="maas-api"
Nov 01 14:47:00.429763 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:00.429747    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9ffca0d-15ec-4324-b8c9-dd23d23898e7" containerName="cleanup"
Nov 01 14:47:00.429763 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:00.429752    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="040af721-9da6-4df4-a334-a5d95c482f3b" containerName="istio-proxy"
Nov 01 14:47:00.434644 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:00.434606    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbac83-76b7b8d447-5chmg"
Nov 01 14:47:00.447676 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:00.447646    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa7cf0a9-06d8-43fc-a81e-bc95b7f1bc80-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-cbac83-76b7b8d447-5chmg\" (UID: \"aa7cf0a9-06d8-43fc-a81e-bc95b7f1bc80\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbac83-76b7b8d447-5chmg"
Nov 01 14:47:00.447987 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:00.447960    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxh4t\" (UniqueName: \"kubernetes.io/projected/aa7cf0a9-06d8-43fc-a81e-bc95b7f1bc80-kube-api-access-bxh4t\") pod \"payload-pre-processing-e2e-ipp-b-cbac83-76b7b8d447-5chmg\" (UID: \"aa7cf0a9-06d8-43fc-a81e-bc95b7f1bc80\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbac83-76b7b8d447-5chmg"
Nov 01 14:47:00.529207 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:00.529155    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-cbac83\""
Nov 01 14:47:00.529470 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:00.529445    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-cbac83-dockercfg-m6l7p\""
Nov 01 14:47:00.535873 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:00.535848    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cbac83-76b7b8d447-5chmg"]
Nov 01 14:47:00.549094 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:00.549060    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxh4t\" (UniqueName: \"kubernetes.io/projected/aa7cf0a9-06d8-43fc-a81e-bc95b7f1bc80-kube-api-access-bxh4t\") pod \"payload-pre-processing-e2e-ipp-b-cbac83-76b7b8d447-5chmg\" (UID: \"aa7cf0a9-06d8-43fc-a81e-bc95b7f1bc80\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbac83-76b7b8d447-5chmg"
Nov 01 14:47:00.549239 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:00.549148    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa7cf0a9-06d8-43fc-a81e-bc95b7f1bc80-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-cbac83-76b7b8d447-5chmg\" (UID: \"aa7cf0a9-06d8-43fc-a81e-bc95b7f1bc80\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbac83-76b7b8d447-5chmg"
Nov 01 14:47:00.549939 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:00.549917    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa7cf0a9-06d8-43fc-a81e-bc95b7f1bc80-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-cbac83-76b7b8d447-5chmg\" (UID: \"aa7cf0a9-06d8-43fc-a81e-bc95b7f1bc80\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbac83-76b7b8d447-5chmg"
Nov 01 14:47:00.603314 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:00.603278    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxh4t\" (UniqueName: \"kubernetes.io/projected/aa7cf0a9-06d8-43fc-a81e-bc95b7f1bc80-kube-api-access-bxh4t\") pod \"payload-pre-processing-e2e-ipp-b-cbac83-76b7b8d447-5chmg\" (UID: \"aa7cf0a9-06d8-43fc-a81e-bc95b7f1bc80\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbac83-76b7b8d447-5chmg"
Nov 01 14:47:00.748343 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:00.748308    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbac83-76b7b8d447-5chmg"
Nov 01 14:47:01.123915 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:01.123834    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cbac83-76b7b8d447-5chmg"]
Nov 01 14:47:01.130514 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:47:01.130488    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa7cf0a9_06d8_43fc_a81e_bc95b7f1bc80.slice/crio-6ef77068c7f930140b4693122ce38f8c187bc1fbc2c49626d5d230b3f6c741b0 WatchSource:0}: Error finding container 6ef77068c7f930140b4693122ce38f8c187bc1fbc2c49626d5d230b3f6c741b0: Status 404 returned error can't find the container with id 6ef77068c7f930140b4693122ce38f8c187bc1fbc2c49626d5d230b3f6c741b0
Nov 01 14:47:01.435000 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:01.434912    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbac83-76b7b8d447-5chmg" event={"ID":"aa7cf0a9-06d8-43fc-a81e-bc95b7f1bc80","Type":"ContainerStarted","Data":"6ef77068c7f930140b4693122ce38f8c187bc1fbc2c49626d5d230b3f6c741b0"}
Nov 01 14:47:03.450828 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:03.450737    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbac83-76b7b8d447-5chmg" event={"ID":"aa7cf0a9-06d8-43fc-a81e-bc95b7f1bc80","Type":"ContainerStarted","Data":"dc8cf4c92c88fc3a7871cab58ac5d5ac6c64cc03b0345975a0383032909818d9"}
Nov 01 14:47:03.450828 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:03.450806    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbac83-76b7b8d447-5chmg"
Nov 01 14:47:03.559815 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:03.559750    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbac83-76b7b8d447-5chmg" podStartSLOduration=1.656547138 podStartE2EDuration="3.559731162s" totalImagesPullingTime="1.903184024s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:47:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:47:03.555685627 +0000 UTC m=+1810.885548360" watchObservedRunningTime="2026-11-01 14:47:03.559731162 +0000 UTC m=+1810.889593903"
Nov 01 14:47:06.018750 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:06.018707    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-be7e2e-785d7fd4ff-sq6wf"]
Nov 01 14:47:06.020232 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:06.020202    4993 cpu_manager.go:414] "containerMap: removing container" podUID="903f1b4f-57e1-4fe4-95ba-14b49e728b9a" containerName="cleanup"
Nov 01 14:47:06.020379 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:06.020363    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="903f1b4f-57e1-4fe4-95ba-14b49e728b9a" containerName="cleanup"
Nov 01 14:47:06.029191 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:06.020618    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="903f1b4f-57e1-4fe4-95ba-14b49e728b9a" containerName="cleanup"
Nov 01 14:47:06.029395 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:06.029374    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9ffca0d-15ec-4324-b8c9-dd23d23898e7" containerName="cleanup"
Nov 01 14:47:06.033677 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:06.033651    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-be7e2e-785d7fd4ff-sq6wf"
Nov 01 14:47:06.090320 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:06.090271    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-be7e2e-785d7fd4ff-sq6wf"]
Nov 01 14:47:06.133194 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:06.119141    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-be7e2e-serving-cert\""
Nov 01 14:47:06.300792 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:06.300695    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e02990c-0339-49b0-af86-17211a267bbc-metrics-certs\") pod \"maas-api-e2e-ipp-b-be7e2e-785d7fd4ff-sq6wf\" (UID: \"8e02990c-0339-49b0-af86-17211a267bbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-be7e2e-785d7fd4ff-sq6wf"
Nov 01 14:47:06.320188 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:06.316479    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e02990c-0339-49b0-af86-17211a267bbc-maas-api-tls\") pod \"maas-api-e2e-ipp-b-be7e2e-785d7fd4ff-sq6wf\" (UID: \"8e02990c-0339-49b0-af86-17211a267bbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-be7e2e-785d7fd4ff-sq6wf"
Nov 01 14:47:06.324225 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:06.320458    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgjpl\" (UniqueName: \"kubernetes.io/projected/8e02990c-0339-49b0-af86-17211a267bbc-kube-api-access-pgjpl\") pod \"maas-api-e2e-ipp-b-be7e2e-785d7fd4ff-sq6wf\" (UID: \"8e02990c-0339-49b0-af86-17211a267bbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-be7e2e-785d7fd4ff-sq6wf"
Nov 01 14:47:06.337296 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:06.337253    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e02990c-0339-49b0-af86-17211a267bbc-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-be7e2e-785d7fd4ff-sq6wf\" (UID: \"8e02990c-0339-49b0-af86-17211a267bbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-be7e2e-785d7fd4ff-sq6wf"
Nov 01 14:47:06.544257 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:06.544223    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e02990c-0339-49b0-af86-17211a267bbc-maas-api-tls\") pod \"maas-api-e2e-ipp-b-be7e2e-785d7fd4ff-sq6wf\" (UID: \"8e02990c-0339-49b0-af86-17211a267bbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-be7e2e-785d7fd4ff-sq6wf"
Nov 01 14:47:06.544561 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:06.544520    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgjpl\" (UniqueName: \"kubernetes.io/projected/8e02990c-0339-49b0-af86-17211a267bbc-kube-api-access-pgjpl\") pod \"maas-api-e2e-ipp-b-be7e2e-785d7fd4ff-sq6wf\" (UID: \"8e02990c-0339-49b0-af86-17211a267bbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-be7e2e-785d7fd4ff-sq6wf"
Nov 01 14:47:06.544809 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:06.544787    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e02990c-0339-49b0-af86-17211a267bbc-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-be7e2e-785d7fd4ff-sq6wf\" (UID: \"8e02990c-0339-49b0-af86-17211a267bbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-be7e2e-785d7fd4ff-sq6wf"
Nov 01 14:47:06.544968 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:47:06.544370    4993 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-be7e2e-serving-cert: secret "maas-api-e2e-ipp-b-be7e2e-serving-cert" not found
Nov 01 14:47:06.545207 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:47:06.545154    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e02990c-0339-49b0-af86-17211a267bbc-maas-api-tls podName:8e02990c-0339-49b0-af86-17211a267bbc nodeName:}" failed. No retries permitted until 2026-11-01 14:47:07.045132147 +0000 UTC m=+1814.374994881 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8e02990c-0339-49b0-af86-17211a267bbc-maas-api-tls") pod "maas-api-e2e-ipp-b-be7e2e-785d7fd4ff-sq6wf" (UID: "8e02990c-0339-49b0-af86-17211a267bbc") : secret "maas-api-e2e-ipp-b-be7e2e-serving-cert" not found
Nov 01 14:47:06.545333 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:06.545007    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e02990c-0339-49b0-af86-17211a267bbc-metrics-certs\") pod \"maas-api-e2e-ipp-b-be7e2e-785d7fd4ff-sq6wf\" (UID: \"8e02990c-0339-49b0-af86-17211a267bbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-be7e2e-785d7fd4ff-sq6wf"
Nov 01 14:47:06.546005 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:06.545977    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e02990c-0339-49b0-af86-17211a267bbc-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-be7e2e-785d7fd4ff-sq6wf\" (UID: \"8e02990c-0339-49b0-af86-17211a267bbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-be7e2e-785d7fd4ff-sq6wf"
Nov 01 14:47:06.569666 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:06.568715    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e02990c-0339-49b0-af86-17211a267bbc-metrics-certs\") pod \"maas-api-e2e-ipp-b-be7e2e-785d7fd4ff-sq6wf\" (UID: \"8e02990c-0339-49b0-af86-17211a267bbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-be7e2e-785d7fd4ff-sq6wf"
Nov 01 14:47:06.615428 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:06.615394    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgjpl\" (UniqueName: \"kubernetes.io/projected/8e02990c-0339-49b0-af86-17211a267bbc-kube-api-access-pgjpl\") pod \"maas-api-e2e-ipp-b-be7e2e-785d7fd4ff-sq6wf\" (UID: \"8e02990c-0339-49b0-af86-17211a267bbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-be7e2e-785d7fd4ff-sq6wf"
Nov 01 14:47:07.051597 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:07.051562    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e02990c-0339-49b0-af86-17211a267bbc-maas-api-tls\") pod \"maas-api-e2e-ipp-b-be7e2e-785d7fd4ff-sq6wf\" (UID: \"8e02990c-0339-49b0-af86-17211a267bbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-be7e2e-785d7fd4ff-sq6wf"
Nov 01 14:47:07.054244 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:07.054215    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e02990c-0339-49b0-af86-17211a267bbc-maas-api-tls\") pod \"maas-api-e2e-ipp-b-be7e2e-785d7fd4ff-sq6wf\" (UID: \"8e02990c-0339-49b0-af86-17211a267bbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-be7e2e-785d7fd4ff-sq6wf"
Nov 01 14:47:07.250862 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:07.250826    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-be7e2e-785d7fd4ff-sq6wf"
Nov 01 14:47:07.628162 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:07.628078    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-be7e2e-785d7fd4ff-sq6wf"]
Nov 01 14:47:07.633575 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:47:07.633552    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e02990c_0339_49b0_af86_17211a267bbc.slice/crio-fb61c1a865fce6e5be8774307a3648cfc9aec46e69ec7d209933e916309dc5c0 WatchSource:0}: Error finding container fb61c1a865fce6e5be8774307a3648cfc9aec46e69ec7d209933e916309dc5c0: Status 404 returned error can't find the container with id fb61c1a865fce6e5be8774307a3648cfc9aec46e69ec7d209933e916309dc5c0
Nov 01 14:47:08.556922 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:08.556884    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-be7e2e-785d7fd4ff-sq6wf" event={"ID":"8e02990c-0339-49b0-af86-17211a267bbc","Type":"ContainerStarted","Data":"fb61c1a865fce6e5be8774307a3648cfc9aec46e69ec7d209933e916309dc5c0"}
Nov 01 14:47:09.565586 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:09.565543    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-be7e2e-785d7fd4ff-sq6wf" event={"ID":"8e02990c-0339-49b0-af86-17211a267bbc","Type":"ContainerStarted","Data":"677cdf51b97afd75c8c15b13afdbaa81d30e7c7ad6470750843c97a78a20a004"}
Nov 01 14:47:09.566004 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:09.565642    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-be7e2e-785d7fd4ff-sq6wf"
Nov 01 14:47:09.674921 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:09.674865    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-be7e2e-785d7fd4ff-sq6wf" podStartSLOduration=3.644118542 podStartE2EDuration="4.674848422s" totalImagesPullingTime="1.03072988s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:47:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:47:09.673360274 +0000 UTC m=+1817.003223011" watchObservedRunningTime="2026-11-01 14:47:09.674848422 +0000 UTC m=+1817.004711162"
Nov 01 14:47:14.461422 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:14.461384    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbac83-76b7b8d447-5chmg"
Nov 01 14:47:15.579111 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:15.579080    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-be7e2e-785d7fd4ff-sq6wf"
Nov 01 14:47:27.501305 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:47:27.501269    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:47:30.542450 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:30.542402    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-34d384-5bbf97d5bb-qks27"]
Nov 01 14:47:30.546557 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:30.546534    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-34d384-5bbf97d5bb-qks27"
Nov 01 14:47:30.594944 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:30.594908    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-34d384-serving-cert\""
Nov 01 14:47:30.645053 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:30.644963    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-34d384-5bbf97d5bb-qks27"]
Nov 01 14:47:30.645272 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:30.645124    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfb14cd9-70b5-442d-a13e-549973735eed-maas-api-tls\") pod \"maas-api-e2e-ipp-a-34d384-5bbf97d5bb-qks27\" (UID: \"bfb14cd9-70b5-442d-a13e-549973735eed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-34d384-5bbf97d5bb-qks27"
Nov 01 14:47:30.645272 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:30.645216    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfb14cd9-70b5-442d-a13e-549973735eed-metrics-certs\") pod \"maas-api-e2e-ipp-a-34d384-5bbf97d5bb-qks27\" (UID: \"bfb14cd9-70b5-442d-a13e-549973735eed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-34d384-5bbf97d5bb-qks27"
Nov 01 14:47:30.645414 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:30.645312    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shfc7\" (UniqueName: \"kubernetes.io/projected/bfb14cd9-70b5-442d-a13e-549973735eed-kube-api-access-shfc7\") pod \"maas-api-e2e-ipp-a-34d384-5bbf97d5bb-qks27\" (UID: \"bfb14cd9-70b5-442d-a13e-549973735eed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-34d384-5bbf97d5bb-qks27"
Nov 01 14:47:30.645477 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:30.645416    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfb14cd9-70b5-442d-a13e-549973735eed-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-34d384-5bbf97d5bb-qks27\" (UID: \"bfb14cd9-70b5-442d-a13e-549973735eed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-34d384-5bbf97d5bb-qks27"
Nov 01 14:47:30.746880 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:30.746842    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfb14cd9-70b5-442d-a13e-549973735eed-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-34d384-5bbf97d5bb-qks27\" (UID: \"bfb14cd9-70b5-442d-a13e-549973735eed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-34d384-5bbf97d5bb-qks27"
Nov 01 14:47:30.747058 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:30.746945    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfb14cd9-70b5-442d-a13e-549973735eed-maas-api-tls\") pod \"maas-api-e2e-ipp-a-34d384-5bbf97d5bb-qks27\" (UID: \"bfb14cd9-70b5-442d-a13e-549973735eed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-34d384-5bbf97d5bb-qks27"
Nov 01 14:47:30.747058 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:30.746971    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfb14cd9-70b5-442d-a13e-549973735eed-metrics-certs\") pod \"maas-api-e2e-ipp-a-34d384-5bbf97d5bb-qks27\" (UID: \"bfb14cd9-70b5-442d-a13e-549973735eed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-34d384-5bbf97d5bb-qks27"
Nov 01 14:47:30.747058 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:30.747036    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shfc7\" (UniqueName: \"kubernetes.io/projected/bfb14cd9-70b5-442d-a13e-549973735eed-kube-api-access-shfc7\") pod \"maas-api-e2e-ipp-a-34d384-5bbf97d5bb-qks27\" (UID: \"bfb14cd9-70b5-442d-a13e-549973735eed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-34d384-5bbf97d5bb-qks27"
Nov 01 14:47:30.747207 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:47:30.747054    4993 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-34d384-serving-cert: secret "maas-api-e2e-ipp-a-34d384-serving-cert" not found
Nov 01 14:47:30.747207 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:47:30.747116    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfb14cd9-70b5-442d-a13e-549973735eed-maas-api-tls podName:bfb14cd9-70b5-442d-a13e-549973735eed nodeName:}" failed. No retries permitted until 2026-11-01 14:47:31.247100634 +0000 UTC m=+1838.576963349 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bfb14cd9-70b5-442d-a13e-549973735eed-maas-api-tls") pod "maas-api-e2e-ipp-a-34d384-5bbf97d5bb-qks27" (UID: "bfb14cd9-70b5-442d-a13e-549973735eed") : secret "maas-api-e2e-ipp-a-34d384-serving-cert" not found
Nov 01 14:47:30.747645 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:30.747626    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfb14cd9-70b5-442d-a13e-549973735eed-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-34d384-5bbf97d5bb-qks27\" (UID: \"bfb14cd9-70b5-442d-a13e-549973735eed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-34d384-5bbf97d5bb-qks27"
Nov 01 14:47:30.749484 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:30.749460    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfb14cd9-70b5-442d-a13e-549973735eed-metrics-certs\") pod \"maas-api-e2e-ipp-a-34d384-5bbf97d5bb-qks27\" (UID: \"bfb14cd9-70b5-442d-a13e-549973735eed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-34d384-5bbf97d5bb-qks27"
Nov 01 14:47:30.804377 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:30.804291    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shfc7\" (UniqueName: \"kubernetes.io/projected/bfb14cd9-70b5-442d-a13e-549973735eed-kube-api-access-shfc7\") pod \"maas-api-e2e-ipp-a-34d384-5bbf97d5bb-qks27\" (UID: \"bfb14cd9-70b5-442d-a13e-549973735eed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-34d384-5bbf97d5bb-qks27"
Nov 01 14:47:31.260201 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:31.260129    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfb14cd9-70b5-442d-a13e-549973735eed-maas-api-tls\") pod \"maas-api-e2e-ipp-a-34d384-5bbf97d5bb-qks27\" (UID: \"bfb14cd9-70b5-442d-a13e-549973735eed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-34d384-5bbf97d5bb-qks27"
Nov 01 14:47:31.262839 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:31.262803    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfb14cd9-70b5-442d-a13e-549973735eed-maas-api-tls\") pod \"maas-api-e2e-ipp-a-34d384-5bbf97d5bb-qks27\" (UID: \"bfb14cd9-70b5-442d-a13e-549973735eed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-34d384-5bbf97d5bb-qks27"
Nov 01 14:47:31.462040 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:31.461998    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-34d384-5bbf97d5bb-qks27"
Nov 01 14:47:31.838967 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:31.838884    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-34d384-5bbf97d5bb-qks27"]
Nov 01 14:47:31.844870 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:47:31.844840    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfb14cd9_70b5_442d_a13e_549973735eed.slice/crio-29fb81111354dd095cd363981cf4e979785e9f6c15e5b4df934fc793d6c41602 WatchSource:0}: Error finding container 29fb81111354dd095cd363981cf4e979785e9f6c15e5b4df934fc793d6c41602: Status 404 returned error can't find the container with id 29fb81111354dd095cd363981cf4e979785e9f6c15e5b4df934fc793d6c41602
Nov 01 14:47:32.735702 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:32.735664    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-34d384-5bbf97d5bb-qks27" event={"ID":"bfb14cd9-70b5-442d-a13e-549973735eed","Type":"ContainerStarted","Data":"29fb81111354dd095cd363981cf4e979785e9f6c15e5b4df934fc793d6c41602"}
Nov 01 14:47:33.743818 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:33.743781    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-34d384-5bbf97d5bb-qks27" event={"ID":"bfb14cd9-70b5-442d-a13e-549973735eed","Type":"ContainerStarted","Data":"33e4afabeab0e5c9de7e9fe7387f50914b9e1b32b4ca61b40c970e9f87d0396b"}
Nov 01 14:47:33.744300 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:33.743863    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-34d384-5bbf97d5bb-qks27"
Nov 01 14:47:33.851014 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:33.850948    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-34d384-5bbf97d5bb-qks27" podStartSLOduration=2.9946469430000002 podStartE2EDuration="3.850931497s" totalImagesPullingTime="856.284554ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:47:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:47:33.850499056 +0000 UTC m=+1841.180361785" watchObservedRunningTime="2026-11-01 14:47:33.850931497 +0000 UTC m=+1841.180794236"
Nov 01 14:47:39.766188 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:47:39.765293    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-34d384-5bbf97d5bb-qks27"
Nov 01 14:47:57.500840 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:47:57.500804    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:48:06.656266 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:48:06.656227    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-bhsml"]
Nov 01 14:48:06.660536 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:48:06.660515    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-bhsml"
Nov 01 14:48:06.685003 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:48:06.684978    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9ffa582-9776-4ee9-86fa-b3039e14d4e1-workload-certs\") pod \"e2e-ipp-a-34d384-openshift-default-5d9f84bc87-bhsml\" (UID: \"d9ffa582-9776-4ee9-86fa-b3039e14d4e1\") " pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-bhsml"
Nov 01 14:48:06.685136 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:48:06.685009    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9ffa582-9776-4ee9-86fa-b3039e14d4e1-istio-podinfo\") pod \"e2e-ipp-a-34d384-openshift-default-5d9f84bc87-bhsml\" (UID: \"d9ffa582-9776-4ee9-86fa-b3039e14d4e1\") " pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-bhsml"
Nov 01 14:48:06.685136 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:48:06.685037    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9ffa582-9776-4ee9-86fa-b3039e14d4e1-credential-socket\") pod \"e2e-ipp-a-34d384-openshift-default-5d9f84bc87-bhsml\" (UID: \"d9ffa582-9776-4ee9-86fa-b3039e14d4e1\") " pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-bhsml"
Nov 01 14:48:06.685136 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:48:06.685104    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9ffa582-9776-4ee9-86fa-b3039e14d4e1-istio-envoy\") pod \"e2e-ipp-a-34d384-openshift-default-5d9f84bc87-bhsml\" (UID: \"d9ffa582-9776-4ee9-86fa-b3039e14d4e1\") " pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-bhsml"
Nov 01 14:48:06.685295 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:48:06.685163    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9ffa582-9776-4ee9-86fa-b3039e14d4e1-workload-socket\") pod \"e2e-ipp-a-34d384-openshift-default-5d9f84bc87-bhsml\" (UID: \"d9ffa582-9776-4ee9-86fa-b3039e14d4e1\") " pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-bhsml"
Nov 01 14:48:06.685295 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:48:06.685258    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9ffa582-9776-4ee9-86fa-b3039e14d4e1-istio-token\") pod \"e2e-ipp-a-34d384-openshift-default-5d9f84bc87-bhsml\" (UID: \"d9ffa582-9776-4ee9-86fa-b3039e14d4e1\") " pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-bhsml"
Nov 01 14:48:06.685295 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:48:06.685276    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9ffa582-9776-4ee9-86fa-b3039e14d4e1-istiod-ca-cert\") pod \"e2e-ipp-a-34d384-openshift-default-5d9f84bc87-bhsml\" (UID: \"d9ffa582-9776-4ee9-86fa-b3039e14d4e1\") " pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-bhsml"
Nov 01 14:48:06.685295 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:48:06.685291    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xs24\" (UniqueName: \"kubernetes.io/projected/d9ffa582-9776-4ee9-86fa-b3039e14d4e1-kube-api-access-6xs24\") pod \"e2e-ipp-a-34d384-openshift-default-5d9f84bc87-bhsml\" (UID: \"d9ffa582-9776-4ee9-86fa-b3039e14d4e1\") " pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-bhsml"
Nov 01 14:48:06.685521 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:48:06.685503    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9ffa582-9776-4ee9-86fa-b3039e14d4e1-istio-data\") pod \"e2e-ipp-a-34d384-openshift-default-5d9f84bc87-bhsml\" (UID: \"d9ffa582-9776-4ee9-86fa-b3039e14d4e1\") " pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-bhsml"
Nov 01 14:48:06.708619 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:48:06.708593    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-34d384-openshift-default-dockercfg-7trgk\""
Nov 01 14:48:06.761058 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:48:06.761019    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-bhsml"]
Nov 01 14:48:06.787070 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:48:06.787033    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9ffa582-9776-4ee9-86fa-b3039e14d4e1-istio-data\") pod \"e2e-ipp-a-34d384-openshift-default-5d9f84bc87-bhsml\" (UID: \"d9ffa582-9776-4ee9-86fa-b3039e14d4e1\") " pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-bhsml"
Nov 01 14:48:06.790346 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:48:06.787866    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9ffa582-9776-4ee9-86fa-b3039e14d4e1-istio-data\") pod \"e2e-ipp-a-34d384-openshift-default-5d9f84bc87-bhsml\" (UID: \"d9ffa582-9776-4ee9-86fa-b3039e14d4e1\") " pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-bhsml"
Nov 01 14:48:06.790618 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:48:06.790315    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9ffa582-9776-4ee9-86fa-b3039e14d4e1-workload-certs\") pod \"e2e-ipp-a-34d384-openshift-default-5d9f84bc87-bhsml\" (UID: \"d9ffa582-9776-4ee9-86fa-b3039e14d4e1\") " pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-bhsml"
Nov 01 14:48:06.790618 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:48:06.790609    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9ffa582-9776-4ee9-86fa-b3039e14d4e1-workload-certs\") pod \"e2e-ipp-a-34d384-openshift-default-5d9f84bc87-bhsml\" (UID: \"d9ffa582-9776-4ee9-86fa-b3039e14d4e1\") " pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-bhsml"
Nov 01 14:48:06.790784 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:48:06.790651    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9ffa582-9776-4ee9-86fa-b3039e14d4e1-istio-podinfo\") pod \"e2e-ipp-a-34d384-openshift-default-5d9f84bc87-bhsml\" (UID: \"d9ffa582-9776-4ee9-86fa-b3039e14d4e1\") " pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-bhsml"
Nov 01 14:48:06.790784 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:48:06.790705    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9ffa582-9776-4ee9-86fa-b3039e14d4e1-credential-socket\") pod \"e2e-ipp-a-34d384-openshift-default-5d9f84bc87-bhsml\" (UID: \"d9ffa582-9776-4ee9-86fa-b3039e14d4e1\") " pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-bhsml"
Nov 01 14:48:06.790784 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:48:06.790732    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9ffa582-9776-4ee9-86fa-b3039e14d4e1-istio-envoy\") pod \"e2e-ipp-a-34d384-openshift-default-5d9f84bc87-bhsml\" (UID: \"d9ffa582-9776-4ee9-86fa-b3039e14d4e1\") " pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-bhsml"
Nov 01 14:48:06.790784 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:48:06.790780    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9ffa582-9776-4ee9-86fa-b3039e14d4e1-workload-socket\") pod \"e2e-ipp-a-34d384-openshift-default-5d9f84bc87-bhsml\" (UID: \"d9ffa582-9776-4ee9-86fa-b3039e14d4e1\") " pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-bhsml"
Nov 01 14:48:06.790992 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:48:06.790895    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9ffa582-9776-4ee9-86fa-b3039e14d4e1-istio-token\") pod \"e2e-ipp-a-34d384-openshift-default-5d9f84bc87-bhsml\" (UID: \"d9ffa582-9776-4ee9-86fa-b3039e14d4e1\") " pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-bhsml"
Nov 01 14:48:06.790992 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:48:06.790920    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9ffa582-9776-4ee9-86fa-b3039e14d4e1-istiod-ca-cert\") pod \"e2e-ipp-a-34d384-openshift-default-5d9f84bc87-bhsml\" (UID: \"d9ffa582-9776-4ee9-86fa-b3039e14d4e1\") " pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-bhsml"
Nov 01 14:48:06.790992 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:48:06.790946    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xs24\" (UniqueName: \"kubernetes.io/projected/d9ffa582-9776-4ee9-86fa-b3039e14d4e1-kube-api-access-6xs24\") pod \"e2e-ipp-a-34d384-openshift-default-5d9f84bc87-bhsml\" (UID: \"d9ffa582-9776-4ee9-86fa-b3039e14d4e1\") " pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-bhsml"
Nov 01 14:48:06.792011 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:48:06.791286    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9ffa582-9776-4ee9-86fa-b3039e14d4e1-workload-socket\") pod \"e2e-ipp-a-34d384-openshift-default-5d9f84bc87-bhsml\" (UID: \"d9ffa582-9776-4ee9-86fa-b3039e14d4e1\") " pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-bhsml"
Nov 01 14:48:06.792011 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:48:06.791547    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9ffa582-9776-4ee9-86fa-b3039e14d4e1-credential-socket\") pod \"e2e-ipp-a-34d384-openshift-default-5d9f84bc87-bhsml\" (UID: \"d9ffa582-9776-4ee9-86fa-b3039e14d4e1\") " pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-bhsml"
Nov 01 14:48:06.792011 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:48:06.791970    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9ffa582-9776-4ee9-86fa-b3039e14d4e1-istiod-ca-cert\") pod \"e2e-ipp-a-34d384-openshift-default-5d9f84bc87-bhsml\" (UID: \"d9ffa582-9776-4ee9-86fa-b3039e14d4e1\") " pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-bhsml"
Nov 01 14:48:06.795315 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:48:06.795288    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9ffa582-9776-4ee9-86fa-b3039e14d4e1-istio-envoy\") pod \"e2e-ipp-a-34d384-openshift-default-5d9f84bc87-bhsml\" (UID: \"d9ffa582-9776-4ee9-86fa-b3039e14d4e1\") " pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-bhsml"
Nov 01 14:48:06.798966 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:48:06.798919    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9ffa582-9776-4ee9-86fa-b3039e14d4e1-istio-podinfo\") pod \"e2e-ipp-a-34d384-openshift-default-5d9f84bc87-bhsml\" (UID: \"d9ffa582-9776-4ee9-86fa-b3039e14d4e1\") " pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-bhsml"
Nov 01 14:48:06.853997 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:48:06.853950    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xs24\" (UniqueName: \"kubernetes.io/projected/d9ffa582-9776-4ee9-86fa-b3039e14d4e1-kube-api-access-6xs24\") pod \"e2e-ipp-a-34d384-openshift-default-5d9f84bc87-bhsml\" (UID: \"d9ffa582-9776-4ee9-86fa-b3039e14d4e1\") " pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-bhsml"
Nov 01 14:48:06.854186 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:48:06.854084    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9ffa582-9776-4ee9-86fa-b3039e14d4e1-istio-token\") pod \"e2e-ipp-a-34d384-openshift-default-5d9f84bc87-bhsml\" (UID: \"d9ffa582-9776-4ee9-86fa-b3039e14d4e1\") " pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-bhsml"
Nov 01 14:48:06.974062 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:48:06.973973    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-bhsml"
Nov 01 14:48:07.400103 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:48:07.400064    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-bhsml"]
Nov 01 14:48:07.406465 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:48:07.406436    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9ffa582_9776_4ee9_86fa_b3039e14d4e1.slice/crio-a6952e8b906eeeda9f4be18ef7bd446e8aa8faa85918ce6a36d4ffb3181f7652 WatchSource:0}: Error finding container a6952e8b906eeeda9f4be18ef7bd446e8aa8faa85918ce6a36d4ffb3181f7652: Status 404 returned error can't find the container with id a6952e8b906eeeda9f4be18ef7bd446e8aa8faa85918ce6a36d4ffb3181f7652
Nov 01 14:48:07.408636 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:48:07.408601    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 14:48:07.408712 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:48:07.408672    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 14:48:07.408712 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:48:07.408701    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 14:48:08.074194 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:48:08.073489    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-bhsml" event={"ID":"d9ffa582-9776-4ee9-86fa-b3039e14d4e1","Type":"ContainerStarted","Data":"2c8899ae0059724a1c43576c8784bfc16bff3a2ee1daab15f7c9c50e62d401ba"}
Nov 01 14:48:08.074194 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:48:08.073569    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-bhsml" event={"ID":"d9ffa582-9776-4ee9-86fa-b3039e14d4e1","Type":"ContainerStarted","Data":"a6952e8b906eeeda9f4be18ef7bd446e8aa8faa85918ce6a36d4ffb3181f7652"}
Nov 01 14:48:08.236826 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:48:08.236773    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-bhsml" podStartSLOduration=2.236760097 podStartE2EDuration="2.236760097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:48:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:48:08.233799536 +0000 UTC m=+1875.563662284" watchObservedRunningTime="2026-11-01 14:48:08.236760097 +0000 UTC m=+1875.566622833"
Nov 01 14:48:08.975988 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:48:08.974494    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-bhsml"
Nov 01 14:48:08.978451 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:48:08.978426    4993 patch_prober.go:29] interesting pod/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-bhsml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Nov 01 14:48:08.978655 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:48:08.978626    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-bhsml" podUID="d9ffa582-9776-4ee9-86fa-b3039e14d4e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Nov 01 14:48:09.974626 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:48:09.974573    4993 patch_prober.go:29] interesting pod/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-bhsml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Nov 01 14:48:09.975303 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:48:09.975258    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-bhsml" podUID="d9ffa582-9776-4ee9-86fa-b3039e14d4e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Nov 01 14:48:10.976562 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:48:10.976523    4993 patch_prober.go:29] interesting pod/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-bhsml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Nov 01 14:48:10.976961 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:48:10.976598    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-bhsml" podUID="d9ffa582-9776-4ee9-86fa-b3039e14d4e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Nov 01 14:48:11.978669 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:48:11.978628    4993 patch_prober.go:29] interesting pod/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-bhsml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Nov 01 14:48:11.979074 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:48:11.978707    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-bhsml" podUID="d9ffa582-9776-4ee9-86fa-b3039e14d4e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Nov 01 14:48:12.974657 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:48:12.974614    4993 patch_prober.go:29] interesting pod/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-bhsml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Nov 01 14:48:12.974970 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:48:12.974935    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-bhsml" podUID="d9ffa582-9776-4ee9-86fa-b3039e14d4e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Nov 01 14:48:13.979066 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:48:13.979026    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-bhsml"
Nov 01 14:48:13.979495 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:48:13.979458    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-bhsml"
Nov 01 14:48:13.980039 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:48:13.980019    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-bhsml"
Nov 01 14:48:27.501869 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:48:27.501824    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:48:57.577076 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:48:57.577035    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:48:57.940877 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:48:57.940797    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cbac83-76b7b8d447-5chmg"]
Nov 01 14:48:57.941034 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:48:57.941003    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbac83-76b7b8d447-5chmg" podUID="aa7cf0a9-06d8-43fc-a81e-bc95b7f1bc80" containerName="payload-pre-processing" containerID="cri-o://dc8cf4c92c88fc3a7871cab58ac5d5ac6c64cc03b0345975a0383032909818d9" gracePeriod=30
Nov 01 14:48:58.225490 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:48:58.225463    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbac83-76b7b8d447-5chmg"
Nov 01 14:48:58.235772 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:48:58.235744    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa7cf0a9-06d8-43fc-a81e-bc95b7f1bc80-kube-api-access-bxh4t\" (UniqueName: \"kubernetes.io/projected/aa7cf0a9-06d8-43fc-a81e-bc95b7f1bc80-kube-api-access-bxh4t\") pod \"aa7cf0a9-06d8-43fc-a81e-bc95b7f1bc80\" (UID: \"aa7cf0a9-06d8-43fc-a81e-bc95b7f1bc80\") "
Nov 01 14:48:58.235895 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:48:58.235879    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa7cf0a9-06d8-43fc-a81e-bc95b7f1bc80-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa7cf0a9-06d8-43fc-a81e-bc95b7f1bc80-plugins-config-volume\") pod \"aa7cf0a9-06d8-43fc-a81e-bc95b7f1bc80\" (UID: \"aa7cf0a9-06d8-43fc-a81e-bc95b7f1bc80\") "
Nov 01 14:48:58.236307 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:48:58.236279    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa7cf0a9-06d8-43fc-a81e-bc95b7f1bc80-plugins-config-volume" pod "aa7cf0a9-06d8-43fc-a81e-bc95b7f1bc80" (UID: "aa7cf0a9-06d8-43fc-a81e-bc95b7f1bc80"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:48:58.236404 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:48:58.236393    4993 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa7cf0a9-06d8-43fc-a81e-bc95b7f1bc80-plugins-config-volume\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:48:58.237858 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:48:58.237824    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa7cf0a9-06d8-43fc-a81e-bc95b7f1bc80-kube-api-access-bxh4t" pod "aa7cf0a9-06d8-43fc-a81e-bc95b7f1bc80" (UID: "aa7cf0a9-06d8-43fc-a81e-bc95b7f1bc80"). InnerVolumeSpecName "kube-api-access-bxh4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:48:58.337947 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:48:58.337913    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxh4t\" (UniqueName: \"kubernetes.io/projected/aa7cf0a9-06d8-43fc-a81e-bc95b7f1bc80-kube-api-access-bxh4t\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:48:58.508814 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:48:58.508780    4993 generic.go:356] "Generic (PLEG): container finished" podID="aa7cf0a9-06d8-43fc-a81e-bc95b7f1bc80" containerID="dc8cf4c92c88fc3a7871cab58ac5d5ac6c64cc03b0345975a0383032909818d9" exitCode=0
Nov 01 14:48:58.509016 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:48:58.508919    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbac83-76b7b8d447-5chmg" event={"ID":"aa7cf0a9-06d8-43fc-a81e-bc95b7f1bc80","Type":"ContainerDied","Data":"dc8cf4c92c88fc3a7871cab58ac5d5ac6c64cc03b0345975a0383032909818d9"}
Nov 01 14:48:58.509016 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:48:58.508980    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbac83-76b7b8d447-5chmg" event={"ID":"aa7cf0a9-06d8-43fc-a81e-bc95b7f1bc80","Type":"ContainerDied","Data":"6ef77068c7f930140b4693122ce38f8c187bc1fbc2c49626d5d230b3f6c741b0"}
Nov 01 14:48:58.509016 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:48:58.509001    4993 scope.go:122] "RemoveContainer" containerID="dc8cf4c92c88fc3a7871cab58ac5d5ac6c64cc03b0345975a0383032909818d9"
Nov 01 14:48:58.509262 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:48:58.509121    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbac83-76b7b8d447-5chmg"
Nov 01 14:48:58.519708 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:48:58.519680    4993 scope.go:122] "RemoveContainer" containerID="dc8cf4c92c88fc3a7871cab58ac5d5ac6c64cc03b0345975a0383032909818d9"
Nov 01 14:48:58.520013 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:48:58.519990    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc8cf4c92c88fc3a7871cab58ac5d5ac6c64cc03b0345975a0383032909818d9\": container with ID starting with dc8cf4c92c88fc3a7871cab58ac5d5ac6c64cc03b0345975a0383032909818d9 not found: ID does not exist" containerID="dc8cf4c92c88fc3a7871cab58ac5d5ac6c64cc03b0345975a0383032909818d9"
Nov 01 14:48:58.520100 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:48:58.520021    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc8cf4c92c88fc3a7871cab58ac5d5ac6c64cc03b0345975a0383032909818d9"} err="failed to get container status \"dc8cf4c92c88fc3a7871cab58ac5d5ac6c64cc03b0345975a0383032909818d9\": rpc error: code = NotFound desc = could not find container \"dc8cf4c92c88fc3a7871cab58ac5d5ac6c64cc03b0345975a0383032909818d9\": container with ID starting with dc8cf4c92c88fc3a7871cab58ac5d5ac6c64cc03b0345975a0383032909818d9 not found: ID does not exist"
Nov 01 14:48:58.661976 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:48:58.661940    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cbac83-76b7b8d447-5chmg"]
Nov 01 14:48:58.665855 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:48:58.665830    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cbac83-76b7b8d447-5chmg"]
Nov 01 14:48:58.795265 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:48:58.795160    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa7cf0a9-06d8-43fc-a81e-bc95b7f1bc80" path="/var/lib/kubelet/pods/aa7cf0a9-06d8-43fc-a81e-bc95b7f1bc80/volumes"
Nov 01 14:49:27.501477 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:49:27.501391    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:49:36.410816 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:49:36.410778    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7"]
Nov 01 14:49:36.413272 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:49:36.410979    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7" podUID="30998c96-4380-4e46-89db-c2f328351f71" containerName="istio-proxy" containerID="cri-o://3395dd973dc9b24731aa32f565c12de9b3aaafdee4aa92189970da6cfa8d1d18" gracePeriod=30
Nov 01 14:49:41.697411 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:49:41.697383    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7"
Nov 01 14:49:41.737837 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:49:41.737800    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30998c96-4380-4e46-89db-c2f328351f71-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30998c96-4380-4e46-89db-c2f328351f71-istio-data\") pod \"30998c96-4380-4e46-89db-c2f328351f71\" (UID: \"30998c96-4380-4e46-89db-c2f328351f71\") "
Nov 01 14:49:41.738037 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:49:41.737865    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30998c96-4380-4e46-89db-c2f328351f71-istio-token\" (UniqueName: \"kubernetes.io/projected/30998c96-4380-4e46-89db-c2f328351f71-istio-token\") pod \"30998c96-4380-4e46-89db-c2f328351f71\" (UID: \"30998c96-4380-4e46-89db-c2f328351f71\") "
Nov 01 14:49:41.738037 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:49:41.738014    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30998c96-4380-4e46-89db-c2f328351f71-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30998c96-4380-4e46-89db-c2f328351f71-istio-envoy\") pod \"30998c96-4380-4e46-89db-c2f328351f71\" (UID: \"30998c96-4380-4e46-89db-c2f328351f71\") "
Nov 01 14:49:41.738204 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:49:41.738054    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30998c96-4380-4e46-89db-c2f328351f71-kube-api-access-wr4gj\" (UniqueName: \"kubernetes.io/projected/30998c96-4380-4e46-89db-c2f328351f71-kube-api-access-wr4gj\") pod \"30998c96-4380-4e46-89db-c2f328351f71\" (UID: \"30998c96-4380-4e46-89db-c2f328351f71\") "
Nov 01 14:49:41.738204 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:49:41.738098    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30998c96-4380-4e46-89db-c2f328351f71-istio-data" pod "30998c96-4380-4e46-89db-c2f328351f71" (UID: "30998c96-4380-4e46-89db-c2f328351f71"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:49:41.738324 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:49:41.738253    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30998c96-4380-4e46-89db-c2f328351f71-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30998c96-4380-4e46-89db-c2f328351f71-workload-socket\") pod \"30998c96-4380-4e46-89db-c2f328351f71\" (UID: \"30998c96-4380-4e46-89db-c2f328351f71\") "
Nov 01 14:49:41.738368 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:49:41.738317    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30998c96-4380-4e46-89db-c2f328351f71-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30998c96-4380-4e46-89db-c2f328351f71-workload-certs\") pod \"30998c96-4380-4e46-89db-c2f328351f71\" (UID: \"30998c96-4380-4e46-89db-c2f328351f71\") "
Nov 01 14:49:41.738368 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:49:41.738347    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/30998c96-4380-4e46-89db-c2f328351f71-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30998c96-4380-4e46-89db-c2f328351f71-istio-podinfo\") pod \"30998c96-4380-4e46-89db-c2f328351f71\" (UID: \"30998c96-4380-4e46-89db-c2f328351f71\") "
Nov 01 14:49:41.738459 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:49:41.738387    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30998c96-4380-4e46-89db-c2f328351f71-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30998c96-4380-4e46-89db-c2f328351f71-credential-socket\") pod \"30998c96-4380-4e46-89db-c2f328351f71\" (UID: \"30998c96-4380-4e46-89db-c2f328351f71\") "
Nov 01 14:49:41.738459 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:49:41.738410    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30998c96-4380-4e46-89db-c2f328351f71-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30998c96-4380-4e46-89db-c2f328351f71-istiod-ca-cert\") pod \"30998c96-4380-4e46-89db-c2f328351f71\" (UID: \"30998c96-4380-4e46-89db-c2f328351f71\") "
Nov 01 14:49:41.738570 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:49:41.738495    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30998c96-4380-4e46-89db-c2f328351f71-workload-socket" pod "30998c96-4380-4e46-89db-c2f328351f71" (UID: "30998c96-4380-4e46-89db-c2f328351f71"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:49:41.738694 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:49:41.738645    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30998c96-4380-4e46-89db-c2f328351f71-workload-certs" pod "30998c96-4380-4e46-89db-c2f328351f71" (UID: "30998c96-4380-4e46-89db-c2f328351f71"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:49:41.738777 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:49:41.738726    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30998c96-4380-4e46-89db-c2f328351f71-credential-socket" pod "30998c96-4380-4e46-89db-c2f328351f71" (UID: "30998c96-4380-4e46-89db-c2f328351f71"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:49:41.738911 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:49:41.738888    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30998c96-4380-4e46-89db-c2f328351f71-istiod-ca-cert" pod "30998c96-4380-4e46-89db-c2f328351f71" (UID: "30998c96-4380-4e46-89db-c2f328351f71"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:49:41.739057 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:49:41.739037    4993 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30998c96-4380-4e46-89db-c2f328351f71-workload-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:49:41.739104 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:49:41.739067    4993 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30998c96-4380-4e46-89db-c2f328351f71-credential-socket\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:49:41.739104 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:49:41.739085    4993 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30998c96-4380-4e46-89db-c2f328351f71-istiod-ca-cert\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:49:41.739104 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:49:41.739099    4993 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30998c96-4380-4e46-89db-c2f328351f71-istio-data\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:49:41.739230 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:49:41.739114    4993 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30998c96-4380-4e46-89db-c2f328351f71-workload-socket\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:49:41.740353 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:49:41.740331    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30998c96-4380-4e46-89db-c2f328351f71-istio-envoy" pod "30998c96-4380-4e46-89db-c2f328351f71" (UID: "30998c96-4380-4e46-89db-c2f328351f71"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:49:41.740454 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:49:41.740357    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30998c96-4380-4e46-89db-c2f328351f71-kube-api-access-wr4gj" pod "30998c96-4380-4e46-89db-c2f328351f71" (UID: "30998c96-4380-4e46-89db-c2f328351f71"). InnerVolumeSpecName "kube-api-access-wr4gj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:49:41.740820 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:49:41.740802    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30998c96-4380-4e46-89db-c2f328351f71-istio-token" pod "30998c96-4380-4e46-89db-c2f328351f71" (UID: "30998c96-4380-4e46-89db-c2f328351f71"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:49:41.740906 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:49:41.740804    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/30998c96-4380-4e46-89db-c2f328351f71-istio-podinfo" pod "30998c96-4380-4e46-89db-c2f328351f71" (UID: "30998c96-4380-4e46-89db-c2f328351f71"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 14:49:41.841390 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:49:41.841355    4993 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30998c96-4380-4e46-89db-c2f328351f71-istio-podinfo\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:49:41.841390 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:49:41.841393    4993 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30998c96-4380-4e46-89db-c2f328351f71-istio-token\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:49:41.841631 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:49:41.841409    4993 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30998c96-4380-4e46-89db-c2f328351f71-istio-envoy\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:49:41.841631 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:49:41.841434    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wr4gj\" (UniqueName: \"kubernetes.io/projected/30998c96-4380-4e46-89db-c2f328351f71-kube-api-access-wr4gj\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:49:41.888365 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:49:41.888333    4993 generic.go:356] "Generic (PLEG): container finished" podID="30998c96-4380-4e46-89db-c2f328351f71" containerID="3395dd973dc9b24731aa32f565c12de9b3aaafdee4aa92189970da6cfa8d1d18" exitCode=0
Nov 01 14:49:41.888543 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:49:41.888385    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7"
Nov 01 14:49:41.888543 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:49:41.888411    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7" event={"ID":"30998c96-4380-4e46-89db-c2f328351f71","Type":"ContainerDied","Data":"3395dd973dc9b24731aa32f565c12de9b3aaafdee4aa92189970da6cfa8d1d18"}
Nov 01 14:49:41.888543 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:49:41.888481    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7" event={"ID":"30998c96-4380-4e46-89db-c2f328351f71","Type":"ContainerDied","Data":"6f6ae2fe410a41d3ee0a79b8da82b65b609860615cb70986d636ac153f3e2cc5"}
Nov 01 14:49:41.888543 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:49:41.888510    4993 scope.go:122] "RemoveContainer" containerID="3395dd973dc9b24731aa32f565c12de9b3aaafdee4aa92189970da6cfa8d1d18"
Nov 01 14:49:41.899405 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:49:41.899383    4993 scope.go:122] "RemoveContainer" containerID="3395dd973dc9b24731aa32f565c12de9b3aaafdee4aa92189970da6cfa8d1d18"
Nov 01 14:49:41.899671 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:49:41.899652    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3395dd973dc9b24731aa32f565c12de9b3aaafdee4aa92189970da6cfa8d1d18\": container with ID starting with 3395dd973dc9b24731aa32f565c12de9b3aaafdee4aa92189970da6cfa8d1d18 not found: ID does not exist" containerID="3395dd973dc9b24731aa32f565c12de9b3aaafdee4aa92189970da6cfa8d1d18"
Nov 01 14:49:41.899722 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:49:41.899682    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3395dd973dc9b24731aa32f565c12de9b3aaafdee4aa92189970da6cfa8d1d18"} err="failed to get container status \"3395dd973dc9b24731aa32f565c12de9b3aaafdee4aa92189970da6cfa8d1d18\": rpc error: code = NotFound desc = could not find container \"3395dd973dc9b24731aa32f565c12de9b3aaafdee4aa92189970da6cfa8d1d18\": container with ID starting with 3395dd973dc9b24731aa32f565c12de9b3aaafdee4aa92189970da6cfa8d1d18 not found: ID does not exist"
Nov 01 14:49:42.042935 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:49:42.042865    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7"]
Nov 01 14:49:42.045948 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:49:42.045919    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4bxp7"]
Nov 01 14:49:42.795428 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:49:42.795393    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30998c96-4380-4e46-89db-c2f328351f71" path="/var/lib/kubelet/pods/30998c96-4380-4e46-89db-c2f328351f71/volumes"
Nov 01 14:49:57.501662 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:49:57.501626    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:50:22.956037 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:50:22.955988    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-71e39a-7bd7c6f974-mcvvx"]
Nov 01 14:50:22.956723 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:50:22.956274    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-71e39a-7bd7c6f974-mcvvx" podUID="1d99efc6-0d6e-4cb5-a65e-6ac9eae07ea9" containerName="maas-api" containerID="cri-o://609d4270e614008808a3b5239e98efd49dd09eb6bdda629d7ea77c18abbc855e" gracePeriod=30
Nov 01 14:50:23.242645 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:50:23.242598    4993 generic.go:356] "Generic (PLEG): container finished" podID="1d99efc6-0d6e-4cb5-a65e-6ac9eae07ea9" containerID="609d4270e614008808a3b5239e98efd49dd09eb6bdda629d7ea77c18abbc855e" exitCode=0
Nov 01 14:50:23.243040 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:50:23.243015    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-71e39a-7bd7c6f974-mcvvx" event={"ID":"1d99efc6-0d6e-4cb5-a65e-6ac9eae07ea9","Type":"ContainerDied","Data":"609d4270e614008808a3b5239e98efd49dd09eb6bdda629d7ea77c18abbc855e"}
Nov 01 14:50:23.307786 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:50:23.307758    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-71e39a-7bd7c6f974-mcvvx"
Nov 01 14:50:23.337185 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:50:23.336750    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d99efc6-0d6e-4cb5-a65e-6ac9eae07ea9-kube-api-access-xr9kk\" (UniqueName: \"kubernetes.io/projected/1d99efc6-0d6e-4cb5-a65e-6ac9eae07ea9-kube-api-access-xr9kk\") pod \"1d99efc6-0d6e-4cb5-a65e-6ac9eae07ea9\" (UID: \"1d99efc6-0d6e-4cb5-a65e-6ac9eae07ea9\") "
Nov 01 14:50:23.337185 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:50:23.336912    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d99efc6-0d6e-4cb5-a65e-6ac9eae07ea9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d99efc6-0d6e-4cb5-a65e-6ac9eae07ea9-metrics-certs\") pod \"1d99efc6-0d6e-4cb5-a65e-6ac9eae07ea9\" (UID: \"1d99efc6-0d6e-4cb5-a65e-6ac9eae07ea9\") "
Nov 01 14:50:23.337185 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:50:23.336973    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d99efc6-0d6e-4cb5-a65e-6ac9eae07ea9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d99efc6-0d6e-4cb5-a65e-6ac9eae07ea9-openshift-service-ca\") pod \"1d99efc6-0d6e-4cb5-a65e-6ac9eae07ea9\" (UID: \"1d99efc6-0d6e-4cb5-a65e-6ac9eae07ea9\") "
Nov 01 14:50:23.337185 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:50:23.337083    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d99efc6-0d6e-4cb5-a65e-6ac9eae07ea9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d99efc6-0d6e-4cb5-a65e-6ac9eae07ea9-maas-api-tls\") pod \"1d99efc6-0d6e-4cb5-a65e-6ac9eae07ea9\" (UID: \"1d99efc6-0d6e-4cb5-a65e-6ac9eae07ea9\") "
Nov 01 14:50:23.342958 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:50:23.342625    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d99efc6-0d6e-4cb5-a65e-6ac9eae07ea9-openshift-service-ca" pod "1d99efc6-0d6e-4cb5-a65e-6ac9eae07ea9" (UID: "1d99efc6-0d6e-4cb5-a65e-6ac9eae07ea9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:50:23.346257 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:50:23.346160    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d99efc6-0d6e-4cb5-a65e-6ac9eae07ea9-kube-api-access-xr9kk" pod "1d99efc6-0d6e-4cb5-a65e-6ac9eae07ea9" (UID: "1d99efc6-0d6e-4cb5-a65e-6ac9eae07ea9"). InnerVolumeSpecName "kube-api-access-xr9kk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:50:23.362342 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:50:23.362316    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d99efc6-0d6e-4cb5-a65e-6ac9eae07ea9-maas-api-tls" pod "1d99efc6-0d6e-4cb5-a65e-6ac9eae07ea9" (UID: "1d99efc6-0d6e-4cb5-a65e-6ac9eae07ea9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:50:23.366593 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:50:23.366296    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d99efc6-0d6e-4cb5-a65e-6ac9eae07ea9-metrics-certs" pod "1d99efc6-0d6e-4cb5-a65e-6ac9eae07ea9" (UID: "1d99efc6-0d6e-4cb5-a65e-6ac9eae07ea9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:50:23.439417 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:50:23.439381    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d99efc6-0d6e-4cb5-a65e-6ac9eae07ea9-maas-api-tls\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:50:23.439417 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:50:23.439413    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xr9kk\" (UniqueName: \"kubernetes.io/projected/1d99efc6-0d6e-4cb5-a65e-6ac9eae07ea9-kube-api-access-xr9kk\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:50:23.439417 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:50:23.439424    4993 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d99efc6-0d6e-4cb5-a65e-6ac9eae07ea9-metrics-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:50:23.439666 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:50:23.439433    4993 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d99efc6-0d6e-4cb5-a65e-6ac9eae07ea9-openshift-service-ca\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:50:24.250392 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:50:24.250363    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-71e39a-7bd7c6f974-mcvvx"
Nov 01 14:50:24.250831 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:50:24.250402    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-71e39a-7bd7c6f974-mcvvx" event={"ID":"1d99efc6-0d6e-4cb5-a65e-6ac9eae07ea9","Type":"ContainerDied","Data":"eed5bfd2cc4607733bc1ccb6a023a4de060d5a334ec4ddf7683c9a9d032aae14"}
Nov 01 14:50:24.250831 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:50:24.250469    4993 scope.go:122] "RemoveContainer" containerID="609d4270e614008808a3b5239e98efd49dd09eb6bdda629d7ea77c18abbc855e"
Nov 01 14:50:24.403504 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:50:24.403473    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-71e39a-7bd7c6f974-mcvvx"]
Nov 01 14:50:24.449959 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:50:24.449926    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-71e39a-7bd7c6f974-mcvvx"]
Nov 01 14:50:24.796754 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:50:24.796718    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d99efc6-0d6e-4cb5-a65e-6ac9eae07ea9" path="/var/lib/kubelet/pods/1d99efc6-0d6e-4cb5-a65e-6ac9eae07ea9/volumes"
Nov 01 14:50:27.503589 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:50:27.502947    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:50:51.795696 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:50:51.795647    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-be7e2e-785d7fd4ff-sq6wf"]
Nov 01 14:50:51.796341 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:50:51.795890    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-be7e2e-785d7fd4ff-sq6wf" podUID="8e02990c-0339-49b0-af86-17211a267bbc" containerName="maas-api" containerID="cri-o://677cdf51b97afd75c8c15b13afdbaa81d30e7c7ad6470750843c97a78a20a004" gracePeriod=30
Nov 01 14:50:52.162535 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:50:52.162498    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-be7e2e-785d7fd4ff-sq6wf"
Nov 01 14:50:52.311987 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:50:52.311901    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e02990c-0339-49b0-af86-17211a267bbc-kube-api-access-pgjpl\" (UniqueName: \"kubernetes.io/projected/8e02990c-0339-49b0-af86-17211a267bbc-kube-api-access-pgjpl\") pod \"8e02990c-0339-49b0-af86-17211a267bbc\" (UID: \"8e02990c-0339-49b0-af86-17211a267bbc\") "
Nov 01 14:50:52.312300 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:50:52.312275    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e02990c-0339-49b0-af86-17211a267bbc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e02990c-0339-49b0-af86-17211a267bbc-maas-api-tls\") pod \"8e02990c-0339-49b0-af86-17211a267bbc\" (UID: \"8e02990c-0339-49b0-af86-17211a267bbc\") "
Nov 01 14:50:52.312544 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:50:52.312527    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e02990c-0339-49b0-af86-17211a267bbc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e02990c-0339-49b0-af86-17211a267bbc-openshift-service-ca\") pod \"8e02990c-0339-49b0-af86-17211a267bbc\" (UID: \"8e02990c-0339-49b0-af86-17211a267bbc\") "
Nov 01 14:50:52.313109 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:50:52.312992    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e02990c-0339-49b0-af86-17211a267bbc-openshift-service-ca" pod "8e02990c-0339-49b0-af86-17211a267bbc" (UID: "8e02990c-0339-49b0-af86-17211a267bbc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:50:52.313328 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:50:52.313301    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e02990c-0339-49b0-af86-17211a267bbc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e02990c-0339-49b0-af86-17211a267bbc-metrics-certs\") pod \"8e02990c-0339-49b0-af86-17211a267bbc\" (UID: \"8e02990c-0339-49b0-af86-17211a267bbc\") "
Nov 01 14:50:52.313891 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:50:52.313869    4993 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e02990c-0339-49b0-af86-17211a267bbc-openshift-service-ca\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:50:52.317638 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:50:52.317614    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e02990c-0339-49b0-af86-17211a267bbc-kube-api-access-pgjpl" pod "8e02990c-0339-49b0-af86-17211a267bbc" (UID: "8e02990c-0339-49b0-af86-17211a267bbc"). InnerVolumeSpecName "kube-api-access-pgjpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:50:52.322363 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:50:52.322335    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e02990c-0339-49b0-af86-17211a267bbc-maas-api-tls" pod "8e02990c-0339-49b0-af86-17211a267bbc" (UID: "8e02990c-0339-49b0-af86-17211a267bbc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:50:52.337968 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:50:52.325867    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e02990c-0339-49b0-af86-17211a267bbc-metrics-certs" pod "8e02990c-0339-49b0-af86-17211a267bbc" (UID: "8e02990c-0339-49b0-af86-17211a267bbc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:50:52.415593 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:50:52.415504    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgjpl\" (UniqueName: \"kubernetes.io/projected/8e02990c-0339-49b0-af86-17211a267bbc-kube-api-access-pgjpl\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:50:52.415593 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:50:52.415544    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e02990c-0339-49b0-af86-17211a267bbc-maas-api-tls\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:50:52.415593 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:50:52.415561    4993 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e02990c-0339-49b0-af86-17211a267bbc-metrics-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:50:52.470298 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:50:52.470264    4993 generic.go:356] "Generic (PLEG): container finished" podID="8e02990c-0339-49b0-af86-17211a267bbc" containerID="677cdf51b97afd75c8c15b13afdbaa81d30e7c7ad6470750843c97a78a20a004" exitCode=0
Nov 01 14:50:52.470488 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:50:52.470334    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-be7e2e-785d7fd4ff-sq6wf"
Nov 01 14:50:52.470488 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:50:52.470359    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-be7e2e-785d7fd4ff-sq6wf" event={"ID":"8e02990c-0339-49b0-af86-17211a267bbc","Type":"ContainerDied","Data":"677cdf51b97afd75c8c15b13afdbaa81d30e7c7ad6470750843c97a78a20a004"}
Nov 01 14:50:52.470488 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:50:52.470424    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-be7e2e-785d7fd4ff-sq6wf" event={"ID":"8e02990c-0339-49b0-af86-17211a267bbc","Type":"ContainerDied","Data":"fb61c1a865fce6e5be8774307a3648cfc9aec46e69ec7d209933e916309dc5c0"}
Nov 01 14:50:52.470488 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:50:52.470462    4993 scope.go:122] "RemoveContainer" containerID="677cdf51b97afd75c8c15b13afdbaa81d30e7c7ad6470750843c97a78a20a004"
Nov 01 14:50:52.483415 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:50:52.483395    4993 scope.go:122] "RemoveContainer" containerID="677cdf51b97afd75c8c15b13afdbaa81d30e7c7ad6470750843c97a78a20a004"
Nov 01 14:50:52.483771 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:50:52.483751    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"677cdf51b97afd75c8c15b13afdbaa81d30e7c7ad6470750843c97a78a20a004\": container with ID starting with 677cdf51b97afd75c8c15b13afdbaa81d30e7c7ad6470750843c97a78a20a004 not found: ID does not exist" containerID="677cdf51b97afd75c8c15b13afdbaa81d30e7c7ad6470750843c97a78a20a004"
Nov 01 14:50:52.483849 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:50:52.483780    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"677cdf51b97afd75c8c15b13afdbaa81d30e7c7ad6470750843c97a78a20a004"} err="failed to get container status \"677cdf51b97afd75c8c15b13afdbaa81d30e7c7ad6470750843c97a78a20a004\": rpc error: code = NotFound desc = could not find container \"677cdf51b97afd75c8c15b13afdbaa81d30e7c7ad6470750843c97a78a20a004\": container with ID starting with 677cdf51b97afd75c8c15b13afdbaa81d30e7c7ad6470750843c97a78a20a004 not found: ID does not exist"
Nov 01 14:50:52.623289 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:50:52.623253    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-be7e2e-785d7fd4ff-sq6wf"]
Nov 01 14:50:52.626001 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:50:52.625972    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-be7e2e-785d7fd4ff-sq6wf"]
Nov 01 14:50:52.799287 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:50:52.799246    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e02990c-0339-49b0-af86-17211a267bbc" path="/var/lib/kubelet/pods/8e02990c-0339-49b0-af86-17211a267bbc/volumes"
Nov 01 14:50:57.500730 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:50:57.500691    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:51:27.501480 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:51:27.501441    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:51:48.797453 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:51:48.797417    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm"]
Nov 01 14:51:48.798073 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:51:48.797666    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm" podUID="6db204f9-b7a9-4b31-ab49-a4406d2b879d" containerName="istio-proxy" containerID="cri-o://1722a1329d811379e417698f11af469d4516c2c279a21625d24741ba481002db" gracePeriod=30
Nov 01 14:51:49.397488 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:51:49.397446    4993 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 14:51:49.397675 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:51:49.397510    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm" podUID="6db204f9-b7a9-4b31-ab49-a4406d2b879d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 14:51:53.085469 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:51:53.085436    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xmwtp_bb59051d-c2df-4362-a157-00f14f915e79/console-operator/3.log"
Nov 01 14:51:53.087105 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:51:53.087077    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xmwtp_bb59051d-c2df-4362-a157-00f14f915e79/console-operator/3.log"
Nov 01 14:51:53.947209 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:51:53.947154    4993 generic.go:356] "Generic (PLEG): container finished" podID="6db204f9-b7a9-4b31-ab49-a4406d2b879d" containerID="1722a1329d811379e417698f11af469d4516c2c279a21625d24741ba481002db" exitCode=0
Nov 01 14:51:53.947392 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:51:53.947218    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm" event={"ID":"6db204f9-b7a9-4b31-ab49-a4406d2b879d","Type":"ContainerDied","Data":"1722a1329d811379e417698f11af469d4516c2c279a21625d24741ba481002db"}
Nov 01 14:51:54.074280 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:51:54.074219    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm"
Nov 01 14:51:54.164062 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:51:54.164030    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6db204f9-b7a9-4b31-ab49-a4406d2b879d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db204f9-b7a9-4b31-ab49-a4406d2b879d-credential-socket\") pod \"6db204f9-b7a9-4b31-ab49-a4406d2b879d\" (UID: \"6db204f9-b7a9-4b31-ab49-a4406d2b879d\") "
Nov 01 14:51:54.164578 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:51:54.164147    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6db204f9-b7a9-4b31-ab49-a4406d2b879d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db204f9-b7a9-4b31-ab49-a4406d2b879d-workload-socket\") pod \"6db204f9-b7a9-4b31-ab49-a4406d2b879d\" (UID: \"6db204f9-b7a9-4b31-ab49-a4406d2b879d\") "
Nov 01 14:51:54.164578 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:51:54.164196    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6db204f9-b7a9-4b31-ab49-a4406d2b879d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6db204f9-b7a9-4b31-ab49-a4406d2b879d-istiod-ca-cert\") pod \"6db204f9-b7a9-4b31-ab49-a4406d2b879d\" (UID: \"6db204f9-b7a9-4b31-ab49-a4406d2b879d\") "
Nov 01 14:51:54.164578 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:51:54.164227    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6db204f9-b7a9-4b31-ab49-a4406d2b879d-kube-api-access-c7qgv\" (UniqueName: \"kubernetes.io/projected/6db204f9-b7a9-4b31-ab49-a4406d2b879d-kube-api-access-c7qgv\") pod \"6db204f9-b7a9-4b31-ab49-a4406d2b879d\" (UID: \"6db204f9-b7a9-4b31-ab49-a4406d2b879d\") "
Nov 01 14:51:54.164578 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:51:54.164281    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6db204f9-b7a9-4b31-ab49-a4406d2b879d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6db204f9-b7a9-4b31-ab49-a4406d2b879d-istio-podinfo\") pod \"6db204f9-b7a9-4b31-ab49-a4406d2b879d\" (UID: \"6db204f9-b7a9-4b31-ab49-a4406d2b879d\") "
Nov 01 14:51:54.164578 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:51:54.164393    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6db204f9-b7a9-4b31-ab49-a4406d2b879d-credential-socket" pod "6db204f9-b7a9-4b31-ab49-a4406d2b879d" (UID: "6db204f9-b7a9-4b31-ab49-a4406d2b879d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:51:54.164578 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:51:54.164403    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6db204f9-b7a9-4b31-ab49-a4406d2b879d-workload-socket" pod "6db204f9-b7a9-4b31-ab49-a4406d2b879d" (UID: "6db204f9-b7a9-4b31-ab49-a4406d2b879d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:51:54.164578 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:51:54.164447    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6db204f9-b7a9-4b31-ab49-a4406d2b879d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6db204f9-b7a9-4b31-ab49-a4406d2b879d-istio-data\") pod \"6db204f9-b7a9-4b31-ab49-a4406d2b879d\" (UID: \"6db204f9-b7a9-4b31-ab49-a4406d2b879d\") "
Nov 01 14:51:54.164578 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:51:54.164582    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6db204f9-b7a9-4b31-ab49-a4406d2b879d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6db204f9-b7a9-4b31-ab49-a4406d2b879d-istio-envoy\") pod \"6db204f9-b7a9-4b31-ab49-a4406d2b879d\" (UID: \"6db204f9-b7a9-4b31-ab49-a4406d2b879d\") "
Nov 01 14:51:54.165013 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:51:54.164626    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6db204f9-b7a9-4b31-ab49-a4406d2b879d-istiod-ca-cert" pod "6db204f9-b7a9-4b31-ab49-a4406d2b879d" (UID: "6db204f9-b7a9-4b31-ab49-a4406d2b879d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:51:54.165013 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:51:54.164682    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6db204f9-b7a9-4b31-ab49-a4406d2b879d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6db204f9-b7a9-4b31-ab49-a4406d2b879d-workload-certs\") pod \"6db204f9-b7a9-4b31-ab49-a4406d2b879d\" (UID: \"6db204f9-b7a9-4b31-ab49-a4406d2b879d\") "
Nov 01 14:51:54.165013 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:51:54.164711    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6db204f9-b7a9-4b31-ab49-a4406d2b879d-istio-token\" (UniqueName: \"kubernetes.io/projected/6db204f9-b7a9-4b31-ab49-a4406d2b879d-istio-token\") pod \"6db204f9-b7a9-4b31-ab49-a4406d2b879d\" (UID: \"6db204f9-b7a9-4b31-ab49-a4406d2b879d\") "
Nov 01 14:51:54.165013 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:51:54.164847    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6db204f9-b7a9-4b31-ab49-a4406d2b879d-istio-data" pod "6db204f9-b7a9-4b31-ab49-a4406d2b879d" (UID: "6db204f9-b7a9-4b31-ab49-a4406d2b879d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:51:54.165013 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:51:54.164952    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6db204f9-b7a9-4b31-ab49-a4406d2b879d-workload-certs" pod "6db204f9-b7a9-4b31-ab49-a4406d2b879d" (UID: "6db204f9-b7a9-4b31-ab49-a4406d2b879d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:51:54.165287 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:51:54.165262    4993 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6db204f9-b7a9-4b31-ab49-a4406d2b879d-workload-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:51:54.165348 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:51:54.165288    4993 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db204f9-b7a9-4b31-ab49-a4406d2b879d-credential-socket\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:51:54.165348 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:51:54.165303    4993 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db204f9-b7a9-4b31-ab49-a4406d2b879d-workload-socket\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:51:54.165348 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:51:54.165318    4993 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6db204f9-b7a9-4b31-ab49-a4406d2b879d-istiod-ca-cert\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:51:54.165348 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:51:54.165332    4993 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6db204f9-b7a9-4b31-ab49-a4406d2b879d-istio-data\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:51:54.167117 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:51:54.167092    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6db204f9-b7a9-4b31-ab49-a4406d2b879d-istio-envoy" pod "6db204f9-b7a9-4b31-ab49-a4406d2b879d" (UID: "6db204f9-b7a9-4b31-ab49-a4406d2b879d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:51:54.167257 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:51:54.167126    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6db204f9-b7a9-4b31-ab49-a4406d2b879d-istio-podinfo" pod "6db204f9-b7a9-4b31-ab49-a4406d2b879d" (UID: "6db204f9-b7a9-4b31-ab49-a4406d2b879d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 14:51:54.167346 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:51:54.167326    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6db204f9-b7a9-4b31-ab49-a4406d2b879d-istio-token" pod "6db204f9-b7a9-4b31-ab49-a4406d2b879d" (UID: "6db204f9-b7a9-4b31-ab49-a4406d2b879d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:51:54.167403 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:51:54.167371    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6db204f9-b7a9-4b31-ab49-a4406d2b879d-kube-api-access-c7qgv" pod "6db204f9-b7a9-4b31-ab49-a4406d2b879d" (UID: "6db204f9-b7a9-4b31-ab49-a4406d2b879d"). InnerVolumeSpecName "kube-api-access-c7qgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:51:54.266565 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:51:54.266520    4993 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6db204f9-b7a9-4b31-ab49-a4406d2b879d-istio-envoy\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:51:54.266565 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:51:54.266564    4993 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6db204f9-b7a9-4b31-ab49-a4406d2b879d-istio-token\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:51:54.266817 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:51:54.266637    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7qgv\" (UniqueName: \"kubernetes.io/projected/6db204f9-b7a9-4b31-ab49-a4406d2b879d-kube-api-access-c7qgv\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:51:54.266817 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:51:54.266657    4993 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6db204f9-b7a9-4b31-ab49-a4406d2b879d-istio-podinfo\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:51:54.954695 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:51:54.954663    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm"
Nov 01 14:51:54.954867 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:51:54.954661    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm" event={"ID":"6db204f9-b7a9-4b31-ab49-a4406d2b879d","Type":"ContainerDied","Data":"87744fe1032d0f5747e2e2315e014584cbf7361fecb51df6dc98fcf19b4e99a8"}
Nov 01 14:51:54.954867 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:51:54.954816    4993 scope.go:122] "RemoveContainer" containerID="1722a1329d811379e417698f11af469d4516c2c279a21625d24741ba481002db"
Nov 01 14:51:55.154078 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:51:55.154001    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm"]
Nov 01 14:51:55.199874 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:51:55.199844    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-lv6fm"]
Nov 01 14:51:56.797714 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:51:56.797674    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6db204f9-b7a9-4b31-ab49-a4406d2b879d" path="/var/lib/kubelet/pods/6db204f9-b7a9-4b31-ab49-a4406d2b879d/volumes"
Nov 01 14:51:57.504297 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:51:57.504262    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:51:59.730273 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:51:59.730236    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-e381b7-59fcfbb485-c7zvt"]
Nov 01 14:51:59.730778 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:51:59.730762    4993 cpu_manager.go:414] "containerMap: removing container" podUID="30998c96-4380-4e46-89db-c2f328351f71" containerName="istio-proxy"
Nov 01 14:51:59.730819 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:51:59.730778    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30998c96-4380-4e46-89db-c2f328351f71" containerName="istio-proxy"
Nov 01 14:51:59.730819 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:51:59.730796    4993 cpu_manager.go:414] "containerMap: removing container" podUID="6db204f9-b7a9-4b31-ab49-a4406d2b879d" containerName="istio-proxy"
Nov 01 14:51:59.730819 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:51:59.730801    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6db204f9-b7a9-4b31-ab49-a4406d2b879d" containerName="istio-proxy"
Nov 01 14:51:59.730819 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:51:59.730810    4993 cpu_manager.go:414] "containerMap: removing container" podUID="1d99efc6-0d6e-4cb5-a65e-6ac9eae07ea9" containerName="maas-api"
Nov 01 14:51:59.730819 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:51:59.730815    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d99efc6-0d6e-4cb5-a65e-6ac9eae07ea9" containerName="maas-api"
Nov 01 14:51:59.730974 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:51:59.730836    4993 cpu_manager.go:414] "containerMap: removing container" podUID="8e02990c-0339-49b0-af86-17211a267bbc" containerName="maas-api"
Nov 01 14:51:59.730974 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:51:59.730841    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e02990c-0339-49b0-af86-17211a267bbc" containerName="maas-api"
Nov 01 14:51:59.730974 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:51:59.730856    4993 cpu_manager.go:414] "containerMap: removing container" podUID="aa7cf0a9-06d8-43fc-a81e-bc95b7f1bc80" containerName="payload-pre-processing"
Nov 01 14:51:59.730974 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:51:59.730861    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa7cf0a9-06d8-43fc-a81e-bc95b7f1bc80" containerName="payload-pre-processing"
Nov 01 14:51:59.730974 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:51:59.730938    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="30998c96-4380-4e46-89db-c2f328351f71" containerName="istio-proxy"
Nov 01 14:51:59.730974 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:51:59.730950    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="6db204f9-b7a9-4b31-ab49-a4406d2b879d" containerName="istio-proxy"
Nov 01 14:51:59.730974 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:51:59.730956    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d99efc6-0d6e-4cb5-a65e-6ac9eae07ea9" containerName="maas-api"
Nov 01 14:51:59.730974 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:51:59.730964    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e02990c-0339-49b0-af86-17211a267bbc" containerName="maas-api"
Nov 01 14:51:59.730974 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:51:59.730972    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa7cf0a9-06d8-43fc-a81e-bc95b7f1bc80" containerName="payload-pre-processing"
Nov 01 14:51:59.735837 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:51:59.735811    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e381b7-59fcfbb485-c7zvt"
Nov 01 14:51:59.784316 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:51:59.784284    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-e381b7-serving-cert\""
Nov 01 14:51:59.805581 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:51:59.805526    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-e381b7-59fcfbb485-c7zvt"]
Nov 01 14:51:59.831181 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:51:59.831119    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxcnn\" (UniqueName: \"kubernetes.io/projected/8e899083-bce2-46b2-a25e-975c8922102f-kube-api-access-qxcnn\") pod \"maas-api-e2e-praxis-b-e381b7-59fcfbb485-c7zvt\" (UID: \"8e899083-bce2-46b2-a25e-975c8922102f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e381b7-59fcfbb485-c7zvt"
Nov 01 14:51:59.831359 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:51:59.831247    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e899083-bce2-46b2-a25e-975c8922102f-metrics-certs\") pod \"maas-api-e2e-praxis-b-e381b7-59fcfbb485-c7zvt\" (UID: \"8e899083-bce2-46b2-a25e-975c8922102f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e381b7-59fcfbb485-c7zvt"
Nov 01 14:51:59.831359 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:51:59.831318    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e899083-bce2-46b2-a25e-975c8922102f-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-e381b7-59fcfbb485-c7zvt\" (UID: \"8e899083-bce2-46b2-a25e-975c8922102f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e381b7-59fcfbb485-c7zvt"
Nov 01 14:51:59.831803 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:51:59.831777    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e899083-bce2-46b2-a25e-975c8922102f-maas-api-tls\") pod \"maas-api-e2e-praxis-b-e381b7-59fcfbb485-c7zvt\" (UID: \"8e899083-bce2-46b2-a25e-975c8922102f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e381b7-59fcfbb485-c7zvt"
Nov 01 14:51:59.933424 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:51:59.933384    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxcnn\" (UniqueName: \"kubernetes.io/projected/8e899083-bce2-46b2-a25e-975c8922102f-kube-api-access-qxcnn\") pod \"maas-api-e2e-praxis-b-e381b7-59fcfbb485-c7zvt\" (UID: \"8e899083-bce2-46b2-a25e-975c8922102f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e381b7-59fcfbb485-c7zvt"
Nov 01 14:51:59.933617 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:51:59.933434    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e899083-bce2-46b2-a25e-975c8922102f-metrics-certs\") pod \"maas-api-e2e-praxis-b-e381b7-59fcfbb485-c7zvt\" (UID: \"8e899083-bce2-46b2-a25e-975c8922102f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e381b7-59fcfbb485-c7zvt"
Nov 01 14:51:59.933617 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:51:59.933469    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e899083-bce2-46b2-a25e-975c8922102f-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-e381b7-59fcfbb485-c7zvt\" (UID: \"8e899083-bce2-46b2-a25e-975c8922102f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e381b7-59fcfbb485-c7zvt"
Nov 01 14:51:59.933617 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:51:59.933589    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e899083-bce2-46b2-a25e-975c8922102f-maas-api-tls\") pod \"maas-api-e2e-praxis-b-e381b7-59fcfbb485-c7zvt\" (UID: \"8e899083-bce2-46b2-a25e-975c8922102f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e381b7-59fcfbb485-c7zvt"
Nov 01 14:51:59.933844 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:51:59.933717    4993 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-e381b7-serving-cert: secret "maas-api-e2e-praxis-b-e381b7-serving-cert" not found
Nov 01 14:51:59.933844 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:51:59.933796    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e899083-bce2-46b2-a25e-975c8922102f-maas-api-tls podName:8e899083-bce2-46b2-a25e-975c8922102f nodeName:}" failed. No retries permitted until 2026-11-01 14:52:00.433776116 +0000 UTC m=+2107.763638848 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8e899083-bce2-46b2-a25e-975c8922102f-maas-api-tls") pod "maas-api-e2e-praxis-b-e381b7-59fcfbb485-c7zvt" (UID: "8e899083-bce2-46b2-a25e-975c8922102f") : secret "maas-api-e2e-praxis-b-e381b7-serving-cert" not found
Nov 01 14:51:59.934307 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:51:59.934286    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e899083-bce2-46b2-a25e-975c8922102f-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-e381b7-59fcfbb485-c7zvt\" (UID: \"8e899083-bce2-46b2-a25e-975c8922102f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e381b7-59fcfbb485-c7zvt"
Nov 01 14:51:59.935924 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:51:59.935899    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e899083-bce2-46b2-a25e-975c8922102f-metrics-certs\") pod \"maas-api-e2e-praxis-b-e381b7-59fcfbb485-c7zvt\" (UID: \"8e899083-bce2-46b2-a25e-975c8922102f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e381b7-59fcfbb485-c7zvt"
Nov 01 14:52:00.008929 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:00.008893    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxcnn\" (UniqueName: \"kubernetes.io/projected/8e899083-bce2-46b2-a25e-975c8922102f-kube-api-access-qxcnn\") pod \"maas-api-e2e-praxis-b-e381b7-59fcfbb485-c7zvt\" (UID: \"8e899083-bce2-46b2-a25e-975c8922102f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e381b7-59fcfbb485-c7zvt"
Nov 01 14:52:00.442274 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:00.442155    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e899083-bce2-46b2-a25e-975c8922102f-maas-api-tls\") pod \"maas-api-e2e-praxis-b-e381b7-59fcfbb485-c7zvt\" (UID: \"8e899083-bce2-46b2-a25e-975c8922102f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e381b7-59fcfbb485-c7zvt"
Nov 01 14:52:00.444716 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:00.444684    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e899083-bce2-46b2-a25e-975c8922102f-maas-api-tls\") pod \"maas-api-e2e-praxis-b-e381b7-59fcfbb485-c7zvt\" (UID: \"8e899083-bce2-46b2-a25e-975c8922102f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e381b7-59fcfbb485-c7zvt"
Nov 01 14:52:00.649958 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:00.649924    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e381b7-59fcfbb485-c7zvt"
Nov 01 14:52:01.038510 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:01.038471    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-e381b7-59fcfbb485-c7zvt"]
Nov 01 14:52:01.044143 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:52:01.044115    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e899083_bce2_46b2_a25e_975c8922102f.slice/crio-84605b2da9225d699c8c359e8665d12886ee7535b6cb08d0ea0a0f4d11490787 WatchSource:0}: Error finding container 84605b2da9225d699c8c359e8665d12886ee7535b6cb08d0ea0a0f4d11490787: Status 404 returned error can't find the container with id 84605b2da9225d699c8c359e8665d12886ee7535b6cb08d0ea0a0f4d11490787
Nov 01 14:52:01.045509 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:01.045489    4993 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 14:52:02.021083 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:02.021047    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e381b7-59fcfbb485-c7zvt" event={"ID":"8e899083-bce2-46b2-a25e-975c8922102f","Type":"ContainerStarted","Data":"93efefaca1914bebdcbafd4c16c8db76280b1ba81809d9aed877351bbb079ce0"}
Nov 01 14:52:02.021275 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:02.021102    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e381b7-59fcfbb485-c7zvt" event={"ID":"8e899083-bce2-46b2-a25e-975c8922102f","Type":"ContainerStarted","Data":"84605b2da9225d699c8c359e8665d12886ee7535b6cb08d0ea0a0f4d11490787"}
Nov 01 14:52:02.021275 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:02.021154    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e381b7-59fcfbb485-c7zvt"
Nov 01 14:52:02.147300 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:02.147203    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e381b7-59fcfbb485-c7zvt" podStartSLOduration=2.312249906 podStartE2EDuration="3.147187703s" totalImagesPullingTime="834.937797ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:51:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:52:02.14434094 +0000 UTC m=+2109.474203677" watchObservedRunningTime="2026-11-01 14:52:02.147187703 +0000 UTC m=+2109.477050439"
Nov 01 14:52:08.033338 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:08.033260    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e381b7-59fcfbb485-c7zvt"
Nov 01 14:52:13.261470 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:13.261427    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l"]
Nov 01 14:52:13.262025 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:13.261721    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l" podUID="5bb22946-1d01-4745-a1e3-519f213c333c" containerName="istio-proxy" containerID="cri-o://92579e6a99b638be7a0253edf0a8aa2f462cd552ff72624764336db62129aacc" gracePeriod=30
Nov 01 14:52:14.276932 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:14.276893    4993 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 14:52:14.277387 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:14.276963    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l" podUID="5bb22946-1d01-4745-a1e3-519f213c333c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 14:52:17.324154 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:17.324109    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-34d384-5bbf97d5bb-qks27"]
Nov 01 14:52:17.324681 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:17.324346    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-34d384-5bbf97d5bb-qks27" podUID="bfb14cd9-70b5-442d-a13e-549973735eed" containerName="maas-api" containerID="cri-o://33e4afabeab0e5c9de7e9fe7387f50914b9e1b32b4ca61b40c970e9f87d0396b" gracePeriod=30
Nov 01 14:52:17.725783 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:17.725748    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-34d384-5bbf97d5bb-qks27"
Nov 01 14:52:17.847697 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:17.847608    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfb14cd9-70b5-442d-a13e-549973735eed-kube-api-access-shfc7\" (UniqueName: \"kubernetes.io/projected/bfb14cd9-70b5-442d-a13e-549973735eed-kube-api-access-shfc7\") pod \"bfb14cd9-70b5-442d-a13e-549973735eed\" (UID: \"bfb14cd9-70b5-442d-a13e-549973735eed\") "
Nov 01 14:52:17.848965 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:17.848937    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfb14cd9-70b5-442d-a13e-549973735eed-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfb14cd9-70b5-442d-a13e-549973735eed-openshift-service-ca\") pod \"bfb14cd9-70b5-442d-a13e-549973735eed\" (UID: \"bfb14cd9-70b5-442d-a13e-549973735eed\") "
Nov 01 14:52:17.849539 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:17.849519    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfb14cd9-70b5-442d-a13e-549973735eed-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfb14cd9-70b5-442d-a13e-549973735eed-maas-api-tls\") pod \"bfb14cd9-70b5-442d-a13e-549973735eed\" (UID: \"bfb14cd9-70b5-442d-a13e-549973735eed\") "
Nov 01 14:52:17.849979 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:17.849959    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfb14cd9-70b5-442d-a13e-549973735eed-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfb14cd9-70b5-442d-a13e-549973735eed-metrics-certs\") pod \"bfb14cd9-70b5-442d-a13e-549973735eed\" (UID: \"bfb14cd9-70b5-442d-a13e-549973735eed\") "
Nov 01 14:52:17.850468 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:17.849452    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfb14cd9-70b5-442d-a13e-549973735eed-openshift-service-ca" pod "bfb14cd9-70b5-442d-a13e-549973735eed" (UID: "bfb14cd9-70b5-442d-a13e-549973735eed"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:52:17.850951 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:17.850925    4993 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfb14cd9-70b5-442d-a13e-549973735eed-openshift-service-ca\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:52:17.854014 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:17.853988    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfb14cd9-70b5-442d-a13e-549973735eed-maas-api-tls" pod "bfb14cd9-70b5-442d-a13e-549973735eed" (UID: "bfb14cd9-70b5-442d-a13e-549973735eed"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:52:17.854112 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:17.854012    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfb14cd9-70b5-442d-a13e-549973735eed-kube-api-access-shfc7" pod "bfb14cd9-70b5-442d-a13e-549973735eed" (UID: "bfb14cd9-70b5-442d-a13e-549973735eed"). InnerVolumeSpecName "kube-api-access-shfc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:52:17.855401 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:17.855379    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfb14cd9-70b5-442d-a13e-549973735eed-metrics-certs" pod "bfb14cd9-70b5-442d-a13e-549973735eed" (UID: "bfb14cd9-70b5-442d-a13e-549973735eed"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:52:17.952300 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:17.952263    4993 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfb14cd9-70b5-442d-a13e-549973735eed-metrics-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:52:17.952300 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:17.952302    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shfc7\" (UniqueName: \"kubernetes.io/projected/bfb14cd9-70b5-442d-a13e-549973735eed-kube-api-access-shfc7\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:52:17.952548 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:17.952319    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfb14cd9-70b5-442d-a13e-549973735eed-maas-api-tls\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:52:18.120350 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:18.120266    4993 generic.go:356] "Generic (PLEG): container finished" podID="bfb14cd9-70b5-442d-a13e-549973735eed" containerID="33e4afabeab0e5c9de7e9fe7387f50914b9e1b32b4ca61b40c970e9f87d0396b" exitCode=0
Nov 01 14:52:18.120350 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:18.120315    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-34d384-5bbf97d5bb-qks27"
Nov 01 14:52:18.120350 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:18.120323    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-34d384-5bbf97d5bb-qks27" event={"ID":"bfb14cd9-70b5-442d-a13e-549973735eed","Type":"ContainerDied","Data":"33e4afabeab0e5c9de7e9fe7387f50914b9e1b32b4ca61b40c970e9f87d0396b"}
Nov 01 14:52:18.120589 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:18.120374    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-34d384-5bbf97d5bb-qks27" event={"ID":"bfb14cd9-70b5-442d-a13e-549973735eed","Type":"ContainerDied","Data":"29fb81111354dd095cd363981cf4e979785e9f6c15e5b4df934fc793d6c41602"}
Nov 01 14:52:18.120589 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:18.120397    4993 scope.go:122] "RemoveContainer" containerID="33e4afabeab0e5c9de7e9fe7387f50914b9e1b32b4ca61b40c970e9f87d0396b"
Nov 01 14:52:18.131528 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:18.131509    4993 scope.go:122] "RemoveContainer" containerID="33e4afabeab0e5c9de7e9fe7387f50914b9e1b32b4ca61b40c970e9f87d0396b"
Nov 01 14:52:18.131786 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:52:18.131761    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33e4afabeab0e5c9de7e9fe7387f50914b9e1b32b4ca61b40c970e9f87d0396b\": container with ID starting with 33e4afabeab0e5c9de7e9fe7387f50914b9e1b32b4ca61b40c970e9f87d0396b not found: ID does not exist" containerID="33e4afabeab0e5c9de7e9fe7387f50914b9e1b32b4ca61b40c970e9f87d0396b"
Nov 01 14:52:18.131864 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:18.131795    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33e4afabeab0e5c9de7e9fe7387f50914b9e1b32b4ca61b40c970e9f87d0396b"} err="failed to get container status \"33e4afabeab0e5c9de7e9fe7387f50914b9e1b32b4ca61b40c970e9f87d0396b\": rpc error: code = NotFound desc = could not find container \"33e4afabeab0e5c9de7e9fe7387f50914b9e1b32b4ca61b40c970e9f87d0396b\": container with ID starting with 33e4afabeab0e5c9de7e9fe7387f50914b9e1b32b4ca61b40c970e9f87d0396b not found: ID does not exist"
Nov 01 14:52:18.370738 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:18.370594    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-34d384-5bbf97d5bb-qks27"]
Nov 01 14:52:18.387361 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:18.387330    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-34d384-5bbf97d5bb-qks27"]
Nov 01 14:52:18.538216 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:18.538163    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l"
Nov 01 14:52:18.662867 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:18.661211    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bb22946-1d01-4745-a1e3-519f213c333c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bb22946-1d01-4745-a1e3-519f213c333c-istio-envoy\") pod \"5bb22946-1d01-4745-a1e3-519f213c333c\" (UID: \"5bb22946-1d01-4745-a1e3-519f213c333c\") "
Nov 01 14:52:18.662867 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:18.661297    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bb22946-1d01-4745-a1e3-519f213c333c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb22946-1d01-4745-a1e3-519f213c333c-workload-socket\") pod \"5bb22946-1d01-4745-a1e3-519f213c333c\" (UID: \"5bb22946-1d01-4745-a1e3-519f213c333c\") "
Nov 01 14:52:18.662867 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:18.661328    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bb22946-1d01-4745-a1e3-519f213c333c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bb22946-1d01-4745-a1e3-519f213c333c-workload-certs\") pod \"5bb22946-1d01-4745-a1e3-519f213c333c\" (UID: \"5bb22946-1d01-4745-a1e3-519f213c333c\") "
Nov 01 14:52:18.662867 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:18.661438    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bb22946-1d01-4745-a1e3-519f213c333c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bb22946-1d01-4745-a1e3-519f213c333c-istio-data\") pod \"5bb22946-1d01-4745-a1e3-519f213c333c\" (UID: \"5bb22946-1d01-4745-a1e3-519f213c333c\") "
Nov 01 14:52:18.662867 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:18.661471    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5bb22946-1d01-4745-a1e3-519f213c333c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bb22946-1d01-4745-a1e3-519f213c333c-istio-podinfo\") pod \"5bb22946-1d01-4745-a1e3-519f213c333c\" (UID: \"5bb22946-1d01-4745-a1e3-519f213c333c\") "
Nov 01 14:52:18.662867 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:18.661548    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bb22946-1d01-4745-a1e3-519f213c333c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb22946-1d01-4745-a1e3-519f213c333c-credential-socket\") pod \"5bb22946-1d01-4745-a1e3-519f213c333c\" (UID: \"5bb22946-1d01-4745-a1e3-519f213c333c\") "
Nov 01 14:52:18.662867 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:18.661616    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bb22946-1d01-4745-a1e3-519f213c333c-istio-token\" (UniqueName: \"kubernetes.io/projected/5bb22946-1d01-4745-a1e3-519f213c333c-istio-token\") pod \"5bb22946-1d01-4745-a1e3-519f213c333c\" (UID: \"5bb22946-1d01-4745-a1e3-519f213c333c\") "
Nov 01 14:52:18.662867 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:18.661690    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bb22946-1d01-4745-a1e3-519f213c333c-kube-api-access-m4lrc\" (UniqueName: \"kubernetes.io/projected/5bb22946-1d01-4745-a1e3-519f213c333c-kube-api-access-m4lrc\") pod \"5bb22946-1d01-4745-a1e3-519f213c333c\" (UID: \"5bb22946-1d01-4745-a1e3-519f213c333c\") "
Nov 01 14:52:18.662867 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:18.661745    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bb22946-1d01-4745-a1e3-519f213c333c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bb22946-1d01-4745-a1e3-519f213c333c-istiod-ca-cert\") pod \"5bb22946-1d01-4745-a1e3-519f213c333c\" (UID: \"5bb22946-1d01-4745-a1e3-519f213c333c\") "
Nov 01 14:52:18.662867 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:18.661922    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bb22946-1d01-4745-a1e3-519f213c333c-istio-data" pod "5bb22946-1d01-4745-a1e3-519f213c333c" (UID: "5bb22946-1d01-4745-a1e3-519f213c333c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:52:18.662867 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:18.662271    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bb22946-1d01-4745-a1e3-519f213c333c-istiod-ca-cert" pod "5bb22946-1d01-4745-a1e3-519f213c333c" (UID: "5bb22946-1d01-4745-a1e3-519f213c333c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:52:18.662867 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:18.662464    4993 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bb22946-1d01-4745-a1e3-519f213c333c-istio-data\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:52:18.662867 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:18.662489    4993 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bb22946-1d01-4745-a1e3-519f213c333c-istiod-ca-cert\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:52:18.662867 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:18.662708    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bb22946-1d01-4745-a1e3-519f213c333c-workload-socket" pod "5bb22946-1d01-4745-a1e3-519f213c333c" (UID: "5bb22946-1d01-4745-a1e3-519f213c333c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:52:18.663640 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:18.662922    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bb22946-1d01-4745-a1e3-519f213c333c-workload-certs" pod "5bb22946-1d01-4745-a1e3-519f213c333c" (UID: "5bb22946-1d01-4745-a1e3-519f213c333c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:52:18.663640 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:18.663355    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bb22946-1d01-4745-a1e3-519f213c333c-credential-socket" pod "5bb22946-1d01-4745-a1e3-519f213c333c" (UID: "5bb22946-1d01-4745-a1e3-519f213c333c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:52:18.665312 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:18.665258    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5bb22946-1d01-4745-a1e3-519f213c333c-istio-podinfo" pod "5bb22946-1d01-4745-a1e3-519f213c333c" (UID: "5bb22946-1d01-4745-a1e3-519f213c333c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 14:52:18.665312 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:18.665272    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bb22946-1d01-4745-a1e3-519f213c333c-istio-token" pod "5bb22946-1d01-4745-a1e3-519f213c333c" (UID: "5bb22946-1d01-4745-a1e3-519f213c333c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:52:18.665583 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:18.665556    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bb22946-1d01-4745-a1e3-519f213c333c-kube-api-access-m4lrc" pod "5bb22946-1d01-4745-a1e3-519f213c333c" (UID: "5bb22946-1d01-4745-a1e3-519f213c333c"). InnerVolumeSpecName "kube-api-access-m4lrc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:52:18.665677 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:18.665640    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bb22946-1d01-4745-a1e3-519f213c333c-istio-envoy" pod "5bb22946-1d01-4745-a1e3-519f213c333c" (UID: "5bb22946-1d01-4745-a1e3-519f213c333c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:52:18.763958 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:18.763923    4993 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bb22946-1d01-4745-a1e3-519f213c333c-istio-envoy\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:52:18.764131 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:18.764003    4993 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb22946-1d01-4745-a1e3-519f213c333c-workload-socket\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:52:18.764131 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:18.764017    4993 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bb22946-1d01-4745-a1e3-519f213c333c-workload-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:52:18.764131 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:18.764047    4993 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bb22946-1d01-4745-a1e3-519f213c333c-istio-podinfo\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:52:18.764131 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:18.764056    4993 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb22946-1d01-4745-a1e3-519f213c333c-credential-socket\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:52:18.764131 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:18.764066    4993 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bb22946-1d01-4745-a1e3-519f213c333c-istio-token\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:52:18.764131 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:18.764076    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4lrc\" (UniqueName: \"kubernetes.io/projected/5bb22946-1d01-4745-a1e3-519f213c333c-kube-api-access-m4lrc\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:52:18.795785 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:18.795750    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfb14cd9-70b5-442d-a13e-549973735eed" path="/var/lib/kubelet/pods/bfb14cd9-70b5-442d-a13e-549973735eed/volumes"
Nov 01 14:52:19.128208 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:19.128156    4993 generic.go:356] "Generic (PLEG): container finished" podID="5bb22946-1d01-4745-a1e3-519f213c333c" containerID="92579e6a99b638be7a0253edf0a8aa2f462cd552ff72624764336db62129aacc" exitCode=0
Nov 01 14:52:19.128208 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:19.128200    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l" event={"ID":"5bb22946-1d01-4745-a1e3-519f213c333c","Type":"ContainerDied","Data":"92579e6a99b638be7a0253edf0a8aa2f462cd552ff72624764336db62129aacc"}
Nov 01 14:52:19.128431 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:19.128240    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l"
Nov 01 14:52:19.128431 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:19.128265    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l" event={"ID":"5bb22946-1d01-4745-a1e3-519f213c333c","Type":"ContainerDied","Data":"d15d8e43191f7e895bce675847ce9cf3f7d2ca727e73882d0a5ef44d24c46169"}
Nov 01 14:52:19.128431 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:19.128299    4993 scope.go:122] "RemoveContainer" containerID="92579e6a99b638be7a0253edf0a8aa2f462cd552ff72624764336db62129aacc"
Nov 01 14:52:19.138061 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:19.138043    4993 scope.go:122] "RemoveContainer" containerID="92579e6a99b638be7a0253edf0a8aa2f462cd552ff72624764336db62129aacc"
Nov 01 14:52:19.138368 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:52:19.138348    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92579e6a99b638be7a0253edf0a8aa2f462cd552ff72624764336db62129aacc\": container with ID starting with 92579e6a99b638be7a0253edf0a8aa2f462cd552ff72624764336db62129aacc not found: ID does not exist" containerID="92579e6a99b638be7a0253edf0a8aa2f462cd552ff72624764336db62129aacc"
Nov 01 14:52:19.138439 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:19.138377    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92579e6a99b638be7a0253edf0a8aa2f462cd552ff72624764336db62129aacc"} err="failed to get container status \"92579e6a99b638be7a0253edf0a8aa2f462cd552ff72624764336db62129aacc\": rpc error: code = NotFound desc = could not find container \"92579e6a99b638be7a0253edf0a8aa2f462cd552ff72624764336db62129aacc\": container with ID starting with 92579e6a99b638be7a0253edf0a8aa2f462cd552ff72624764336db62129aacc not found: ID does not exist"
Nov 01 14:52:19.319162 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:19.319129    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l"]
Nov 01 14:52:19.364504 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:19.364472    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-jmf6l"]
Nov 01 14:52:20.796546 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:20.796512    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bb22946-1d01-4745-a1e3-519f213c333c" path="/var/lib/kubelet/pods/5bb22946-1d01-4745-a1e3-519f213c333c/volumes"
Nov 01 14:52:21.279191 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:21.275088    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-bhsml"]
Nov 01 14:52:21.279191 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:21.276681    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-bhsml" podUID="d9ffa582-9776-4ee9-86fa-b3039e14d4e1" containerName="istio-proxy" containerID="cri-o://2c8899ae0059724a1c43576c8784bfc16bff3a2ee1daab15f7c9c50e62d401ba" gracePeriod=30
Nov 01 14:52:26.542069 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:26.542043    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-bhsml"
Nov 01 14:52:26.598936 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:26.598846    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9ffa582-9776-4ee9-86fa-b3039e14d4e1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9ffa582-9776-4ee9-86fa-b3039e14d4e1-credential-socket\") pod \"d9ffa582-9776-4ee9-86fa-b3039e14d4e1\" (UID: \"d9ffa582-9776-4ee9-86fa-b3039e14d4e1\") "
Nov 01 14:52:26.598936 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:26.598926    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9ffa582-9776-4ee9-86fa-b3039e14d4e1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9ffa582-9776-4ee9-86fa-b3039e14d4e1-istiod-ca-cert\") pod \"d9ffa582-9776-4ee9-86fa-b3039e14d4e1\" (UID: \"d9ffa582-9776-4ee9-86fa-b3039e14d4e1\") "
Nov 01 14:52:26.599158 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:26.598957    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9ffa582-9776-4ee9-86fa-b3039e14d4e1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9ffa582-9776-4ee9-86fa-b3039e14d4e1-istio-data\") pod \"d9ffa582-9776-4ee9-86fa-b3039e14d4e1\" (UID: \"d9ffa582-9776-4ee9-86fa-b3039e14d4e1\") "
Nov 01 14:52:26.599158 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:26.599064    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9ffa582-9776-4ee9-86fa-b3039e14d4e1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9ffa582-9776-4ee9-86fa-b3039e14d4e1-workload-certs\") pod \"d9ffa582-9776-4ee9-86fa-b3039e14d4e1\" (UID: \"d9ffa582-9776-4ee9-86fa-b3039e14d4e1\") "
Nov 01 14:52:26.599158 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:26.599093    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9ffa582-9776-4ee9-86fa-b3039e14d4e1-istio-token\" (UniqueName: \"kubernetes.io/projected/d9ffa582-9776-4ee9-86fa-b3039e14d4e1-istio-token\") pod \"d9ffa582-9776-4ee9-86fa-b3039e14d4e1\" (UID: \"d9ffa582-9776-4ee9-86fa-b3039e14d4e1\") "
Nov 01 14:52:26.599158 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:26.599128    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9ffa582-9776-4ee9-86fa-b3039e14d4e1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9ffa582-9776-4ee9-86fa-b3039e14d4e1-istio-envoy\") pod \"d9ffa582-9776-4ee9-86fa-b3039e14d4e1\" (UID: \"d9ffa582-9776-4ee9-86fa-b3039e14d4e1\") "
Nov 01 14:52:26.599382 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:26.599200    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9ffa582-9776-4ee9-86fa-b3039e14d4e1-kube-api-access-6xs24\" (UniqueName: \"kubernetes.io/projected/d9ffa582-9776-4ee9-86fa-b3039e14d4e1-kube-api-access-6xs24\") pod \"d9ffa582-9776-4ee9-86fa-b3039e14d4e1\" (UID: \"d9ffa582-9776-4ee9-86fa-b3039e14d4e1\") "
Nov 01 14:52:26.599382 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:26.599211    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9ffa582-9776-4ee9-86fa-b3039e14d4e1-credential-socket" pod "d9ffa582-9776-4ee9-86fa-b3039e14d4e1" (UID: "d9ffa582-9776-4ee9-86fa-b3039e14d4e1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:52:26.599382 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:26.599255    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d9ffa582-9776-4ee9-86fa-b3039e14d4e1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9ffa582-9776-4ee9-86fa-b3039e14d4e1-istio-podinfo\") pod \"d9ffa582-9776-4ee9-86fa-b3039e14d4e1\" (UID: \"d9ffa582-9776-4ee9-86fa-b3039e14d4e1\") "
Nov 01 14:52:26.599382 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:26.599326    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9ffa582-9776-4ee9-86fa-b3039e14d4e1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9ffa582-9776-4ee9-86fa-b3039e14d4e1-workload-socket\") pod \"d9ffa582-9776-4ee9-86fa-b3039e14d4e1\" (UID: \"d9ffa582-9776-4ee9-86fa-b3039e14d4e1\") "
Nov 01 14:52:26.599574 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:26.599420    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9ffa582-9776-4ee9-86fa-b3039e14d4e1-istiod-ca-cert" pod "d9ffa582-9776-4ee9-86fa-b3039e14d4e1" (UID: "d9ffa582-9776-4ee9-86fa-b3039e14d4e1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:52:26.599761 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:26.599733    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9ffa582-9776-4ee9-86fa-b3039e14d4e1-workload-certs" pod "d9ffa582-9776-4ee9-86fa-b3039e14d4e1" (UID: "d9ffa582-9776-4ee9-86fa-b3039e14d4e1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:52:26.599823 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:26.599800    4993 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9ffa582-9776-4ee9-86fa-b3039e14d4e1-credential-socket\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:52:26.599874 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:26.599819    4993 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9ffa582-9776-4ee9-86fa-b3039e14d4e1-istiod-ca-cert\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:52:26.599926 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:26.599874    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9ffa582-9776-4ee9-86fa-b3039e14d4e1-workload-socket" pod "d9ffa582-9776-4ee9-86fa-b3039e14d4e1" (UID: "d9ffa582-9776-4ee9-86fa-b3039e14d4e1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:52:26.599984 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:26.599935    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9ffa582-9776-4ee9-86fa-b3039e14d4e1-istio-data" pod "d9ffa582-9776-4ee9-86fa-b3039e14d4e1" (UID: "d9ffa582-9776-4ee9-86fa-b3039e14d4e1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:52:26.601692 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:26.601665    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9ffa582-9776-4ee9-86fa-b3039e14d4e1-istio-token" pod "d9ffa582-9776-4ee9-86fa-b3039e14d4e1" (UID: "d9ffa582-9776-4ee9-86fa-b3039e14d4e1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:52:26.601865 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:26.601845    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9ffa582-9776-4ee9-86fa-b3039e14d4e1-istio-envoy" pod "d9ffa582-9776-4ee9-86fa-b3039e14d4e1" (UID: "d9ffa582-9776-4ee9-86fa-b3039e14d4e1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:52:26.602011 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:26.601982    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d9ffa582-9776-4ee9-86fa-b3039e14d4e1-istio-podinfo" pod "d9ffa582-9776-4ee9-86fa-b3039e14d4e1" (UID: "d9ffa582-9776-4ee9-86fa-b3039e14d4e1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 14:52:26.602011 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:26.602001    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9ffa582-9776-4ee9-86fa-b3039e14d4e1-kube-api-access-6xs24" pod "d9ffa582-9776-4ee9-86fa-b3039e14d4e1" (UID: "d9ffa582-9776-4ee9-86fa-b3039e14d4e1"). InnerVolumeSpecName "kube-api-access-6xs24". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:52:26.701240 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:26.701206    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xs24\" (UniqueName: \"kubernetes.io/projected/d9ffa582-9776-4ee9-86fa-b3039e14d4e1-kube-api-access-6xs24\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:52:26.701240 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:26.701243    4993 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9ffa582-9776-4ee9-86fa-b3039e14d4e1-istio-podinfo\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:52:26.701432 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:26.701259    4993 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9ffa582-9776-4ee9-86fa-b3039e14d4e1-workload-socket\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:52:26.701432 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:26.701273    4993 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9ffa582-9776-4ee9-86fa-b3039e14d4e1-istio-data\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:52:26.701432 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:26.701287    4993 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9ffa582-9776-4ee9-86fa-b3039e14d4e1-workload-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:52:26.701432 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:26.701300    4993 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9ffa582-9776-4ee9-86fa-b3039e14d4e1-istio-token\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:52:26.701432 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:26.701313    4993 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9ffa582-9776-4ee9-86fa-b3039e14d4e1-istio-envoy\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:52:27.213848 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:27.213814    4993 generic.go:356] "Generic (PLEG): container finished" podID="d9ffa582-9776-4ee9-86fa-b3039e14d4e1" containerID="2c8899ae0059724a1c43576c8784bfc16bff3a2ee1daab15f7c9c50e62d401ba" exitCode=0
Nov 01 14:52:27.214033 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:27.213863    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-bhsml"
Nov 01 14:52:27.214033 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:27.213898    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-bhsml" event={"ID":"d9ffa582-9776-4ee9-86fa-b3039e14d4e1","Type":"ContainerDied","Data":"2c8899ae0059724a1c43576c8784bfc16bff3a2ee1daab15f7c9c50e62d401ba"}
Nov 01 14:52:27.214033 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:27.213953    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-bhsml" event={"ID":"d9ffa582-9776-4ee9-86fa-b3039e14d4e1","Type":"ContainerDied","Data":"a6952e8b906eeeda9f4be18ef7bd446e8aa8faa85918ce6a36d4ffb3181f7652"}
Nov 01 14:52:27.214033 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:27.213979    4993 scope.go:122] "RemoveContainer" containerID="2c8899ae0059724a1c43576c8784bfc16bff3a2ee1daab15f7c9c50e62d401ba"
Nov 01 14:52:27.223712 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:27.223683    4993 scope.go:122] "RemoveContainer" containerID="2c8899ae0059724a1c43576c8784bfc16bff3a2ee1daab15f7c9c50e62d401ba"
Nov 01 14:52:27.223980 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:52:27.223963    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c8899ae0059724a1c43576c8784bfc16bff3a2ee1daab15f7c9c50e62d401ba\": container with ID starting with 2c8899ae0059724a1c43576c8784bfc16bff3a2ee1daab15f7c9c50e62d401ba not found: ID does not exist" containerID="2c8899ae0059724a1c43576c8784bfc16bff3a2ee1daab15f7c9c50e62d401ba"
Nov 01 14:52:27.224031 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:27.223988    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c8899ae0059724a1c43576c8784bfc16bff3a2ee1daab15f7c9c50e62d401ba"} err="failed to get container status \"2c8899ae0059724a1c43576c8784bfc16bff3a2ee1daab15f7c9c50e62d401ba\": rpc error: code = NotFound desc = could not find container \"2c8899ae0059724a1c43576c8784bfc16bff3a2ee1daab15f7c9c50e62d401ba\": container with ID starting with 2c8899ae0059724a1c43576c8784bfc16bff3a2ee1daab15f7c9c50e62d401ba not found: ID does not exist"
Nov 01 14:52:27.412791 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:27.412760    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-bhsml"]
Nov 01 14:52:27.415234 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:27.415205    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-bhsml"]
Nov 01 14:52:27.501357 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:52:27.501320    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:52:28.796575 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:52:28.796529    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9ffa582-9776-4ee9-86fa-b3039e14d4e1" path="/var/lib/kubelet/pods/d9ffa582-9776-4ee9-86fa-b3039e14d4e1/volumes"
Nov 01 14:52:57.502122 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:52:57.502080    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:53:20.846101 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:53:20.846067    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-288f1e-84fdbc4bd9-wvt6v"]
Nov 01 14:53:20.846616 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:53:20.846586    4993 cpu_manager.go:414] "containerMap: removing container" podUID="5bb22946-1d01-4745-a1e3-519f213c333c" containerName="istio-proxy"
Nov 01 14:53:20.846616 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:53:20.846598    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bb22946-1d01-4745-a1e3-519f213c333c" containerName="istio-proxy"
Nov 01 14:53:20.846695 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:53:20.846640    4993 cpu_manager.go:414] "containerMap: removing container" podUID="bfb14cd9-70b5-442d-a13e-549973735eed" containerName="maas-api"
Nov 01 14:53:20.846695 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:53:20.846646    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfb14cd9-70b5-442d-a13e-549973735eed" containerName="maas-api"
Nov 01 14:53:20.846695 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:53:20.846653    4993 cpu_manager.go:414] "containerMap: removing container" podUID="d9ffa582-9776-4ee9-86fa-b3039e14d4e1" containerName="istio-proxy"
Nov 01 14:53:20.846695 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:53:20.846658    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9ffa582-9776-4ee9-86fa-b3039e14d4e1" containerName="istio-proxy"
Nov 01 14:53:20.846812 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:53:20.846744    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bb22946-1d01-4745-a1e3-519f213c333c" containerName="istio-proxy"
Nov 01 14:53:20.846812 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:53:20.846760    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfb14cd9-70b5-442d-a13e-549973735eed" containerName="maas-api"
Nov 01 14:53:20.846812 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:53:20.846766    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9ffa582-9776-4ee9-86fa-b3039e14d4e1" containerName="istio-proxy"
Nov 01 14:53:20.850192 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:53:20.850147    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-288f1e-84fdbc4bd9-wvt6v"
Nov 01 14:53:20.908544 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:53:20.908505    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-288f1e-84fdbc4bd9-wvt6v"]
Nov 01 14:53:20.910293 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:53:20.910269    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-288f1e-serving-cert\""
Nov 01 14:53:20.950136 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:53:20.950097    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce6e8049-31b5-4750-994b-83a4f0533715-metrics-certs\") pod \"maas-api-e2e-ipp-b-288f1e-84fdbc4bd9-wvt6v\" (UID: \"ce6e8049-31b5-4750-994b-83a4f0533715\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-288f1e-84fdbc4bd9-wvt6v"
Nov 01 14:53:20.950322 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:53:20.950288    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4btbt\" (UniqueName: \"kubernetes.io/projected/ce6e8049-31b5-4750-994b-83a4f0533715-kube-api-access-4btbt\") pod \"maas-api-e2e-ipp-b-288f1e-84fdbc4bd9-wvt6v\" (UID: \"ce6e8049-31b5-4750-994b-83a4f0533715\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-288f1e-84fdbc4bd9-wvt6v"
Nov 01 14:53:20.950385 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:53:20.950352    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce6e8049-31b5-4750-994b-83a4f0533715-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-288f1e-84fdbc4bd9-wvt6v\" (UID: \"ce6e8049-31b5-4750-994b-83a4f0533715\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-288f1e-84fdbc4bd9-wvt6v"
Nov 01 14:53:20.950472 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:53:20.950451    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce6e8049-31b5-4750-994b-83a4f0533715-maas-api-tls\") pod \"maas-api-e2e-ipp-b-288f1e-84fdbc4bd9-wvt6v\" (UID: \"ce6e8049-31b5-4750-994b-83a4f0533715\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-288f1e-84fdbc4bd9-wvt6v"
Nov 01 14:53:21.052383 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:53:21.052341    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce6e8049-31b5-4750-994b-83a4f0533715-metrics-certs\") pod \"maas-api-e2e-ipp-b-288f1e-84fdbc4bd9-wvt6v\" (UID: \"ce6e8049-31b5-4750-994b-83a4f0533715\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-288f1e-84fdbc4bd9-wvt6v"
Nov 01 14:53:21.052598 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:53:21.052473    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4btbt\" (UniqueName: \"kubernetes.io/projected/ce6e8049-31b5-4750-994b-83a4f0533715-kube-api-access-4btbt\") pod \"maas-api-e2e-ipp-b-288f1e-84fdbc4bd9-wvt6v\" (UID: \"ce6e8049-31b5-4750-994b-83a4f0533715\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-288f1e-84fdbc4bd9-wvt6v"
Nov 01 14:53:21.052598 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:53:21.052532    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce6e8049-31b5-4750-994b-83a4f0533715-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-288f1e-84fdbc4bd9-wvt6v\" (UID: \"ce6e8049-31b5-4750-994b-83a4f0533715\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-288f1e-84fdbc4bd9-wvt6v"
Nov 01 14:53:21.052598 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:53:21.052575    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce6e8049-31b5-4750-994b-83a4f0533715-maas-api-tls\") pod \"maas-api-e2e-ipp-b-288f1e-84fdbc4bd9-wvt6v\" (UID: \"ce6e8049-31b5-4750-994b-83a4f0533715\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-288f1e-84fdbc4bd9-wvt6v"
Nov 01 14:53:21.052800 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:53:21.052781    4993 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-288f1e-serving-cert: secret "maas-api-e2e-ipp-b-288f1e-serving-cert" not found
Nov 01 14:53:21.052874 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:53:21.052864    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce6e8049-31b5-4750-994b-83a4f0533715-maas-api-tls podName:ce6e8049-31b5-4750-994b-83a4f0533715 nodeName:}" failed. No retries permitted until 2026-11-01 14:53:21.552839876 +0000 UTC m=+2188.882702606 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ce6e8049-31b5-4750-994b-83a4f0533715-maas-api-tls") pod "maas-api-e2e-ipp-b-288f1e-84fdbc4bd9-wvt6v" (UID: "ce6e8049-31b5-4750-994b-83a4f0533715") : secret "maas-api-e2e-ipp-b-288f1e-serving-cert" not found
Nov 01 14:53:21.053692 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:53:21.053662    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce6e8049-31b5-4750-994b-83a4f0533715-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-288f1e-84fdbc4bd9-wvt6v\" (UID: \"ce6e8049-31b5-4750-994b-83a4f0533715\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-288f1e-84fdbc4bd9-wvt6v"
Nov 01 14:53:21.055409 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:53:21.055377    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce6e8049-31b5-4750-994b-83a4f0533715-metrics-certs\") pod \"maas-api-e2e-ipp-b-288f1e-84fdbc4bd9-wvt6v\" (UID: \"ce6e8049-31b5-4750-994b-83a4f0533715\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-288f1e-84fdbc4bd9-wvt6v"
Nov 01 14:53:21.107252 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:53:21.107122    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4btbt\" (UniqueName: \"kubernetes.io/projected/ce6e8049-31b5-4750-994b-83a4f0533715-kube-api-access-4btbt\") pod \"maas-api-e2e-ipp-b-288f1e-84fdbc4bd9-wvt6v\" (UID: \"ce6e8049-31b5-4750-994b-83a4f0533715\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-288f1e-84fdbc4bd9-wvt6v"
Nov 01 14:53:21.559651 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:53:21.559607    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce6e8049-31b5-4750-994b-83a4f0533715-maas-api-tls\") pod \"maas-api-e2e-ipp-b-288f1e-84fdbc4bd9-wvt6v\" (UID: \"ce6e8049-31b5-4750-994b-83a4f0533715\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-288f1e-84fdbc4bd9-wvt6v"
Nov 01 14:53:21.562487 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:53:21.562460    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce6e8049-31b5-4750-994b-83a4f0533715-maas-api-tls\") pod \"maas-api-e2e-ipp-b-288f1e-84fdbc4bd9-wvt6v\" (UID: \"ce6e8049-31b5-4750-994b-83a4f0533715\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-288f1e-84fdbc4bd9-wvt6v"
Nov 01 14:53:21.763999 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:53:21.763967    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-288f1e-84fdbc4bd9-wvt6v"
Nov 01 14:53:22.132938 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:53:22.132860    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-288f1e-84fdbc4bd9-wvt6v"]
Nov 01 14:53:22.139183 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:53:22.139142    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce6e8049_31b5_4750_994b_83a4f0533715.slice/crio-eb385f9e011b5e3d0a05ad9226a902e63b82e7246de66f3e557b09cc4d0bc2cf WatchSource:0}: Error finding container eb385f9e011b5e3d0a05ad9226a902e63b82e7246de66f3e557b09cc4d0bc2cf: Status 404 returned error can't find the container with id eb385f9e011b5e3d0a05ad9226a902e63b82e7246de66f3e557b09cc4d0bc2cf
Nov 01 14:53:22.572246 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:53:22.572202    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-288f1e-84fdbc4bd9-wvt6v" event={"ID":"ce6e8049-31b5-4750-994b-83a4f0533715","Type":"ContainerStarted","Data":"eb385f9e011b5e3d0a05ad9226a902e63b82e7246de66f3e557b09cc4d0bc2cf"}
Nov 01 14:53:23.580711 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:53:23.580672    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-288f1e-84fdbc4bd9-wvt6v" event={"ID":"ce6e8049-31b5-4750-994b-83a4f0533715","Type":"ContainerStarted","Data":"95d0f0a1b825ee4acc8bb7ce60533a046338a0dace25216cecee813c15beb0d0"}
Nov 01 14:53:23.581157 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:53:23.580760    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-288f1e-84fdbc4bd9-wvt6v"
Nov 01 14:53:23.690487 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:53:23.690429    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-288f1e-84fdbc4bd9-wvt6v" podStartSLOduration=2.681412284 podStartE2EDuration="3.690412006s" totalImagesPullingTime="1.008999722s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:53:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:53:23.687796246 +0000 UTC m=+2191.017658982" watchObservedRunningTime="2026-11-01 14:53:23.690412006 +0000 UTC m=+2191.020274744"
Nov 01 14:53:27.501887 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:53:27.501848    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:53:29.593328 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:53:29.593294    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-288f1e-84fdbc4bd9-wvt6v"
Nov 01 14:53:44.558028 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:53:44.557943    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-305290-788f694d7b-qpw5h"]
Nov 01 14:53:44.562546 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:53:44.562520    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-305290-788f694d7b-qpw5h"
Nov 01 14:53:44.610714 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:53:44.610685    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-305290-serving-cert\""
Nov 01 14:53:44.657509 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:53:44.657462    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69833224-a389-47b6-a069-002612808b6b-metrics-certs\") pod \"maas-api-e2e-praxis-b-305290-788f694d7b-qpw5h\" (UID: \"69833224-a389-47b6-a069-002612808b6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-305290-788f694d7b-qpw5h"
Nov 01 14:53:44.657729 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:53:44.657545    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69833224-a389-47b6-a069-002612808b6b-maas-api-tls\") pod \"maas-api-e2e-praxis-b-305290-788f694d7b-qpw5h\" (UID: \"69833224-a389-47b6-a069-002612808b6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-305290-788f694d7b-qpw5h"
Nov 01 14:53:44.657729 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:53:44.657599    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69833224-a389-47b6-a069-002612808b6b-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-305290-788f694d7b-qpw5h\" (UID: \"69833224-a389-47b6-a069-002612808b6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-305290-788f694d7b-qpw5h"
Nov 01 14:53:44.657861 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:53:44.657741    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tctjn\" (UniqueName: \"kubernetes.io/projected/69833224-a389-47b6-a069-002612808b6b-kube-api-access-tctjn\") pod \"maas-api-e2e-praxis-b-305290-788f694d7b-qpw5h\" (UID: \"69833224-a389-47b6-a069-002612808b6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-305290-788f694d7b-qpw5h"
Nov 01 14:53:44.661002 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:53:44.660965    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-305290-788f694d7b-qpw5h"]
Nov 01 14:53:44.758963 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:53:44.758914    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69833224-a389-47b6-a069-002612808b6b-maas-api-tls\") pod \"maas-api-e2e-praxis-b-305290-788f694d7b-qpw5h\" (UID: \"69833224-a389-47b6-a069-002612808b6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-305290-788f694d7b-qpw5h"
Nov 01 14:53:44.759123 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:53:44.758978    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69833224-a389-47b6-a069-002612808b6b-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-305290-788f694d7b-qpw5h\" (UID: \"69833224-a389-47b6-a069-002612808b6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-305290-788f694d7b-qpw5h"
Nov 01 14:53:44.759123 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:53:44.759084    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tctjn\" (UniqueName: \"kubernetes.io/projected/69833224-a389-47b6-a069-002612808b6b-kube-api-access-tctjn\") pod \"maas-api-e2e-praxis-b-305290-788f694d7b-qpw5h\" (UID: \"69833224-a389-47b6-a069-002612808b6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-305290-788f694d7b-qpw5h"
Nov 01 14:53:44.759123 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:53:44.759102    4993 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-305290-serving-cert: secret "maas-api-e2e-praxis-b-305290-serving-cert" not found
Nov 01 14:53:44.759304 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:53:44.759210    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69833224-a389-47b6-a069-002612808b6b-maas-api-tls podName:69833224-a389-47b6-a069-002612808b6b nodeName:}" failed. No retries permitted until 2026-11-01 14:53:45.25915475 +0000 UTC m=+2212.589017487 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/69833224-a389-47b6-a069-002612808b6b-maas-api-tls") pod "maas-api-e2e-praxis-b-305290-788f694d7b-qpw5h" (UID: "69833224-a389-47b6-a069-002612808b6b") : secret "maas-api-e2e-praxis-b-305290-serving-cert" not found
Nov 01 14:53:44.759304 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:53:44.759257    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69833224-a389-47b6-a069-002612808b6b-metrics-certs\") pod \"maas-api-e2e-praxis-b-305290-788f694d7b-qpw5h\" (UID: \"69833224-a389-47b6-a069-002612808b6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-305290-788f694d7b-qpw5h"
Nov 01 14:53:44.759826 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:53:44.759803    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69833224-a389-47b6-a069-002612808b6b-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-305290-788f694d7b-qpw5h\" (UID: \"69833224-a389-47b6-a069-002612808b6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-305290-788f694d7b-qpw5h"
Nov 01 14:53:44.761802 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:53:44.761777    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69833224-a389-47b6-a069-002612808b6b-metrics-certs\") pod \"maas-api-e2e-praxis-b-305290-788f694d7b-qpw5h\" (UID: \"69833224-a389-47b6-a069-002612808b6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-305290-788f694d7b-qpw5h"
Nov 01 14:53:44.816915 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:53:44.816751    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tctjn\" (UniqueName: \"kubernetes.io/projected/69833224-a389-47b6-a069-002612808b6b-kube-api-access-tctjn\") pod \"maas-api-e2e-praxis-b-305290-788f694d7b-qpw5h\" (UID: \"69833224-a389-47b6-a069-002612808b6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-305290-788f694d7b-qpw5h"
Nov 01 14:53:45.286424 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:53:45.286390    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69833224-a389-47b6-a069-002612808b6b-maas-api-tls\") pod \"maas-api-e2e-praxis-b-305290-788f694d7b-qpw5h\" (UID: \"69833224-a389-47b6-a069-002612808b6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-305290-788f694d7b-qpw5h"
Nov 01 14:53:45.289037 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:53:45.289011    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69833224-a389-47b6-a069-002612808b6b-maas-api-tls\") pod \"maas-api-e2e-praxis-b-305290-788f694d7b-qpw5h\" (UID: \"69833224-a389-47b6-a069-002612808b6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-305290-788f694d7b-qpw5h"
Nov 01 14:53:45.480555 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:53:45.480512    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-305290-788f694d7b-qpw5h"
Nov 01 14:53:45.855919 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:53:45.855838    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-305290-788f694d7b-qpw5h"]
Nov 01 14:53:45.861912 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:53:45.861878    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69833224_a389_47b6_a069_002612808b6b.slice/crio-8db3934bee521bc7c557786e91f722df76ef984bc960fa03264aa4d5f6bddb72 WatchSource:0}: Error finding container 8db3934bee521bc7c557786e91f722df76ef984bc960fa03264aa4d5f6bddb72: Status 404 returned error can't find the container with id 8db3934bee521bc7c557786e91f722df76ef984bc960fa03264aa4d5f6bddb72
Nov 01 14:53:46.728238 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:53:46.728197    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-305290-788f694d7b-qpw5h" event={"ID":"69833224-a389-47b6-a069-002612808b6b","Type":"ContainerStarted","Data":"8db3934bee521bc7c557786e91f722df76ef984bc960fa03264aa4d5f6bddb72"}
Nov 01 14:53:47.735813 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:53:47.735769    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-305290-788f694d7b-qpw5h" event={"ID":"69833224-a389-47b6-a069-002612808b6b","Type":"ContainerStarted","Data":"42bd62b2a67d4d1c09cefcd6b8bacf1f18f774cdee49780709d7905fb6b852be"}
Nov 01 14:53:47.736325 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:53:47.735958    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-305290-788f694d7b-qpw5h"
Nov 01 14:53:47.842991 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:53:47.842929    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-305290-788f694d7b-qpw5h" podStartSLOduration=2.830316183 podStartE2EDuration="3.842916211s" totalImagesPullingTime="1.012600028s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:53:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:53:47.84221748 +0000 UTC m=+2215.172080211" watchObservedRunningTime="2026-11-01 14:53:47.842916211 +0000 UTC m=+2215.172778947"
Nov 01 14:53:53.751483 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:53:53.751447    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-305290-788f694d7b-qpw5h"
Nov 01 14:53:57.504375 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:53:57.504333    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:53:59.017708 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:53:59.017674    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-e381b7-59fcfbb485-c7zvt"]
Nov 01 14:53:59.018110 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:53:59.017925    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e381b7-59fcfbb485-c7zvt" podUID="8e899083-bce2-46b2-a25e-975c8922102f" containerName="maas-api" containerID="cri-o://93efefaca1914bebdcbafd4c16c8db76280b1ba81809d9aed877351bbb079ce0" gracePeriod=30
Nov 01 14:53:59.352601 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:53:59.352571    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e381b7-59fcfbb485-c7zvt"
Nov 01 14:53:59.364856 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:53:59.364831    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e899083-bce2-46b2-a25e-975c8922102f-kube-api-access-qxcnn\" (UniqueName: \"kubernetes.io/projected/8e899083-bce2-46b2-a25e-975c8922102f-kube-api-access-qxcnn\") pod \"8e899083-bce2-46b2-a25e-975c8922102f\" (UID: \"8e899083-bce2-46b2-a25e-975c8922102f\") "
Nov 01 14:53:59.365014 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:53:59.364869    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e899083-bce2-46b2-a25e-975c8922102f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e899083-bce2-46b2-a25e-975c8922102f-maas-api-tls\") pod \"8e899083-bce2-46b2-a25e-975c8922102f\" (UID: \"8e899083-bce2-46b2-a25e-975c8922102f\") "
Nov 01 14:53:59.365014 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:53:59.364974    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e899083-bce2-46b2-a25e-975c8922102f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e899083-bce2-46b2-a25e-975c8922102f-openshift-service-ca\") pod \"8e899083-bce2-46b2-a25e-975c8922102f\" (UID: \"8e899083-bce2-46b2-a25e-975c8922102f\") "
Nov 01 14:53:59.365014 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:53:59.365010    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e899083-bce2-46b2-a25e-975c8922102f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e899083-bce2-46b2-a25e-975c8922102f-metrics-certs\") pod \"8e899083-bce2-46b2-a25e-975c8922102f\" (UID: \"8e899083-bce2-46b2-a25e-975c8922102f\") "
Nov 01 14:53:59.365578 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:53:59.365544    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e899083-bce2-46b2-a25e-975c8922102f-openshift-service-ca" pod "8e899083-bce2-46b2-a25e-975c8922102f" (UID: "8e899083-bce2-46b2-a25e-975c8922102f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:53:59.367893 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:53:59.367862    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e899083-bce2-46b2-a25e-975c8922102f-kube-api-access-qxcnn" pod "8e899083-bce2-46b2-a25e-975c8922102f" (UID: "8e899083-bce2-46b2-a25e-975c8922102f"). InnerVolumeSpecName "kube-api-access-qxcnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:53:59.368105 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:53:59.368084    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e899083-bce2-46b2-a25e-975c8922102f-metrics-certs" pod "8e899083-bce2-46b2-a25e-975c8922102f" (UID: "8e899083-bce2-46b2-a25e-975c8922102f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:53:59.369450 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:53:59.369429    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e899083-bce2-46b2-a25e-975c8922102f-maas-api-tls" pod "8e899083-bce2-46b2-a25e-975c8922102f" (UID: "8e899083-bce2-46b2-a25e-975c8922102f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:53:59.470199 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:53:59.469353    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxcnn\" (UniqueName: \"kubernetes.io/projected/8e899083-bce2-46b2-a25e-975c8922102f-kube-api-access-qxcnn\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:53:59.470199 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:53:59.469392    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e899083-bce2-46b2-a25e-975c8922102f-maas-api-tls\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:53:59.470199 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:53:59.469408    4993 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e899083-bce2-46b2-a25e-975c8922102f-openshift-service-ca\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:53:59.470199 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:53:59.469422    4993 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e899083-bce2-46b2-a25e-975c8922102f-metrics-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:53:59.818893 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:53:59.818860    4993 generic.go:356] "Generic (PLEG): container finished" podID="8e899083-bce2-46b2-a25e-975c8922102f" containerID="93efefaca1914bebdcbafd4c16c8db76280b1ba81809d9aed877351bbb079ce0" exitCode=0
Nov 01 14:53:59.819084 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:53:59.818934    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e381b7-59fcfbb485-c7zvt" event={"ID":"8e899083-bce2-46b2-a25e-975c8922102f","Type":"ContainerDied","Data":"93efefaca1914bebdcbafd4c16c8db76280b1ba81809d9aed877351bbb079ce0"}
Nov 01 14:53:59.819084 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:53:59.818973    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e381b7-59fcfbb485-c7zvt" event={"ID":"8e899083-bce2-46b2-a25e-975c8922102f","Type":"ContainerDied","Data":"84605b2da9225d699c8c359e8665d12886ee7535b6cb08d0ea0a0f4d11490787"}
Nov 01 14:53:59.819084 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:53:59.819023    4993 scope.go:122] "RemoveContainer" containerID="93efefaca1914bebdcbafd4c16c8db76280b1ba81809d9aed877351bbb079ce0"
Nov 01 14:53:59.819316 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:53:59.819157    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e381b7-59fcfbb485-c7zvt"
Nov 01 14:53:59.833078 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:53:59.833053    4993 scope.go:122] "RemoveContainer" containerID="93efefaca1914bebdcbafd4c16c8db76280b1ba81809d9aed877351bbb079ce0"
Nov 01 14:53:59.833367 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:53:59.833337    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93efefaca1914bebdcbafd4c16c8db76280b1ba81809d9aed877351bbb079ce0\": container with ID starting with 93efefaca1914bebdcbafd4c16c8db76280b1ba81809d9aed877351bbb079ce0 not found: ID does not exist" containerID="93efefaca1914bebdcbafd4c16c8db76280b1ba81809d9aed877351bbb079ce0"
Nov 01 14:53:59.833466 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:53:59.833376    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93efefaca1914bebdcbafd4c16c8db76280b1ba81809d9aed877351bbb079ce0"} err="failed to get container status \"93efefaca1914bebdcbafd4c16c8db76280b1ba81809d9aed877351bbb079ce0\": rpc error: code = NotFound desc = could not find container \"93efefaca1914bebdcbafd4c16c8db76280b1ba81809d9aed877351bbb079ce0\": container with ID starting with 93efefaca1914bebdcbafd4c16c8db76280b1ba81809d9aed877351bbb079ce0 not found: ID does not exist"
Nov 01 14:53:59.975286 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:53:59.975250    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-e381b7-59fcfbb485-c7zvt"]
Nov 01 14:54:00.021830 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:54:00.021789    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-e381b7-59fcfbb485-c7zvt"]
Nov 01 14:54:00.799446 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:54:00.799406    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e899083-bce2-46b2-a25e-975c8922102f" path="/var/lib/kubelet/pods/8e899083-bce2-46b2-a25e-975c8922102f/volumes"
Nov 01 14:54:23.598667 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:54:23.598631    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc"]
Nov 01 14:54:23.599082 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:54:23.598824    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc" podUID="fefc13c7-831b-45aa-b73c-69814cc7e5b4" containerName="istio-proxy" containerID="cri-o://938a947427a8543f64ed0ffe58b39891827dec8ed66a6a5babb6fbc166a373bc" gracePeriod=30
Nov 01 14:54:26.456569 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:54:26.456528    4993 patch_prober.go:29] interesting pod/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 14:54:26.457144 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:54:26.456631    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc" podUID="fefc13c7-831b-45aa-b73c-69814cc7e5b4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 14:54:27.503880 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:54:27.503844    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:54:28.871466 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:54:28.871400    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc"
Nov 01 14:54:28.940638 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:54:28.940595    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fefc13c7-831b-45aa-b73c-69814cc7e5b4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fefc13c7-831b-45aa-b73c-69814cc7e5b4-workload-certs\") pod \"fefc13c7-831b-45aa-b73c-69814cc7e5b4\" (UID: \"fefc13c7-831b-45aa-b73c-69814cc7e5b4\") "
Nov 01 14:54:28.940801 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:54:28.940648    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fefc13c7-831b-45aa-b73c-69814cc7e5b4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fefc13c7-831b-45aa-b73c-69814cc7e5b4-credential-socket\") pod \"fefc13c7-831b-45aa-b73c-69814cc7e5b4\" (UID: \"fefc13c7-831b-45aa-b73c-69814cc7e5b4\") "
Nov 01 14:54:28.940801 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:54:28.940687    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fefc13c7-831b-45aa-b73c-69814cc7e5b4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fefc13c7-831b-45aa-b73c-69814cc7e5b4-istiod-ca-cert\") pod \"fefc13c7-831b-45aa-b73c-69814cc7e5b4\" (UID: \"fefc13c7-831b-45aa-b73c-69814cc7e5b4\") "
Nov 01 14:54:28.940801 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:54:28.940718    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fefc13c7-831b-45aa-b73c-69814cc7e5b4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fefc13c7-831b-45aa-b73c-69814cc7e5b4-workload-socket\") pod \"fefc13c7-831b-45aa-b73c-69814cc7e5b4\" (UID: \"fefc13c7-831b-45aa-b73c-69814cc7e5b4\") "
Nov 01 14:54:28.940801 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:54:28.940759    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fefc13c7-831b-45aa-b73c-69814cc7e5b4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fefc13c7-831b-45aa-b73c-69814cc7e5b4-istio-envoy\") pod \"fefc13c7-831b-45aa-b73c-69814cc7e5b4\" (UID: \"fefc13c7-831b-45aa-b73c-69814cc7e5b4\") "
Nov 01 14:54:28.941047 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:54:28.940841    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fefc13c7-831b-45aa-b73c-69814cc7e5b4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fefc13c7-831b-45aa-b73c-69814cc7e5b4-istio-data\") pod \"fefc13c7-831b-45aa-b73c-69814cc7e5b4\" (UID: \"fefc13c7-831b-45aa-b73c-69814cc7e5b4\") "
Nov 01 14:54:28.941047 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:54:28.940858    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fefc13c7-831b-45aa-b73c-69814cc7e5b4-workload-certs" pod "fefc13c7-831b-45aa-b73c-69814cc7e5b4" (UID: "fefc13c7-831b-45aa-b73c-69814cc7e5b4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:54:28.941047 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:54:28.940915    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fefc13c7-831b-45aa-b73c-69814cc7e5b4-istio-token\" (UniqueName: \"kubernetes.io/projected/fefc13c7-831b-45aa-b73c-69814cc7e5b4-istio-token\") pod \"fefc13c7-831b-45aa-b73c-69814cc7e5b4\" (UID: \"fefc13c7-831b-45aa-b73c-69814cc7e5b4\") "
Nov 01 14:54:28.941047 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:54:28.940962    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fefc13c7-831b-45aa-b73c-69814cc7e5b4-credential-socket" pod "fefc13c7-831b-45aa-b73c-69814cc7e5b4" (UID: "fefc13c7-831b-45aa-b73c-69814cc7e5b4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:54:28.941047 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:54:28.940993    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fefc13c7-831b-45aa-b73c-69814cc7e5b4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fefc13c7-831b-45aa-b73c-69814cc7e5b4-istio-podinfo\") pod \"fefc13c7-831b-45aa-b73c-69814cc7e5b4\" (UID: \"fefc13c7-831b-45aa-b73c-69814cc7e5b4\") "
Nov 01 14:54:28.941047 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:54:28.941045    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fefc13c7-831b-45aa-b73c-69814cc7e5b4-kube-api-access-kqljm\" (UniqueName: \"kubernetes.io/projected/fefc13c7-831b-45aa-b73c-69814cc7e5b4-kube-api-access-kqljm\") pod \"fefc13c7-831b-45aa-b73c-69814cc7e5b4\" (UID: \"fefc13c7-831b-45aa-b73c-69814cc7e5b4\") "
Nov 01 14:54:28.941373 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:54:28.941083    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fefc13c7-831b-45aa-b73c-69814cc7e5b4-istiod-ca-cert" pod "fefc13c7-831b-45aa-b73c-69814cc7e5b4" (UID: "fefc13c7-831b-45aa-b73c-69814cc7e5b4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:54:28.941373 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:54:28.941210    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fefc13c7-831b-45aa-b73c-69814cc7e5b4-istio-data" pod "fefc13c7-831b-45aa-b73c-69814cc7e5b4" (UID: "fefc13c7-831b-45aa-b73c-69814cc7e5b4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:54:28.941373 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:54:28.941323    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fefc13c7-831b-45aa-b73c-69814cc7e5b4-workload-socket" pod "fefc13c7-831b-45aa-b73c-69814cc7e5b4" (UID: "fefc13c7-831b-45aa-b73c-69814cc7e5b4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:54:28.942000 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:54:28.941975    4993 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fefc13c7-831b-45aa-b73c-69814cc7e5b4-istiod-ca-cert\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:54:28.942112 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:54:28.942005    4993 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fefc13c7-831b-45aa-b73c-69814cc7e5b4-workload-socket\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:54:28.942112 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:54:28.942021    4993 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fefc13c7-831b-45aa-b73c-69814cc7e5b4-istio-data\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:54:28.942112 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:54:28.942035    4993 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fefc13c7-831b-45aa-b73c-69814cc7e5b4-workload-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:54:28.942112 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:54:28.942051    4993 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fefc13c7-831b-45aa-b73c-69814cc7e5b4-credential-socket\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:54:28.943133 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:54:28.943110    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fefc13c7-831b-45aa-b73c-69814cc7e5b4-istio-envoy" pod "fefc13c7-831b-45aa-b73c-69814cc7e5b4" (UID: "fefc13c7-831b-45aa-b73c-69814cc7e5b4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:54:28.943262 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:54:28.943238    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fefc13c7-831b-45aa-b73c-69814cc7e5b4-istio-podinfo" pod "fefc13c7-831b-45aa-b73c-69814cc7e5b4" (UID: "fefc13c7-831b-45aa-b73c-69814cc7e5b4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 14:54:28.943262 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:54:28.943242    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fefc13c7-831b-45aa-b73c-69814cc7e5b4-kube-api-access-kqljm" pod "fefc13c7-831b-45aa-b73c-69814cc7e5b4" (UID: "fefc13c7-831b-45aa-b73c-69814cc7e5b4"). InnerVolumeSpecName "kube-api-access-kqljm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:54:28.943356 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:54:28.943271    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fefc13c7-831b-45aa-b73c-69814cc7e5b4-istio-token" pod "fefc13c7-831b-45aa-b73c-69814cc7e5b4" (UID: "fefc13c7-831b-45aa-b73c-69814cc7e5b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:54:29.016999 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:54:29.016961    4993 generic.go:356] "Generic (PLEG): container finished" podID="fefc13c7-831b-45aa-b73c-69814cc7e5b4" containerID="938a947427a8543f64ed0ffe58b39891827dec8ed66a6a5babb6fbc166a373bc" exitCode=0
Nov 01 14:54:29.017276 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:54:29.017013    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc"
Nov 01 14:54:29.017276 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:54:29.017046    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc" event={"ID":"fefc13c7-831b-45aa-b73c-69814cc7e5b4","Type":"ContainerDied","Data":"938a947427a8543f64ed0ffe58b39891827dec8ed66a6a5babb6fbc166a373bc"}
Nov 01 14:54:29.017276 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:54:29.017113    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc" event={"ID":"fefc13c7-831b-45aa-b73c-69814cc7e5b4","Type":"ContainerDied","Data":"c2dc3942611b1c63caa413ab6ad3b89f60c6079cd3e5dd3224f4531dae0a3f06"}
Nov 01 14:54:29.017276 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:54:29.017144    4993 scope.go:122] "RemoveContainer" containerID="938a947427a8543f64ed0ffe58b39891827dec8ed66a6a5babb6fbc166a373bc"
Nov 01 14:54:29.028211 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:54:29.028150    4993 scope.go:122] "RemoveContainer" containerID="938a947427a8543f64ed0ffe58b39891827dec8ed66a6a5babb6fbc166a373bc"
Nov 01 14:54:29.028519 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:54:29.028496    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"938a947427a8543f64ed0ffe58b39891827dec8ed66a6a5babb6fbc166a373bc\": container with ID starting with 938a947427a8543f64ed0ffe58b39891827dec8ed66a6a5babb6fbc166a373bc not found: ID does not exist" containerID="938a947427a8543f64ed0ffe58b39891827dec8ed66a6a5babb6fbc166a373bc"
Nov 01 14:54:29.028601 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:54:29.028529    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"938a947427a8543f64ed0ffe58b39891827dec8ed66a6a5babb6fbc166a373bc"} err="failed to get container status \"938a947427a8543f64ed0ffe58b39891827dec8ed66a6a5babb6fbc166a373bc\": rpc error: code = NotFound desc = could not find container \"938a947427a8543f64ed0ffe58b39891827dec8ed66a6a5babb6fbc166a373bc\": container with ID starting with 938a947427a8543f64ed0ffe58b39891827dec8ed66a6a5babb6fbc166a373bc not found: ID does not exist"
Nov 01 14:54:29.043289 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:54:29.043263    4993 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fefc13c7-831b-45aa-b73c-69814cc7e5b4-istio-podinfo\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:54:29.043289 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:54:29.043289    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqljm\" (UniqueName: \"kubernetes.io/projected/fefc13c7-831b-45aa-b73c-69814cc7e5b4-kube-api-access-kqljm\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:54:29.043445 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:54:29.043304    4993 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fefc13c7-831b-45aa-b73c-69814cc7e5b4-istio-envoy\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:54:29.043445 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:54:29.043313    4993 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fefc13c7-831b-45aa-b73c-69814cc7e5b4-istio-token\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:54:29.134100 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:54:29.133994    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc"]
Nov 01 14:54:29.142145 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:54:29.142113    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-4pkbc"]
Nov 01 14:54:30.798017 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:54:30.797976    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fefc13c7-831b-45aa-b73c-69814cc7e5b4" path="/var/lib/kubelet/pods/fefc13c7-831b-45aa-b73c-69814cc7e5b4/volumes"
Nov 01 14:54:36.905954 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:54:36.905912    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-2048c1/ipp-route-2048c1-kserve-689d44696f-7p5jp"]
Nov 01 14:54:36.908394 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:54:36.906606    4993 cpu_manager.go:414] "containerMap: removing container" podUID="fefc13c7-831b-45aa-b73c-69814cc7e5b4" containerName="istio-proxy"
Nov 01 14:54:36.908394 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:54:36.906624    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fefc13c7-831b-45aa-b73c-69814cc7e5b4" containerName="istio-proxy"
Nov 01 14:54:36.908394 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:54:36.906664    4993 cpu_manager.go:414] "containerMap: removing container" podUID="8e899083-bce2-46b2-a25e-975c8922102f" containerName="maas-api"
Nov 01 14:54:36.908394 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:54:36.906670    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e899083-bce2-46b2-a25e-975c8922102f" containerName="maas-api"
Nov 01 14:54:36.908394 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:54:36.906742    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e899083-bce2-46b2-a25e-975c8922102f" containerName="maas-api"
Nov 01 14:54:36.908394 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:54:36.906753    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="fefc13c7-831b-45aa-b73c-69814cc7e5b4" containerName="istio-proxy"
Nov 01 14:54:36.909964 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:54:36.909944    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-2048c1/ipp-route-2048c1-kserve-689d44696f-7p5jp"
Nov 01 14:54:36.958753 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:54:36.958716    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-2048c1\"/\"openshift-service-ca.crt\""
Nov 01 14:54:37.004789 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:54:37.004748    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-2048c1\"/\"kube-root-ca.crt\""
Nov 01 14:54:37.004789 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:54:37.004753    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-2048c1\"/\"default-dockercfg-xl9sw\""
Nov 01 14:54:37.005132 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:54:37.004990    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-2048c1\"/\"ipp-route-2048c1-kserve-self-signed-certs\""
Nov 01 14:54:37.009651 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:54:37.009618    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-2048c1/ipp-route-2048c1-kserve-689d44696f-7p5jp"]
Nov 01 14:54:37.060054 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:54:37.059957    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwmbw\" (UniqueName: \"kubernetes.io/projected/94b390b0-5705-4584-a23b-1e19e7cb7ca3-kube-api-access-kwmbw\") pod \"ipp-route-2048c1-kserve-689d44696f-7p5jp\" (UID: \"94b390b0-5705-4584-a23b-1e19e7cb7ca3\") " pod="ai-tenant-e2e-ipp-a-2048c1/ipp-route-2048c1-kserve-689d44696f-7p5jp"
Nov 01 14:54:37.060401 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:54:37.060114    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94b390b0-5705-4584-a23b-1e19e7cb7ca3-dshm\") pod \"ipp-route-2048c1-kserve-689d44696f-7p5jp\" (UID: \"94b390b0-5705-4584-a23b-1e19e7cb7ca3\") " pod="ai-tenant-e2e-ipp-a-2048c1/ipp-route-2048c1-kserve-689d44696f-7p5jp"
Nov 01 14:54:37.060401 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:54:37.060262    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94b390b0-5705-4584-a23b-1e19e7cb7ca3-tls-certs\") pod \"ipp-route-2048c1-kserve-689d44696f-7p5jp\" (UID: \"94b390b0-5705-4584-a23b-1e19e7cb7ca3\") " pod="ai-tenant-e2e-ipp-a-2048c1/ipp-route-2048c1-kserve-689d44696f-7p5jp"
Nov 01 14:54:37.060401 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:54:37.060351    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94b390b0-5705-4584-a23b-1e19e7cb7ca3-tmp-dir\") pod \"ipp-route-2048c1-kserve-689d44696f-7p5jp\" (UID: \"94b390b0-5705-4584-a23b-1e19e7cb7ca3\") " pod="ai-tenant-e2e-ipp-a-2048c1/ipp-route-2048c1-kserve-689d44696f-7p5jp"
Nov 01 14:54:37.060639 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:54:37.060420    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94b390b0-5705-4584-a23b-1e19e7cb7ca3-home\") pod \"ipp-route-2048c1-kserve-689d44696f-7p5jp\" (UID: \"94b390b0-5705-4584-a23b-1e19e7cb7ca3\") " pod="ai-tenant-e2e-ipp-a-2048c1/ipp-route-2048c1-kserve-689d44696f-7p5jp"
Nov 01 14:54:37.060639 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:54:37.060444    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94b390b0-5705-4584-a23b-1e19e7cb7ca3-model-cache\") pod \"ipp-route-2048c1-kserve-689d44696f-7p5jp\" (UID: \"94b390b0-5705-4584-a23b-1e19e7cb7ca3\") " pod="ai-tenant-e2e-ipp-a-2048c1/ipp-route-2048c1-kserve-689d44696f-7p5jp"
Nov 01 14:54:37.161594 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:54:37.161488    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwmbw\" (UniqueName: \"kubernetes.io/projected/94b390b0-5705-4584-a23b-1e19e7cb7ca3-kube-api-access-kwmbw\") pod \"ipp-route-2048c1-kserve-689d44696f-7p5jp\" (UID: \"94b390b0-5705-4584-a23b-1e19e7cb7ca3\") " pod="ai-tenant-e2e-ipp-a-2048c1/ipp-route-2048c1-kserve-689d44696f-7p5jp"
Nov 01 14:54:37.161594 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:54:37.161570    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94b390b0-5705-4584-a23b-1e19e7cb7ca3-dshm\") pod \"ipp-route-2048c1-kserve-689d44696f-7p5jp\" (UID: \"94b390b0-5705-4584-a23b-1e19e7cb7ca3\") " pod="ai-tenant-e2e-ipp-a-2048c1/ipp-route-2048c1-kserve-689d44696f-7p5jp"
Nov 01 14:54:37.161830 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:54:37.161622    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94b390b0-5705-4584-a23b-1e19e7cb7ca3-tls-certs\") pod \"ipp-route-2048c1-kserve-689d44696f-7p5jp\" (UID: \"94b390b0-5705-4584-a23b-1e19e7cb7ca3\") " pod="ai-tenant-e2e-ipp-a-2048c1/ipp-route-2048c1-kserve-689d44696f-7p5jp"
Nov 01 14:54:37.161830 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:54:37.161674    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94b390b0-5705-4584-a23b-1e19e7cb7ca3-tmp-dir\") pod \"ipp-route-2048c1-kserve-689d44696f-7p5jp\" (UID: \"94b390b0-5705-4584-a23b-1e19e7cb7ca3\") " pod="ai-tenant-e2e-ipp-a-2048c1/ipp-route-2048c1-kserve-689d44696f-7p5jp"
Nov 01 14:54:37.161830 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:54:37.161717    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94b390b0-5705-4584-a23b-1e19e7cb7ca3-home\") pod \"ipp-route-2048c1-kserve-689d44696f-7p5jp\" (UID: \"94b390b0-5705-4584-a23b-1e19e7cb7ca3\") " pod="ai-tenant-e2e-ipp-a-2048c1/ipp-route-2048c1-kserve-689d44696f-7p5jp"
Nov 01 14:54:37.161830 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:54:37.161739    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94b390b0-5705-4584-a23b-1e19e7cb7ca3-model-cache\") pod \"ipp-route-2048c1-kserve-689d44696f-7p5jp\" (UID: \"94b390b0-5705-4584-a23b-1e19e7cb7ca3\") " pod="ai-tenant-e2e-ipp-a-2048c1/ipp-route-2048c1-kserve-689d44696f-7p5jp"
Nov 01 14:54:37.162249 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:54:37.162223    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94b390b0-5705-4584-a23b-1e19e7cb7ca3-model-cache\") pod \"ipp-route-2048c1-kserve-689d44696f-7p5jp\" (UID: \"94b390b0-5705-4584-a23b-1e19e7cb7ca3\") " pod="ai-tenant-e2e-ipp-a-2048c1/ipp-route-2048c1-kserve-689d44696f-7p5jp"
Nov 01 14:54:37.162556 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:54:37.162529    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94b390b0-5705-4584-a23b-1e19e7cb7ca3-tmp-dir\") pod \"ipp-route-2048c1-kserve-689d44696f-7p5jp\" (UID: \"94b390b0-5705-4584-a23b-1e19e7cb7ca3\") " pod="ai-tenant-e2e-ipp-a-2048c1/ipp-route-2048c1-kserve-689d44696f-7p5jp"
Nov 01 14:54:37.162954 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:54:37.162929    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94b390b0-5705-4584-a23b-1e19e7cb7ca3-home\") pod \"ipp-route-2048c1-kserve-689d44696f-7p5jp\" (UID: \"94b390b0-5705-4584-a23b-1e19e7cb7ca3\") " pod="ai-tenant-e2e-ipp-a-2048c1/ipp-route-2048c1-kserve-689d44696f-7p5jp"
Nov 01 14:54:37.164682 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:54:37.164659    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94b390b0-5705-4584-a23b-1e19e7cb7ca3-dshm\") pod \"ipp-route-2048c1-kserve-689d44696f-7p5jp\" (UID: \"94b390b0-5705-4584-a23b-1e19e7cb7ca3\") " pod="ai-tenant-e2e-ipp-a-2048c1/ipp-route-2048c1-kserve-689d44696f-7p5jp"
Nov 01 14:54:37.165277 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:54:37.165192    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94b390b0-5705-4584-a23b-1e19e7cb7ca3-tls-certs\") pod \"ipp-route-2048c1-kserve-689d44696f-7p5jp\" (UID: \"94b390b0-5705-4584-a23b-1e19e7cb7ca3\") " pod="ai-tenant-e2e-ipp-a-2048c1/ipp-route-2048c1-kserve-689d44696f-7p5jp"
Nov 01 14:54:37.214380 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:54:37.214350    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwmbw\" (UniqueName: \"kubernetes.io/projected/94b390b0-5705-4584-a23b-1e19e7cb7ca3-kube-api-access-kwmbw\") pod \"ipp-route-2048c1-kserve-689d44696f-7p5jp\" (UID: \"94b390b0-5705-4584-a23b-1e19e7cb7ca3\") " pod="ai-tenant-e2e-ipp-a-2048c1/ipp-route-2048c1-kserve-689d44696f-7p5jp"
Nov 01 14:54:37.222330 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:54:37.222304    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-2048c1/ipp-route-2048c1-kserve-689d44696f-7p5jp"
Nov 01 14:54:37.604003 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:54:37.603966    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-2048c1/ipp-route-2048c1-kserve-689d44696f-7p5jp"]
Nov 01 14:54:37.609485 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:54:37.609457    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod94b390b0_5705_4584_a23b_1e19e7cb7ca3.slice/crio-2c7437bd052813a797778a6a5c0aba05ad4f31cc48928701437203a8798e6546 WatchSource:0}: Error finding container 2c7437bd052813a797778a6a5c0aba05ad4f31cc48928701437203a8798e6546: Status 404 returned error can't find the container with id 2c7437bd052813a797778a6a5c0aba05ad4f31cc48928701437203a8798e6546
Nov 01 14:54:38.083828 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:54:38.083783    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-2048c1/ipp-route-2048c1-kserve-689d44696f-7p5jp" event={"ID":"94b390b0-5705-4584-a23b-1e19e7cb7ca3","Type":"ContainerStarted","Data":"193041fcd4004c00651f31f8d05aa076f90b711a4473d2a0f585701cd93176af"}
Nov 01 14:54:38.084325 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:54:38.083834    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-2048c1/ipp-route-2048c1-kserve-689d44696f-7p5jp" event={"ID":"94b390b0-5705-4584-a23b-1e19e7cb7ca3","Type":"ContainerStarted","Data":"2c7437bd052813a797778a6a5c0aba05ad4f31cc48928701437203a8798e6546"}
Nov 01 14:54:38.191614 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:54:38.191553    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-2048c1/ipp-route-2048c1-kserve-689d44696f-7p5jp" podStartSLOduration=2.1915388 podStartE2EDuration="2.1915388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:54:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:54:38.189889283 +0000 UTC m=+2265.519752021" watchObservedRunningTime="2026-11-01 14:54:38.1915388 +0000 UTC m=+2265.521401541"
Nov 01 14:54:38.222590 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:54:38.222558    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-2048c1/ipp-route-2048c1-kserve-689d44696f-7p5jp"
Nov 01 14:54:47.223141 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:54:47.223106    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-2048c1/ipp-route-2048c1-kserve-689d44696f-7p5jp"
Nov 01 14:54:47.235471 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:54:47.235441    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-2048c1/ipp-route-2048c1-kserve-689d44696f-7p5jp"
Nov 01 14:54:48.160534 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:54:48.160496    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-2048c1/ipp-route-2048c1-kserve-689d44696f-7p5jp"
Nov 01 14:54:57.500601 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:54:57.500558    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:55:20.475051 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:20.474965    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb"]
Nov 01 14:55:20.479762 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:20.479738    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb"
Nov 01 14:55:20.528223 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:20.528194    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-536a87-openshift-default-dockercfg-588rv\""
Nov 01 14:55:20.583377 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:20.583337    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb"]
Nov 01 14:55:20.620681 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:20.620638    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4196a39a-6cde-4b40-b2b2-c4792aceb17e-workload-socket\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb\" (UID: \"4196a39a-6cde-4b40-b2b2-c4792aceb17e\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb"
Nov 01 14:55:20.620860 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:20.620837    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4196a39a-6cde-4b40-b2b2-c4792aceb17e-istiod-ca-cert\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb\" (UID: \"4196a39a-6cde-4b40-b2b2-c4792aceb17e\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb"
Nov 01 14:55:20.620926 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:20.620913    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfncq\" (UniqueName: \"kubernetes.io/projected/4196a39a-6cde-4b40-b2b2-c4792aceb17e-kube-api-access-dfncq\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb\" (UID: \"4196a39a-6cde-4b40-b2b2-c4792aceb17e\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb"
Nov 01 14:55:20.621003 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:20.620983    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4196a39a-6cde-4b40-b2b2-c4792aceb17e-istio-data\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb\" (UID: \"4196a39a-6cde-4b40-b2b2-c4792aceb17e\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb"
Nov 01 14:55:20.621070 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:20.621046    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4196a39a-6cde-4b40-b2b2-c4792aceb17e-credential-socket\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb\" (UID: \"4196a39a-6cde-4b40-b2b2-c4792aceb17e\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb"
Nov 01 14:55:20.621130 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:20.621086    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4196a39a-6cde-4b40-b2b2-c4792aceb17e-istio-podinfo\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb\" (UID: \"4196a39a-6cde-4b40-b2b2-c4792aceb17e\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb"
Nov 01 14:55:20.621224 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:20.621135    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4196a39a-6cde-4b40-b2b2-c4792aceb17e-istio-token\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb\" (UID: \"4196a39a-6cde-4b40-b2b2-c4792aceb17e\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb"
Nov 01 14:55:20.621295 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:20.621262    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4196a39a-6cde-4b40-b2b2-c4792aceb17e-workload-certs\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb\" (UID: \"4196a39a-6cde-4b40-b2b2-c4792aceb17e\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb"
Nov 01 14:55:20.621350 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:20.621301    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4196a39a-6cde-4b40-b2b2-c4792aceb17e-istio-envoy\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb\" (UID: \"4196a39a-6cde-4b40-b2b2-c4792aceb17e\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb"
Nov 01 14:55:20.722968 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:20.722928    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4196a39a-6cde-4b40-b2b2-c4792aceb17e-workload-socket\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb\" (UID: \"4196a39a-6cde-4b40-b2b2-c4792aceb17e\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb"
Nov 01 14:55:20.723211 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:20.723057    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4196a39a-6cde-4b40-b2b2-c4792aceb17e-istiod-ca-cert\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb\" (UID: \"4196a39a-6cde-4b40-b2b2-c4792aceb17e\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb"
Nov 01 14:55:20.723211 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:20.723114    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfncq\" (UniqueName: \"kubernetes.io/projected/4196a39a-6cde-4b40-b2b2-c4792aceb17e-kube-api-access-dfncq\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb\" (UID: \"4196a39a-6cde-4b40-b2b2-c4792aceb17e\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb"
Nov 01 14:55:20.723211 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:20.723145    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4196a39a-6cde-4b40-b2b2-c4792aceb17e-istio-data\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb\" (UID: \"4196a39a-6cde-4b40-b2b2-c4792aceb17e\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb"
Nov 01 14:55:20.723211 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:20.723208    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4196a39a-6cde-4b40-b2b2-c4792aceb17e-credential-socket\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb\" (UID: \"4196a39a-6cde-4b40-b2b2-c4792aceb17e\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb"
Nov 01 14:55:20.723491 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:20.723327    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4196a39a-6cde-4b40-b2b2-c4792aceb17e-istio-podinfo\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb\" (UID: \"4196a39a-6cde-4b40-b2b2-c4792aceb17e\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb"
Nov 01 14:55:20.723491 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:20.723373    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4196a39a-6cde-4b40-b2b2-c4792aceb17e-istio-token\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb\" (UID: \"4196a39a-6cde-4b40-b2b2-c4792aceb17e\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb"
Nov 01 14:55:20.723491 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:20.723405    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4196a39a-6cde-4b40-b2b2-c4792aceb17e-workload-socket\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb\" (UID: \"4196a39a-6cde-4b40-b2b2-c4792aceb17e\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb"
Nov 01 14:55:20.723491 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:20.723466    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4196a39a-6cde-4b40-b2b2-c4792aceb17e-workload-certs\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb\" (UID: \"4196a39a-6cde-4b40-b2b2-c4792aceb17e\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb"
Nov 01 14:55:20.723693 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:20.723501    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4196a39a-6cde-4b40-b2b2-c4792aceb17e-istio-envoy\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb\" (UID: \"4196a39a-6cde-4b40-b2b2-c4792aceb17e\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb"
Nov 01 14:55:20.723693 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:20.723524    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4196a39a-6cde-4b40-b2b2-c4792aceb17e-credential-socket\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb\" (UID: \"4196a39a-6cde-4b40-b2b2-c4792aceb17e\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb"
Nov 01 14:55:20.723693 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:20.723682    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4196a39a-6cde-4b40-b2b2-c4792aceb17e-istio-data\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb\" (UID: \"4196a39a-6cde-4b40-b2b2-c4792aceb17e\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb"
Nov 01 14:55:20.723899 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:20.723870    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4196a39a-6cde-4b40-b2b2-c4792aceb17e-workload-certs\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb\" (UID: \"4196a39a-6cde-4b40-b2b2-c4792aceb17e\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb"
Nov 01 14:55:20.724092 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:20.724073    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4196a39a-6cde-4b40-b2b2-c4792aceb17e-istiod-ca-cert\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb\" (UID: \"4196a39a-6cde-4b40-b2b2-c4792aceb17e\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb"
Nov 01 14:55:20.725860 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:20.725798    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4196a39a-6cde-4b40-b2b2-c4792aceb17e-istio-envoy\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb\" (UID: \"4196a39a-6cde-4b40-b2b2-c4792aceb17e\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb"
Nov 01 14:55:20.725959 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:20.725876    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4196a39a-6cde-4b40-b2b2-c4792aceb17e-istio-podinfo\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb\" (UID: \"4196a39a-6cde-4b40-b2b2-c4792aceb17e\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb"
Nov 01 14:55:20.776234 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:20.776199    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4196a39a-6cde-4b40-b2b2-c4792aceb17e-istio-token\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb\" (UID: \"4196a39a-6cde-4b40-b2b2-c4792aceb17e\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb"
Nov 01 14:55:20.776387 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:20.776333    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfncq\" (UniqueName: \"kubernetes.io/projected/4196a39a-6cde-4b40-b2b2-c4792aceb17e-kube-api-access-dfncq\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb\" (UID: \"4196a39a-6cde-4b40-b2b2-c4792aceb17e\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb"
Nov 01 14:55:20.793953 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:20.793920    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb"
Nov 01 14:55:21.218345 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:21.218304    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb"]
Nov 01 14:55:21.223594 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:55:21.223564    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4196a39a_6cde_4b40_b2b2_c4792aceb17e.slice/crio-e2e6b07f5d475f23200e246afa786edb71207437dcc83c9dc94d9694a917a5e0 WatchSource:0}: Error finding container e2e6b07f5d475f23200e246afa786edb71207437dcc83c9dc94d9694a917a5e0: Status 404 returned error can't find the container with id e2e6b07f5d475f23200e246afa786edb71207437dcc83c9dc94d9694a917a5e0
Nov 01 14:55:21.226129 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:21.226095    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 14:55:21.226240 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:21.226176    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 14:55:21.226240 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:21.226209    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 14:55:21.379073 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:21.379031    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb" event={"ID":"4196a39a-6cde-4b40-b2b2-c4792aceb17e","Type":"ContainerStarted","Data":"39bc79121cbef6faffb528cc2034b9a978a4707319f0bcfda02e52ee6b334d97"}
Nov 01 14:55:21.379379 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:21.379099    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb" event={"ID":"4196a39a-6cde-4b40-b2b2-c4792aceb17e","Type":"ContainerStarted","Data":"e2e6b07f5d475f23200e246afa786edb71207437dcc83c9dc94d9694a917a5e0"}
Nov 01 14:55:21.489032 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:21.488978    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb" podStartSLOduration=1.488963055 podStartE2EDuration="1.488963055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:55:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:55:21.486588277 +0000 UTC m=+2308.816451015" watchObservedRunningTime="2026-11-01 14:55:21.488963055 +0000 UTC m=+2308.818825793"
Nov 01 14:55:21.794989 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:21.794909    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb"
Nov 01 14:55:22.796712 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:22.796675    4993 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Nov 01 14:55:22.797244 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:22.796727    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb" podUID="4196a39a-6cde-4b40-b2b2-c4792aceb17e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Nov 01 14:55:23.795305 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:23.795271    4993 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Nov 01 14:55:23.795480 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:23.795330    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb" podUID="4196a39a-6cde-4b40-b2b2-c4792aceb17e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Nov 01 14:55:24.795353 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:24.795310    4993 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Nov 01 14:55:24.795868 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:24.795374    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb" podUID="4196a39a-6cde-4b40-b2b2-c4792aceb17e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Nov 01 14:55:25.794867 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:25.794827    4993 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Nov 01 14:55:25.795072 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:25.794891    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb" podUID="4196a39a-6cde-4b40-b2b2-c4792aceb17e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Nov 01 14:55:26.794549 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:26.794516    4993 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Nov 01 14:55:26.794953 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:26.794570    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb" podUID="4196a39a-6cde-4b40-b2b2-c4792aceb17e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Nov 01 14:55:27.511474 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:55:27.511436    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:55:27.795299 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:27.795209    4993 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Nov 01 14:55:27.795299 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:27.795283    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb" podUID="4196a39a-6cde-4b40-b2b2-c4792aceb17e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Nov 01 14:55:28.990059 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:28.990023    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb"
Nov 01 14:55:28.991254 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:28.991220    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb"
Nov 01 14:55:28.991963 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:28.991944    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb"
Nov 01 14:55:38.415143 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:38.415105    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-praxis-a-f764c7/praxis-route-f764c7-kserve-74b89fbc9f-pfk5c"]
Nov 01 14:55:38.419790 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:38.419766    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-f764c7/praxis-route-f764c7-kserve-74b89fbc9f-pfk5c"
Nov 01 14:55:38.514499 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:38.514455    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-f764c7\"/\"default-dockercfg-9hrf9\""
Nov 01 14:55:38.514499 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:38.514455    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-f764c7\"/\"praxis-route-f764c7-kserve-self-signed-certs\""
Nov 01 14:55:38.514499 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:38.514458    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-f764c7\"/\"openshift-service-ca.crt\""
Nov 01 14:55:38.514499 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:38.514458    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-f764c7\"/\"kube-root-ca.crt\""
Nov 01 14:55:38.522710 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:38.522683    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-f764c7/praxis-route-f764c7-kserve-74b89fbc9f-pfk5c"]
Nov 01 14:55:38.605618 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:38.605579    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1bfefdf-9beb-4c02-b992-685efb079724-home\") pod \"praxis-route-f764c7-kserve-74b89fbc9f-pfk5c\" (UID: \"f1bfefdf-9beb-4c02-b992-685efb079724\") " pod="ai-tenant-e2e-praxis-a-f764c7/praxis-route-f764c7-kserve-74b89fbc9f-pfk5c"
Nov 01 14:55:38.605618 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:38.605614    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1bfefdf-9beb-4c02-b992-685efb079724-dshm\") pod \"praxis-route-f764c7-kserve-74b89fbc9f-pfk5c\" (UID: \"f1bfefdf-9beb-4c02-b992-685efb079724\") " pod="ai-tenant-e2e-praxis-a-f764c7/praxis-route-f764c7-kserve-74b89fbc9f-pfk5c"
Nov 01 14:55:38.605841 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:38.605684    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1bfefdf-9beb-4c02-b992-685efb079724-tls-certs\") pod \"praxis-route-f764c7-kserve-74b89fbc9f-pfk5c\" (UID: \"f1bfefdf-9beb-4c02-b992-685efb079724\") " pod="ai-tenant-e2e-praxis-a-f764c7/praxis-route-f764c7-kserve-74b89fbc9f-pfk5c"
Nov 01 14:55:38.605841 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:38.605781    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1bfefdf-9beb-4c02-b992-685efb079724-model-cache\") pod \"praxis-route-f764c7-kserve-74b89fbc9f-pfk5c\" (UID: \"f1bfefdf-9beb-4c02-b992-685efb079724\") " pod="ai-tenant-e2e-praxis-a-f764c7/praxis-route-f764c7-kserve-74b89fbc9f-pfk5c"
Nov 01 14:55:38.605841 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:38.605829    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1bfefdf-9beb-4c02-b992-685efb079724-tmp-dir\") pod \"praxis-route-f764c7-kserve-74b89fbc9f-pfk5c\" (UID: \"f1bfefdf-9beb-4c02-b992-685efb079724\") " pod="ai-tenant-e2e-praxis-a-f764c7/praxis-route-f764c7-kserve-74b89fbc9f-pfk5c"
Nov 01 14:55:38.605942 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:38.605845    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2928\" (UniqueName: \"kubernetes.io/projected/f1bfefdf-9beb-4c02-b992-685efb079724-kube-api-access-s2928\") pod \"praxis-route-f764c7-kserve-74b89fbc9f-pfk5c\" (UID: \"f1bfefdf-9beb-4c02-b992-685efb079724\") " pod="ai-tenant-e2e-praxis-a-f764c7/praxis-route-f764c7-kserve-74b89fbc9f-pfk5c"
Nov 01 14:55:38.707714 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:38.707626    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1bfefdf-9beb-4c02-b992-685efb079724-tmp-dir\") pod \"praxis-route-f764c7-kserve-74b89fbc9f-pfk5c\" (UID: \"f1bfefdf-9beb-4c02-b992-685efb079724\") " pod="ai-tenant-e2e-praxis-a-f764c7/praxis-route-f764c7-kserve-74b89fbc9f-pfk5c"
Nov 01 14:55:38.707714 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:38.707668    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2928\" (UniqueName: \"kubernetes.io/projected/f1bfefdf-9beb-4c02-b992-685efb079724-kube-api-access-s2928\") pod \"praxis-route-f764c7-kserve-74b89fbc9f-pfk5c\" (UID: \"f1bfefdf-9beb-4c02-b992-685efb079724\") " pod="ai-tenant-e2e-praxis-a-f764c7/praxis-route-f764c7-kserve-74b89fbc9f-pfk5c"
Nov 01 14:55:38.707906 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:38.707869    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1bfefdf-9beb-4c02-b992-685efb079724-home\") pod \"praxis-route-f764c7-kserve-74b89fbc9f-pfk5c\" (UID: \"f1bfefdf-9beb-4c02-b992-685efb079724\") " pod="ai-tenant-e2e-praxis-a-f764c7/praxis-route-f764c7-kserve-74b89fbc9f-pfk5c"
Nov 01 14:55:38.707906 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:38.707898    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1bfefdf-9beb-4c02-b992-685efb079724-dshm\") pod \"praxis-route-f764c7-kserve-74b89fbc9f-pfk5c\" (UID: \"f1bfefdf-9beb-4c02-b992-685efb079724\") " pod="ai-tenant-e2e-praxis-a-f764c7/praxis-route-f764c7-kserve-74b89fbc9f-pfk5c"
Nov 01 14:55:38.707997 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:38.707960    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1bfefdf-9beb-4c02-b992-685efb079724-tls-certs\") pod \"praxis-route-f764c7-kserve-74b89fbc9f-pfk5c\" (UID: \"f1bfefdf-9beb-4c02-b992-685efb079724\") " pod="ai-tenant-e2e-praxis-a-f764c7/praxis-route-f764c7-kserve-74b89fbc9f-pfk5c"
Nov 01 14:55:38.708047 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:38.707992    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1bfefdf-9beb-4c02-b992-685efb079724-model-cache\") pod \"praxis-route-f764c7-kserve-74b89fbc9f-pfk5c\" (UID: \"f1bfefdf-9beb-4c02-b992-685efb079724\") " pod="ai-tenant-e2e-praxis-a-f764c7/praxis-route-f764c7-kserve-74b89fbc9f-pfk5c"
Nov 01 14:55:38.708103 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:38.708081    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1bfefdf-9beb-4c02-b992-685efb079724-tmp-dir\") pod \"praxis-route-f764c7-kserve-74b89fbc9f-pfk5c\" (UID: \"f1bfefdf-9beb-4c02-b992-685efb079724\") " pod="ai-tenant-e2e-praxis-a-f764c7/praxis-route-f764c7-kserve-74b89fbc9f-pfk5c"
Nov 01 14:55:38.708337 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:38.708317    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1bfefdf-9beb-4c02-b992-685efb079724-model-cache\") pod \"praxis-route-f764c7-kserve-74b89fbc9f-pfk5c\" (UID: \"f1bfefdf-9beb-4c02-b992-685efb079724\") " pod="ai-tenant-e2e-praxis-a-f764c7/praxis-route-f764c7-kserve-74b89fbc9f-pfk5c"
Nov 01 14:55:38.708447 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:38.708319    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1bfefdf-9beb-4c02-b992-685efb079724-home\") pod \"praxis-route-f764c7-kserve-74b89fbc9f-pfk5c\" (UID: \"f1bfefdf-9beb-4c02-b992-685efb079724\") " pod="ai-tenant-e2e-praxis-a-f764c7/praxis-route-f764c7-kserve-74b89fbc9f-pfk5c"
Nov 01 14:55:38.710874 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:38.710846    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1bfefdf-9beb-4c02-b992-685efb079724-dshm\") pod \"praxis-route-f764c7-kserve-74b89fbc9f-pfk5c\" (UID: \"f1bfefdf-9beb-4c02-b992-685efb079724\") " pod="ai-tenant-e2e-praxis-a-f764c7/praxis-route-f764c7-kserve-74b89fbc9f-pfk5c"
Nov 01 14:55:38.711654 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:38.711631    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1bfefdf-9beb-4c02-b992-685efb079724-tls-certs\") pod \"praxis-route-f764c7-kserve-74b89fbc9f-pfk5c\" (UID: \"f1bfefdf-9beb-4c02-b992-685efb079724\") " pod="ai-tenant-e2e-praxis-a-f764c7/praxis-route-f764c7-kserve-74b89fbc9f-pfk5c"
Nov 01 14:55:38.761125 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:38.761088    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2928\" (UniqueName: \"kubernetes.io/projected/f1bfefdf-9beb-4c02-b992-685efb079724-kube-api-access-s2928\") pod \"praxis-route-f764c7-kserve-74b89fbc9f-pfk5c\" (UID: \"f1bfefdf-9beb-4c02-b992-685efb079724\") " pod="ai-tenant-e2e-praxis-a-f764c7/praxis-route-f764c7-kserve-74b89fbc9f-pfk5c"
Nov 01 14:55:39.033059 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:39.033027    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-f764c7/praxis-route-f764c7-kserve-74b89fbc9f-pfk5c"
Nov 01 14:55:39.418016 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:39.417919    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-f764c7/praxis-route-f764c7-kserve-74b89fbc9f-pfk5c"]
Nov 01 14:55:39.423412 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:55:39.423373    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf1bfefdf_9beb_4c02_b992_685efb079724.slice/crio-a23c60d36a99c0513e8d7b3e5ab0e850b6ad3761420294311fed5ccd9c6fc141 WatchSource:0}: Error finding container a23c60d36a99c0513e8d7b3e5ab0e850b6ad3761420294311fed5ccd9c6fc141: Status 404 returned error can't find the container with id a23c60d36a99c0513e8d7b3e5ab0e850b6ad3761420294311fed5ccd9c6fc141
Nov 01 14:55:39.514186 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:39.513552    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-f764c7/praxis-route-f764c7-kserve-74b89fbc9f-pfk5c" event={"ID":"f1bfefdf-9beb-4c02-b992-685efb079724","Type":"ContainerStarted","Data":"249f38c8c542de654addf7338ee6015761006a1681ba880c442f9a7687fcb7b8"}
Nov 01 14:55:39.514186 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:39.513604    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-f764c7/praxis-route-f764c7-kserve-74b89fbc9f-pfk5c" event={"ID":"f1bfefdf-9beb-4c02-b992-685efb079724","Type":"ContainerStarted","Data":"a23c60d36a99c0513e8d7b3e5ab0e850b6ad3761420294311fed5ccd9c6fc141"}
Nov 01 14:55:39.621379 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:39.621326    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-praxis-a-f764c7/praxis-route-f764c7-kserve-74b89fbc9f-pfk5c" podStartSLOduration=1.6213126199999999 podStartE2EDuration="1.62131262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:55:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:55:39.619254142 +0000 UTC m=+2326.949116890" watchObservedRunningTime="2026-11-01 14:55:39.62131262 +0000 UTC m=+2326.951175357"
Nov 01 14:55:40.033770 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:40.033739    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-f764c7/praxis-route-f764c7-kserve-74b89fbc9f-pfk5c"
Nov 01 14:55:49.033911 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:49.033876    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-praxis-a-f764c7/praxis-route-f764c7-kserve-74b89fbc9f-pfk5c"
Nov 01 14:55:49.046917 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:49.046892    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-praxis-a-f764c7/praxis-route-f764c7-kserve-74b89fbc9f-pfk5c"
Nov 01 14:55:49.602189 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:55:49.602131    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-f764c7/praxis-route-f764c7-kserve-74b89fbc9f-pfk5c"
Nov 01 14:55:57.501718 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:55:57.501676    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:56:27.503286 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:56:27.503244    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:56:36.037049 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:36.036967    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-drj66"]
Nov 01 14:56:36.037502 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:36.037195    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-drj66" podUID="de2b3fa1-8060-4944-b43a-1d9501fa8cda" containerName="istio-proxy" containerID="cri-o://a1a01cae9973a2f7065735c885ffaab8307168ec1b37bdc544b2e61bedc4b194" gracePeriod=30
Nov 01 14:56:36.083009 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:36.082976    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq"]
Nov 01 14:56:36.087261 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:36.087240    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq"
Nov 01 14:56:36.161205 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:36.161152    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq"]
Nov 01 14:56:36.214599 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:36.214565    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e-istio-envoy\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq\" (UID: \"bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq"
Nov 01 14:56:36.214778 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:36.214618    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e-istio-podinfo\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq\" (UID: \"bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq"
Nov 01 14:56:36.214778 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:36.214714    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e-credential-socket\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq\" (UID: \"bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq"
Nov 01 14:56:36.214868 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:36.214785    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e-workload-certs\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq\" (UID: \"bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq"
Nov 01 14:56:36.214868 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:36.214823    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e-workload-socket\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq\" (UID: \"bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq"
Nov 01 14:56:36.214947 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:36.214877    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e-istio-token\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq\" (UID: \"bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq"
Nov 01 14:56:36.214947 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:36.214927    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mk2z\" (UniqueName: \"kubernetes.io/projected/bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e-kube-api-access-7mk2z\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq\" (UID: \"bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq"
Nov 01 14:56:36.215060 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:36.215029    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e-istio-data\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq\" (UID: \"bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq"
Nov 01 14:56:36.215103 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:36.215061    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e-istiod-ca-cert\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq\" (UID: \"bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq"
Nov 01 14:56:36.315974 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:36.315877    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e-workload-certs\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq\" (UID: \"bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq"
Nov 01 14:56:36.315974 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:36.315956    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e-workload-socket\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq\" (UID: \"bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq"
Nov 01 14:56:36.316251 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:36.315986    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e-istio-token\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq\" (UID: \"bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq"
Nov 01 14:56:36.316251 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:36.316050    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mk2z\" (UniqueName: \"kubernetes.io/projected/bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e-kube-api-access-7mk2z\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq\" (UID: \"bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq"
Nov 01 14:56:36.316422 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:36.316397    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e-istio-data\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq\" (UID: \"bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq"
Nov 01 14:56:36.316507 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:36.316436    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e-istiod-ca-cert\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq\" (UID: \"bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq"
Nov 01 14:56:36.316507 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:36.316405    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e-workload-certs\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq\" (UID: \"bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq"
Nov 01 14:56:36.316616 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:36.316517    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e-istio-envoy\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq\" (UID: \"bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq"
Nov 01 14:56:36.316616 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:36.316578    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e-istio-podinfo\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq\" (UID: \"bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq"
Nov 01 14:56:36.316616 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:36.316527    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e-workload-socket\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq\" (UID: \"bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq"
Nov 01 14:56:36.316781 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:36.316620    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e-credential-socket\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq\" (UID: \"bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq"
Nov 01 14:56:36.316781 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:36.316650    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e-istio-data\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq\" (UID: \"bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq"
Nov 01 14:56:36.316949 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:36.316928    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e-credential-socket\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq\" (UID: \"bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq"
Nov 01 14:56:36.317218 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:36.317153    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e-istiod-ca-cert\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq\" (UID: \"bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq"
Nov 01 14:56:36.318885 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:36.318864    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e-istio-envoy\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq\" (UID: \"bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq"
Nov 01 14:56:36.319261 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:36.319241    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e-istio-podinfo\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq\" (UID: \"bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq"
Nov 01 14:56:36.370114 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:36.370079    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mk2z\" (UniqueName: \"kubernetes.io/projected/bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e-kube-api-access-7mk2z\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq\" (UID: \"bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq"
Nov 01 14:56:36.370340 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:36.370320    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e-istio-token\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq\" (UID: \"bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq"
Nov 01 14:56:36.403130 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:36.403100    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq"
Nov 01 14:56:36.831311 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:36.831278    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq"]
Nov 01 14:56:36.837086 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:56:36.837059    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc9b2c86_eb1c_4ac3_8270_7b2a4da4946e.slice/crio-29f89407d84fbd487c792a2b39b375397fccab82d64954e01df480aaa70d9228 WatchSource:0}: Error finding container 29f89407d84fbd487c792a2b39b375397fccab82d64954e01df480aaa70d9228: Status 404 returned error can't find the container with id 29f89407d84fbd487c792a2b39b375397fccab82d64954e01df480aaa70d9228
Nov 01 14:56:36.839745 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:36.839710    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 14:56:36.839830 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:36.839780    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 14:56:36.839830 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:36.839814    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 14:56:36.944009 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:36.943966    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq" event={"ID":"bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e","Type":"ContainerStarted","Data":"65a5002b2d0ed05bad4537af8054c47a895fe65569c006c25f46c2fdaf7926e1"}
Nov 01 14:56:36.944209 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:36.944070    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq" event={"ID":"bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e","Type":"ContainerStarted","Data":"29f89407d84fbd487c792a2b39b375397fccab82d64954e01df480aaa70d9228"}
Nov 01 14:56:37.056138 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:37.056088    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq" podStartSLOduration=1.056070726 podStartE2EDuration="1.056070726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:56:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:56:37.054485354 +0000 UTC m=+2384.384348091" watchObservedRunningTime="2026-11-01 14:56:37.056070726 +0000 UTC m=+2384.385933463"
Nov 01 14:56:37.302559 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:37.302515    4993 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-drj66 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 14:56:37.302734 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:37.302577    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-drj66" podUID="de2b3fa1-8060-4944-b43a-1d9501fa8cda" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 14:56:37.403802 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:37.403754    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq"
Nov 01 14:56:37.405616 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:37.405585    4993 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Nov 01 14:56:37.405764 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:37.405637    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq" podUID="bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Nov 01 14:56:38.404113 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:38.404075    4993 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Nov 01 14:56:38.404607 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:38.404144    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq" podUID="bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Nov 01 14:56:39.403634 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:39.403600    4993 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Nov 01 14:56:39.403823 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:39.403654    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq" podUID="bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Nov 01 14:56:40.404398 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:40.404359    4993 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Nov 01 14:56:40.404815 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:40.404440    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq" podUID="bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Nov 01 14:56:41.399974 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:41.399941    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-drj66"
Nov 01 14:56:41.404033 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:41.404004    4993 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Nov 01 14:56:41.404197 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:41.404057    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq" podUID="bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Nov 01 14:56:41.415280 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:41.415252    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/de2b3fa1-8060-4944-b43a-1d9501fa8cda-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de2b3fa1-8060-4944-b43a-1d9501fa8cda-istio-podinfo\") pod \"de2b3fa1-8060-4944-b43a-1d9501fa8cda\" (UID: \"de2b3fa1-8060-4944-b43a-1d9501fa8cda\") "
Nov 01 14:56:41.415624 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:41.415287    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de2b3fa1-8060-4944-b43a-1d9501fa8cda-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de2b3fa1-8060-4944-b43a-1d9501fa8cda-workload-certs\") pod \"de2b3fa1-8060-4944-b43a-1d9501fa8cda\" (UID: \"de2b3fa1-8060-4944-b43a-1d9501fa8cda\") "
Nov 01 14:56:41.415624 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:41.415312    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de2b3fa1-8060-4944-b43a-1d9501fa8cda-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de2b3fa1-8060-4944-b43a-1d9501fa8cda-istio-data\") pod \"de2b3fa1-8060-4944-b43a-1d9501fa8cda\" (UID: \"de2b3fa1-8060-4944-b43a-1d9501fa8cda\") "
Nov 01 14:56:41.415624 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:41.415343    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de2b3fa1-8060-4944-b43a-1d9501fa8cda-istio-token\" (UniqueName: \"kubernetes.io/projected/de2b3fa1-8060-4944-b43a-1d9501fa8cda-istio-token\") pod \"de2b3fa1-8060-4944-b43a-1d9501fa8cda\" (UID: \"de2b3fa1-8060-4944-b43a-1d9501fa8cda\") "
Nov 01 14:56:41.415624 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:41.415392    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de2b3fa1-8060-4944-b43a-1d9501fa8cda-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de2b3fa1-8060-4944-b43a-1d9501fa8cda-istio-envoy\") pod \"de2b3fa1-8060-4944-b43a-1d9501fa8cda\" (UID: \"de2b3fa1-8060-4944-b43a-1d9501fa8cda\") "
Nov 01 14:56:41.415624 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:41.415466    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de2b3fa1-8060-4944-b43a-1d9501fa8cda-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de2b3fa1-8060-4944-b43a-1d9501fa8cda-credential-socket\") pod \"de2b3fa1-8060-4944-b43a-1d9501fa8cda\" (UID: \"de2b3fa1-8060-4944-b43a-1d9501fa8cda\") "
Nov 01 14:56:41.415624 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:41.415559    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de2b3fa1-8060-4944-b43a-1d9501fa8cda-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de2b3fa1-8060-4944-b43a-1d9501fa8cda-istiod-ca-cert\") pod \"de2b3fa1-8060-4944-b43a-1d9501fa8cda\" (UID: \"de2b3fa1-8060-4944-b43a-1d9501fa8cda\") "
Nov 01 14:56:41.415624 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:41.415589    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de2b3fa1-8060-4944-b43a-1d9501fa8cda-workload-certs" pod "de2b3fa1-8060-4944-b43a-1d9501fa8cda" (UID: "de2b3fa1-8060-4944-b43a-1d9501fa8cda"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:56:41.415624 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:41.415604    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de2b3fa1-8060-4944-b43a-1d9501fa8cda-istio-data" pod "de2b3fa1-8060-4944-b43a-1d9501fa8cda" (UID: "de2b3fa1-8060-4944-b43a-1d9501fa8cda"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:56:41.416059 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:41.415633    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de2b3fa1-8060-4944-b43a-1d9501fa8cda-kube-api-access-2f8qd\" (UniqueName: \"kubernetes.io/projected/de2b3fa1-8060-4944-b43a-1d9501fa8cda-kube-api-access-2f8qd\") pod \"de2b3fa1-8060-4944-b43a-1d9501fa8cda\" (UID: \"de2b3fa1-8060-4944-b43a-1d9501fa8cda\") "
Nov 01 14:56:41.416059 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:41.415700    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de2b3fa1-8060-4944-b43a-1d9501fa8cda-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de2b3fa1-8060-4944-b43a-1d9501fa8cda-workload-socket\") pod \"de2b3fa1-8060-4944-b43a-1d9501fa8cda\" (UID: \"de2b3fa1-8060-4944-b43a-1d9501fa8cda\") "
Nov 01 14:56:41.416059 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:41.415814    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de2b3fa1-8060-4944-b43a-1d9501fa8cda-credential-socket" pod "de2b3fa1-8060-4944-b43a-1d9501fa8cda" (UID: "de2b3fa1-8060-4944-b43a-1d9501fa8cda"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:56:41.416059 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:41.415933    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de2b3fa1-8060-4944-b43a-1d9501fa8cda-istiod-ca-cert" pod "de2b3fa1-8060-4944-b43a-1d9501fa8cda" (UID: "de2b3fa1-8060-4944-b43a-1d9501fa8cda"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:56:41.416292 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:41.416070    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de2b3fa1-8060-4944-b43a-1d9501fa8cda-workload-socket" pod "de2b3fa1-8060-4944-b43a-1d9501fa8cda" (UID: "de2b3fa1-8060-4944-b43a-1d9501fa8cda"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:56:41.416395 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:41.416371    4993 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de2b3fa1-8060-4944-b43a-1d9501fa8cda-workload-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:56:41.416462 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:41.416403    4993 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de2b3fa1-8060-4944-b43a-1d9501fa8cda-istio-data\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:56:41.416462 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:41.416420    4993 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de2b3fa1-8060-4944-b43a-1d9501fa8cda-credential-socket\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:56:41.416462 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:41.416434    4993 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de2b3fa1-8060-4944-b43a-1d9501fa8cda-istiod-ca-cert\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:56:41.416462 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:41.416451    4993 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de2b3fa1-8060-4944-b43a-1d9501fa8cda-workload-socket\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:56:41.417720 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:41.417694    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de2b3fa1-8060-4944-b43a-1d9501fa8cda-istio-token" pod "de2b3fa1-8060-4944-b43a-1d9501fa8cda" (UID: "de2b3fa1-8060-4944-b43a-1d9501fa8cda"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:56:41.417720 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:41.417701    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de2b3fa1-8060-4944-b43a-1d9501fa8cda-istio-envoy" pod "de2b3fa1-8060-4944-b43a-1d9501fa8cda" (UID: "de2b3fa1-8060-4944-b43a-1d9501fa8cda"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:56:41.417896 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:41.417872    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/de2b3fa1-8060-4944-b43a-1d9501fa8cda-istio-podinfo" pod "de2b3fa1-8060-4944-b43a-1d9501fa8cda" (UID: "de2b3fa1-8060-4944-b43a-1d9501fa8cda"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 14:56:41.417997 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:41.417935    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de2b3fa1-8060-4944-b43a-1d9501fa8cda-kube-api-access-2f8qd" pod "de2b3fa1-8060-4944-b43a-1d9501fa8cda" (UID: "de2b3fa1-8060-4944-b43a-1d9501fa8cda"). InnerVolumeSpecName "kube-api-access-2f8qd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:56:41.517568 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:41.517535    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2f8qd\" (UniqueName: \"kubernetes.io/projected/de2b3fa1-8060-4944-b43a-1d9501fa8cda-kube-api-access-2f8qd\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:56:41.517568 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:41.517565    4993 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de2b3fa1-8060-4944-b43a-1d9501fa8cda-istio-podinfo\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:56:41.517568 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:41.517575    4993 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de2b3fa1-8060-4944-b43a-1d9501fa8cda-istio-token\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:56:41.517809 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:41.517583    4993 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de2b3fa1-8060-4944-b43a-1d9501fa8cda-istio-envoy\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:56:42.005565 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:42.005527    4993 generic.go:356] "Generic (PLEG): container finished" podID="de2b3fa1-8060-4944-b43a-1d9501fa8cda" containerID="a1a01cae9973a2f7065735c885ffaab8307168ec1b37bdc544b2e61bedc4b194" exitCode=0
Nov 01 14:56:42.005772 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:42.005643    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-drj66" event={"ID":"de2b3fa1-8060-4944-b43a-1d9501fa8cda","Type":"ContainerDied","Data":"a1a01cae9973a2f7065735c885ffaab8307168ec1b37bdc544b2e61bedc4b194"}
Nov 01 14:56:42.005772 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:42.005724    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-drj66" event={"ID":"de2b3fa1-8060-4944-b43a-1d9501fa8cda","Type":"ContainerDied","Data":"77bd80675c9e09a1a2b3e5092b463bb1955d19924a7984520eab16e3a0e7d56b"}
Nov 01 14:56:42.005772 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:42.005753    4993 scope.go:122] "RemoveContainer" containerID="a1a01cae9973a2f7065735c885ffaab8307168ec1b37bdc544b2e61bedc4b194"
Nov 01 14:56:42.005935 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:42.005921    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-drj66"
Nov 01 14:56:42.017702 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:42.017683    4993 scope.go:122] "RemoveContainer" containerID="a1a01cae9973a2f7065735c885ffaab8307168ec1b37bdc544b2e61bedc4b194"
Nov 01 14:56:42.017923 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:56:42.017903    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1a01cae9973a2f7065735c885ffaab8307168ec1b37bdc544b2e61bedc4b194\": container with ID starting with a1a01cae9973a2f7065735c885ffaab8307168ec1b37bdc544b2e61bedc4b194 not found: ID does not exist" containerID="a1a01cae9973a2f7065735c885ffaab8307168ec1b37bdc544b2e61bedc4b194"
Nov 01 14:56:42.017978 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:42.017931    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1a01cae9973a2f7065735c885ffaab8307168ec1b37bdc544b2e61bedc4b194"} err="failed to get container status \"a1a01cae9973a2f7065735c885ffaab8307168ec1b37bdc544b2e61bedc4b194\": rpc error: code = NotFound desc = could not find container \"a1a01cae9973a2f7065735c885ffaab8307168ec1b37bdc544b2e61bedc4b194\": container with ID starting with a1a01cae9973a2f7065735c885ffaab8307168ec1b37bdc544b2e61bedc4b194 not found: ID does not exist"
Nov 01 14:56:42.187308 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:42.187271    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-drj66"]
Nov 01 14:56:42.191917 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:42.191885    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-drj66"]
Nov 01 14:56:42.404321 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:42.404238    4993 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Nov 01 14:56:42.404321 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:42.404305    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq" podUID="bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Nov 01 14:56:42.798582 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:42.798548    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de2b3fa1-8060-4944-b43a-1d9501fa8cda" path="/var/lib/kubelet/pods/de2b3fa1-8060-4944-b43a-1d9501fa8cda/volumes"
Nov 01 14:56:43.403800 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:43.403761    4993 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Nov 01 14:56:43.403977 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:43.403839    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq" podUID="bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Nov 01 14:56:44.408794 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:44.408761    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq"
Nov 01 14:56:44.409203 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:44.409052    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq"
Nov 01 14:56:44.409654 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:44.409638    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq"
Nov 01 14:56:50.005759 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:50.005720    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-305290-788f694d7b-qpw5h"]
Nov 01 14:56:50.006243 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:50.005954    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-305290-788f694d7b-qpw5h" podUID="69833224-a389-47b6-a069-002612808b6b" containerName="maas-api" containerID="cri-o://42bd62b2a67d4d1c09cefcd6b8bacf1f18f774cdee49780709d7905fb6b852be" gracePeriod=30
Nov 01 14:56:50.316744 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:50.316715    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-305290-788f694d7b-qpw5h"
Nov 01 14:56:50.428396 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:50.428363    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69833224-a389-47b6-a069-002612808b6b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69833224-a389-47b6-a069-002612808b6b-maas-api-tls\") pod \"69833224-a389-47b6-a069-002612808b6b\" (UID: \"69833224-a389-47b6-a069-002612808b6b\") "
Nov 01 14:56:50.428623 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:50.428430    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69833224-a389-47b6-a069-002612808b6b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69833224-a389-47b6-a069-002612808b6b-openshift-service-ca\") pod \"69833224-a389-47b6-a069-002612808b6b\" (UID: \"69833224-a389-47b6-a069-002612808b6b\") "
Nov 01 14:56:50.428623 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:50.428484    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69833224-a389-47b6-a069-002612808b6b-kube-api-access-tctjn\" (UniqueName: \"kubernetes.io/projected/69833224-a389-47b6-a069-002612808b6b-kube-api-access-tctjn\") pod \"69833224-a389-47b6-a069-002612808b6b\" (UID: \"69833224-a389-47b6-a069-002612808b6b\") "
Nov 01 14:56:50.428623 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:50.428520    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69833224-a389-47b6-a069-002612808b6b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/69833224-a389-47b6-a069-002612808b6b-metrics-certs\") pod \"69833224-a389-47b6-a069-002612808b6b\" (UID: \"69833224-a389-47b6-a069-002612808b6b\") "
Nov 01 14:56:50.429146 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:50.428855    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69833224-a389-47b6-a069-002612808b6b-openshift-service-ca" pod "69833224-a389-47b6-a069-002612808b6b" (UID: "69833224-a389-47b6-a069-002612808b6b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:56:50.429146 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:50.429041    4993 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69833224-a389-47b6-a069-002612808b6b-openshift-service-ca\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:56:50.431034 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:50.430988    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69833224-a389-47b6-a069-002612808b6b-metrics-certs" pod "69833224-a389-47b6-a069-002612808b6b" (UID: "69833224-a389-47b6-a069-002612808b6b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:56:50.431264 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:50.431230    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69833224-a389-47b6-a069-002612808b6b-kube-api-access-tctjn" pod "69833224-a389-47b6-a069-002612808b6b" (UID: "69833224-a389-47b6-a069-002612808b6b"). InnerVolumeSpecName "kube-api-access-tctjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:56:50.431264 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:50.431241    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69833224-a389-47b6-a069-002612808b6b-maas-api-tls" pod "69833224-a389-47b6-a069-002612808b6b" (UID: "69833224-a389-47b6-a069-002612808b6b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:56:50.530107 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:50.530073    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69833224-a389-47b6-a069-002612808b6b-maas-api-tls\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:56:50.530107 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:50.530102    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tctjn\" (UniqueName: \"kubernetes.io/projected/69833224-a389-47b6-a069-002612808b6b-kube-api-access-tctjn\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:56:50.530107 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:50.530113    4993 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69833224-a389-47b6-a069-002612808b6b-metrics-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:56:51.082716 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:51.082624    4993 generic.go:356] "Generic (PLEG): container finished" podID="69833224-a389-47b6-a069-002612808b6b" containerID="42bd62b2a67d4d1c09cefcd6b8bacf1f18f774cdee49780709d7905fb6b852be" exitCode=0
Nov 01 14:56:51.082716 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:51.082670    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-305290-788f694d7b-qpw5h"
Nov 01 14:56:51.083209 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:51.082708    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-305290-788f694d7b-qpw5h" event={"ID":"69833224-a389-47b6-a069-002612808b6b","Type":"ContainerDied","Data":"42bd62b2a67d4d1c09cefcd6b8bacf1f18f774cdee49780709d7905fb6b852be"}
Nov 01 14:56:51.083209 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:51.082778    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-305290-788f694d7b-qpw5h" event={"ID":"69833224-a389-47b6-a069-002612808b6b","Type":"ContainerDied","Data":"8db3934bee521bc7c557786e91f722df76ef984bc960fa03264aa4d5f6bddb72"}
Nov 01 14:56:51.083209 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:51.082810    4993 scope.go:122] "RemoveContainer" containerID="42bd62b2a67d4d1c09cefcd6b8bacf1f18f774cdee49780709d7905fb6b852be"
Nov 01 14:56:51.092789 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:51.092757    4993 scope.go:122] "RemoveContainer" containerID="42bd62b2a67d4d1c09cefcd6b8bacf1f18f774cdee49780709d7905fb6b852be"
Nov 01 14:56:51.093085 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:56:51.093064    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42bd62b2a67d4d1c09cefcd6b8bacf1f18f774cdee49780709d7905fb6b852be\": container with ID starting with 42bd62b2a67d4d1c09cefcd6b8bacf1f18f774cdee49780709d7905fb6b852be not found: ID does not exist" containerID="42bd62b2a67d4d1c09cefcd6b8bacf1f18f774cdee49780709d7905fb6b852be"
Nov 01 14:56:51.093208 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:51.093093    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42bd62b2a67d4d1c09cefcd6b8bacf1f18f774cdee49780709d7905fb6b852be"} err="failed to get container status \"42bd62b2a67d4d1c09cefcd6b8bacf1f18f774cdee49780709d7905fb6b852be\": rpc error: code = NotFound desc = could not find container \"42bd62b2a67d4d1c09cefcd6b8bacf1f18f774cdee49780709d7905fb6b852be\": container with ID starting with 42bd62b2a67d4d1c09cefcd6b8bacf1f18f774cdee49780709d7905fb6b852be not found: ID does not exist"
Nov 01 14:56:51.238386 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:51.238354    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-305290-788f694d7b-qpw5h"]
Nov 01 14:56:51.284524 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:51.284490    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-305290-788f694d7b-qpw5h"]
Nov 01 14:56:52.796833 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:52.796797    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69833224-a389-47b6-a069-002612808b6b" path="/var/lib/kubelet/pods/69833224-a389-47b6-a069-002612808b6b/volumes"
Nov 01 14:56:53.135812 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:53.135728    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xmwtp_bb59051d-c2df-4362-a157-00f14f915e79/console-operator/3.log"
Nov 01 14:56:53.136296 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:56:53.136270    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xmwtp_bb59051d-c2df-4362-a157-00f14f915e79/console-operator/3.log"
Nov 01 14:56:57.505206 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:56:57.505147    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:57:02.746756 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:02.746713    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-2048c1/ipp-route-2048c1-kserve-689d44696f-7p5jp"]
Nov 01 14:57:17.767828 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:17.767740    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-2048c1/ipp-route-2048c1-kserve-689d44696f-7p5jp" podUID="94b390b0-5705-4584-a23b-1e19e7cb7ca3" containerName="main" containerID="cri-o://193041fcd4004c00651f31f8d05aa076f90b711a4473d2a0f585701cd93176af" gracePeriod=45
Nov 01 14:57:18.047056 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:18.047028    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-2048c1/ipp-route-2048c1-kserve-689d44696f-7p5jp"
Nov 01 14:57:18.162083 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:18.162035    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94b390b0-5705-4584-a23b-1e19e7cb7ca3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/94b390b0-5705-4584-a23b-1e19e7cb7ca3-dshm\") pod \"94b390b0-5705-4584-a23b-1e19e7cb7ca3\" (UID: \"94b390b0-5705-4584-a23b-1e19e7cb7ca3\") "
Nov 01 14:57:18.162083 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:18.162083    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94b390b0-5705-4584-a23b-1e19e7cb7ca3-tls-certs\" (UniqueName: \"kubernetes.io/secret/94b390b0-5705-4584-a23b-1e19e7cb7ca3-tls-certs\") pod \"94b390b0-5705-4584-a23b-1e19e7cb7ca3\" (UID: \"94b390b0-5705-4584-a23b-1e19e7cb7ca3\") "
Nov 01 14:57:18.162375 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:18.162214    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94b390b0-5705-4584-a23b-1e19e7cb7ca3-kube-api-access-kwmbw\" (UniqueName: \"kubernetes.io/projected/94b390b0-5705-4584-a23b-1e19e7cb7ca3-kube-api-access-kwmbw\") pod \"94b390b0-5705-4584-a23b-1e19e7cb7ca3\" (UID: \"94b390b0-5705-4584-a23b-1e19e7cb7ca3\") "
Nov 01 14:57:18.162375 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:18.162259    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94b390b0-5705-4584-a23b-1e19e7cb7ca3-home\" (UniqueName: \"kubernetes.io/empty-dir/94b390b0-5705-4584-a23b-1e19e7cb7ca3-home\") pod \"94b390b0-5705-4584-a23b-1e19e7cb7ca3\" (UID: \"94b390b0-5705-4584-a23b-1e19e7cb7ca3\") "
Nov 01 14:57:18.162375 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:18.162292    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94b390b0-5705-4584-a23b-1e19e7cb7ca3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94b390b0-5705-4584-a23b-1e19e7cb7ca3-model-cache\") pod \"94b390b0-5705-4584-a23b-1e19e7cb7ca3\" (UID: \"94b390b0-5705-4584-a23b-1e19e7cb7ca3\") "
Nov 01 14:57:18.162375 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:18.162354    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94b390b0-5705-4584-a23b-1e19e7cb7ca3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94b390b0-5705-4584-a23b-1e19e7cb7ca3-tmp-dir\") pod \"94b390b0-5705-4584-a23b-1e19e7cb7ca3\" (UID: \"94b390b0-5705-4584-a23b-1e19e7cb7ca3\") "
Nov 01 14:57:18.163121 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:18.163052    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94b390b0-5705-4584-a23b-1e19e7cb7ca3-model-cache" pod "94b390b0-5705-4584-a23b-1e19e7cb7ca3" (UID: "94b390b0-5705-4584-a23b-1e19e7cb7ca3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:57:18.163121 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:18.163063    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94b390b0-5705-4584-a23b-1e19e7cb7ca3-tmp-dir" pod "94b390b0-5705-4584-a23b-1e19e7cb7ca3" (UID: "94b390b0-5705-4584-a23b-1e19e7cb7ca3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:57:18.163596 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:18.163564    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94b390b0-5705-4584-a23b-1e19e7cb7ca3-home" pod "94b390b0-5705-4584-a23b-1e19e7cb7ca3" (UID: "94b390b0-5705-4584-a23b-1e19e7cb7ca3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:57:18.167372 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:18.167316    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94b390b0-5705-4584-a23b-1e19e7cb7ca3-dshm" pod "94b390b0-5705-4584-a23b-1e19e7cb7ca3" (UID: "94b390b0-5705-4584-a23b-1e19e7cb7ca3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:57:18.167372 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:18.167358    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94b390b0-5705-4584-a23b-1e19e7cb7ca3-kube-api-access-kwmbw" pod "94b390b0-5705-4584-a23b-1e19e7cb7ca3" (UID: "94b390b0-5705-4584-a23b-1e19e7cb7ca3"). InnerVolumeSpecName "kube-api-access-kwmbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:57:18.167566 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:18.167316    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94b390b0-5705-4584-a23b-1e19e7cb7ca3-tls-certs" pod "94b390b0-5705-4584-a23b-1e19e7cb7ca3" (UID: "94b390b0-5705-4584-a23b-1e19e7cb7ca3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:57:18.263799 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:18.263763    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwmbw\" (UniqueName: \"kubernetes.io/projected/94b390b0-5705-4584-a23b-1e19e7cb7ca3-kube-api-access-kwmbw\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:57:18.263799 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:18.263797    4993 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94b390b0-5705-4584-a23b-1e19e7cb7ca3-home\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:57:18.263995 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:18.263808    4993 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94b390b0-5705-4584-a23b-1e19e7cb7ca3-model-cache\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:57:18.263995 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:18.263818    4993 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94b390b0-5705-4584-a23b-1e19e7cb7ca3-tmp-dir\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:57:18.263995 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:18.263827    4993 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94b390b0-5705-4584-a23b-1e19e7cb7ca3-dshm\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:57:18.263995 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:18.263835    4993 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94b390b0-5705-4584-a23b-1e19e7cb7ca3-tls-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:57:18.275827 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:18.275733    4993 generic.go:356] "Generic (PLEG): container finished" podID="94b390b0-5705-4584-a23b-1e19e7cb7ca3" containerID="193041fcd4004c00651f31f8d05aa076f90b711a4473d2a0f585701cd93176af" exitCode=0
Nov 01 14:57:18.275827 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:18.275771    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-2048c1/ipp-route-2048c1-kserve-689d44696f-7p5jp"
Nov 01 14:57:18.276009 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:18.275814    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-2048c1/ipp-route-2048c1-kserve-689d44696f-7p5jp" event={"ID":"94b390b0-5705-4584-a23b-1e19e7cb7ca3","Type":"ContainerDied","Data":"193041fcd4004c00651f31f8d05aa076f90b711a4473d2a0f585701cd93176af"}
Nov 01 14:57:18.276009 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:18.275867    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-2048c1/ipp-route-2048c1-kserve-689d44696f-7p5jp" event={"ID":"94b390b0-5705-4584-a23b-1e19e7cb7ca3","Type":"ContainerDied","Data":"2c7437bd052813a797778a6a5c0aba05ad4f31cc48928701437203a8798e6546"}
Nov 01 14:57:18.276009 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:18.275890    4993 scope.go:122] "RemoveContainer" containerID="193041fcd4004c00651f31f8d05aa076f90b711a4473d2a0f585701cd93176af"
Nov 01 14:57:18.288559 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:18.288536    4993 scope.go:122] "RemoveContainer" containerID="193041fcd4004c00651f31f8d05aa076f90b711a4473d2a0f585701cd93176af"
Nov 01 14:57:18.288879 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:57:18.288854    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"193041fcd4004c00651f31f8d05aa076f90b711a4473d2a0f585701cd93176af\": container with ID starting with 193041fcd4004c00651f31f8d05aa076f90b711a4473d2a0f585701cd93176af not found: ID does not exist" containerID="193041fcd4004c00651f31f8d05aa076f90b711a4473d2a0f585701cd93176af"
Nov 01 14:57:18.288950 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:18.288892    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"193041fcd4004c00651f31f8d05aa076f90b711a4473d2a0f585701cd93176af"} err="failed to get container status \"193041fcd4004c00651f31f8d05aa076f90b711a4473d2a0f585701cd93176af\": rpc error: code = NotFound desc = could not find container \"193041fcd4004c00651f31f8d05aa076f90b711a4473d2a0f585701cd93176af\": container with ID starting with 193041fcd4004c00651f31f8d05aa076f90b711a4473d2a0f585701cd93176af not found: ID does not exist"
Nov 01 14:57:18.429060 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:18.429026    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-2048c1/ipp-route-2048c1-kserve-689d44696f-7p5jp"]
Nov 01 14:57:18.435701 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:18.435669    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-2048c1/ipp-route-2048c1-kserve-689d44696f-7p5jp"]
Nov 01 14:57:18.797045 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:18.797004    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94b390b0-5705-4584-a23b-1e19e7cb7ca3" path="/var/lib/kubelet/pods/94b390b0-5705-4584-a23b-1e19e7cb7ca3/volumes"
Nov 01 14:57:21.464089 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:21.463563    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl"]
Nov 01 14:57:21.476004 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:21.475410    4993 cpu_manager.go:414] "containerMap: removing container" podUID="94b390b0-5705-4584-a23b-1e19e7cb7ca3" containerName="main"
Nov 01 14:57:21.476004 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:21.475445    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94b390b0-5705-4584-a23b-1e19e7cb7ca3" containerName="main"
Nov 01 14:57:21.476004 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:21.475462    4993 cpu_manager.go:414] "containerMap: removing container" podUID="69833224-a389-47b6-a069-002612808b6b" containerName="maas-api"
Nov 01 14:57:21.476004 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:21.475469    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69833224-a389-47b6-a069-002612808b6b" containerName="maas-api"
Nov 01 14:57:21.476004 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:21.475508    4993 cpu_manager.go:414] "containerMap: removing container" podUID="de2b3fa1-8060-4944-b43a-1d9501fa8cda" containerName="istio-proxy"
Nov 01 14:57:21.476004 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:21.475517    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de2b3fa1-8060-4944-b43a-1d9501fa8cda" containerName="istio-proxy"
Nov 01 14:57:21.476004 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:21.475776    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="94b390b0-5705-4584-a23b-1e19e7cb7ca3" containerName="main"
Nov 01 14:57:21.476004 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:21.475798    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="69833224-a389-47b6-a069-002612808b6b" containerName="maas-api"
Nov 01 14:57:21.476004 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:21.475811    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="de2b3fa1-8060-4944-b43a-1d9501fa8cda" containerName="istio-proxy"
Nov 01 14:57:21.482138 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:21.482070    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl"
Nov 01 14:57:21.558961 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:21.558899    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl"]
Nov 01 14:57:21.612056 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:21.612019    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48-istio-podinfo\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl\" (UID: \"0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl"
Nov 01 14:57:21.612279 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:21.612082    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48-workload-certs\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl\" (UID: \"0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl"
Nov 01 14:57:21.612279 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:21.612228    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48-istio-data\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl\" (UID: \"0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl"
Nov 01 14:57:21.612405 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:21.612331    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48-istio-envoy\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl\" (UID: \"0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl"
Nov 01 14:57:21.612405 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:21.612356    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48-istio-token\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl\" (UID: \"0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl"
Nov 01 14:57:21.612505 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:21.612441    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48-workload-socket\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl\" (UID: \"0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl"
Nov 01 14:57:21.612505 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:21.612496    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48-istiod-ca-cert\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl\" (UID: \"0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl"
Nov 01 14:57:21.612607 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:21.612582    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48-credential-socket\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl\" (UID: \"0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl"
Nov 01 14:57:21.612665 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:21.612625    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmrmz\" (UniqueName: \"kubernetes.io/projected/0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48-kube-api-access-tmrmz\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl\" (UID: \"0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl"
Nov 01 14:57:21.713888 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:21.713843    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48-credential-socket\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl\" (UID: \"0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl"
Nov 01 14:57:21.714075 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:21.713895    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmrmz\" (UniqueName: \"kubernetes.io/projected/0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48-kube-api-access-tmrmz\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl\" (UID: \"0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl"
Nov 01 14:57:21.714075 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:21.713992    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48-istio-podinfo\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl\" (UID: \"0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl"
Nov 01 14:57:21.714237 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:21.714071    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48-workload-certs\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl\" (UID: \"0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl"
Nov 01 14:57:21.714237 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:21.714110    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48-istio-data\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl\" (UID: \"0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl"
Nov 01 14:57:21.714237 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:21.714213    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48-istio-envoy\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl\" (UID: \"0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl"
Nov 01 14:57:21.714408 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:21.714242    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48-istio-token\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl\" (UID: \"0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl"
Nov 01 14:57:21.714408 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:21.714308    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48-workload-socket\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl\" (UID: \"0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl"
Nov 01 14:57:21.714408 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:21.714348    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48-credential-socket\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl\" (UID: \"0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl"
Nov 01 14:57:21.714408 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:21.714354    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48-istiod-ca-cert\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl\" (UID: \"0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl"
Nov 01 14:57:21.714621 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:21.714556    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48-istio-data\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl\" (UID: \"0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl"
Nov 01 14:57:21.714889 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:21.714859    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48-workload-socket\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl\" (UID: \"0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl"
Nov 01 14:57:21.714997 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:21.714947    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48-workload-certs\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl\" (UID: \"0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl"
Nov 01 14:57:21.715117 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:21.715091    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48-istiod-ca-cert\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl\" (UID: \"0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl"
Nov 01 14:57:21.716548 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:21.716523    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48-istio-envoy\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl\" (UID: \"0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl"
Nov 01 14:57:21.716921 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:21.716897    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48-istio-podinfo\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl\" (UID: \"0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl"
Nov 01 14:57:21.769046 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:21.769015    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmrmz\" (UniqueName: \"kubernetes.io/projected/0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48-kube-api-access-tmrmz\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl\" (UID: \"0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl"
Nov 01 14:57:21.769231 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:21.769020    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48-istio-token\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl\" (UID: \"0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl"
Nov 01 14:57:21.805051 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:21.805016    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl"
Nov 01 14:57:22.308266 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:22.293112    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl"]
Nov 01 14:57:22.376769 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:22.375183    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 14:57:22.376769 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:22.375286    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 14:57:22.376769 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:22.375332    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 14:57:23.417023 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:23.416980    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl" event={"ID":"0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48","Type":"ContainerStarted","Data":"5cc2aa36c36438f325003aa99c9fa79d127cd36a8f67ee1aa8b9788a5aa34ba9"}
Nov 01 14:57:23.417505 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:23.417089    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl" event={"ID":"0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48","Type":"ContainerStarted","Data":"52deff652672642d3fb2a6a47ea34f25b386720653068ddfd64087e404ba97f6"}
Nov 01 14:57:23.639656 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:23.639603    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl" podStartSLOduration=2.639588621 podStartE2EDuration="2.639588621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:57:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:57:23.590665394 +0000 UTC m=+2430.920528133" watchObservedRunningTime="2026-11-01 14:57:23.639588621 +0000 UTC m=+2430.969451357"
Nov 01 14:57:23.806209 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:23.805532    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl"
Nov 01 14:57:23.807813 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:23.807786    4993 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 01 14:57:23.808012 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:23.807988    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl" podUID="0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 01 14:57:24.805513 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:24.805479    4993 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 01 14:57:24.805917 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:24.805547    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl" podUID="0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 01 14:57:25.806163 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:25.806118    4993 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 01 14:57:25.806596 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:25.806208    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl" podUID="0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 01 14:57:26.806232 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:26.806191    4993 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 01 14:57:26.806623 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:26.806258    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl" podUID="0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 01 14:57:27.503245 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:57:27.503206    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:57:27.805451 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:27.805367    4993 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 01 14:57:27.805451 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:27.805416    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl" podUID="0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 01 14:57:28.806030 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:28.805995    4993 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 01 14:57:28.806460 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:28.806052    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl" podUID="0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 01 14:57:29.878115 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:29.878073    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl"
Nov 01 14:57:29.878584 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:29.878536    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl"
Nov 01 14:57:29.879204 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:29.879183    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl"
Nov 01 14:57:49.778238 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:57:49.778197    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-f764c7/praxis-route-f764c7-kserve-74b89fbc9f-pfk5c"]
Nov 01 14:57:57.501458 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:57:57.501418    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:58:04.799440 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:04.799354    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-praxis-a-f764c7/praxis-route-f764c7-kserve-74b89fbc9f-pfk5c" podUID="f1bfefdf-9beb-4c02-b992-685efb079724" containerName="main" containerID="cri-o://249f38c8c542de654addf7338ee6015761006a1681ba880c442f9a7687fcb7b8" gracePeriod=45
Nov 01 14:58:05.094177 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:05.094136    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-f764c7/praxis-route-f764c7-kserve-74b89fbc9f-pfk5c"
Nov 01 14:58:05.137889 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:05.137857    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1bfefdf-9beb-4c02-b992-685efb079724-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1bfefdf-9beb-4c02-b992-685efb079724-tmp-dir\") pod \"f1bfefdf-9beb-4c02-b992-685efb079724\" (UID: \"f1bfefdf-9beb-4c02-b992-685efb079724\") "
Nov 01 14:58:05.138042 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:05.137949    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1bfefdf-9beb-4c02-b992-685efb079724-tls-certs\" (UniqueName: \"kubernetes.io/secret/f1bfefdf-9beb-4c02-b992-685efb079724-tls-certs\") pod \"f1bfefdf-9beb-4c02-b992-685efb079724\" (UID: \"f1bfefdf-9beb-4c02-b992-685efb079724\") "
Nov 01 14:58:05.138098 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:05.138037    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1bfefdf-9beb-4c02-b992-685efb079724-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1bfefdf-9beb-4c02-b992-685efb079724-model-cache\") pod \"f1bfefdf-9beb-4c02-b992-685efb079724\" (UID: \"f1bfefdf-9beb-4c02-b992-685efb079724\") "
Nov 01 14:58:05.138098 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:05.138093    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1bfefdf-9beb-4c02-b992-685efb079724-home\" (UniqueName: \"kubernetes.io/empty-dir/f1bfefdf-9beb-4c02-b992-685efb079724-home\") pod \"f1bfefdf-9beb-4c02-b992-685efb079724\" (UID: \"f1bfefdf-9beb-4c02-b992-685efb079724\") "
Nov 01 14:58:05.138258 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:05.138097    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1bfefdf-9beb-4c02-b992-685efb079724-tmp-dir" pod "f1bfefdf-9beb-4c02-b992-685efb079724" (UID: "f1bfefdf-9beb-4c02-b992-685efb079724"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:58:05.138258 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:05.138238    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1bfefdf-9beb-4c02-b992-685efb079724-kube-api-access-s2928\" (UniqueName: \"kubernetes.io/projected/f1bfefdf-9beb-4c02-b992-685efb079724-kube-api-access-s2928\") pod \"f1bfefdf-9beb-4c02-b992-685efb079724\" (UID: \"f1bfefdf-9beb-4c02-b992-685efb079724\") "
Nov 01 14:58:05.138368 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:05.138313    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1bfefdf-9beb-4c02-b992-685efb079724-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1bfefdf-9beb-4c02-b992-685efb079724-dshm\") pod \"f1bfefdf-9beb-4c02-b992-685efb079724\" (UID: \"f1bfefdf-9beb-4c02-b992-685efb079724\") "
Nov 01 14:58:05.138368 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:05.138339    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1bfefdf-9beb-4c02-b992-685efb079724-model-cache" pod "f1bfefdf-9beb-4c02-b992-685efb079724" (UID: "f1bfefdf-9beb-4c02-b992-685efb079724"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:58:05.138900 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:05.138367    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1bfefdf-9beb-4c02-b992-685efb079724-home" pod "f1bfefdf-9beb-4c02-b992-685efb079724" (UID: "f1bfefdf-9beb-4c02-b992-685efb079724"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:58:05.138900 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:05.138751    4993 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1bfefdf-9beb-4c02-b992-685efb079724-model-cache\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:58:05.138900 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:05.138766    4993 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1bfefdf-9beb-4c02-b992-685efb079724-home\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:58:05.138900 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:05.138775    4993 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1bfefdf-9beb-4c02-b992-685efb079724-tmp-dir\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:58:05.140520 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:05.140498    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1bfefdf-9beb-4c02-b992-685efb079724-dshm" pod "f1bfefdf-9beb-4c02-b992-685efb079724" (UID: "f1bfefdf-9beb-4c02-b992-685efb079724"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:58:05.140520 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:05.140508    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1bfefdf-9beb-4c02-b992-685efb079724-tls-certs" pod "f1bfefdf-9beb-4c02-b992-685efb079724" (UID: "f1bfefdf-9beb-4c02-b992-685efb079724"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:58:05.140708 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:05.140689    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1bfefdf-9beb-4c02-b992-685efb079724-kube-api-access-s2928" pod "f1bfefdf-9beb-4c02-b992-685efb079724" (UID: "f1bfefdf-9beb-4c02-b992-685efb079724"). InnerVolumeSpecName "kube-api-access-s2928". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:58:05.252202 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:05.241097    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2928\" (UniqueName: \"kubernetes.io/projected/f1bfefdf-9beb-4c02-b992-685efb079724-kube-api-access-s2928\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:58:05.252202 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:05.241134    4993 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1bfefdf-9beb-4c02-b992-685efb079724-dshm\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:58:05.252202 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:05.241159    4993 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1bfefdf-9beb-4c02-b992-685efb079724-tls-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:58:05.717949 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:05.717914    4993 generic.go:356] "Generic (PLEG): container finished" podID="f1bfefdf-9beb-4c02-b992-685efb079724" containerID="249f38c8c542de654addf7338ee6015761006a1681ba880c442f9a7687fcb7b8" exitCode=0
Nov 01 14:58:05.718148 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:05.717979    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-f764c7/praxis-route-f764c7-kserve-74b89fbc9f-pfk5c"
Nov 01 14:58:05.718148 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:05.718000    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-f764c7/praxis-route-f764c7-kserve-74b89fbc9f-pfk5c" event={"ID":"f1bfefdf-9beb-4c02-b992-685efb079724","Type":"ContainerDied","Data":"249f38c8c542de654addf7338ee6015761006a1681ba880c442f9a7687fcb7b8"}
Nov 01 14:58:05.718148 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:05.718047    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-f764c7/praxis-route-f764c7-kserve-74b89fbc9f-pfk5c" event={"ID":"f1bfefdf-9beb-4c02-b992-685efb079724","Type":"ContainerDied","Data":"a23c60d36a99c0513e8d7b3e5ab0e850b6ad3761420294311fed5ccd9c6fc141"}
Nov 01 14:58:05.718148 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:05.718065    4993 scope.go:122] "RemoveContainer" containerID="249f38c8c542de654addf7338ee6015761006a1681ba880c442f9a7687fcb7b8"
Nov 01 14:58:05.728197 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:05.728177    4993 scope.go:122] "RemoveContainer" containerID="249f38c8c542de654addf7338ee6015761006a1681ba880c442f9a7687fcb7b8"
Nov 01 14:58:05.728483 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:58:05.728459    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"249f38c8c542de654addf7338ee6015761006a1681ba880c442f9a7687fcb7b8\": container with ID starting with 249f38c8c542de654addf7338ee6015761006a1681ba880c442f9a7687fcb7b8 not found: ID does not exist" containerID="249f38c8c542de654addf7338ee6015761006a1681ba880c442f9a7687fcb7b8"
Nov 01 14:58:05.728557 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:05.728494    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"249f38c8c542de654addf7338ee6015761006a1681ba880c442f9a7687fcb7b8"} err="failed to get container status \"249f38c8c542de654addf7338ee6015761006a1681ba880c442f9a7687fcb7b8\": rpc error: code = NotFound desc = could not find container \"249f38c8c542de654addf7338ee6015761006a1681ba880c442f9a7687fcb7b8\": container with ID starting with 249f38c8c542de654addf7338ee6015761006a1681ba880c442f9a7687fcb7b8 not found: ID does not exist"
Nov 01 14:58:05.871758 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:05.871725    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-f764c7/praxis-route-f764c7-kserve-74b89fbc9f-pfk5c"]
Nov 01 14:58:05.917697 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:05.917657    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-praxis-a-f764c7/praxis-route-f764c7-kserve-74b89fbc9f-pfk5c"]
Nov 01 14:58:06.798748 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:06.798707    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1bfefdf-9beb-4c02-b992-685efb079724" path="/var/lib/kubelet/pods/f1bfefdf-9beb-4c02-b992-685efb079724/volumes"
Nov 01 14:58:25.623624 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:25.623586    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl"]
Nov 01 14:58:25.624120 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:25.624105    4993 cpu_manager.go:414] "containerMap: removing container" podUID="f1bfefdf-9beb-4c02-b992-685efb079724" containerName="main"
Nov 01 14:58:25.624195 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:25.624122    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1bfefdf-9beb-4c02-b992-685efb079724" containerName="main"
Nov 01 14:58:25.624255 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:25.624245    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1bfefdf-9beb-4c02-b992-685efb079724" containerName="main"
Nov 01 14:58:25.627786 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:25.627768    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl"
Nov 01 14:58:25.654646 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:25.654608    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfdb4\" (UniqueName: \"kubernetes.io/projected/4e1cf19c-563c-45d0-9a6a-a10fdb869f4d-kube-api-access-cfdb4\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl\" (UID: \"4e1cf19c-563c-45d0-9a6a-a10fdb869f4d\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl"
Nov 01 14:58:25.654762 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:25.654725    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e1cf19c-563c-45d0-9a6a-a10fdb869f4d-istio-data\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl\" (UID: \"4e1cf19c-563c-45d0-9a6a-a10fdb869f4d\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl"
Nov 01 14:58:25.654762 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:25.654752    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e1cf19c-563c-45d0-9a6a-a10fdb869f4d-istio-token\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl\" (UID: \"4e1cf19c-563c-45d0-9a6a-a10fdb869f4d\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl"
Nov 01 14:58:25.654848 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:25.654808    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e1cf19c-563c-45d0-9a6a-a10fdb869f4d-istio-podinfo\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl\" (UID: \"4e1cf19c-563c-45d0-9a6a-a10fdb869f4d\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl"
Nov 01 14:58:25.654848 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:25.654839    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e1cf19c-563c-45d0-9a6a-a10fdb869f4d-workload-socket\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl\" (UID: \"4e1cf19c-563c-45d0-9a6a-a10fdb869f4d\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl"
Nov 01 14:58:25.654981 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:25.654959    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e1cf19c-563c-45d0-9a6a-a10fdb869f4d-istiod-ca-cert\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl\" (UID: \"4e1cf19c-563c-45d0-9a6a-a10fdb869f4d\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl"
Nov 01 14:58:25.655019 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:25.654996    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e1cf19c-563c-45d0-9a6a-a10fdb869f4d-workload-certs\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl\" (UID: \"4e1cf19c-563c-45d0-9a6a-a10fdb869f4d\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl"
Nov 01 14:58:25.655076 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:25.655041    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e1cf19c-563c-45d0-9a6a-a10fdb869f4d-credential-socket\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl\" (UID: \"4e1cf19c-563c-45d0-9a6a-a10fdb869f4d\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl"
Nov 01 14:58:25.655122 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:25.655077    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e1cf19c-563c-45d0-9a6a-a10fdb869f4d-istio-envoy\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl\" (UID: \"4e1cf19c-563c-45d0-9a6a-a10fdb869f4d\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl"
Nov 01 14:58:25.745195 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:25.745145    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl"]
Nov 01 14:58:25.755787 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:25.755755    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e1cf19c-563c-45d0-9a6a-a10fdb869f4d-istiod-ca-cert\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl\" (UID: \"4e1cf19c-563c-45d0-9a6a-a10fdb869f4d\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl"
Nov 01 14:58:25.755957 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:25.755803    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e1cf19c-563c-45d0-9a6a-a10fdb869f4d-workload-certs\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl\" (UID: \"4e1cf19c-563c-45d0-9a6a-a10fdb869f4d\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl"
Nov 01 14:58:25.755957 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:25.755836    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e1cf19c-563c-45d0-9a6a-a10fdb869f4d-credential-socket\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl\" (UID: \"4e1cf19c-563c-45d0-9a6a-a10fdb869f4d\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl"
Nov 01 14:58:25.755957 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:25.755864    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e1cf19c-563c-45d0-9a6a-a10fdb869f4d-istio-envoy\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl\" (UID: \"4e1cf19c-563c-45d0-9a6a-a10fdb869f4d\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl"
Nov 01 14:58:25.755957 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:25.755918    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfdb4\" (UniqueName: \"kubernetes.io/projected/4e1cf19c-563c-45d0-9a6a-a10fdb869f4d-kube-api-access-cfdb4\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl\" (UID: \"4e1cf19c-563c-45d0-9a6a-a10fdb869f4d\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl"
Nov 01 14:58:25.756193 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:25.756004    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e1cf19c-563c-45d0-9a6a-a10fdb869f4d-istio-data\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl\" (UID: \"4e1cf19c-563c-45d0-9a6a-a10fdb869f4d\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl"
Nov 01 14:58:25.756193 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:25.756051    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e1cf19c-563c-45d0-9a6a-a10fdb869f4d-istio-token\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl\" (UID: \"4e1cf19c-563c-45d0-9a6a-a10fdb869f4d\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl"
Nov 01 14:58:25.756428 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:25.756399    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e1cf19c-563c-45d0-9a6a-a10fdb869f4d-istio-podinfo\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl\" (UID: \"4e1cf19c-563c-45d0-9a6a-a10fdb869f4d\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl"
Nov 01 14:58:25.756570 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:25.756481    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e1cf19c-563c-45d0-9a6a-a10fdb869f4d-workload-socket\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl\" (UID: \"4e1cf19c-563c-45d0-9a6a-a10fdb869f4d\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl"
Nov 01 14:58:25.756968 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:25.756945    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e1cf19c-563c-45d0-9a6a-a10fdb869f4d-workload-certs\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl\" (UID: \"4e1cf19c-563c-45d0-9a6a-a10fdb869f4d\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl"
Nov 01 14:58:25.757098 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:25.756981    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e1cf19c-563c-45d0-9a6a-a10fdb869f4d-istio-data\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl\" (UID: \"4e1cf19c-563c-45d0-9a6a-a10fdb869f4d\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl"
Nov 01 14:58:25.757237 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:25.757127    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e1cf19c-563c-45d0-9a6a-a10fdb869f4d-credential-socket\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl\" (UID: \"4e1cf19c-563c-45d0-9a6a-a10fdb869f4d\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl"
Nov 01 14:58:25.757357 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:25.757282    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e1cf19c-563c-45d0-9a6a-a10fdb869f4d-workload-socket\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl\" (UID: \"4e1cf19c-563c-45d0-9a6a-a10fdb869f4d\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl"
Nov 01 14:58:25.757357 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:25.757343    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e1cf19c-563c-45d0-9a6a-a10fdb869f4d-istiod-ca-cert\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl\" (UID: \"4e1cf19c-563c-45d0-9a6a-a10fdb869f4d\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl"
Nov 01 14:58:25.758813 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:25.758769    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e1cf19c-563c-45d0-9a6a-a10fdb869f4d-istio-envoy\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl\" (UID: \"4e1cf19c-563c-45d0-9a6a-a10fdb869f4d\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl"
Nov 01 14:58:25.759980 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:25.759577    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e1cf19c-563c-45d0-9a6a-a10fdb869f4d-istio-podinfo\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl\" (UID: \"4e1cf19c-563c-45d0-9a6a-a10fdb869f4d\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl"
Nov 01 14:58:25.814006 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:25.813976    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e1cf19c-563c-45d0-9a6a-a10fdb869f4d-istio-token\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl\" (UID: \"4e1cf19c-563c-45d0-9a6a-a10fdb869f4d\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl"
Nov 01 14:58:25.816579 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:25.816549    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfdb4\" (UniqueName: \"kubernetes.io/projected/4e1cf19c-563c-45d0-9a6a-a10fdb869f4d-kube-api-access-cfdb4\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl\" (UID: \"4e1cf19c-563c-45d0-9a6a-a10fdb869f4d\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl"
Nov 01 14:58:25.941041 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:25.940951    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl"
Nov 01 14:58:26.401674 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:26.401633    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl"]
Nov 01 14:58:26.407667 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:58:26.407628    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e1cf19c_563c_45d0_9a6a_a10fdb869f4d.slice/crio-d3bc823cbf69364db6976a1b8f50b01be9413edd95073d289bf16fb19d08a28a WatchSource:0}: Error finding container d3bc823cbf69364db6976a1b8f50b01be9413edd95073d289bf16fb19d08a28a: Status 404 returned error can't find the container with id d3bc823cbf69364db6976a1b8f50b01be9413edd95073d289bf16fb19d08a28a
Nov 01 14:58:26.410129 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:26.410093    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 14:58:26.410246 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:26.410160    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 14:58:26.410246 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:26.410208    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 14:58:26.872125 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:26.872080    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl" event={"ID":"4e1cf19c-563c-45d0-9a6a-a10fdb869f4d","Type":"ContainerStarted","Data":"cebabe451d28b38d6560c657aa57ca8aa09065ee1fe3849ba558e83fbc430e11"}
Nov 01 14:58:26.872645 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:26.872154    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl" event={"ID":"4e1cf19c-563c-45d0-9a6a-a10fdb869f4d","Type":"ContainerStarted","Data":"d3bc823cbf69364db6976a1b8f50b01be9413edd95073d289bf16fb19d08a28a"}
Nov 01 14:58:26.942084 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:26.942049    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl"
Nov 01 14:58:26.992998 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:26.991829    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl" podStartSLOduration=1.991808764 podStartE2EDuration="1.991808764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:58:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:58:26.988406384 +0000 UTC m=+2494.318269123" watchObservedRunningTime="2026-11-01 14:58:26.991808764 +0000 UTC m=+2494.321671500"
Nov 01 14:58:27.514213 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:58:27.513891    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:58:27.943257 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:27.943154    4993 patch_prober.go:29] interesting pod/e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Nov 01 14:58:27.943257 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:27.943228    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl" podUID="4e1cf19c-563c-45d0-9a6a-a10fdb869f4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Nov 01 14:58:28.941284 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:28.941239    4993 patch_prober.go:29] interesting pod/e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Nov 01 14:58:28.941476 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:28.941302    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl" podUID="4e1cf19c-563c-45d0-9a6a-a10fdb869f4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Nov 01 14:58:29.941949 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:29.941912    4993 patch_prober.go:29] interesting pod/e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Nov 01 14:58:29.942362 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:29.941975    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl" podUID="4e1cf19c-563c-45d0-9a6a-a10fdb869f4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Nov 01 14:58:30.941441 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:30.941408    4993 patch_prober.go:29] interesting pod/e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Nov 01 14:58:30.941628 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:30.941460    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl" podUID="4e1cf19c-563c-45d0-9a6a-a10fdb869f4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Nov 01 14:58:31.942057 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:31.942025    4993 patch_prober.go:29] interesting pod/e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Nov 01 14:58:31.942464 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:31.942077    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl" podUID="4e1cf19c-563c-45d0-9a6a-a10fdb869f4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Nov 01 14:58:32.941916 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:32.941864    4993 patch_prober.go:29] interesting pod/e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Nov 01 14:58:32.942130 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:32.941959    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl" podUID="4e1cf19c-563c-45d0-9a6a-a10fdb869f4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Nov 01 14:58:33.945836 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:33.945804    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl"
Nov 01 14:58:33.946246 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:33.946085    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl"
Nov 01 14:58:33.946782 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:33.946761    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl"
Nov 01 14:58:45.697434 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:45.696865    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx"]
Nov 01 14:58:45.698465 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:45.698423    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx" podUID="c3b714f6-ea83-4197-8587-1407bbf7907a" containerName="istio-proxy" containerID="cri-o://6c0c9edbd0c327e92f5929626048fdbc2f861312ff98a9d55da1a2750e368a7e" gracePeriod=30
Nov 01 14:58:51.079058 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:51.079028    4993 generic.go:356] "Generic (PLEG): container finished" podID="c3b714f6-ea83-4197-8587-1407bbf7907a" containerID="6c0c9edbd0c327e92f5929626048fdbc2f861312ff98a9d55da1a2750e368a7e" exitCode=0
Nov 01 14:58:51.079416 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:51.079110    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx" event={"ID":"c3b714f6-ea83-4197-8587-1407bbf7907a","Type":"ContainerDied","Data":"6c0c9edbd0c327e92f5929626048fdbc2f861312ff98a9d55da1a2750e368a7e"}
Nov 01 14:58:51.079416 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:51.079159    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx" event={"ID":"c3b714f6-ea83-4197-8587-1407bbf7907a","Type":"ContainerDied","Data":"a6f82c9bd79b72962ecca9fa95d46463710be9f65766240d7f2b01beb6464e0d"}
Nov 01 14:58:51.079416 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:51.079197    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6f82c9bd79b72962ecca9fa95d46463710be9f65766240d7f2b01beb6464e0d"
Nov 01 14:58:51.085615 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:51.085594    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx"
Nov 01 14:58:51.180018 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:51.179978    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3b714f6-ea83-4197-8587-1407bbf7907a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3b714f6-ea83-4197-8587-1407bbf7907a-credential-socket\") pod \"c3b714f6-ea83-4197-8587-1407bbf7907a\" (UID: \"c3b714f6-ea83-4197-8587-1407bbf7907a\") "
Nov 01 14:58:51.180286 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:51.180070    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3b714f6-ea83-4197-8587-1407bbf7907a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3b714f6-ea83-4197-8587-1407bbf7907a-workload-certs\") pod \"c3b714f6-ea83-4197-8587-1407bbf7907a\" (UID: \"c3b714f6-ea83-4197-8587-1407bbf7907a\") "
Nov 01 14:58:51.180286 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:51.180120    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3b714f6-ea83-4197-8587-1407bbf7907a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3b714f6-ea83-4197-8587-1407bbf7907a-istio-data\") pod \"c3b714f6-ea83-4197-8587-1407bbf7907a\" (UID: \"c3b714f6-ea83-4197-8587-1407bbf7907a\") "
Nov 01 14:58:51.180435 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:51.180340    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3b714f6-ea83-4197-8587-1407bbf7907a-credential-socket" pod "c3b714f6-ea83-4197-8587-1407bbf7907a" (UID: "c3b714f6-ea83-4197-8587-1407bbf7907a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:58:51.180435 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:51.180267    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3b714f6-ea83-4197-8587-1407bbf7907a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3b714f6-ea83-4197-8587-1407bbf7907a-workload-socket\") pod \"c3b714f6-ea83-4197-8587-1407bbf7907a\" (UID: \"c3b714f6-ea83-4197-8587-1407bbf7907a\") "
Nov 01 14:58:51.180550 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:51.180519    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3b714f6-ea83-4197-8587-1407bbf7907a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3b714f6-ea83-4197-8587-1407bbf7907a-istiod-ca-cert\") pod \"c3b714f6-ea83-4197-8587-1407bbf7907a\" (UID: \"c3b714f6-ea83-4197-8587-1407bbf7907a\") "
Nov 01 14:58:51.180550 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:51.180536    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3b714f6-ea83-4197-8587-1407bbf7907a-workload-socket" pod "c3b714f6-ea83-4197-8587-1407bbf7907a" (UID: "c3b714f6-ea83-4197-8587-1407bbf7907a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:58:51.180668 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:51.180611    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3b714f6-ea83-4197-8587-1407bbf7907a-istio-token\" (UniqueName: \"kubernetes.io/projected/c3b714f6-ea83-4197-8587-1407bbf7907a-istio-token\") pod \"c3b714f6-ea83-4197-8587-1407bbf7907a\" (UID: \"c3b714f6-ea83-4197-8587-1407bbf7907a\") "
Nov 01 14:58:51.180731 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:51.180703    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3b714f6-ea83-4197-8587-1407bbf7907a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3b714f6-ea83-4197-8587-1407bbf7907a-istio-envoy\") pod \"c3b714f6-ea83-4197-8587-1407bbf7907a\" (UID: \"c3b714f6-ea83-4197-8587-1407bbf7907a\") "
Nov 01 14:58:51.180797 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:51.180729    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3b714f6-ea83-4197-8587-1407bbf7907a-workload-certs" pod "c3b714f6-ea83-4197-8587-1407bbf7907a" (UID: "c3b714f6-ea83-4197-8587-1407bbf7907a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:58:51.180797 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:51.180761    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c3b714f6-ea83-4197-8587-1407bbf7907a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3b714f6-ea83-4197-8587-1407bbf7907a-istio-podinfo\") pod \"c3b714f6-ea83-4197-8587-1407bbf7907a\" (UID: \"c3b714f6-ea83-4197-8587-1407bbf7907a\") "
Nov 01 14:58:51.180903 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:51.180846    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3b714f6-ea83-4197-8587-1407bbf7907a-kube-api-access-x6l2j\" (UniqueName: \"kubernetes.io/projected/c3b714f6-ea83-4197-8587-1407bbf7907a-kube-api-access-x6l2j\") pod \"c3b714f6-ea83-4197-8587-1407bbf7907a\" (UID: \"c3b714f6-ea83-4197-8587-1407bbf7907a\") "
Nov 01 14:58:51.180964 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:51.180910    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3b714f6-ea83-4197-8587-1407bbf7907a-istio-data" pod "c3b714f6-ea83-4197-8587-1407bbf7907a" (UID: "c3b714f6-ea83-4197-8587-1407bbf7907a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:58:51.181554 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:51.181494    4993 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3b714f6-ea83-4197-8587-1407bbf7907a-workload-socket\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:58:51.182776 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:51.181990    4993 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3b714f6-ea83-4197-8587-1407bbf7907a-credential-socket\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:58:51.182960 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:51.182930    4993 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3b714f6-ea83-4197-8587-1407bbf7907a-workload-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:58:51.182960 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:51.182961    4993 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3b714f6-ea83-4197-8587-1407bbf7907a-istio-data\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:58:51.183144 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:51.182445    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3b714f6-ea83-4197-8587-1407bbf7907a-istiod-ca-cert" pod "c3b714f6-ea83-4197-8587-1407bbf7907a" (UID: "c3b714f6-ea83-4197-8587-1407bbf7907a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:58:51.183802 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:51.183762    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3b714f6-ea83-4197-8587-1407bbf7907a-istio-token" pod "c3b714f6-ea83-4197-8587-1407bbf7907a" (UID: "c3b714f6-ea83-4197-8587-1407bbf7907a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:58:51.183912 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:51.183817    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3b714f6-ea83-4197-8587-1407bbf7907a-istio-envoy" pod "c3b714f6-ea83-4197-8587-1407bbf7907a" (UID: "c3b714f6-ea83-4197-8587-1407bbf7907a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:58:51.184078 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:51.184054    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c3b714f6-ea83-4197-8587-1407bbf7907a-istio-podinfo" pod "c3b714f6-ea83-4197-8587-1407bbf7907a" (UID: "c3b714f6-ea83-4197-8587-1407bbf7907a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 14:58:51.185546 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:51.185508    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3b714f6-ea83-4197-8587-1407bbf7907a-kube-api-access-x6l2j" pod "c3b714f6-ea83-4197-8587-1407bbf7907a" (UID: "c3b714f6-ea83-4197-8587-1407bbf7907a"). InnerVolumeSpecName "kube-api-access-x6l2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:58:51.284197 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:51.284088    4993 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3b714f6-ea83-4197-8587-1407bbf7907a-istiod-ca-cert\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:58:51.284197 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:51.284122    4993 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3b714f6-ea83-4197-8587-1407bbf7907a-istio-token\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:58:51.284197 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:51.284135    4993 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3b714f6-ea83-4197-8587-1407bbf7907a-istio-envoy\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:58:51.284197 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:51.284150    4993 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3b714f6-ea83-4197-8587-1407bbf7907a-istio-podinfo\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:58:51.284197 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:51.284163    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6l2j\" (UniqueName: \"kubernetes.io/projected/c3b714f6-ea83-4197-8587-1407bbf7907a-kube-api-access-x6l2j\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:58:52.088237 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:52.085516    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx"
Nov 01 14:58:52.287421 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:52.287386    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx"]
Nov 01 14:58:52.340234 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:52.340118    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-nzrjx"]
Nov 01 14:58:52.797531 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:52.797491    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3b714f6-ea83-4197-8587-1407bbf7907a" path="/var/lib/kubelet/pods/c3b714f6-ea83-4197-8587-1407bbf7907a/volumes"
Nov 01 14:58:53.536342 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:58:53.536306    4993 scope.go:122] "RemoveContainer" containerID="6c0c9edbd0c327e92f5929626048fdbc2f861312ff98a9d55da1a2750e368a7e"
Nov 01 14:58:57.506829 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:58:57.506783    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:59:01.987268 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:01.987228    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-288f1e-84fdbc4bd9-wvt6v"]
Nov 01 14:59:01.987739 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:01.987487    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-288f1e-84fdbc4bd9-wvt6v" podUID="ce6e8049-31b5-4750-994b-83a4f0533715" containerName="maas-api" containerID="cri-o://95d0f0a1b825ee4acc8bb7ce60533a046338a0dace25216cecee813c15beb0d0" gracePeriod=30
Nov 01 14:59:02.201683 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:02.201642    4993 generic.go:356] "Generic (PLEG): container finished" podID="ce6e8049-31b5-4750-994b-83a4f0533715" containerID="95d0f0a1b825ee4acc8bb7ce60533a046338a0dace25216cecee813c15beb0d0" exitCode=0
Nov 01 14:59:02.201863 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:02.201741    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-288f1e-84fdbc4bd9-wvt6v" event={"ID":"ce6e8049-31b5-4750-994b-83a4f0533715","Type":"ContainerDied","Data":"95d0f0a1b825ee4acc8bb7ce60533a046338a0dace25216cecee813c15beb0d0"}
Nov 01 14:59:02.321782 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:02.321744    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-288f1e-84fdbc4bd9-wvt6v"
Nov 01 14:59:02.475228 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:02.473359    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce6e8049-31b5-4750-994b-83a4f0533715-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce6e8049-31b5-4750-994b-83a4f0533715-maas-api-tls\") pod \"ce6e8049-31b5-4750-994b-83a4f0533715\" (UID: \"ce6e8049-31b5-4750-994b-83a4f0533715\") "
Nov 01 14:59:02.475228 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:02.473415    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce6e8049-31b5-4750-994b-83a4f0533715-kube-api-access-4btbt\" (UniqueName: \"kubernetes.io/projected/ce6e8049-31b5-4750-994b-83a4f0533715-kube-api-access-4btbt\") pod \"ce6e8049-31b5-4750-994b-83a4f0533715\" (UID: \"ce6e8049-31b5-4750-994b-83a4f0533715\") "
Nov 01 14:59:02.475228 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:02.473498    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce6e8049-31b5-4750-994b-83a4f0533715-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce6e8049-31b5-4750-994b-83a4f0533715-openshift-service-ca\") pod \"ce6e8049-31b5-4750-994b-83a4f0533715\" (UID: \"ce6e8049-31b5-4750-994b-83a4f0533715\") "
Nov 01 14:59:02.475228 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:02.473553    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce6e8049-31b5-4750-994b-83a4f0533715-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce6e8049-31b5-4750-994b-83a4f0533715-metrics-certs\") pod \"ce6e8049-31b5-4750-994b-83a4f0533715\" (UID: \"ce6e8049-31b5-4750-994b-83a4f0533715\") "
Nov 01 14:59:02.479233 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:02.476065    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce6e8049-31b5-4750-994b-83a4f0533715-metrics-certs" pod "ce6e8049-31b5-4750-994b-83a4f0533715" (UID: "ce6e8049-31b5-4750-994b-83a4f0533715"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:59:02.479233 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:02.476402    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce6e8049-31b5-4750-994b-83a4f0533715-openshift-service-ca" pod "ce6e8049-31b5-4750-994b-83a4f0533715" (UID: "ce6e8049-31b5-4750-994b-83a4f0533715"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:59:02.483203 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:02.479628    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce6e8049-31b5-4750-994b-83a4f0533715-maas-api-tls" pod "ce6e8049-31b5-4750-994b-83a4f0533715" (UID: "ce6e8049-31b5-4750-994b-83a4f0533715"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 14:59:02.483203 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:02.482769    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce6e8049-31b5-4750-994b-83a4f0533715-kube-api-access-4btbt" pod "ce6e8049-31b5-4750-994b-83a4f0533715" (UID: "ce6e8049-31b5-4750-994b-83a4f0533715"). InnerVolumeSpecName "kube-api-access-4btbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:59:02.575984 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:02.575886    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce6e8049-31b5-4750-994b-83a4f0533715-maas-api-tls\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:59:02.575984 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:02.575936    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4btbt\" (UniqueName: \"kubernetes.io/projected/ce6e8049-31b5-4750-994b-83a4f0533715-kube-api-access-4btbt\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:59:02.575984 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:02.575954    4993 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce6e8049-31b5-4750-994b-83a4f0533715-openshift-service-ca\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:59:02.575984 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:02.575971    4993 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce6e8049-31b5-4750-994b-83a4f0533715-metrics-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:59:03.246499 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:03.246470    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-288f1e-84fdbc4bd9-wvt6v"
Nov 01 14:59:03.247959 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:03.246515    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-288f1e-84fdbc4bd9-wvt6v" event={"ID":"ce6e8049-31b5-4750-994b-83a4f0533715","Type":"ContainerDied","Data":"eb385f9e011b5e3d0a05ad9226a902e63b82e7246de66f3e557b09cc4d0bc2cf"}
Nov 01 14:59:03.248160 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:03.248146    4993 scope.go:122] "RemoveContainer" containerID="95d0f0a1b825ee4acc8bb7ce60533a046338a0dace25216cecee813c15beb0d0"
Nov 01 14:59:03.401844 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:03.401809    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-288f1e-84fdbc4bd9-wvt6v"]
Nov 01 14:59:03.447340 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:03.447310    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-288f1e-84fdbc4bd9-wvt6v"]
Nov 01 14:59:04.796755 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:04.796719    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce6e8049-31b5-4750-994b-83a4f0533715" path="/var/lib/kubelet/pods/ce6e8049-31b5-4750-994b-83a4f0533715/volumes"
Nov 01 14:59:27.531179 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:59:27.531123    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:59:29.838812 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:29.838761    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fvgkd"]
Nov 01 14:59:29.840146 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:29.840079    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fvgkd" podUID="08df1013-bce6-4645-bba5-736db953ce2b" containerName="istio-proxy" containerID="cri-o://822f62b290c685dc242f9fd99fa8106970551d2853d7aa42feca3b98f04ceed6" gracePeriod=30
Nov 01 14:59:35.134881 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:35.134812    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fvgkd"
Nov 01 14:59:35.283918 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:35.283883    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08df1013-bce6-4645-bba5-736db953ce2b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08df1013-bce6-4645-bba5-736db953ce2b-istiod-ca-cert\") pod \"08df1013-bce6-4645-bba5-736db953ce2b\" (UID: \"08df1013-bce6-4645-bba5-736db953ce2b\") "
Nov 01 14:59:35.284110 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:35.283941    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/08df1013-bce6-4645-bba5-736db953ce2b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08df1013-bce6-4645-bba5-736db953ce2b-istio-podinfo\") pod \"08df1013-bce6-4645-bba5-736db953ce2b\" (UID: \"08df1013-bce6-4645-bba5-736db953ce2b\") "
Nov 01 14:59:35.284110 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:35.283978    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08df1013-bce6-4645-bba5-736db953ce2b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08df1013-bce6-4645-bba5-736db953ce2b-workload-socket\") pod \"08df1013-bce6-4645-bba5-736db953ce2b\" (UID: \"08df1013-bce6-4645-bba5-736db953ce2b\") "
Nov 01 14:59:35.284258 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:35.284117    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08df1013-bce6-4645-bba5-736db953ce2b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08df1013-bce6-4645-bba5-736db953ce2b-workload-certs\") pod \"08df1013-bce6-4645-bba5-736db953ce2b\" (UID: \"08df1013-bce6-4645-bba5-736db953ce2b\") "
Nov 01 14:59:35.284258 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:35.284155    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08df1013-bce6-4645-bba5-736db953ce2b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08df1013-bce6-4645-bba5-736db953ce2b-istio-data\") pod \"08df1013-bce6-4645-bba5-736db953ce2b\" (UID: \"08df1013-bce6-4645-bba5-736db953ce2b\") "
Nov 01 14:59:35.284258 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:35.284240    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08df1013-bce6-4645-bba5-736db953ce2b-kube-api-access-jwmgk\" (UniqueName: \"kubernetes.io/projected/08df1013-bce6-4645-bba5-736db953ce2b-kube-api-access-jwmgk\") pod \"08df1013-bce6-4645-bba5-736db953ce2b\" (UID: \"08df1013-bce6-4645-bba5-736db953ce2b\") "
Nov 01 14:59:35.284430 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:35.284266    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08df1013-bce6-4645-bba5-736db953ce2b-workload-socket" pod "08df1013-bce6-4645-bba5-736db953ce2b" (UID: "08df1013-bce6-4645-bba5-736db953ce2b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:59:35.284430 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:35.284397    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08df1013-bce6-4645-bba5-736db953ce2b-workload-certs" pod "08df1013-bce6-4645-bba5-736db953ce2b" (UID: "08df1013-bce6-4645-bba5-736db953ce2b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:59:35.284526 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:35.284455    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08df1013-bce6-4645-bba5-736db953ce2b-istiod-ca-cert" pod "08df1013-bce6-4645-bba5-736db953ce2b" (UID: "08df1013-bce6-4645-bba5-736db953ce2b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 14:59:35.284526 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:35.284470    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08df1013-bce6-4645-bba5-736db953ce2b-istio-data" pod "08df1013-bce6-4645-bba5-736db953ce2b" (UID: "08df1013-bce6-4645-bba5-736db953ce2b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:59:35.284701 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:35.284681    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08df1013-bce6-4645-bba5-736db953ce2b-istio-token\" (UniqueName: \"kubernetes.io/projected/08df1013-bce6-4645-bba5-736db953ce2b-istio-token\") pod \"08df1013-bce6-4645-bba5-736db953ce2b\" (UID: \"08df1013-bce6-4645-bba5-736db953ce2b\") "
Nov 01 14:59:35.284795 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:35.284732    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08df1013-bce6-4645-bba5-736db953ce2b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08df1013-bce6-4645-bba5-736db953ce2b-istio-envoy\") pod \"08df1013-bce6-4645-bba5-736db953ce2b\" (UID: \"08df1013-bce6-4645-bba5-736db953ce2b\") "
Nov 01 14:59:35.284795 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:35.284765    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08df1013-bce6-4645-bba5-736db953ce2b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08df1013-bce6-4645-bba5-736db953ce2b-credential-socket\") pod \"08df1013-bce6-4645-bba5-736db953ce2b\" (UID: \"08df1013-bce6-4645-bba5-736db953ce2b\") "
Nov 01 14:59:35.285158 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:35.285132    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08df1013-bce6-4645-bba5-736db953ce2b-credential-socket" pod "08df1013-bce6-4645-bba5-736db953ce2b" (UID: "08df1013-bce6-4645-bba5-736db953ce2b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:59:35.285390 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:35.285371    4993 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08df1013-bce6-4645-bba5-736db953ce2b-credential-socket\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:59:35.285390 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:35.285394    4993 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08df1013-bce6-4645-bba5-736db953ce2b-istiod-ca-cert\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:59:35.285512 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:35.285408    4993 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08df1013-bce6-4645-bba5-736db953ce2b-workload-socket\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:59:35.285512 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:35.285423    4993 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08df1013-bce6-4645-bba5-736db953ce2b-workload-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:59:35.285512 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:35.285437    4993 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08df1013-bce6-4645-bba5-736db953ce2b-istio-data\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:59:35.286724 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:35.286696    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/08df1013-bce6-4645-bba5-736db953ce2b-istio-podinfo" pod "08df1013-bce6-4645-bba5-736db953ce2b" (UID: "08df1013-bce6-4645-bba5-736db953ce2b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 14:59:35.286961 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:35.286935    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08df1013-bce6-4645-bba5-736db953ce2b-kube-api-access-jwmgk" pod "08df1013-bce6-4645-bba5-736db953ce2b" (UID: "08df1013-bce6-4645-bba5-736db953ce2b"). InnerVolumeSpecName "kube-api-access-jwmgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:59:35.287089 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:35.286971    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08df1013-bce6-4645-bba5-736db953ce2b-istio-token" pod "08df1013-bce6-4645-bba5-736db953ce2b" (UID: "08df1013-bce6-4645-bba5-736db953ce2b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 14:59:35.287153 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:35.287141    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08df1013-bce6-4645-bba5-736db953ce2b-istio-envoy" pod "08df1013-bce6-4645-bba5-736db953ce2b" (UID: "08df1013-bce6-4645-bba5-736db953ce2b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 14:59:35.386972 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:35.386887    4993 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08df1013-bce6-4645-bba5-736db953ce2b-istio-token\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:59:35.386972 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:35.386924    4993 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08df1013-bce6-4645-bba5-736db953ce2b-istio-envoy\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:59:35.386972 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:35.386934    4993 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08df1013-bce6-4645-bba5-736db953ce2b-istio-podinfo\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:59:35.386972 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:35.386944    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwmgk\" (UniqueName: \"kubernetes.io/projected/08df1013-bce6-4645-bba5-736db953ce2b-kube-api-access-jwmgk\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 14:59:35.686856 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:35.686755    4993 generic.go:356] "Generic (PLEG): container finished" podID="08df1013-bce6-4645-bba5-736db953ce2b" containerID="822f62b290c685dc242f9fd99fa8106970551d2853d7aa42feca3b98f04ceed6" exitCode=0
Nov 01 14:59:35.686856 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:35.686828    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fvgkd"
Nov 01 14:59:35.687079 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:35.686857    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fvgkd" event={"ID":"08df1013-bce6-4645-bba5-736db953ce2b","Type":"ContainerDied","Data":"822f62b290c685dc242f9fd99fa8106970551d2853d7aa42feca3b98f04ceed6"}
Nov 01 14:59:35.687079 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:35.686939    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fvgkd" event={"ID":"08df1013-bce6-4645-bba5-736db953ce2b","Type":"ContainerDied","Data":"1ad40de17405e109687aeae6694757973ac90767db2990ebe982c9f828ce2198"}
Nov 01 14:59:35.687079 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:35.686955    4993 scope.go:122] "RemoveContainer" containerID="822f62b290c685dc242f9fd99fa8106970551d2853d7aa42feca3b98f04ceed6"
Nov 01 14:59:35.700028 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:35.700006    4993 scope.go:122] "RemoveContainer" containerID="822f62b290c685dc242f9fd99fa8106970551d2853d7aa42feca3b98f04ceed6"
Nov 01 14:59:35.700371 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:59:35.700350    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"822f62b290c685dc242f9fd99fa8106970551d2853d7aa42feca3b98f04ceed6\": container with ID starting with 822f62b290c685dc242f9fd99fa8106970551d2853d7aa42feca3b98f04ceed6 not found: ID does not exist" containerID="822f62b290c685dc242f9fd99fa8106970551d2853d7aa42feca3b98f04ceed6"
Nov 01 14:59:35.700445 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:35.700386    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"822f62b290c685dc242f9fd99fa8106970551d2853d7aa42feca3b98f04ceed6"} err="failed to get container status \"822f62b290c685dc242f9fd99fa8106970551d2853d7aa42feca3b98f04ceed6\": rpc error: code = NotFound desc = could not find container \"822f62b290c685dc242f9fd99fa8106970551d2853d7aa42feca3b98f04ceed6\": container with ID starting with 822f62b290c685dc242f9fd99fa8106970551d2853d7aa42feca3b98f04ceed6 not found: ID does not exist"
Nov 01 14:59:35.845016 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:35.844982    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fvgkd"]
Nov 01 14:59:35.849421 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:35.849393    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-fvgkd"]
Nov 01 14:59:36.796506 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:36.796464    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08df1013-bce6-4645-bba5-736db953ce2b" path="/var/lib/kubelet/pods/08df1013-bce6-4645-bba5-736db953ce2b/volumes"
Nov 01 14:59:56.578800 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:56.578761    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-0c9d0a-6588775df-ggnn6"]
Nov 01 14:59:56.579325 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:56.579310    4993 cpu_manager.go:414] "containerMap: removing container" podUID="ce6e8049-31b5-4750-994b-83a4f0533715" containerName="maas-api"
Nov 01 14:59:56.579378 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:56.579325    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce6e8049-31b5-4750-994b-83a4f0533715" containerName="maas-api"
Nov 01 14:59:56.579378 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:56.579339    4993 cpu_manager.go:414] "containerMap: removing container" podUID="c3b714f6-ea83-4197-8587-1407bbf7907a" containerName="istio-proxy"
Nov 01 14:59:56.579378 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:56.579345    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3b714f6-ea83-4197-8587-1407bbf7907a" containerName="istio-proxy"
Nov 01 14:59:56.579378 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:56.579377    4993 cpu_manager.go:414] "containerMap: removing container" podUID="08df1013-bce6-4645-bba5-736db953ce2b" containerName="istio-proxy"
Nov 01 14:59:56.579543 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:56.579382    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08df1013-bce6-4645-bba5-736db953ce2b" containerName="istio-proxy"
Nov 01 14:59:56.579543 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:56.579467    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="08df1013-bce6-4645-bba5-736db953ce2b" containerName="istio-proxy"
Nov 01 14:59:56.579543 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:56.579477    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce6e8049-31b5-4750-994b-83a4f0533715" containerName="maas-api"
Nov 01 14:59:56.579543 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:56.579483    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3b714f6-ea83-4197-8587-1407bbf7907a" containerName="istio-proxy"
Nov 01 14:59:56.584327 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:56.584304    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0c9d0a-6588775df-ggnn6"
Nov 01 14:59:56.632818 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:56.632784    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-0c9d0a-serving-cert\""
Nov 01 14:59:56.683599 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:56.683558    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-0c9d0a-6588775df-ggnn6"]
Nov 01 14:59:56.746505 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:56.746467    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2bf57b9-f35a-46c6-91e4-bbabd4908195-metrics-certs\") pod \"maas-api-e2e-praxis-a-0c9d0a-6588775df-ggnn6\" (UID: \"e2bf57b9-f35a-46c6-91e4-bbabd4908195\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0c9d0a-6588775df-ggnn6"
Nov 01 14:59:56.746724 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:56.746668    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94bnv\" (UniqueName: \"kubernetes.io/projected/e2bf57b9-f35a-46c6-91e4-bbabd4908195-kube-api-access-94bnv\") pod \"maas-api-e2e-praxis-a-0c9d0a-6588775df-ggnn6\" (UID: \"e2bf57b9-f35a-46c6-91e4-bbabd4908195\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0c9d0a-6588775df-ggnn6"
Nov 01 14:59:56.746786 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:56.746719    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2bf57b9-f35a-46c6-91e4-bbabd4908195-maas-api-tls\") pod \"maas-api-e2e-praxis-a-0c9d0a-6588775df-ggnn6\" (UID: \"e2bf57b9-f35a-46c6-91e4-bbabd4908195\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0c9d0a-6588775df-ggnn6"
Nov 01 14:59:56.746887 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:56.746864    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2bf57b9-f35a-46c6-91e4-bbabd4908195-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-0c9d0a-6588775df-ggnn6\" (UID: \"e2bf57b9-f35a-46c6-91e4-bbabd4908195\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0c9d0a-6588775df-ggnn6"
Nov 01 14:59:56.848389 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:56.848283    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2bf57b9-f35a-46c6-91e4-bbabd4908195-metrics-certs\") pod \"maas-api-e2e-praxis-a-0c9d0a-6588775df-ggnn6\" (UID: \"e2bf57b9-f35a-46c6-91e4-bbabd4908195\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0c9d0a-6588775df-ggnn6"
Nov 01 14:59:56.848578 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:56.848417    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94bnv\" (UniqueName: \"kubernetes.io/projected/e2bf57b9-f35a-46c6-91e4-bbabd4908195-kube-api-access-94bnv\") pod \"maas-api-e2e-praxis-a-0c9d0a-6588775df-ggnn6\" (UID: \"e2bf57b9-f35a-46c6-91e4-bbabd4908195\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0c9d0a-6588775df-ggnn6"
Nov 01 14:59:56.848578 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:56.848550    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2bf57b9-f35a-46c6-91e4-bbabd4908195-maas-api-tls\") pod \"maas-api-e2e-praxis-a-0c9d0a-6588775df-ggnn6\" (UID: \"e2bf57b9-f35a-46c6-91e4-bbabd4908195\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0c9d0a-6588775df-ggnn6"
Nov 01 14:59:56.849180 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:56.848637    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2bf57b9-f35a-46c6-91e4-bbabd4908195-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-0c9d0a-6588775df-ggnn6\" (UID: \"e2bf57b9-f35a-46c6-91e4-bbabd4908195\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0c9d0a-6588775df-ggnn6"
Nov 01 14:59:56.849180 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:59:56.848824    4993 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-0c9d0a-serving-cert: secret "maas-api-e2e-praxis-a-0c9d0a-serving-cert" not found
Nov 01 14:59:56.849180 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:59:56.848898    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2bf57b9-f35a-46c6-91e4-bbabd4908195-maas-api-tls podName:e2bf57b9-f35a-46c6-91e4-bbabd4908195 nodeName:}" failed. No retries permitted until 2026-11-01 14:59:57.348880955 +0000 UTC m=+2584.678743683 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e2bf57b9-f35a-46c6-91e4-bbabd4908195-maas-api-tls") pod "maas-api-e2e-praxis-a-0c9d0a-6588775df-ggnn6" (UID: "e2bf57b9-f35a-46c6-91e4-bbabd4908195") : secret "maas-api-e2e-praxis-a-0c9d0a-serving-cert" not found
Nov 01 14:59:56.850112 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:56.849654    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2bf57b9-f35a-46c6-91e4-bbabd4908195-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-0c9d0a-6588775df-ggnn6\" (UID: \"e2bf57b9-f35a-46c6-91e4-bbabd4908195\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0c9d0a-6588775df-ggnn6"
Nov 01 14:59:56.851459 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:56.851430    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2bf57b9-f35a-46c6-91e4-bbabd4908195-metrics-certs\") pod \"maas-api-e2e-praxis-a-0c9d0a-6588775df-ggnn6\" (UID: \"e2bf57b9-f35a-46c6-91e4-bbabd4908195\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0c9d0a-6588775df-ggnn6"
Nov 01 14:59:56.906936 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:56.906898    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94bnv\" (UniqueName: \"kubernetes.io/projected/e2bf57b9-f35a-46c6-91e4-bbabd4908195-kube-api-access-94bnv\") pod \"maas-api-e2e-praxis-a-0c9d0a-6588775df-ggnn6\" (UID: \"e2bf57b9-f35a-46c6-91e4-bbabd4908195\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0c9d0a-6588775df-ggnn6"
Nov 01 14:59:57.355642 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:57.355606    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2bf57b9-f35a-46c6-91e4-bbabd4908195-maas-api-tls\") pod \"maas-api-e2e-praxis-a-0c9d0a-6588775df-ggnn6\" (UID: \"e2bf57b9-f35a-46c6-91e4-bbabd4908195\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0c9d0a-6588775df-ggnn6"
Nov 01 14:59:57.358155 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:57.358125    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2bf57b9-f35a-46c6-91e4-bbabd4908195-maas-api-tls\") pod \"maas-api-e2e-praxis-a-0c9d0a-6588775df-ggnn6\" (UID: \"e2bf57b9-f35a-46c6-91e4-bbabd4908195\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0c9d0a-6588775df-ggnn6"
Nov 01 14:59:57.497534 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:57.497501    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0c9d0a-6588775df-ggnn6"
Nov 01 14:59:57.500643 ip-10-0-134-37 kubenswrapper[4993]: E1101 14:59:57.500609    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 14:59:57.867407 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:57.867376    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-0c9d0a-6588775df-ggnn6"]
Nov 01 14:59:57.872796 ip-10-0-134-37 kubenswrapper[4993]: W1101 14:59:57.872754    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2bf57b9_f35a_46c6_91e4_bbabd4908195.slice/crio-2ee12b3373d27017d2a755c1cc65e73296bbc4dd60533d4273bde053e760a980 WatchSource:0}: Error finding container 2ee12b3373d27017d2a755c1cc65e73296bbc4dd60533d4273bde053e760a980: Status 404 returned error can't find the container with id 2ee12b3373d27017d2a755c1cc65e73296bbc4dd60533d4273bde053e760a980
Nov 01 14:59:57.874325 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:57.874303    4993 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 14:59:58.839091 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:58.839050    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0c9d0a-6588775df-ggnn6" event={"ID":"e2bf57b9-f35a-46c6-91e4-bbabd4908195","Type":"ContainerStarted","Data":"3b13040974f021b7c43680fd1aeb16e7820fa6149609fd93c2c7f389b07dd372"}
Nov 01 14:59:58.839319 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:58.839102    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0c9d0a-6588775df-ggnn6" event={"ID":"e2bf57b9-f35a-46c6-91e4-bbabd4908195","Type":"ContainerStarted","Data":"2ee12b3373d27017d2a755c1cc65e73296bbc4dd60533d4273bde053e760a980"}
Nov 01 14:59:58.839319 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:58.839299    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0c9d0a-6588775df-ggnn6"
Nov 01 14:59:58.994240 ip-10-0-134-37 kubenswrapper[4993]: I1101 14:59:58.994182    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0c9d0a-6588775df-ggnn6" podStartSLOduration=2.1548481 podStartE2EDuration="2.994149292s" totalImagesPullingTime="839.301192ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:59:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:59:58.991093059 +0000 UTC m=+2586.320955807" watchObservedRunningTime="2026-11-01 14:59:58.994149292 +0000 UTC m=+2586.324012028"
Nov 01 15:00:00.169448 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:00.169411    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892420-c5b42"]
Nov 01 15:00:00.174431 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:00.174402    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892420-c5b42"
Nov 01 15:00:00.195086 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:00.195054    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-0c9d0a-29892420-ln9zk"]
Nov 01 15:00:00.199652 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:00.199629    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892420-5jxkv"]
Nov 01 15:00:00.199772 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:00.199744    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-0c9d0a-29892420-ln9zk"
Nov 01 15:00:00.203421 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:00.203399    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892420-5jxkv"
Nov 01 15:00:00.209088 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:00.209063    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f5bf8a3-874b-4026-a9c8-8d8eedcfc23d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892420-5jxkv\" (UID: \"5f5bf8a3-874b-4026-a9c8-8d8eedcfc23d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892420-5jxkv"
Nov 01 15:00:00.209237 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:00.209163    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4xdp\" (UniqueName: \"kubernetes.io/projected/5f5bf8a3-874b-4026-a9c8-8d8eedcfc23d-kube-api-access-p4xdp\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892420-5jxkv\" (UID: \"5f5bf8a3-874b-4026-a9c8-8d8eedcfc23d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892420-5jxkv"
Nov 01 15:00:00.209237 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:00.209223    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4438535-151d-4848-a5e1-2d0809e32e1e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-0c9d0a-29892420-ln9zk\" (UID: \"a4438535-151d-4848-a5e1-2d0809e32e1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-0c9d0a-29892420-ln9zk"
Nov 01 15:00:00.209365 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:00.209258    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffvfr\" (UniqueName: \"kubernetes.io/projected/a4438535-151d-4848-a5e1-2d0809e32e1e-kube-api-access-ffvfr\") pod \"maas-api-key-cleanup-e2e-praxis-a-0c9d0a-29892420-ln9zk\" (UID: \"a4438535-151d-4848-a5e1-2d0809e32e1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-0c9d0a-29892420-ln9zk"
Nov 01 15:00:00.209445 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:00.209424    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a7e100d-e267-447c-b21d-2de78d044d92-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892420-c5b42\" (UID: \"4a7e100d-e267-447c-b21d-2de78d044d92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892420-c5b42"
Nov 01 15:00:00.209515 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:00.209471    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whtbm\" (UniqueName: \"kubernetes.io/projected/4a7e100d-e267-447c-b21d-2de78d044d92-kube-api-access-whtbm\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892420-c5b42\" (UID: \"4a7e100d-e267-447c-b21d-2de78d044d92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892420-c5b42"
Nov 01 15:00:00.246658 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:00.246613    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892420-c5b42"]
Nov 01 15:00:00.285439 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:00.285400    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892420-5jxkv"]
Nov 01 15:00:00.287222 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:00.287190    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-0c9d0a-29892420-ln9zk"]
Nov 01 15:00:00.310108 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:00.310072    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4xdp\" (UniqueName: \"kubernetes.io/projected/5f5bf8a3-874b-4026-a9c8-8d8eedcfc23d-kube-api-access-p4xdp\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892420-5jxkv\" (UID: \"5f5bf8a3-874b-4026-a9c8-8d8eedcfc23d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892420-5jxkv"
Nov 01 15:00:00.310108 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:00.310117    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4438535-151d-4848-a5e1-2d0809e32e1e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-0c9d0a-29892420-ln9zk\" (UID: \"a4438535-151d-4848-a5e1-2d0809e32e1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-0c9d0a-29892420-ln9zk"
Nov 01 15:00:00.310404 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:00.310148    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffvfr\" (UniqueName: \"kubernetes.io/projected/a4438535-151d-4848-a5e1-2d0809e32e1e-kube-api-access-ffvfr\") pod \"maas-api-key-cleanup-e2e-praxis-a-0c9d0a-29892420-ln9zk\" (UID: \"a4438535-151d-4848-a5e1-2d0809e32e1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-0c9d0a-29892420-ln9zk"
Nov 01 15:00:00.310404 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:00.310229    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a7e100d-e267-447c-b21d-2de78d044d92-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892420-c5b42\" (UID: \"4a7e100d-e267-447c-b21d-2de78d044d92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892420-c5b42"
Nov 01 15:00:00.310404 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:00.310268    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whtbm\" (UniqueName: \"kubernetes.io/projected/4a7e100d-e267-447c-b21d-2de78d044d92-kube-api-access-whtbm\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892420-c5b42\" (UID: \"4a7e100d-e267-447c-b21d-2de78d044d92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892420-c5b42"
Nov 01 15:00:00.310404 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:00.310386    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f5bf8a3-874b-4026-a9c8-8d8eedcfc23d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892420-5jxkv\" (UID: \"5f5bf8a3-874b-4026-a9c8-8d8eedcfc23d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892420-5jxkv"
Nov 01 15:00:00.310996 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:00.310964    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4438535-151d-4848-a5e1-2d0809e32e1e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-0c9d0a-29892420-ln9zk\" (UID: \"a4438535-151d-4848-a5e1-2d0809e32e1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-0c9d0a-29892420-ln9zk"
Nov 01 15:00:00.311124 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:00.311102    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f5bf8a3-874b-4026-a9c8-8d8eedcfc23d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892420-5jxkv\" (UID: \"5f5bf8a3-874b-4026-a9c8-8d8eedcfc23d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892420-5jxkv"
Nov 01 15:00:00.311750 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:00.311724    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a7e100d-e267-447c-b21d-2de78d044d92-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892420-c5b42\" (UID: \"4a7e100d-e267-447c-b21d-2de78d044d92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892420-c5b42"
Nov 01 15:00:00.324474 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:00.324437    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\""
Nov 01 15:00:00.367029 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:00.366993    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4xdp\" (UniqueName: \"kubernetes.io/projected/5f5bf8a3-874b-4026-a9c8-8d8eedcfc23d-kube-api-access-p4xdp\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892420-5jxkv\" (UID: \"5f5bf8a3-874b-4026-a9c8-8d8eedcfc23d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892420-5jxkv"
Nov 01 15:00:00.368958 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:00.368905    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whtbm\" (UniqueName: \"kubernetes.io/projected/4a7e100d-e267-447c-b21d-2de78d044d92-kube-api-access-whtbm\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892420-c5b42\" (UID: \"4a7e100d-e267-447c-b21d-2de78d044d92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892420-c5b42"
Nov 01 15:00:00.371646 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:00.371616    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffvfr\" (UniqueName: \"kubernetes.io/projected/a4438535-151d-4848-a5e1-2d0809e32e1e-kube-api-access-ffvfr\") pod \"maas-api-key-cleanup-e2e-praxis-a-0c9d0a-29892420-ln9zk\" (UID: \"a4438535-151d-4848-a5e1-2d0809e32e1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-0c9d0a-29892420-ln9zk"
Nov 01 15:00:00.488599 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:00.488563    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892420-c5b42"
Nov 01 15:00:00.523022 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:00.522989    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-0c9d0a-29892420-ln9zk"
Nov 01 15:00:00.558651 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:00.558312    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892420-5jxkv"
Nov 01 15:00:00.872555 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:00.872367    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892420-c5b42"]
Nov 01 15:00:00.875783 ip-10-0-134-37 kubenswrapper[4993]: W1101 15:00:00.875744    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a7e100d_e267_447c_b21d_2de78d044d92.slice/crio-5c0f4ebefbbfafc1aebe76b386c1f6096bb680d28fea05450d3ec9d338a0a6f6 WatchSource:0}: Error finding container 5c0f4ebefbbfafc1aebe76b386c1f6096bb680d28fea05450d3ec9d338a0a6f6: Status 404 returned error can't find the container with id 5c0f4ebefbbfafc1aebe76b386c1f6096bb680d28fea05450d3ec9d338a0a6f6
Nov 01 15:00:00.983052 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:00.982996    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-0c9d0a-29892420-ln9zk"]
Nov 01 15:00:00.988202 ip-10-0-134-37 kubenswrapper[4993]: W1101 15:00:00.988161    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4438535_151d_4848_a5e1_2d0809e32e1e.slice/crio-a93bbf15466bcce2fff6234c005147611fd177faa416712d4b65d30b78d0ea71 WatchSource:0}: Error finding container a93bbf15466bcce2fff6234c005147611fd177faa416712d4b65d30b78d0ea71: Status 404 returned error can't find the container with id a93bbf15466bcce2fff6234c005147611fd177faa416712d4b65d30b78d0ea71
Nov 01 15:00:01.002646 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:01.002602    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892420-5jxkv"]
Nov 01 15:00:01.010606 ip-10-0-134-37 kubenswrapper[4993]: W1101 15:00:01.010570    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f5bf8a3_874b_4026_a9c8_8d8eedcfc23d.slice/crio-081c94c371080625e34a4685421ee31e5973b4c9275bfe9dc9d6f46d79a4f44e WatchSource:0}: Error finding container 081c94c371080625e34a4685421ee31e5973b4c9275bfe9dc9d6f46d79a4f44e: Status 404 returned error can't find the container with id 081c94c371080625e34a4685421ee31e5973b4c9275bfe9dc9d6f46d79a4f44e
Nov 01 15:00:01.858754 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:01.858711    4993 generic.go:356] "Generic (PLEG): container finished" podID="4a7e100d-e267-447c-b21d-2de78d044d92" containerID="de991d5e698e3017f97eedd04f7d5bfa469136dacd1800e7d9619b6f85cb60bb" exitCode=60
Nov 01 15:00:01.859256 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:01.858790    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892420-c5b42" event={"ID":"4a7e100d-e267-447c-b21d-2de78d044d92","Type":"ContainerDied","Data":"de991d5e698e3017f97eedd04f7d5bfa469136dacd1800e7d9619b6f85cb60bb"}
Nov 01 15:00:01.859256 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:01.858845    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892420-c5b42" event={"ID":"4a7e100d-e267-447c-b21d-2de78d044d92","Type":"ContainerStarted","Data":"5c0f4ebefbbfafc1aebe76b386c1f6096bb680d28fea05450d3ec9d338a0a6f6"}
Nov 01 15:00:01.859256 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:01.859070    4993 scope.go:122] "RemoveContainer" containerID="de991d5e698e3017f97eedd04f7d5bfa469136dacd1800e7d9619b6f85cb60bb"
Nov 01 15:00:01.860571 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:01.860544    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892420-5jxkv" event={"ID":"5f5bf8a3-874b-4026-a9c8-8d8eedcfc23d","Type":"ContainerStarted","Data":"66493b05ce52a8acd0b3dc667f6798996dd1b5f266522783aa1f8e0e2b5633f2"}
Nov 01 15:00:01.860778 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:01.860591    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892420-5jxkv" event={"ID":"5f5bf8a3-874b-4026-a9c8-8d8eedcfc23d","Type":"ContainerStarted","Data":"081c94c371080625e34a4685421ee31e5973b4c9275bfe9dc9d6f46d79a4f44e"}
Nov 01 15:00:01.862208 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:01.862102    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-0c9d0a-29892420-ln9zk" event={"ID":"a4438535-151d-4848-a5e1-2d0809e32e1e","Type":"ContainerStarted","Data":"3fcfda00832b5cf464781faba0d63f89c342d935baae750aa709dbfe5e96e60a"}
Nov 01 15:00:01.862208 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:01.862164    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-0c9d0a-29892420-ln9zk" event={"ID":"a4438535-151d-4848-a5e1-2d0809e32e1e","Type":"ContainerStarted","Data":"a93bbf15466bcce2fff6234c005147611fd177faa416712d4b65d30b78d0ea71"}
Nov 01 15:00:02.072345 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:02.072277    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-0c9d0a-29892420-ln9zk" podStartSLOduration=2.07225587 podStartE2EDuration="2.07225587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:00:02.070296931 +0000 UTC m=+2589.400159669" watchObservedRunningTime="2026-11-01 15:00:02.07225587 +0000 UTC m=+2589.402118608"
Nov 01 15:00:02.175151 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:02.175105    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892420-5jxkv" podStartSLOduration=2.175088803 podStartE2EDuration="2.175088803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:00:02.174647387 +0000 UTC m=+2589.504510102" watchObservedRunningTime="2026-11-01 15:00:02.175088803 +0000 UTC m=+2589.504951540"
Nov 01 15:00:02.870461 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:02.870430    4993 generic.go:356] "Generic (PLEG): container finished" podID="4a7e100d-e267-447c-b21d-2de78d044d92" containerID="73daf43deb189a5030c505fea8c49420cc6073e3f6180ab8c9f8119c475e3b02" exitCode=60
Nov 01 15:00:02.870919 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:02.870517    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892420-c5b42" event={"ID":"4a7e100d-e267-447c-b21d-2de78d044d92","Type":"ContainerDied","Data":"73daf43deb189a5030c505fea8c49420cc6073e3f6180ab8c9f8119c475e3b02"}
Nov 01 15:00:02.870919 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:02.870580    4993 scope.go:122] "RemoveContainer" containerID="de991d5e698e3017f97eedd04f7d5bfa469136dacd1800e7d9619b6f85cb60bb"
Nov 01 15:00:02.870919 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:02.870888    4993 scope.go:122] "RemoveContainer" containerID="73daf43deb189a5030c505fea8c49420cc6073e3f6180ab8c9f8119c475e3b02"
Nov 01 15:00:02.871247 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:00:02.871226    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892420-c5b42_odh-ai-gateway-infra(4a7e100d-e267-447c-b21d-2de78d044d92)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892420-c5b42" podUID="4a7e100d-e267-447c-b21d-2de78d044d92"
Nov 01 15:00:02.872210 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:02.872189    4993 generic.go:356] "Generic (PLEG): container finished" podID="5f5bf8a3-874b-4026-a9c8-8d8eedcfc23d" containerID="66493b05ce52a8acd0b3dc667f6798996dd1b5f266522783aa1f8e0e2b5633f2" exitCode=60
Nov 01 15:00:02.872311 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:02.872260    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892420-5jxkv" event={"ID":"5f5bf8a3-874b-4026-a9c8-8d8eedcfc23d","Type":"ContainerDied","Data":"66493b05ce52a8acd0b3dc667f6798996dd1b5f266522783aa1f8e0e2b5633f2"}
Nov 01 15:00:02.872545 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:02.872526    4993 scope.go:122] "RemoveContainer" containerID="66493b05ce52a8acd0b3dc667f6798996dd1b5f266522783aa1f8e0e2b5633f2"
Nov 01 15:00:02.874026 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:02.873952    4993 generic.go:356] "Generic (PLEG): container finished" podID="a4438535-151d-4848-a5e1-2d0809e32e1e" containerID="3fcfda00832b5cf464781faba0d63f89c342d935baae750aa709dbfe5e96e60a" exitCode=7
Nov 01 15:00:02.874081 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:02.874033    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-0c9d0a-29892420-ln9zk" event={"ID":"a4438535-151d-4848-a5e1-2d0809e32e1e","Type":"ContainerDied","Data":"3fcfda00832b5cf464781faba0d63f89c342d935baae750aa709dbfe5e96e60a"}
Nov 01 15:00:02.874273 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:02.874258    4993 scope.go:122] "RemoveContainer" containerID="3fcfda00832b5cf464781faba0d63f89c342d935baae750aa709dbfe5e96e60a"
Nov 01 15:00:03.881977 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:03.881950    4993 scope.go:122] "RemoveContainer" containerID="73daf43deb189a5030c505fea8c49420cc6073e3f6180ab8c9f8119c475e3b02"
Nov 01 15:00:03.882399 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:00:03.882295    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892420-c5b42_odh-ai-gateway-infra(4a7e100d-e267-447c-b21d-2de78d044d92)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892420-c5b42" podUID="4a7e100d-e267-447c-b21d-2de78d044d92"
Nov 01 15:00:03.883591 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:03.883566    4993 generic.go:356] "Generic (PLEG): container finished" podID="5f5bf8a3-874b-4026-a9c8-8d8eedcfc23d" containerID="62db9516ffc621bed2df28962be6675d2bb01fee90236e90cfeaa49e183c4b6e" exitCode=60
Nov 01 15:00:03.883709 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:03.883649    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892420-5jxkv" event={"ID":"5f5bf8a3-874b-4026-a9c8-8d8eedcfc23d","Type":"ContainerDied","Data":"62db9516ffc621bed2df28962be6675d2bb01fee90236e90cfeaa49e183c4b6e"}
Nov 01 15:00:03.883776 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:03.883720    4993 scope.go:122] "RemoveContainer" containerID="66493b05ce52a8acd0b3dc667f6798996dd1b5f266522783aa1f8e0e2b5633f2"
Nov 01 15:00:03.883920 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:03.883896    4993 scope.go:122] "RemoveContainer" containerID="62db9516ffc621bed2df28962be6675d2bb01fee90236e90cfeaa49e183c4b6e"
Nov 01 15:00:03.884274 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:00:03.884250    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892420-5jxkv_odh-ai-gateway-infra(5f5bf8a3-874b-4026-a9c8-8d8eedcfc23d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892420-5jxkv" podUID="5f5bf8a3-874b-4026-a9c8-8d8eedcfc23d"
Nov 01 15:00:03.885625 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:03.885605    4993 generic.go:356] "Generic (PLEG): container finished" podID="a4438535-151d-4848-a5e1-2d0809e32e1e" containerID="3d5924fb1ac1147414f4b7ff457dbae783fb57805bface08587fdde13b387532" exitCode=7
Nov 01 15:00:03.885724 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:03.885683    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-0c9d0a-29892420-ln9zk" event={"ID":"a4438535-151d-4848-a5e1-2d0809e32e1e","Type":"ContainerDied","Data":"3d5924fb1ac1147414f4b7ff457dbae783fb57805bface08587fdde13b387532"}
Nov 01 15:00:03.885918 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:03.885904    4993 scope.go:122] "RemoveContainer" containerID="3d5924fb1ac1147414f4b7ff457dbae783fb57805bface08587fdde13b387532"
Nov 01 15:00:03.886127 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:00:03.886111    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-0c9d0a-29892420-ln9zk_odh-ai-gateway-infra(a4438535-151d-4848-a5e1-2d0809e32e1e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-0c9d0a-29892420-ln9zk" podUID="a4438535-151d-4848-a5e1-2d0809e32e1e"
Nov 01 15:00:03.925679 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:03.925658    4993 scope.go:122] "RemoveContainer" containerID="3fcfda00832b5cf464781faba0d63f89c342d935baae750aa709dbfe5e96e60a"
Nov 01 15:00:04.851779 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:04.851750    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0c9d0a-6588775df-ggnn6"
Nov 01 15:00:04.894249 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:04.894220    4993 scope.go:122] "RemoveContainer" containerID="62db9516ffc621bed2df28962be6675d2bb01fee90236e90cfeaa49e183c4b6e"
Nov 01 15:00:04.895109 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:00:04.895080    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892420-5jxkv_odh-ai-gateway-infra(5f5bf8a3-874b-4026-a9c8-8d8eedcfc23d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892420-5jxkv" podUID="5f5bf8a3-874b-4026-a9c8-8d8eedcfc23d"
Nov 01 15:00:04.898891 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:04.896103    4993 scope.go:122] "RemoveContainer" containerID="3d5924fb1ac1147414f4b7ff457dbae783fb57805bface08587fdde13b387532"
Nov 01 15:00:04.898891 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:00:04.896341    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-0c9d0a-29892420-ln9zk_odh-ai-gateway-infra(a4438535-151d-4848-a5e1-2d0809e32e1e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-0c9d0a-29892420-ln9zk" podUID="a4438535-151d-4848-a5e1-2d0809e32e1e"
Nov 01 15:00:14.788819 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:14.788779    4993 scope.go:122] "RemoveContainer" containerID="73daf43deb189a5030c505fea8c49420cc6073e3f6180ab8c9f8119c475e3b02"
Nov 01 15:00:15.788070 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:15.788036    4993 scope.go:122] "RemoveContainer" containerID="3d5924fb1ac1147414f4b7ff457dbae783fb57805bface08587fdde13b387532"
Nov 01 15:00:15.997215 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:15.997179    4993 generic.go:356] "Generic (PLEG): container finished" podID="4a7e100d-e267-447c-b21d-2de78d044d92" containerID="5ee1b4d4e8640b2ede989af437323e36f76cd97f978cfc43e958a76ccd22a789" exitCode=60
Nov 01 15:00:15.997215 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:15.997202    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892420-c5b42" event={"ID":"4a7e100d-e267-447c-b21d-2de78d044d92","Type":"ContainerDied","Data":"5ee1b4d4e8640b2ede989af437323e36f76cd97f978cfc43e958a76ccd22a789"}
Nov 01 15:00:15.997718 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:15.997277    4993 scope.go:122] "RemoveContainer" containerID="73daf43deb189a5030c505fea8c49420cc6073e3f6180ab8c9f8119c475e3b02"
Nov 01 15:00:15.997718 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:15.997700    4993 scope.go:122] "RemoveContainer" containerID="5ee1b4d4e8640b2ede989af437323e36f76cd97f978cfc43e958a76ccd22a789"
Nov 01 15:00:15.998046 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:00:15.998020    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892420-c5b42_odh-ai-gateway-infra(4a7e100d-e267-447c-b21d-2de78d044d92)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892420-c5b42" podUID="4a7e100d-e267-447c-b21d-2de78d044d92"
Nov 01 15:00:16.788177 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:16.788140    4993 scope.go:122] "RemoveContainer" containerID="62db9516ffc621bed2df28962be6675d2bb01fee90236e90cfeaa49e183c4b6e"
Nov 01 15:00:17.007096 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:17.007061    4993 generic.go:356] "Generic (PLEG): container finished" podID="a4438535-151d-4848-a5e1-2d0809e32e1e" containerID="26281dac43fdbe9f00dda8473fed041a809d4ec74456c4dc9ff7e27e41b0a461" exitCode=60
Nov 01 15:00:17.007528 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:17.007131    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-0c9d0a-29892420-ln9zk" event={"ID":"a4438535-151d-4848-a5e1-2d0809e32e1e","Type":"ContainerDied","Data":"26281dac43fdbe9f00dda8473fed041a809d4ec74456c4dc9ff7e27e41b0a461"}
Nov 01 15:00:17.007528 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:17.007207    4993 scope.go:122] "RemoveContainer" containerID="3d5924fb1ac1147414f4b7ff457dbae783fb57805bface08587fdde13b387532"
Nov 01 15:00:17.007627 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:17.007610    4993 scope.go:122] "RemoveContainer" containerID="26281dac43fdbe9f00dda8473fed041a809d4ec74456c4dc9ff7e27e41b0a461"
Nov 01 15:00:17.007945 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:00:17.007922    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-0c9d0a-29892420-ln9zk_odh-ai-gateway-infra(a4438535-151d-4848-a5e1-2d0809e32e1e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-0c9d0a-29892420-ln9zk" podUID="a4438535-151d-4848-a5e1-2d0809e32e1e"
Nov 01 15:00:17.164042 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:17.163955    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892420-c5b42"]
Nov 01 15:00:17.334033 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:17.334006    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892420-c5b42"
Nov 01 15:00:17.435445 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:17.435343    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a7e100d-e267-447c-b21d-2de78d044d92-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a7e100d-e267-447c-b21d-2de78d044d92-maas-api-service-ca\") pod \"4a7e100d-e267-447c-b21d-2de78d044d92\" (UID: \"4a7e100d-e267-447c-b21d-2de78d044d92\") "
Nov 01 15:00:17.435445 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:17.435410    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a7e100d-e267-447c-b21d-2de78d044d92-kube-api-access-whtbm\" (UniqueName: \"kubernetes.io/projected/4a7e100d-e267-447c-b21d-2de78d044d92-kube-api-access-whtbm\") pod \"4a7e100d-e267-447c-b21d-2de78d044d92\" (UID: \"4a7e100d-e267-447c-b21d-2de78d044d92\") "
Nov 01 15:00:17.435707 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:17.435681    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a7e100d-e267-447c-b21d-2de78d044d92-maas-api-service-ca" pod "4a7e100d-e267-447c-b21d-2de78d044d92" (UID: "4a7e100d-e267-447c-b21d-2de78d044d92"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:00:17.435859 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:17.435838    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a7e100d-e267-447c-b21d-2de78d044d92-maas-api-service-ca\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:00:17.437611 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:17.437590    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a7e100d-e267-447c-b21d-2de78d044d92-kube-api-access-whtbm" pod "4a7e100d-e267-447c-b21d-2de78d044d92" (UID: "4a7e100d-e267-447c-b21d-2de78d044d92"). InnerVolumeSpecName "kube-api-access-whtbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:00:17.537591 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:17.537555    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whtbm\" (UniqueName: \"kubernetes.io/projected/4a7e100d-e267-447c-b21d-2de78d044d92-kube-api-access-whtbm\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:00:18.014182 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:18.014143    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892420-c5b42"
Nov 01 15:00:18.014182 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:18.014152    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892420-c5b42" event={"ID":"4a7e100d-e267-447c-b21d-2de78d044d92","Type":"ContainerDied","Data":"5c0f4ebefbbfafc1aebe76b386c1f6096bb680d28fea05450d3ec9d338a0a6f6"}
Nov 01 15:00:18.014742 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:18.014231    4993 scope.go:122] "RemoveContainer" containerID="5ee1b4d4e8640b2ede989af437323e36f76cd97f978cfc43e958a76ccd22a789"
Nov 01 15:00:18.016406 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:18.016378    4993 generic.go:356] "Generic (PLEG): container finished" podID="5f5bf8a3-874b-4026-a9c8-8d8eedcfc23d" containerID="fab4c004e2482111a3b1a9834f8de37b08bbc1187a71f8ac882043e23ac9d3fc" exitCode=60
Nov 01 15:00:18.016547 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:18.016442    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892420-5jxkv" event={"ID":"5f5bf8a3-874b-4026-a9c8-8d8eedcfc23d","Type":"ContainerDied","Data":"fab4c004e2482111a3b1a9834f8de37b08bbc1187a71f8ac882043e23ac9d3fc"}
Nov 01 15:00:18.016889 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:18.016867    4993 scope.go:122] "RemoveContainer" containerID="fab4c004e2482111a3b1a9834f8de37b08bbc1187a71f8ac882043e23ac9d3fc"
Nov 01 15:00:18.017229 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:00:18.017207    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892420-5jxkv_odh-ai-gateway-infra(5f5bf8a3-874b-4026-a9c8-8d8eedcfc23d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892420-5jxkv" podUID="5f5bf8a3-874b-4026-a9c8-8d8eedcfc23d"
Nov 01 15:00:18.025856 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:18.025835    4993 scope.go:122] "RemoveContainer" containerID="62db9516ffc621bed2df28962be6675d2bb01fee90236e90cfeaa49e183c4b6e"
Nov 01 15:00:18.124370 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:18.124336    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-0c9d0a-29892420-ln9zk"]
Nov 01 15:00:18.274427 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:18.274395    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892420-c5b42"]
Nov 01 15:00:18.279182 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:18.279132    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-d06633-29892420-c5b42"]
Nov 01 15:00:18.289810 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:18.289784    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-0c9d0a-29892420-ln9zk"
Nov 01 15:00:18.447762 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:18.447734    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4438535-151d-4848-a5e1-2d0809e32e1e-kube-api-access-ffvfr\" (UniqueName: \"kubernetes.io/projected/a4438535-151d-4848-a5e1-2d0809e32e1e-kube-api-access-ffvfr\") pod \"a4438535-151d-4848-a5e1-2d0809e32e1e\" (UID: \"a4438535-151d-4848-a5e1-2d0809e32e1e\") "
Nov 01 15:00:18.447927 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:18.447798    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4438535-151d-4848-a5e1-2d0809e32e1e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4438535-151d-4848-a5e1-2d0809e32e1e-maas-api-service-ca\") pod \"a4438535-151d-4848-a5e1-2d0809e32e1e\" (UID: \"a4438535-151d-4848-a5e1-2d0809e32e1e\") "
Nov 01 15:00:18.448126 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:18.448102    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4438535-151d-4848-a5e1-2d0809e32e1e-maas-api-service-ca" pod "a4438535-151d-4848-a5e1-2d0809e32e1e" (UID: "a4438535-151d-4848-a5e1-2d0809e32e1e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:00:18.449849 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:18.449824    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4438535-151d-4848-a5e1-2d0809e32e1e-kube-api-access-ffvfr" pod "a4438535-151d-4848-a5e1-2d0809e32e1e" (UID: "a4438535-151d-4848-a5e1-2d0809e32e1e"). InnerVolumeSpecName "kube-api-access-ffvfr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:00:18.549134 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:18.549056    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffvfr\" (UniqueName: \"kubernetes.io/projected/a4438535-151d-4848-a5e1-2d0809e32e1e-kube-api-access-ffvfr\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:00:18.549134 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:18.549087    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4438535-151d-4848-a5e1-2d0809e32e1e-maas-api-service-ca\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:00:18.797243 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:18.797206    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a7e100d-e267-447c-b21d-2de78d044d92" path="/var/lib/kubelet/pods/4a7e100d-e267-447c-b21d-2de78d044d92/volumes"
Nov 01 15:00:19.029350 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:19.029309    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-0c9d0a-29892420-ln9zk" event={"ID":"a4438535-151d-4848-a5e1-2d0809e32e1e","Type":"ContainerDied","Data":"a93bbf15466bcce2fff6234c005147611fd177faa416712d4b65d30b78d0ea71"}
Nov 01 15:00:19.029760 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:19.029375    4993 scope.go:122] "RemoveContainer" containerID="26281dac43fdbe9f00dda8473fed041a809d4ec74456c4dc9ff7e27e41b0a461"
Nov 01 15:00:19.029760 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:19.029317    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-0c9d0a-29892420-ln9zk"
Nov 01 15:00:19.181738 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:19.181651    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892420-5jxkv"]
Nov 01 15:00:19.186214 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:19.186182    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-0c9d0a-29892420-ln9zk"]
Nov 01 15:00:19.226153 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:19.226116    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-0c9d0a-29892420-ln9zk"]
Nov 01 15:00:19.373791 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:19.373759    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892420-5jxkv"
Nov 01 15:00:19.459504 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:19.459429    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f5bf8a3-874b-4026-a9c8-8d8eedcfc23d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f5bf8a3-874b-4026-a9c8-8d8eedcfc23d-maas-api-service-ca\") pod \"5f5bf8a3-874b-4026-a9c8-8d8eedcfc23d\" (UID: \"5f5bf8a3-874b-4026-a9c8-8d8eedcfc23d\") "
Nov 01 15:00:19.459504 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:19.459474    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f5bf8a3-874b-4026-a9c8-8d8eedcfc23d-kube-api-access-p4xdp\" (UniqueName: \"kubernetes.io/projected/5f5bf8a3-874b-4026-a9c8-8d8eedcfc23d-kube-api-access-p4xdp\") pod \"5f5bf8a3-874b-4026-a9c8-8d8eedcfc23d\" (UID: \"5f5bf8a3-874b-4026-a9c8-8d8eedcfc23d\") "
Nov 01 15:00:19.459888 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:19.459861    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f5bf8a3-874b-4026-a9c8-8d8eedcfc23d-maas-api-service-ca" pod "5f5bf8a3-874b-4026-a9c8-8d8eedcfc23d" (UID: "5f5bf8a3-874b-4026-a9c8-8d8eedcfc23d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:00:19.461791 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:19.461760    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f5bf8a3-874b-4026-a9c8-8d8eedcfc23d-kube-api-access-p4xdp" pod "5f5bf8a3-874b-4026-a9c8-8d8eedcfc23d" (UID: "5f5bf8a3-874b-4026-a9c8-8d8eedcfc23d"). InnerVolumeSpecName "kube-api-access-p4xdp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:00:19.561006 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:19.560965    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f5bf8a3-874b-4026-a9c8-8d8eedcfc23d-maas-api-service-ca\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:00:19.561006 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:19.560999    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4xdp\" (UniqueName: \"kubernetes.io/projected/5f5bf8a3-874b-4026-a9c8-8d8eedcfc23d-kube-api-access-p4xdp\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:00:20.036504 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:20.036469    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892420-5jxkv"
Nov 01 15:00:20.037026 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:20.036470    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892420-5jxkv" event={"ID":"5f5bf8a3-874b-4026-a9c8-8d8eedcfc23d","Type":"ContainerDied","Data":"081c94c371080625e34a4685421ee31e5973b4c9275bfe9dc9d6f46d79a4f44e"}
Nov 01 15:00:20.037026 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:20.036623    4993 scope.go:122] "RemoveContainer" containerID="fab4c004e2482111a3b1a9834f8de37b08bbc1187a71f8ac882043e23ac9d3fc"
Nov 01 15:00:20.213967 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:20.213921    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892420-5jxkv"]
Nov 01 15:00:20.232764 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:20.232726    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892420-5jxkv"]
Nov 01 15:00:20.811639 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:20.811604    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f5bf8a3-874b-4026-a9c8-8d8eedcfc23d" path="/var/lib/kubelet/pods/5f5bf8a3-874b-4026-a9c8-8d8eedcfc23d/volumes"
Nov 01 15:00:20.812039 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:20.812025    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4438535-151d-4848-a5e1-2d0809e32e1e" path="/var/lib/kubelet/pods/a4438535-151d-4848-a5e1-2d0809e32e1e/volumes"
Nov 01 15:00:27.576283 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:00:27.576182    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:00:55.024463 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:55.024426    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-6fcb7b-5488d47698-fw2ch"]
Nov 01 15:00:55.024942 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:55.024928    4993 cpu_manager.go:414] "containerMap: removing container" podUID="5f5bf8a3-874b-4026-a9c8-8d8eedcfc23d" containerName="cleanup"
Nov 01 15:00:55.024993 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:55.024945    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f5bf8a3-874b-4026-a9c8-8d8eedcfc23d" containerName="cleanup"
Nov 01 15:00:55.024993 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:55.024963    4993 cpu_manager.go:414] "containerMap: removing container" podUID="a4438535-151d-4848-a5e1-2d0809e32e1e" containerName="cleanup"
Nov 01 15:00:55.024993 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:55.024969    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4438535-151d-4848-a5e1-2d0809e32e1e" containerName="cleanup"
Nov 01 15:00:55.024993 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:55.024985    4993 cpu_manager.go:414] "containerMap: removing container" podUID="a4438535-151d-4848-a5e1-2d0809e32e1e" containerName="cleanup"
Nov 01 15:00:55.024993 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:55.024991    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4438535-151d-4848-a5e1-2d0809e32e1e" containerName="cleanup"
Nov 01 15:00:55.025200 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:55.024999    4993 cpu_manager.go:414] "containerMap: removing container" podUID="4a7e100d-e267-447c-b21d-2de78d044d92" containerName="cleanup"
Nov 01 15:00:55.025200 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:55.025005    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a7e100d-e267-447c-b21d-2de78d044d92" containerName="cleanup"
Nov 01 15:00:55.025200 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:55.025013    4993 cpu_manager.go:414] "containerMap: removing container" podUID="5f5bf8a3-874b-4026-a9c8-8d8eedcfc23d" containerName="cleanup"
Nov 01 15:00:55.025200 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:55.025017    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f5bf8a3-874b-4026-a9c8-8d8eedcfc23d" containerName="cleanup"
Nov 01 15:00:55.025200 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:55.025025    4993 cpu_manager.go:414] "containerMap: removing container" podUID="4a7e100d-e267-447c-b21d-2de78d044d92" containerName="cleanup"
Nov 01 15:00:55.025200 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:55.025029    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a7e100d-e267-447c-b21d-2de78d044d92" containerName="cleanup"
Nov 01 15:00:55.025200 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:55.025043    4993 cpu_manager.go:414] "containerMap: removing container" podUID="5f5bf8a3-874b-4026-a9c8-8d8eedcfc23d" containerName="cleanup"
Nov 01 15:00:55.025200 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:55.025048    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f5bf8a3-874b-4026-a9c8-8d8eedcfc23d" containerName="cleanup"
Nov 01 15:00:55.025200 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:55.025123    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4438535-151d-4848-a5e1-2d0809e32e1e" containerName="cleanup"
Nov 01 15:00:55.025200 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:55.025137    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a7e100d-e267-447c-b21d-2de78d044d92" containerName="cleanup"
Nov 01 15:00:55.025200 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:55.025146    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f5bf8a3-874b-4026-a9c8-8d8eedcfc23d" containerName="cleanup"
Nov 01 15:00:55.025200 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:55.025155    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a7e100d-e267-447c-b21d-2de78d044d92" containerName="cleanup"
Nov 01 15:00:55.025200 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:55.025191    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4438535-151d-4848-a5e1-2d0809e32e1e" containerName="cleanup"
Nov 01 15:00:55.025200 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:55.025201    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f5bf8a3-874b-4026-a9c8-8d8eedcfc23d" containerName="cleanup"
Nov 01 15:00:55.030015 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:55.029995    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-6fcb7b-5488d47698-fw2ch"
Nov 01 15:00:55.076999 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:55.076957    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69gtl\" (UniqueName: \"kubernetes.io/projected/cf03339d-d720-4474-b495-2d149d09d33a-kube-api-access-69gtl\") pod \"maas-api-e2e-autores-6fcb7b-5488d47698-fw2ch\" (UID: \"cf03339d-d720-4474-b495-2d149d09d33a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-6fcb7b-5488d47698-fw2ch"
Nov 01 15:00:55.077233 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:55.077210    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf03339d-d720-4474-b495-2d149d09d33a-metrics-certs\") pod \"maas-api-e2e-autores-6fcb7b-5488d47698-fw2ch\" (UID: \"cf03339d-d720-4474-b495-2d149d09d33a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-6fcb7b-5488d47698-fw2ch"
Nov 01 15:00:55.077332 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:55.077312    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf03339d-d720-4474-b495-2d149d09d33a-openshift-service-ca\") pod \"maas-api-e2e-autores-6fcb7b-5488d47698-fw2ch\" (UID: \"cf03339d-d720-4474-b495-2d149d09d33a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-6fcb7b-5488d47698-fw2ch"
Nov 01 15:00:55.077388 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:55.077374    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf03339d-d720-4474-b495-2d149d09d33a-maas-api-tls\") pod \"maas-api-e2e-autores-6fcb7b-5488d47698-fw2ch\" (UID: \"cf03339d-d720-4474-b495-2d149d09d33a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-6fcb7b-5488d47698-fw2ch"
Nov 01 15:00:55.080614 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:55.080592    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-6fcb7b-serving-cert\""
Nov 01 15:00:55.130389 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:55.130353    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-6fcb7b-5488d47698-fw2ch"]
Nov 01 15:00:55.178618 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:55.178581    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf03339d-d720-4474-b495-2d149d09d33a-openshift-service-ca\") pod \"maas-api-e2e-autores-6fcb7b-5488d47698-fw2ch\" (UID: \"cf03339d-d720-4474-b495-2d149d09d33a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-6fcb7b-5488d47698-fw2ch"
Nov 01 15:00:55.178825 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:55.178672    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf03339d-d720-4474-b495-2d149d09d33a-maas-api-tls\") pod \"maas-api-e2e-autores-6fcb7b-5488d47698-fw2ch\" (UID: \"cf03339d-d720-4474-b495-2d149d09d33a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-6fcb7b-5488d47698-fw2ch"
Nov 01 15:00:55.178825 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:55.178779    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69gtl\" (UniqueName: \"kubernetes.io/projected/cf03339d-d720-4474-b495-2d149d09d33a-kube-api-access-69gtl\") pod \"maas-api-e2e-autores-6fcb7b-5488d47698-fw2ch\" (UID: \"cf03339d-d720-4474-b495-2d149d09d33a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-6fcb7b-5488d47698-fw2ch"
Nov 01 15:00:55.178944 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:55.178924    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf03339d-d720-4474-b495-2d149d09d33a-metrics-certs\") pod \"maas-api-e2e-autores-6fcb7b-5488d47698-fw2ch\" (UID: \"cf03339d-d720-4474-b495-2d149d09d33a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-6fcb7b-5488d47698-fw2ch"
Nov 01 15:00:55.180137 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:55.179514    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf03339d-d720-4474-b495-2d149d09d33a-openshift-service-ca\") pod \"maas-api-e2e-autores-6fcb7b-5488d47698-fw2ch\" (UID: \"cf03339d-d720-4474-b495-2d149d09d33a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-6fcb7b-5488d47698-fw2ch"
Nov 01 15:00:55.180137 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:00:55.179629    4993 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-6fcb7b-serving-cert: secret "maas-api-e2e-autores-6fcb7b-serving-cert" not found
Nov 01 15:00:55.180137 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:00:55.179694    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf03339d-d720-4474-b495-2d149d09d33a-maas-api-tls podName:cf03339d-d720-4474-b495-2d149d09d33a nodeName:}" failed. No retries permitted until 2026-11-01 15:00:55.679675478 +0000 UTC m=+2643.009538192 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cf03339d-d720-4474-b495-2d149d09d33a-maas-api-tls") pod "maas-api-e2e-autores-6fcb7b-5488d47698-fw2ch" (UID: "cf03339d-d720-4474-b495-2d149d09d33a") : secret "maas-api-e2e-autores-6fcb7b-serving-cert" not found
Nov 01 15:00:55.183692 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:55.183327    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf03339d-d720-4474-b495-2d149d09d33a-metrics-certs\") pod \"maas-api-e2e-autores-6fcb7b-5488d47698-fw2ch\" (UID: \"cf03339d-d720-4474-b495-2d149d09d33a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-6fcb7b-5488d47698-fw2ch"
Nov 01 15:00:55.242786 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:55.242762    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69gtl\" (UniqueName: \"kubernetes.io/projected/cf03339d-d720-4474-b495-2d149d09d33a-kube-api-access-69gtl\") pod \"maas-api-e2e-autores-6fcb7b-5488d47698-fw2ch\" (UID: \"cf03339d-d720-4474-b495-2d149d09d33a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-6fcb7b-5488d47698-fw2ch"
Nov 01 15:00:55.688909 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:55.688865    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf03339d-d720-4474-b495-2d149d09d33a-maas-api-tls\") pod \"maas-api-e2e-autores-6fcb7b-5488d47698-fw2ch\" (UID: \"cf03339d-d720-4474-b495-2d149d09d33a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-6fcb7b-5488d47698-fw2ch"
Nov 01 15:00:55.691341 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:55.691310    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf03339d-d720-4474-b495-2d149d09d33a-maas-api-tls\") pod \"maas-api-e2e-autores-6fcb7b-5488d47698-fw2ch\" (UID: \"cf03339d-d720-4474-b495-2d149d09d33a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-6fcb7b-5488d47698-fw2ch"
Nov 01 15:00:55.945680 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:55.945590    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-6fcb7b-5488d47698-fw2ch"
Nov 01 15:00:56.334104 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:56.334070    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-6fcb7b-5488d47698-fw2ch"]
Nov 01 15:00:56.343650 ip-10-0-134-37 kubenswrapper[4993]: W1101 15:00:56.343614    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf03339d_d720_4474_b495_2d149d09d33a.slice/crio-3550d88839a9d5549f2f29d4950080a6b00d902e8267de11fb0252fad762cd82 WatchSource:0}: Error finding container 3550d88839a9d5549f2f29d4950080a6b00d902e8267de11fb0252fad762cd82: Status 404 returned error can't find the container with id 3550d88839a9d5549f2f29d4950080a6b00d902e8267de11fb0252fad762cd82
Nov 01 15:00:57.295708 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:57.295626    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-6fcb7b-5488d47698-fw2ch" event={"ID":"cf03339d-d720-4474-b495-2d149d09d33a","Type":"ContainerStarted","Data":"3550d88839a9d5549f2f29d4950080a6b00d902e8267de11fb0252fad762cd82"}
Nov 01 15:00:57.502333 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:00:57.502295    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:00:58.307887 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:58.307839    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-6fcb7b-5488d47698-fw2ch" event={"ID":"cf03339d-d720-4474-b495-2d149d09d33a","Type":"ContainerStarted","Data":"65c372fee81064e2ed8f2b5839fde3ec607792571b393718c4e82548618cb214"}
Nov 01 15:00:58.308122 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:58.307921    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-6fcb7b-5488d47698-fw2ch"
Nov 01 15:00:58.439655 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:58.439599    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-6fcb7b-5488d47698-fw2ch" podStartSLOduration=3.555148335 podStartE2EDuration="4.439580875s" totalImagesPullingTime="884.43254ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:00:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:00:58.438641768 +0000 UTC m=+2645.768504502" watchObservedRunningTime="2026-11-01 15:00:58.439580875 +0000 UTC m=+2645.769443647"
Nov 01 15:00:58.879053 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:58.879020    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-b3f06c-76587fff79-m6j9s"]
Nov 01 15:00:58.879835 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:58.879782    4993 cpu_manager.go:414] "containerMap: removing container" podUID="a4438535-151d-4848-a5e1-2d0809e32e1e" containerName="cleanup"
Nov 01 15:00:58.880050 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:58.880031    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4438535-151d-4848-a5e1-2d0809e32e1e" containerName="cleanup"
Nov 01 15:00:58.880199 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:58.880184    4993 cpu_manager.go:414] "containerMap: removing container" podUID="4a7e100d-e267-447c-b21d-2de78d044d92" containerName="cleanup"
Nov 01 15:00:58.880293 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:58.880281    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a7e100d-e267-447c-b21d-2de78d044d92" containerName="cleanup"
Nov 01 15:00:58.880641 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:58.880619    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f5bf8a3-874b-4026-a9c8-8d8eedcfc23d" containerName="cleanup"
Nov 01 15:00:58.880774 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:58.880759    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a7e100d-e267-447c-b21d-2de78d044d92" containerName="cleanup"
Nov 01 15:00:58.880874 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:58.880862    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4438535-151d-4848-a5e1-2d0809e32e1e" containerName="cleanup"
Nov 01 15:00:58.891220 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:58.891192    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b3f06c-76587fff79-m6j9s"
Nov 01 15:00:58.944563 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:58.944525    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-b3f06c\""
Nov 01 15:00:58.944752 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:58.944732    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-b3f06c-dockercfg-kbb9q\""
Nov 01 15:00:58.947621 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:58.947597    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-b3f06c-76587fff79-m6j9s"]
Nov 01 15:00:58.949598 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:58.949565    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tzbh\" (UniqueName: \"kubernetes.io/projected/7e3a0173-3e79-4e57-a9bb-0b9eb792da3a-kube-api-access-8tzbh\") pod \"payload-processing-e2e-shared-b-w4-b3f06c-76587fff79-m6j9s\" (UID: \"7e3a0173-3e79-4e57-a9bb-0b9eb792da3a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b3f06c-76587fff79-m6j9s"
Nov 01 15:00:58.949888 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:58.949783    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e3a0173-3e79-4e57-a9bb-0b9eb792da3a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-b3f06c-76587fff79-m6j9s\" (UID: \"7e3a0173-3e79-4e57-a9bb-0b9eb792da3a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b3f06c-76587fff79-m6j9s"
Nov 01 15:00:59.051021 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:59.050985    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tzbh\" (UniqueName: \"kubernetes.io/projected/7e3a0173-3e79-4e57-a9bb-0b9eb792da3a-kube-api-access-8tzbh\") pod \"payload-processing-e2e-shared-b-w4-b3f06c-76587fff79-m6j9s\" (UID: \"7e3a0173-3e79-4e57-a9bb-0b9eb792da3a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b3f06c-76587fff79-m6j9s"
Nov 01 15:00:59.051238 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:59.051088    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e3a0173-3e79-4e57-a9bb-0b9eb792da3a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-b3f06c-76587fff79-m6j9s\" (UID: \"7e3a0173-3e79-4e57-a9bb-0b9eb792da3a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b3f06c-76587fff79-m6j9s"
Nov 01 15:00:59.052028 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:59.051997    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e3a0173-3e79-4e57-a9bb-0b9eb792da3a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-b3f06c-76587fff79-m6j9s\" (UID: \"7e3a0173-3e79-4e57-a9bb-0b9eb792da3a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b3f06c-76587fff79-m6j9s"
Nov 01 15:00:59.104924 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:59.104887    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tzbh\" (UniqueName: \"kubernetes.io/projected/7e3a0173-3e79-4e57-a9bb-0b9eb792da3a-kube-api-access-8tzbh\") pod \"payload-processing-e2e-shared-b-w4-b3f06c-76587fff79-m6j9s\" (UID: \"7e3a0173-3e79-4e57-a9bb-0b9eb792da3a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b3f06c-76587fff79-m6j9s"
Nov 01 15:00:59.216082 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:59.216002    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b3f06c-76587fff79-m6j9s"
Nov 01 15:00:59.594435 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:00:59.594402    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-b3f06c-76587fff79-m6j9s"]
Nov 01 15:00:59.601529 ip-10-0-134-37 kubenswrapper[4993]: W1101 15:00:59.601499    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e3a0173_3e79_4e57_a9bb_0b9eb792da3a.slice/crio-e735c611ecbad2532c5759db56e19f59253f5e0dabe297d4ae3311296aee734c WatchSource:0}: Error finding container e735c611ecbad2532c5759db56e19f59253f5e0dabe297d4ae3311296aee734c: Status 404 returned error can't find the container with id e735c611ecbad2532c5759db56e19f59253f5e0dabe297d4ae3311296aee734c
Nov 01 15:01:00.327389 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:01:00.327352    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b3f06c-76587fff79-m6j9s" event={"ID":"7e3a0173-3e79-4e57-a9bb-0b9eb792da3a","Type":"ContainerStarted","Data":"69d10cbf8d5bac171dfd74182175ac5dfe74e476b4059eadf53c8f9465e73ab0"}
Nov 01 15:01:00.327789 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:01:00.327404    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b3f06c-76587fff79-m6j9s" event={"ID":"7e3a0173-3e79-4e57-a9bb-0b9eb792da3a","Type":"ContainerStarted","Data":"e735c611ecbad2532c5759db56e19f59253f5e0dabe297d4ae3311296aee734c"}
Nov 01 15:01:00.327789 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:01:00.327436    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b3f06c-76587fff79-m6j9s"
Nov 01 15:01:00.436644 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:01:00.436592    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b3f06c-76587fff79-m6j9s" podStartSLOduration=2.436579106 podStartE2EDuration="2.436579106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:00:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:01:00.433971099 +0000 UTC m=+2647.763833836" watchObservedRunningTime="2026-11-01 15:01:00.436579106 +0000 UTC m=+2647.766441842"
Nov 01 15:01:04.327518 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:01:04.327429    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-6fcb7b-5488d47698-fw2ch"
Nov 01 15:01:11.336429 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:01:11.336400    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b3f06c-76587fff79-m6j9s"
Nov 01 15:01:27.501454 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:01:27.501405    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:01:27.558201 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:01:27.558152    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca0f9e-645d8b74c-4lpp2"]
Nov 01 15:01:27.562319 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:01:27.562296    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca0f9e-645d8b74c-4lpp2"
Nov 01 15:01:27.605039 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:01:27.605009    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e34d9fc-4d6c-496b-95db-6ddd2b2f2c10-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-ca0f9e-645d8b74c-4lpp2\" (UID: \"0e34d9fc-4d6c-496b-95db-6ddd2b2f2c10\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca0f9e-645d8b74c-4lpp2"
Nov 01 15:01:27.605160 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:01:27.605042    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx85s\" (UniqueName: \"kubernetes.io/projected/0e34d9fc-4d6c-496b-95db-6ddd2b2f2c10-kube-api-access-hx85s\") pod \"maas-api-e2e-shared-b-w2-ca0f9e-645d8b74c-4lpp2\" (UID: \"0e34d9fc-4d6c-496b-95db-6ddd2b2f2c10\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca0f9e-645d8b74c-4lpp2"
Nov 01 15:01:27.605241 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:01:27.605159    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e34d9fc-4d6c-496b-95db-6ddd2b2f2c10-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-ca0f9e-645d8b74c-4lpp2\" (UID: \"0e34d9fc-4d6c-496b-95db-6ddd2b2f2c10\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca0f9e-645d8b74c-4lpp2"
Nov 01 15:01:27.605286 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:01:27.605250    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e34d9fc-4d6c-496b-95db-6ddd2b2f2c10-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-ca0f9e-645d8b74c-4lpp2\" (UID: \"0e34d9fc-4d6c-496b-95db-6ddd2b2f2c10\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca0f9e-645d8b74c-4lpp2"
Nov 01 15:01:27.659377 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:01:27.659343    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-ca0f9e-serving-cert\""
Nov 01 15:01:27.662387 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:01:27.662359    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca0f9e-645d8b74c-4lpp2"]
Nov 01 15:01:27.706240 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:01:27.706209    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e34d9fc-4d6c-496b-95db-6ddd2b2f2c10-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-ca0f9e-645d8b74c-4lpp2\" (UID: \"0e34d9fc-4d6c-496b-95db-6ddd2b2f2c10\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca0f9e-645d8b74c-4lpp2"
Nov 01 15:01:27.706424 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:01:27.706268    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e34d9fc-4d6c-496b-95db-6ddd2b2f2c10-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-ca0f9e-645d8b74c-4lpp2\" (UID: \"0e34d9fc-4d6c-496b-95db-6ddd2b2f2c10\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca0f9e-645d8b74c-4lpp2"
Nov 01 15:01:27.706424 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:01:27.706405    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e34d9fc-4d6c-496b-95db-6ddd2b2f2c10-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-ca0f9e-645d8b74c-4lpp2\" (UID: \"0e34d9fc-4d6c-496b-95db-6ddd2b2f2c10\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca0f9e-645d8b74c-4lpp2"
Nov 01 15:01:27.706556 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:01:27.706431    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx85s\" (UniqueName: \"kubernetes.io/projected/0e34d9fc-4d6c-496b-95db-6ddd2b2f2c10-kube-api-access-hx85s\") pod \"maas-api-e2e-shared-b-w2-ca0f9e-645d8b74c-4lpp2\" (UID: \"0e34d9fc-4d6c-496b-95db-6ddd2b2f2c10\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca0f9e-645d8b74c-4lpp2"
Nov 01 15:01:27.706851 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:01:27.706813    4993 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca0f9e-serving-cert: secret "maas-api-e2e-shared-b-w2-ca0f9e-serving-cert" not found
Nov 01 15:01:27.706979 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:01:27.706917    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e34d9fc-4d6c-496b-95db-6ddd2b2f2c10-maas-api-tls podName:0e34d9fc-4d6c-496b-95db-6ddd2b2f2c10 nodeName:}" failed. No retries permitted until 2026-11-01 15:01:28.206894989 +0000 UTC m=+2675.536757721 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0e34d9fc-4d6c-496b-95db-6ddd2b2f2c10-maas-api-tls") pod "maas-api-e2e-shared-b-w2-ca0f9e-645d8b74c-4lpp2" (UID: "0e34d9fc-4d6c-496b-95db-6ddd2b2f2c10") : secret "maas-api-e2e-shared-b-w2-ca0f9e-serving-cert" not found
Nov 01 15:01:27.707058 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:01:27.707042    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e34d9fc-4d6c-496b-95db-6ddd2b2f2c10-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-ca0f9e-645d8b74c-4lpp2\" (UID: \"0e34d9fc-4d6c-496b-95db-6ddd2b2f2c10\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca0f9e-645d8b74c-4lpp2"
Nov 01 15:01:27.709070 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:01:27.709045    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e34d9fc-4d6c-496b-95db-6ddd2b2f2c10-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-ca0f9e-645d8b74c-4lpp2\" (UID: \"0e34d9fc-4d6c-496b-95db-6ddd2b2f2c10\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca0f9e-645d8b74c-4lpp2"
Nov 01 15:01:27.760592 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:01:27.760528    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx85s\" (UniqueName: \"kubernetes.io/projected/0e34d9fc-4d6c-496b-95db-6ddd2b2f2c10-kube-api-access-hx85s\") pod \"maas-api-e2e-shared-b-w2-ca0f9e-645d8b74c-4lpp2\" (UID: \"0e34d9fc-4d6c-496b-95db-6ddd2b2f2c10\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca0f9e-645d8b74c-4lpp2"
Nov 01 15:01:28.212348 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:01:28.212261    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e34d9fc-4d6c-496b-95db-6ddd2b2f2c10-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-ca0f9e-645d8b74c-4lpp2\" (UID: \"0e34d9fc-4d6c-496b-95db-6ddd2b2f2c10\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca0f9e-645d8b74c-4lpp2"
Nov 01 15:01:28.214693 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:01:28.214665    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e34d9fc-4d6c-496b-95db-6ddd2b2f2c10-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-ca0f9e-645d8b74c-4lpp2\" (UID: \"0e34d9fc-4d6c-496b-95db-6ddd2b2f2c10\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca0f9e-645d8b74c-4lpp2"
Nov 01 15:01:28.476559 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:01:28.476455    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca0f9e-645d8b74c-4lpp2"
Nov 01 15:01:28.855473 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:01:28.855435    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca0f9e-645d8b74c-4lpp2"]
Nov 01 15:01:28.860928 ip-10-0-134-37 kubenswrapper[4993]: W1101 15:01:28.860900    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e34d9fc_4d6c_496b_95db_6ddd2b2f2c10.slice/crio-dfc75b0d95a4e50afd138d77a58b879e2f01e87b098bd710377a177002939d28 WatchSource:0}: Error finding container dfc75b0d95a4e50afd138d77a58b879e2f01e87b098bd710377a177002939d28: Status 404 returned error can't find the container with id dfc75b0d95a4e50afd138d77a58b879e2f01e87b098bd710377a177002939d28
Nov 01 15:01:29.512358 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:01:29.512309    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca0f9e-645d8b74c-4lpp2" event={"ID":"0e34d9fc-4d6c-496b-95db-6ddd2b2f2c10","Type":"ContainerStarted","Data":"dfc75b0d95a4e50afd138d77a58b879e2f01e87b098bd710377a177002939d28"}
Nov 01 15:01:30.520346 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:01:30.520302    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca0f9e-645d8b74c-4lpp2" event={"ID":"0e34d9fc-4d6c-496b-95db-6ddd2b2f2c10","Type":"ContainerStarted","Data":"23768c861603b16576ab042e9b5f0ec433d10efa64c5a78faf44f76dc216d9d2"}
Nov 01 15:01:30.520843 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:01:30.520515    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca0f9e-645d8b74c-4lpp2"
Nov 01 15:01:30.631561 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:01:30.631508    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca0f9e-645d8b74c-4lpp2" podStartSLOduration=2.809522143 podStartE2EDuration="3.631492977s" totalImagesPullingTime="821.970834ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:01:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:01:30.628834751 +0000 UTC m=+2677.958697546" watchObservedRunningTime="2026-11-01 15:01:30.631492977 +0000 UTC m=+2677.961355714"
Nov 01 15:01:36.535136 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:01:36.535102    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca0f9e-645d8b74c-4lpp2"
Nov 01 15:01:36.549533 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:01:36.549507    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-52fffa-5cfddf5d94-6mlwc"]
Nov 01 15:01:36.553783 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:01:36.553764    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-52fffa-5cfddf5d94-6mlwc"
Nov 01 15:01:36.618202 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:01:36.607761    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-52fffa-5cfddf5d94-6mlwc"]
Nov 01 15:01:36.636322 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:01:36.636292    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-52fffa-serving-cert\""
Nov 01 15:01:36.718850 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:01:36.718816    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c28e8f1-bdac-437b-9c48-82171b8754f7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-52fffa-5cfddf5d94-6mlwc\" (UID: \"2c28e8f1-bdac-437b-9c48-82171b8754f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-52fffa-5cfddf5d94-6mlwc"
Nov 01 15:01:36.718850 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:01:36.718858    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c28e8f1-bdac-437b-9c48-82171b8754f7-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-52fffa-5cfddf5d94-6mlwc\" (UID: \"2c28e8f1-bdac-437b-9c48-82171b8754f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-52fffa-5cfddf5d94-6mlwc"
Nov 01 15:01:36.719111 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:01:36.718980    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr4dl\" (UniqueName: \"kubernetes.io/projected/2c28e8f1-bdac-437b-9c48-82171b8754f7-kube-api-access-jr4dl\") pod \"maas-api-e2e-shared-b-w1-52fffa-5cfddf5d94-6mlwc\" (UID: \"2c28e8f1-bdac-437b-9c48-82171b8754f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-52fffa-5cfddf5d94-6mlwc"
Nov 01 15:01:36.719111 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:01:36.719027    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c28e8f1-bdac-437b-9c48-82171b8754f7-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-52fffa-5cfddf5d94-6mlwc\" (UID: \"2c28e8f1-bdac-437b-9c48-82171b8754f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-52fffa-5cfddf5d94-6mlwc"
Nov 01 15:01:36.820286 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:01:36.820195    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c28e8f1-bdac-437b-9c48-82171b8754f7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-52fffa-5cfddf5d94-6mlwc\" (UID: \"2c28e8f1-bdac-437b-9c48-82171b8754f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-52fffa-5cfddf5d94-6mlwc"
Nov 01 15:01:36.820286 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:01:36.820257    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c28e8f1-bdac-437b-9c48-82171b8754f7-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-52fffa-5cfddf5d94-6mlwc\" (UID: \"2c28e8f1-bdac-437b-9c48-82171b8754f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-52fffa-5cfddf5d94-6mlwc"
Nov 01 15:01:36.820480 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:01:36.820313    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr4dl\" (UniqueName: \"kubernetes.io/projected/2c28e8f1-bdac-437b-9c48-82171b8754f7-kube-api-access-jr4dl\") pod \"maas-api-e2e-shared-b-w1-52fffa-5cfddf5d94-6mlwc\" (UID: \"2c28e8f1-bdac-437b-9c48-82171b8754f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-52fffa-5cfddf5d94-6mlwc"
Nov 01 15:01:36.820480 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:01:36.820354    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c28e8f1-bdac-437b-9c48-82171b8754f7-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-52fffa-5cfddf5d94-6mlwc\" (UID: \"2c28e8f1-bdac-437b-9c48-82171b8754f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-52fffa-5cfddf5d94-6mlwc"
Nov 01 15:01:36.820579 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:01:36.820553    4993 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-52fffa-serving-cert: secret "maas-api-e2e-shared-b-w1-52fffa-serving-cert" not found
Nov 01 15:01:36.820674 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:01:36.820649    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c28e8f1-bdac-437b-9c48-82171b8754f7-maas-api-tls podName:2c28e8f1-bdac-437b-9c48-82171b8754f7 nodeName:}" failed. No retries permitted until 2026-11-01 15:01:37.320628564 +0000 UTC m=+2684.650491282 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2c28e8f1-bdac-437b-9c48-82171b8754f7-maas-api-tls") pod "maas-api-e2e-shared-b-w1-52fffa-5cfddf5d94-6mlwc" (UID: "2c28e8f1-bdac-437b-9c48-82171b8754f7") : secret "maas-api-e2e-shared-b-w1-52fffa-serving-cert" not found
Nov 01 15:01:36.820924 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:01:36.820904    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c28e8f1-bdac-437b-9c48-82171b8754f7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-52fffa-5cfddf5d94-6mlwc\" (UID: \"2c28e8f1-bdac-437b-9c48-82171b8754f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-52fffa-5cfddf5d94-6mlwc"
Nov 01 15:01:36.823014 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:01:36.822992    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c28e8f1-bdac-437b-9c48-82171b8754f7-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-52fffa-5cfddf5d94-6mlwc\" (UID: \"2c28e8f1-bdac-437b-9c48-82171b8754f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-52fffa-5cfddf5d94-6mlwc"
Nov 01 15:01:36.878639 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:01:36.878596    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr4dl\" (UniqueName: \"kubernetes.io/projected/2c28e8f1-bdac-437b-9c48-82171b8754f7-kube-api-access-jr4dl\") pod \"maas-api-e2e-shared-b-w1-52fffa-5cfddf5d94-6mlwc\" (UID: \"2c28e8f1-bdac-437b-9c48-82171b8754f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-52fffa-5cfddf5d94-6mlwc"
Nov 01 15:01:37.328009 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:01:37.327974    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c28e8f1-bdac-437b-9c48-82171b8754f7-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-52fffa-5cfddf5d94-6mlwc\" (UID: \"2c28e8f1-bdac-437b-9c48-82171b8754f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-52fffa-5cfddf5d94-6mlwc"
Nov 01 15:01:37.330438 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:01:37.330412    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c28e8f1-bdac-437b-9c48-82171b8754f7-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-52fffa-5cfddf5d94-6mlwc\" (UID: \"2c28e8f1-bdac-437b-9c48-82171b8754f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-52fffa-5cfddf5d94-6mlwc"
Nov 01 15:01:37.468406 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:01:37.468358    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-52fffa-5cfddf5d94-6mlwc"
Nov 01 15:01:37.848432 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:01:37.848398    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-52fffa-5cfddf5d94-6mlwc"]
Nov 01 15:01:37.854036 ip-10-0-134-37 kubenswrapper[4993]: W1101 15:01:37.854002    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c28e8f1_bdac_437b_9c48_82171b8754f7.slice/crio-9fc89839f54aec63c2e39a043146c614bb324e0e57fc3ed821b36f030e197149 WatchSource:0}: Error finding container 9fc89839f54aec63c2e39a043146c614bb324e0e57fc3ed821b36f030e197149: Status 404 returned error can't find the container with id 9fc89839f54aec63c2e39a043146c614bb324e0e57fc3ed821b36f030e197149
Nov 01 15:01:38.593054 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:01:38.593005    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-52fffa-5cfddf5d94-6mlwc" event={"ID":"2c28e8f1-bdac-437b-9c48-82171b8754f7","Type":"ContainerStarted","Data":"9fc89839f54aec63c2e39a043146c614bb324e0e57fc3ed821b36f030e197149"}
Nov 01 15:01:39.601995 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:01:39.601953    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-52fffa-5cfddf5d94-6mlwc" event={"ID":"2c28e8f1-bdac-437b-9c48-82171b8754f7","Type":"ContainerStarted","Data":"77b3d270038d4af4b48b2308c0866a495230ee71106247effbcc6cc8d4a458d1"}
Nov 01 15:01:39.602872 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:01:39.602850    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-52fffa-5cfddf5d94-6mlwc"
Nov 01 15:01:39.709336 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:01:39.709284    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-52fffa-5cfddf5d94-6mlwc" podStartSLOduration=2.774325186 podStartE2EDuration="3.709271841s" totalImagesPullingTime="934.946655ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:01:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:01:39.708555857 +0000 UTC m=+2687.038418594" watchObservedRunningTime="2026-11-01 15:01:39.709271841 +0000 UTC m=+2687.039134578"
Nov 01 15:01:46.619357 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:01:46.619322    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-52fffa-5cfddf5d94-6mlwc"
Nov 01 15:01:53.181423 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:01:53.181386    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xmwtp_bb59051d-c2df-4362-a157-00f14f915e79/console-operator/3.log"
Nov 01 15:01:53.181926 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:01:53.181617    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xmwtp_bb59051d-c2df-4362-a157-00f14f915e79/console-operator/3.log"
Nov 01 15:01:54.501347 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:01:54.501313    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b3f06c-76c7d95c95-mkfqr"]
Nov 01 15:01:54.505968 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:01:54.505941    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b3f06c-76c7d95c95-mkfqr"
Nov 01 15:01:54.536775 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:01:54.536743    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6h9q\" (UniqueName: \"kubernetes.io/projected/04ff309b-01de-48c8-a36b-412e5f244b4e-kube-api-access-x6h9q\") pod \"maas-api-e2e-shared-b-w4-b3f06c-76c7d95c95-mkfqr\" (UID: \"04ff309b-01de-48c8-a36b-412e5f244b4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b3f06c-76c7d95c95-mkfqr"
Nov 01 15:01:54.537015 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:01:54.536994    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04ff309b-01de-48c8-a36b-412e5f244b4e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-b3f06c-76c7d95c95-mkfqr\" (UID: \"04ff309b-01de-48c8-a36b-412e5f244b4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b3f06c-76c7d95c95-mkfqr"
Nov 01 15:01:54.537104 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:01:54.537090    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04ff309b-01de-48c8-a36b-412e5f244b4e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-b3f06c-76c7d95c95-mkfqr\" (UID: \"04ff309b-01de-48c8-a36b-412e5f244b4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b3f06c-76c7d95c95-mkfqr"
Nov 01 15:01:54.537163 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:01:54.537119    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04ff309b-01de-48c8-a36b-412e5f244b4e-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-b3f06c-76c7d95c95-mkfqr\" (UID: \"04ff309b-01de-48c8-a36b-412e5f244b4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b3f06c-76c7d95c95-mkfqr"
Nov 01 15:01:54.555831 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:01:54.555806    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-b3f06c-serving-cert\""
Nov 01 15:01:54.605237 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:01:54.605206    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b3f06c-76c7d95c95-mkfqr"]
Nov 01 15:01:54.638234 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:01:54.638195    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04ff309b-01de-48c8-a36b-412e5f244b4e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-b3f06c-76c7d95c95-mkfqr\" (UID: \"04ff309b-01de-48c8-a36b-412e5f244b4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b3f06c-76c7d95c95-mkfqr"
Nov 01 15:01:54.638400 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:01:54.638373    4993 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b3f06c-serving-cert: secret "maas-api-e2e-shared-b-w4-b3f06c-serving-cert" not found
Nov 01 15:01:54.638463 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:01:54.638445    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04ff309b-01de-48c8-a36b-412e5f244b4e-maas-api-tls podName:04ff309b-01de-48c8-a36b-412e5f244b4e nodeName:}" failed. No retries permitted until 2026-11-01 15:01:55.138423799 +0000 UTC m=+2702.468286537 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/04ff309b-01de-48c8-a36b-412e5f244b4e-maas-api-tls") pod "maas-api-e2e-shared-b-w4-b3f06c-76c7d95c95-mkfqr" (UID: "04ff309b-01de-48c8-a36b-412e5f244b4e") : secret "maas-api-e2e-shared-b-w4-b3f06c-serving-cert" not found
Nov 01 15:01:54.638516 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:01:54.638489    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04ff309b-01de-48c8-a36b-412e5f244b4e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-b3f06c-76c7d95c95-mkfqr\" (UID: \"04ff309b-01de-48c8-a36b-412e5f244b4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b3f06c-76c7d95c95-mkfqr"
Nov 01 15:01:54.638558 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:01:54.638528    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04ff309b-01de-48c8-a36b-412e5f244b4e-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-b3f06c-76c7d95c95-mkfqr\" (UID: \"04ff309b-01de-48c8-a36b-412e5f244b4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b3f06c-76c7d95c95-mkfqr"
Nov 01 15:01:54.638621 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:01:54.638597    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6h9q\" (UniqueName: \"kubernetes.io/projected/04ff309b-01de-48c8-a36b-412e5f244b4e-kube-api-access-x6h9q\") pod \"maas-api-e2e-shared-b-w4-b3f06c-76c7d95c95-mkfqr\" (UID: \"04ff309b-01de-48c8-a36b-412e5f244b4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b3f06c-76c7d95c95-mkfqr"
Nov 01 15:01:54.639948 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:01:54.639924    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04ff309b-01de-48c8-a36b-412e5f244b4e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-b3f06c-76c7d95c95-mkfqr\" (UID: \"04ff309b-01de-48c8-a36b-412e5f244b4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b3f06c-76c7d95c95-mkfqr"
Nov 01 15:01:54.642908 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:01:54.642881    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04ff309b-01de-48c8-a36b-412e5f244b4e-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-b3f06c-76c7d95c95-mkfqr\" (UID: \"04ff309b-01de-48c8-a36b-412e5f244b4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b3f06c-76c7d95c95-mkfqr"
Nov 01 15:01:54.692136 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:01:54.692105    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6h9q\" (UniqueName: \"kubernetes.io/projected/04ff309b-01de-48c8-a36b-412e5f244b4e-kube-api-access-x6h9q\") pod \"maas-api-e2e-shared-b-w4-b3f06c-76c7d95c95-mkfqr\" (UID: \"04ff309b-01de-48c8-a36b-412e5f244b4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b3f06c-76c7d95c95-mkfqr"
Nov 01 15:01:55.147046 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:01:55.147015    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04ff309b-01de-48c8-a36b-412e5f244b4e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-b3f06c-76c7d95c95-mkfqr\" (UID: \"04ff309b-01de-48c8-a36b-412e5f244b4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b3f06c-76c7d95c95-mkfqr"
Nov 01 15:01:55.149436 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:01:55.149410    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04ff309b-01de-48c8-a36b-412e5f244b4e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-b3f06c-76c7d95c95-mkfqr\" (UID: \"04ff309b-01de-48c8-a36b-412e5f244b4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b3f06c-76c7d95c95-mkfqr"
Nov 01 15:01:55.419757 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:01:55.419677    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b3f06c-76c7d95c95-mkfqr"
Nov 01 15:01:55.800627 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:01:55.800589    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b3f06c-76c7d95c95-mkfqr"]
Nov 01 15:01:55.806068 ip-10-0-134-37 kubenswrapper[4993]: W1101 15:01:55.806042    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04ff309b_01de_48c8_a36b_412e5f244b4e.slice/crio-2f94b190a5b6e6baacc4451f7b2e9f6a6e8a5605d3ea5db8cb661bc800172ec4 WatchSource:0}: Error finding container 2f94b190a5b6e6baacc4451f7b2e9f6a6e8a5605d3ea5db8cb661bc800172ec4: Status 404 returned error can't find the container with id 2f94b190a5b6e6baacc4451f7b2e9f6a6e8a5605d3ea5db8cb661bc800172ec4
Nov 01 15:01:56.713147 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:01:56.713090    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b3f06c-76c7d95c95-mkfqr" event={"ID":"04ff309b-01de-48c8-a36b-412e5f244b4e","Type":"ContainerStarted","Data":"2f94b190a5b6e6baacc4451f7b2e9f6a6e8a5605d3ea5db8cb661bc800172ec4"}
Nov 01 15:01:57.500663 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:01:57.500630    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:01:57.723186 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:01:57.723130    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b3f06c-76c7d95c95-mkfqr" event={"ID":"04ff309b-01de-48c8-a36b-412e5f244b4e","Type":"ContainerStarted","Data":"0ac590fc478b3fe1be621c931b5ed4cf8eb4cf8b3dc12f1f2744e21067ab1e0d"}
Nov 01 15:01:57.723360 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:01:57.723219    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b3f06c-76c7d95c95-mkfqr"
Nov 01 15:01:57.831730 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:01:57.831630    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b3f06c-76c7d95c95-mkfqr" podStartSLOduration=2.887189248 podStartE2EDuration="3.831615613s" totalImagesPullingTime="944.426365ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:01:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:01:57.829431466 +0000 UTC m=+2705.159294203" watchObservedRunningTime="2026-11-01 15:01:57.831615613 +0000 UTC m=+2705.161478350"
Nov 01 15:02:03.627578 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:02:03.627543    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-8d6ea2-5f9d54c9dc-ffcvt"]
Nov 01 15:02:03.633066 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:02:03.633042    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8d6ea2-5f9d54c9dc-ffcvt"
Nov 01 15:02:03.681937 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:02:03.681901    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-8d6ea2-serving-cert\""
Nov 01 15:02:03.726426 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:02:03.726394    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-8d6ea2-5f9d54c9dc-ffcvt"]
Nov 01 15:02:03.741373 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:02:03.741337    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b3f06c-76c7d95c95-mkfqr"
Nov 01 15:02:03.750674 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:02:03.750635    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq9jp\" (UniqueName: \"kubernetes.io/projected/58304650-1c70-4e2d-9ef7-d0065e8940b7-kube-api-access-mq9jp\") pod \"maas-api-e2e-praxis-b-8d6ea2-5f9d54c9dc-ffcvt\" (UID: \"58304650-1c70-4e2d-9ef7-d0065e8940b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8d6ea2-5f9d54c9dc-ffcvt"
Nov 01 15:02:03.750849 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:02:03.750689    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58304650-1c70-4e2d-9ef7-d0065e8940b7-metrics-certs\") pod \"maas-api-e2e-praxis-b-8d6ea2-5f9d54c9dc-ffcvt\" (UID: \"58304650-1c70-4e2d-9ef7-d0065e8940b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8d6ea2-5f9d54c9dc-ffcvt"
Nov 01 15:02:03.750849 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:02:03.750801    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58304650-1c70-4e2d-9ef7-d0065e8940b7-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-8d6ea2-5f9d54c9dc-ffcvt\" (UID: \"58304650-1c70-4e2d-9ef7-d0065e8940b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8d6ea2-5f9d54c9dc-ffcvt"
Nov 01 15:02:03.751046 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:02:03.751013    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58304650-1c70-4e2d-9ef7-d0065e8940b7-maas-api-tls\") pod \"maas-api-e2e-praxis-b-8d6ea2-5f9d54c9dc-ffcvt\" (UID: \"58304650-1c70-4e2d-9ef7-d0065e8940b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8d6ea2-5f9d54c9dc-ffcvt"
Nov 01 15:02:03.851846 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:02:03.851808    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq9jp\" (UniqueName: \"kubernetes.io/projected/58304650-1c70-4e2d-9ef7-d0065e8940b7-kube-api-access-mq9jp\") pod \"maas-api-e2e-praxis-b-8d6ea2-5f9d54c9dc-ffcvt\" (UID: \"58304650-1c70-4e2d-9ef7-d0065e8940b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8d6ea2-5f9d54c9dc-ffcvt"
Nov 01 15:02:03.852692 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:02:03.852117    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58304650-1c70-4e2d-9ef7-d0065e8940b7-metrics-certs\") pod \"maas-api-e2e-praxis-b-8d6ea2-5f9d54c9dc-ffcvt\" (UID: \"58304650-1c70-4e2d-9ef7-d0065e8940b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8d6ea2-5f9d54c9dc-ffcvt"
Nov 01 15:02:03.852692 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:02:03.852310    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58304650-1c70-4e2d-9ef7-d0065e8940b7-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-8d6ea2-5f9d54c9dc-ffcvt\" (UID: \"58304650-1c70-4e2d-9ef7-d0065e8940b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8d6ea2-5f9d54c9dc-ffcvt"
Nov 01 15:02:03.852692 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:02:03.852500    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58304650-1c70-4e2d-9ef7-d0065e8940b7-maas-api-tls\") pod \"maas-api-e2e-praxis-b-8d6ea2-5f9d54c9dc-ffcvt\" (UID: \"58304650-1c70-4e2d-9ef7-d0065e8940b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8d6ea2-5f9d54c9dc-ffcvt"
Nov 01 15:02:03.852692 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:02:03.852683    4993 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-8d6ea2-serving-cert: secret "maas-api-e2e-praxis-b-8d6ea2-serving-cert" not found
Nov 01 15:02:03.852910 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:02:03.852761    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58304650-1c70-4e2d-9ef7-d0065e8940b7-maas-api-tls podName:58304650-1c70-4e2d-9ef7-d0065e8940b7 nodeName:}" failed. No retries permitted until 2026-11-01 15:02:04.352738445 +0000 UTC m=+2711.682601175 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/58304650-1c70-4e2d-9ef7-d0065e8940b7-maas-api-tls") pod "maas-api-e2e-praxis-b-8d6ea2-5f9d54c9dc-ffcvt" (UID: "58304650-1c70-4e2d-9ef7-d0065e8940b7") : secret "maas-api-e2e-praxis-b-8d6ea2-serving-cert" not found
Nov 01 15:02:03.853152 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:02:03.853122    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58304650-1c70-4e2d-9ef7-d0065e8940b7-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-8d6ea2-5f9d54c9dc-ffcvt\" (UID: \"58304650-1c70-4e2d-9ef7-d0065e8940b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8d6ea2-5f9d54c9dc-ffcvt"
Nov 01 15:02:03.855245 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:02:03.855220    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58304650-1c70-4e2d-9ef7-d0065e8940b7-metrics-certs\") pod \"maas-api-e2e-praxis-b-8d6ea2-5f9d54c9dc-ffcvt\" (UID: \"58304650-1c70-4e2d-9ef7-d0065e8940b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8d6ea2-5f9d54c9dc-ffcvt"
Nov 01 15:02:03.923556 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:02:03.923461    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq9jp\" (UniqueName: \"kubernetes.io/projected/58304650-1c70-4e2d-9ef7-d0065e8940b7-kube-api-access-mq9jp\") pod \"maas-api-e2e-praxis-b-8d6ea2-5f9d54c9dc-ffcvt\" (UID: \"58304650-1c70-4e2d-9ef7-d0065e8940b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8d6ea2-5f9d54c9dc-ffcvt"
Nov 01 15:02:04.360047 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:02:04.360007    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58304650-1c70-4e2d-9ef7-d0065e8940b7-maas-api-tls\") pod \"maas-api-e2e-praxis-b-8d6ea2-5f9d54c9dc-ffcvt\" (UID: \"58304650-1c70-4e2d-9ef7-d0065e8940b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8d6ea2-5f9d54c9dc-ffcvt"
Nov 01 15:02:04.363080 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:02:04.363045    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58304650-1c70-4e2d-9ef7-d0065e8940b7-maas-api-tls\") pod \"maas-api-e2e-praxis-b-8d6ea2-5f9d54c9dc-ffcvt\" (UID: \"58304650-1c70-4e2d-9ef7-d0065e8940b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8d6ea2-5f9d54c9dc-ffcvt"
Nov 01 15:02:04.546525 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:02:04.546491    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8d6ea2-5f9d54c9dc-ffcvt"
Nov 01 15:02:04.923574 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:02:04.923541    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-8d6ea2-5f9d54c9dc-ffcvt"]
Nov 01 15:02:04.928746 ip-10-0-134-37 kubenswrapper[4993]: W1101 15:02:04.928712    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58304650_1c70_4e2d_9ef7_d0065e8940b7.slice/crio-d342fce623431b19f7c7c0f211660c2be38f35a876481e5ce6f878fad59ec390 WatchSource:0}: Error finding container d342fce623431b19f7c7c0f211660c2be38f35a876481e5ce6f878fad59ec390: Status 404 returned error can't find the container with id d342fce623431b19f7c7c0f211660c2be38f35a876481e5ce6f878fad59ec390
Nov 01 15:02:05.790885 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:02:05.790829    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8d6ea2-5f9d54c9dc-ffcvt" event={"ID":"58304650-1c70-4e2d-9ef7-d0065e8940b7","Type":"ContainerStarted","Data":"d342fce623431b19f7c7c0f211660c2be38f35a876481e5ce6f878fad59ec390"}
Nov 01 15:02:06.800444 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:02:06.800404    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8d6ea2-5f9d54c9dc-ffcvt" event={"ID":"58304650-1c70-4e2d-9ef7-d0065e8940b7","Type":"ContainerStarted","Data":"1dc8f12973464ae1aa7b6e1fa5ac94b6aa590f0f254aeb2a3f008771c2d36112"}
Nov 01 15:02:06.800811 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:02:06.800492    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8d6ea2-5f9d54c9dc-ffcvt"
Nov 01 15:02:06.908367 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:02:06.908313    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8d6ea2-5f9d54c9dc-ffcvt" podStartSLOduration=3.058241677 podStartE2EDuration="3.908300155s" totalImagesPullingTime="850.058478ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:02:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:02:06.907290563 +0000 UTC m=+2714.237153296" watchObservedRunningTime="2026-11-01 15:02:06.908300155 +0000 UTC m=+2714.238162937"
Nov 01 15:02:12.812639 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:02:12.812606    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8d6ea2-5f9d54c9dc-ffcvt"
Nov 01 15:02:27.502154 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:02:27.502101    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:02:57.501503 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:02:57.501452    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:03:09.803188 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:03:09.803134    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-4e07cb/auth-test-model-4e07cb-kserve-66869df457-8z2kf"]
Nov 01 15:03:09.807671 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:03:09.807646    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-4e07cb/auth-test-model-4e07cb-kserve-66869df457-8z2kf"
Nov 01 15:03:09.903034 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:03:09.902997    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-4e07cb\"/\"openshift-service-ca.crt\""
Nov 01 15:03:09.903034 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:03:09.902997    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-4e07cb\"/\"kube-root-ca.crt\""
Nov 01 15:03:09.903296 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:03:09.903254    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-4e07cb\"/\"default-dockercfg-vsng9\""
Nov 01 15:03:09.903296 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:03:09.903259    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-4e07cb\"/\"auth-test-model-4e07cb-kserve-self-signed-certs\""
Nov 01 15:03:09.907702 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:03:09.907675    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-4e07cb/auth-test-model-4e07cb-kserve-66869df457-8z2kf"]
Nov 01 15:03:09.909943 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:03:09.909918    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec23a9b1-8adf-4be6-88f9-298452111158-dshm\") pod \"auth-test-model-4e07cb-kserve-66869df457-8z2kf\" (UID: \"ec23a9b1-8adf-4be6-88f9-298452111158\") " pod="ai-tenant-e2e-shared-a-w2-4e07cb/auth-test-model-4e07cb-kserve-66869df457-8z2kf"
Nov 01 15:03:09.910058 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:03:09.909954    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec23a9b1-8adf-4be6-88f9-298452111158-model-cache\") pod \"auth-test-model-4e07cb-kserve-66869df457-8z2kf\" (UID: \"ec23a9b1-8adf-4be6-88f9-298452111158\") " pod="ai-tenant-e2e-shared-a-w2-4e07cb/auth-test-model-4e07cb-kserve-66869df457-8z2kf"
Nov 01 15:03:09.910122 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:03:09.910091    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec23a9b1-8adf-4be6-88f9-298452111158-tls-certs\") pod \"auth-test-model-4e07cb-kserve-66869df457-8z2kf\" (UID: \"ec23a9b1-8adf-4be6-88f9-298452111158\") " pod="ai-tenant-e2e-shared-a-w2-4e07cb/auth-test-model-4e07cb-kserve-66869df457-8z2kf"
Nov 01 15:03:09.910265 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:03:09.910249    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec23a9b1-8adf-4be6-88f9-298452111158-home\") pod \"auth-test-model-4e07cb-kserve-66869df457-8z2kf\" (UID: \"ec23a9b1-8adf-4be6-88f9-298452111158\") " pod="ai-tenant-e2e-shared-a-w2-4e07cb/auth-test-model-4e07cb-kserve-66869df457-8z2kf"
Nov 01 15:03:09.910354 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:03:09.910339    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fw9s\" (UniqueName: \"kubernetes.io/projected/ec23a9b1-8adf-4be6-88f9-298452111158-kube-api-access-6fw9s\") pod \"auth-test-model-4e07cb-kserve-66869df457-8z2kf\" (UID: \"ec23a9b1-8adf-4be6-88f9-298452111158\") " pod="ai-tenant-e2e-shared-a-w2-4e07cb/auth-test-model-4e07cb-kserve-66869df457-8z2kf"
Nov 01 15:03:09.910473 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:03:09.910460    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec23a9b1-8adf-4be6-88f9-298452111158-tmp-dir\") pod \"auth-test-model-4e07cb-kserve-66869df457-8z2kf\" (UID: \"ec23a9b1-8adf-4be6-88f9-298452111158\") " pod="ai-tenant-e2e-shared-a-w2-4e07cb/auth-test-model-4e07cb-kserve-66869df457-8z2kf"
Nov 01 15:03:10.011606 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:03:10.011571    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec23a9b1-8adf-4be6-88f9-298452111158-tls-certs\") pod \"auth-test-model-4e07cb-kserve-66869df457-8z2kf\" (UID: \"ec23a9b1-8adf-4be6-88f9-298452111158\") " pod="ai-tenant-e2e-shared-a-w2-4e07cb/auth-test-model-4e07cb-kserve-66869df457-8z2kf"
Nov 01 15:03:10.011812 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:03:10.011663    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec23a9b1-8adf-4be6-88f9-298452111158-home\") pod \"auth-test-model-4e07cb-kserve-66869df457-8z2kf\" (UID: \"ec23a9b1-8adf-4be6-88f9-298452111158\") " pod="ai-tenant-e2e-shared-a-w2-4e07cb/auth-test-model-4e07cb-kserve-66869df457-8z2kf"
Nov 01 15:03:10.011812 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:03:10.011752    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fw9s\" (UniqueName: \"kubernetes.io/projected/ec23a9b1-8adf-4be6-88f9-298452111158-kube-api-access-6fw9s\") pod \"auth-test-model-4e07cb-kserve-66869df457-8z2kf\" (UID: \"ec23a9b1-8adf-4be6-88f9-298452111158\") " pod="ai-tenant-e2e-shared-a-w2-4e07cb/auth-test-model-4e07cb-kserve-66869df457-8z2kf"
Nov 01 15:03:10.011929 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:03:10.011835    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec23a9b1-8adf-4be6-88f9-298452111158-tmp-dir\") pod \"auth-test-model-4e07cb-kserve-66869df457-8z2kf\" (UID: \"ec23a9b1-8adf-4be6-88f9-298452111158\") " pod="ai-tenant-e2e-shared-a-w2-4e07cb/auth-test-model-4e07cb-kserve-66869df457-8z2kf"
Nov 01 15:03:10.011985 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:03:10.011950    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec23a9b1-8adf-4be6-88f9-298452111158-dshm\") pod \"auth-test-model-4e07cb-kserve-66869df457-8z2kf\" (UID: \"ec23a9b1-8adf-4be6-88f9-298452111158\") " pod="ai-tenant-e2e-shared-a-w2-4e07cb/auth-test-model-4e07cb-kserve-66869df457-8z2kf"
Nov 01 15:03:10.011985 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:03:10.011978    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec23a9b1-8adf-4be6-88f9-298452111158-model-cache\") pod \"auth-test-model-4e07cb-kserve-66869df457-8z2kf\" (UID: \"ec23a9b1-8adf-4be6-88f9-298452111158\") " pod="ai-tenant-e2e-shared-a-w2-4e07cb/auth-test-model-4e07cb-kserve-66869df457-8z2kf"
Nov 01 15:03:10.012513 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:03:10.012476    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec23a9b1-8adf-4be6-88f9-298452111158-model-cache\") pod \"auth-test-model-4e07cb-kserve-66869df457-8z2kf\" (UID: \"ec23a9b1-8adf-4be6-88f9-298452111158\") " pod="ai-tenant-e2e-shared-a-w2-4e07cb/auth-test-model-4e07cb-kserve-66869df457-8z2kf"
Nov 01 15:03:10.012780 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:03:10.012758    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec23a9b1-8adf-4be6-88f9-298452111158-tmp-dir\") pod \"auth-test-model-4e07cb-kserve-66869df457-8z2kf\" (UID: \"ec23a9b1-8adf-4be6-88f9-298452111158\") " pod="ai-tenant-e2e-shared-a-w2-4e07cb/auth-test-model-4e07cb-kserve-66869df457-8z2kf"
Nov 01 15:03:10.013363 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:03:10.013326    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec23a9b1-8adf-4be6-88f9-298452111158-home\") pod \"auth-test-model-4e07cb-kserve-66869df457-8z2kf\" (UID: \"ec23a9b1-8adf-4be6-88f9-298452111158\") " pod="ai-tenant-e2e-shared-a-w2-4e07cb/auth-test-model-4e07cb-kserve-66869df457-8z2kf"
Nov 01 15:03:10.015924 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:03:10.015898    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec23a9b1-8adf-4be6-88f9-298452111158-dshm\") pod \"auth-test-model-4e07cb-kserve-66869df457-8z2kf\" (UID: \"ec23a9b1-8adf-4be6-88f9-298452111158\") " pod="ai-tenant-e2e-shared-a-w2-4e07cb/auth-test-model-4e07cb-kserve-66869df457-8z2kf"
Nov 01 15:03:10.020163 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:03:10.020136    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec23a9b1-8adf-4be6-88f9-298452111158-tls-certs\") pod \"auth-test-model-4e07cb-kserve-66869df457-8z2kf\" (UID: \"ec23a9b1-8adf-4be6-88f9-298452111158\") " pod="ai-tenant-e2e-shared-a-w2-4e07cb/auth-test-model-4e07cb-kserve-66869df457-8z2kf"
Nov 01 15:03:10.065925 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:03:10.065848    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fw9s\" (UniqueName: \"kubernetes.io/projected/ec23a9b1-8adf-4be6-88f9-298452111158-kube-api-access-6fw9s\") pod \"auth-test-model-4e07cb-kserve-66869df457-8z2kf\" (UID: \"ec23a9b1-8adf-4be6-88f9-298452111158\") " pod="ai-tenant-e2e-shared-a-w2-4e07cb/auth-test-model-4e07cb-kserve-66869df457-8z2kf"
Nov 01 15:03:10.120036 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:03:10.119987    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-4e07cb/auth-test-model-4e07cb-kserve-66869df457-8z2kf"
Nov 01 15:03:10.508393 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:03:10.508345    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-4e07cb/auth-test-model-4e07cb-kserve-66869df457-8z2kf"]
Nov 01 15:03:10.513208 ip-10-0-134-37 kubenswrapper[4993]: W1101 15:03:10.513175    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podec23a9b1_8adf_4be6_88f9_298452111158.slice/crio-f83fb4bcf316cb0a2c137b6b4812b93a1266bc56f37353c6772d8010f3f2e782 WatchSource:0}: Error finding container f83fb4bcf316cb0a2c137b6b4812b93a1266bc56f37353c6772d8010f3f2e782: Status 404 returned error can't find the container with id f83fb4bcf316cb0a2c137b6b4812b93a1266bc56f37353c6772d8010f3f2e782
Nov 01 15:03:11.262091 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:03:11.262053    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-4e07cb/auth-test-model-4e07cb-kserve-66869df457-8z2kf" event={"ID":"ec23a9b1-8adf-4be6-88f9-298452111158","Type":"ContainerStarted","Data":"cb1bbc705193b7fbc92258bbb4f7d6564a2df3107f17740170b3faf36c6bb9d9"}
Nov 01 15:03:11.262091 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:03:11.262097    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-4e07cb/auth-test-model-4e07cb-kserve-66869df457-8z2kf" event={"ID":"ec23a9b1-8adf-4be6-88f9-298452111158","Type":"ContainerStarted","Data":"f83fb4bcf316cb0a2c137b6b4812b93a1266bc56f37353c6772d8010f3f2e782"}
Nov 01 15:03:11.386936 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:03:11.386877    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-4e07cb/auth-test-model-4e07cb-kserve-66869df457-8z2kf" podStartSLOduration=2.386862242 podStartE2EDuration="2.386862242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:03:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:03:11.384684035 +0000 UTC m=+2778.714546772" watchObservedRunningTime="2026-11-01 15:03:11.386862242 +0000 UTC m=+2778.716724980"
Nov 01 15:03:12.120425 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:03:12.120388    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-4e07cb/auth-test-model-4e07cb-kserve-66869df457-8z2kf"
Nov 01 15:03:20.120424 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:03:20.120379    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-4e07cb/auth-test-model-4e07cb-kserve-66869df457-8z2kf"
Nov 01 15:03:20.133491 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:03:20.133466    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-4e07cb/auth-test-model-4e07cb-kserve-66869df457-8z2kf"
Nov 01 15:03:20.347053 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:03:20.347018    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-4e07cb/auth-test-model-4e07cb-kserve-66869df457-8z2kf"
Nov 01 15:03:27.501256 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:03:27.501212    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:03:57.502675 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:03:57.502634    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:04:01.619543 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:01.619508    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-0c9d0a-6588775df-ggnn6"]
Nov 01 15:04:01.619994 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:01.619751    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0c9d0a-6588775df-ggnn6" podUID="e2bf57b9-f35a-46c6-91e4-bbabd4908195" containerName="maas-api" containerID="cri-o://3b13040974f021b7c43680fd1aeb16e7820fa6149609fd93c2c7f389b07dd372" gracePeriod=30
Nov 01 15:04:01.938616 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:01.938581    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0c9d0a-6588775df-ggnn6"
Nov 01 15:04:02.080675 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:02.078803    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2bf57b9-f35a-46c6-91e4-bbabd4908195-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2bf57b9-f35a-46c6-91e4-bbabd4908195-maas-api-tls\") pod \"e2bf57b9-f35a-46c6-91e4-bbabd4908195\" (UID: \"e2bf57b9-f35a-46c6-91e4-bbabd4908195\") "
Nov 01 15:04:02.080675 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:02.078910    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2bf57b9-f35a-46c6-91e4-bbabd4908195-kube-api-access-94bnv\" (UniqueName: \"kubernetes.io/projected/e2bf57b9-f35a-46c6-91e4-bbabd4908195-kube-api-access-94bnv\") pod \"e2bf57b9-f35a-46c6-91e4-bbabd4908195\" (UID: \"e2bf57b9-f35a-46c6-91e4-bbabd4908195\") "
Nov 01 15:04:02.080675 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:02.078948    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2bf57b9-f35a-46c6-91e4-bbabd4908195-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2bf57b9-f35a-46c6-91e4-bbabd4908195-metrics-certs\") pod \"e2bf57b9-f35a-46c6-91e4-bbabd4908195\" (UID: \"e2bf57b9-f35a-46c6-91e4-bbabd4908195\") "
Nov 01 15:04:02.080675 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:02.079199    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2bf57b9-f35a-46c6-91e4-bbabd4908195-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2bf57b9-f35a-46c6-91e4-bbabd4908195-openshift-service-ca\") pod \"e2bf57b9-f35a-46c6-91e4-bbabd4908195\" (UID: \"e2bf57b9-f35a-46c6-91e4-bbabd4908195\") "
Nov 01 15:04:02.081867 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:02.081837    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2bf57b9-f35a-46c6-91e4-bbabd4908195-openshift-service-ca" pod "e2bf57b9-f35a-46c6-91e4-bbabd4908195" (UID: "e2bf57b9-f35a-46c6-91e4-bbabd4908195"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:04:02.085878 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:02.085580    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2bf57b9-f35a-46c6-91e4-bbabd4908195-kube-api-access-94bnv" pod "e2bf57b9-f35a-46c6-91e4-bbabd4908195" (UID: "e2bf57b9-f35a-46c6-91e4-bbabd4908195"). InnerVolumeSpecName "kube-api-access-94bnv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:04:02.093449 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:02.093273    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2bf57b9-f35a-46c6-91e4-bbabd4908195-metrics-certs" pod "e2bf57b9-f35a-46c6-91e4-bbabd4908195" (UID: "e2bf57b9-f35a-46c6-91e4-bbabd4908195"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:04:02.093449 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:02.093399    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2bf57b9-f35a-46c6-91e4-bbabd4908195-maas-api-tls" pod "e2bf57b9-f35a-46c6-91e4-bbabd4908195" (UID: "e2bf57b9-f35a-46c6-91e4-bbabd4908195"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:04:02.182019 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:02.181913    4993 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2bf57b9-f35a-46c6-91e4-bbabd4908195-openshift-service-ca\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:04:02.182019 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:02.181947    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2bf57b9-f35a-46c6-91e4-bbabd4908195-maas-api-tls\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:04:02.182019 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:02.181958    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94bnv\" (UniqueName: \"kubernetes.io/projected/e2bf57b9-f35a-46c6-91e4-bbabd4908195-kube-api-access-94bnv\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:04:02.182019 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:02.181973    4993 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2bf57b9-f35a-46c6-91e4-bbabd4908195-metrics-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:04:02.630538 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:02.630443    4993 generic.go:356] "Generic (PLEG): container finished" podID="e2bf57b9-f35a-46c6-91e4-bbabd4908195" containerID="3b13040974f021b7c43680fd1aeb16e7820fa6149609fd93c2c7f389b07dd372" exitCode=0
Nov 01 15:04:02.630538 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:02.630509    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0c9d0a-6588775df-ggnn6"
Nov 01 15:04:02.630538 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:02.630520    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0c9d0a-6588775df-ggnn6" event={"ID":"e2bf57b9-f35a-46c6-91e4-bbabd4908195","Type":"ContainerDied","Data":"3b13040974f021b7c43680fd1aeb16e7820fa6149609fd93c2c7f389b07dd372"}
Nov 01 15:04:02.631147 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:02.630613    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0c9d0a-6588775df-ggnn6" event={"ID":"e2bf57b9-f35a-46c6-91e4-bbabd4908195","Type":"ContainerDied","Data":"2ee12b3373d27017d2a755c1cc65e73296bbc4dd60533d4273bde053e760a980"}
Nov 01 15:04:02.631147 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:02.630638    4993 scope.go:122] "RemoveContainer" containerID="3b13040974f021b7c43680fd1aeb16e7820fa6149609fd93c2c7f389b07dd372"
Nov 01 15:04:02.641229 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:02.641212    4993 scope.go:122] "RemoveContainer" containerID="3b13040974f021b7c43680fd1aeb16e7820fa6149609fd93c2c7f389b07dd372"
Nov 01 15:04:02.641491 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:04:02.641473    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b13040974f021b7c43680fd1aeb16e7820fa6149609fd93c2c7f389b07dd372\": container with ID starting with 3b13040974f021b7c43680fd1aeb16e7820fa6149609fd93c2c7f389b07dd372 not found: ID does not exist" containerID="3b13040974f021b7c43680fd1aeb16e7820fa6149609fd93c2c7f389b07dd372"
Nov 01 15:04:02.641529 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:02.641500    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b13040974f021b7c43680fd1aeb16e7820fa6149609fd93c2c7f389b07dd372"} err="failed to get container status \"3b13040974f021b7c43680fd1aeb16e7820fa6149609fd93c2c7f389b07dd372\": rpc error: code = NotFound desc = could not find container \"3b13040974f021b7c43680fd1aeb16e7820fa6149609fd93c2c7f389b07dd372\": container with ID starting with 3b13040974f021b7c43680fd1aeb16e7820fa6149609fd93c2c7f389b07dd372 not found: ID does not exist"
Nov 01 15:04:02.784541 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:02.784506    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-0c9d0a-6588775df-ggnn6"]
Nov 01 15:04:02.830558 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:02.830527    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-0c9d0a-6588775df-ggnn6"]
Nov 01 15:04:04.796907 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:04.796875    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2bf57b9-f35a-46c6-91e4-bbabd4908195" path="/var/lib/kubelet/pods/e2bf57b9-f35a-46c6-91e4-bbabd4908195/volumes"
Nov 01 15:04:12.147208 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:12.147156    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-6fcb7b-5488d47698-fw2ch"]
Nov 01 15:04:12.147713 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:12.147440    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-6fcb7b-5488d47698-fw2ch" podUID="cf03339d-d720-4474-b495-2d149d09d33a" containerName="maas-api" containerID="cri-o://65c372fee81064e2ed8f2b5839fde3ec607792571b393718c4e82548618cb214" gracePeriod=30
Nov 01 15:04:12.488450 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:12.488418    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-6fcb7b-5488d47698-fw2ch"
Nov 01 15:04:12.507533 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:12.507499    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf03339d-d720-4474-b495-2d149d09d33a-kube-api-access-69gtl\" (UniqueName: \"kubernetes.io/projected/cf03339d-d720-4474-b495-2d149d09d33a-kube-api-access-69gtl\") pod \"cf03339d-d720-4474-b495-2d149d09d33a\" (UID: \"cf03339d-d720-4474-b495-2d149d09d33a\") "
Nov 01 15:04:12.507713 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:12.507566    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf03339d-d720-4474-b495-2d149d09d33a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf03339d-d720-4474-b495-2d149d09d33a-openshift-service-ca\") pod \"cf03339d-d720-4474-b495-2d149d09d33a\" (UID: \"cf03339d-d720-4474-b495-2d149d09d33a\") "
Nov 01 15:04:12.507713 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:12.507644    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf03339d-d720-4474-b495-2d149d09d33a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf03339d-d720-4474-b495-2d149d09d33a-maas-api-tls\") pod \"cf03339d-d720-4474-b495-2d149d09d33a\" (UID: \"cf03339d-d720-4474-b495-2d149d09d33a\") "
Nov 01 15:04:12.507840 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:12.507731    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf03339d-d720-4474-b495-2d149d09d33a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf03339d-d720-4474-b495-2d149d09d33a-metrics-certs\") pod \"cf03339d-d720-4474-b495-2d149d09d33a\" (UID: \"cf03339d-d720-4474-b495-2d149d09d33a\") "
Nov 01 15:04:12.507984 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:12.507953    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf03339d-d720-4474-b495-2d149d09d33a-openshift-service-ca" pod "cf03339d-d720-4474-b495-2d149d09d33a" (UID: "cf03339d-d720-4474-b495-2d149d09d33a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:04:12.508358 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:12.508332    4993 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf03339d-d720-4474-b495-2d149d09d33a-openshift-service-ca\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:04:12.510024 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:12.509991    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf03339d-d720-4474-b495-2d149d09d33a-maas-api-tls" pod "cf03339d-d720-4474-b495-2d149d09d33a" (UID: "cf03339d-d720-4474-b495-2d149d09d33a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:04:12.510577 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:12.510539    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf03339d-d720-4474-b495-2d149d09d33a-kube-api-access-69gtl" pod "cf03339d-d720-4474-b495-2d149d09d33a" (UID: "cf03339d-d720-4474-b495-2d149d09d33a"). InnerVolumeSpecName "kube-api-access-69gtl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:04:12.510739 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:12.510717    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf03339d-d720-4474-b495-2d149d09d33a-metrics-certs" pod "cf03339d-d720-4474-b495-2d149d09d33a" (UID: "cf03339d-d720-4474-b495-2d149d09d33a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:04:12.619191 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:12.610035    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69gtl\" (UniqueName: \"kubernetes.io/projected/cf03339d-d720-4474-b495-2d149d09d33a-kube-api-access-69gtl\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:04:12.619191 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:12.610074    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf03339d-d720-4474-b495-2d149d09d33a-maas-api-tls\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:04:12.619191 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:12.610091    4993 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf03339d-d720-4474-b495-2d149d09d33a-metrics-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:04:12.702820 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:12.702787    4993 generic.go:356] "Generic (PLEG): container finished" podID="cf03339d-d720-4474-b495-2d149d09d33a" containerID="65c372fee81064e2ed8f2b5839fde3ec607792571b393718c4e82548618cb214" exitCode=0
Nov 01 15:04:12.703020 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:12.702860    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-6fcb7b-5488d47698-fw2ch"
Nov 01 15:04:12.703020 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:12.702875    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-6fcb7b-5488d47698-fw2ch" event={"ID":"cf03339d-d720-4474-b495-2d149d09d33a","Type":"ContainerDied","Data":"65c372fee81064e2ed8f2b5839fde3ec607792571b393718c4e82548618cb214"}
Nov 01 15:04:12.703020 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:12.702938    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-6fcb7b-5488d47698-fw2ch" event={"ID":"cf03339d-d720-4474-b495-2d149d09d33a","Type":"ContainerDied","Data":"3550d88839a9d5549f2f29d4950080a6b00d902e8267de11fb0252fad762cd82"}
Nov 01 15:04:12.703020 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:12.702960    4993 scope.go:122] "RemoveContainer" containerID="65c372fee81064e2ed8f2b5839fde3ec607792571b393718c4e82548618cb214"
Nov 01 15:04:12.716415 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:12.716392    4993 scope.go:122] "RemoveContainer" containerID="65c372fee81064e2ed8f2b5839fde3ec607792571b393718c4e82548618cb214"
Nov 01 15:04:12.717641 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:04:12.716869    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65c372fee81064e2ed8f2b5839fde3ec607792571b393718c4e82548618cb214\": container with ID starting with 65c372fee81064e2ed8f2b5839fde3ec607792571b393718c4e82548618cb214 not found: ID does not exist" containerID="65c372fee81064e2ed8f2b5839fde3ec607792571b393718c4e82548618cb214"
Nov 01 15:04:12.717641 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:12.716912    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65c372fee81064e2ed8f2b5839fde3ec607792571b393718c4e82548618cb214"} err="failed to get container status \"65c372fee81064e2ed8f2b5839fde3ec607792571b393718c4e82548618cb214\": rpc error: code = NotFound desc = could not find container \"65c372fee81064e2ed8f2b5839fde3ec607792571b393718c4e82548618cb214\": container with ID starting with 65c372fee81064e2ed8f2b5839fde3ec607792571b393718c4e82548618cb214 not found: ID does not exist"
Nov 01 15:04:12.857467 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:12.857422    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-6fcb7b-5488d47698-fw2ch"]
Nov 01 15:04:12.860872 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:12.860844    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-6fcb7b-5488d47698-fw2ch"]
Nov 01 15:04:13.194006 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:13.193976    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-8d6ea2-5f9d54c9dc-ffcvt"]
Nov 01 15:04:13.194515 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:13.194257    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8d6ea2-5f9d54c9dc-ffcvt" podUID="58304650-1c70-4e2d-9ef7-d0065e8940b7" containerName="maas-api" containerID="cri-o://1dc8f12973464ae1aa7b6e1fa5ac94b6aa590f0f254aeb2a3f008771c2d36112" gracePeriod=30
Nov 01 15:04:13.541556 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:13.541526    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8d6ea2-5f9d54c9dc-ffcvt"
Nov 01 15:04:13.647763 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:13.647723    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58304650-1c70-4e2d-9ef7-d0065e8940b7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/58304650-1c70-4e2d-9ef7-d0065e8940b7-metrics-certs\") pod \"58304650-1c70-4e2d-9ef7-d0065e8940b7\" (UID: \"58304650-1c70-4e2d-9ef7-d0065e8940b7\") "
Nov 01 15:04:13.647981 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:13.647798    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58304650-1c70-4e2d-9ef7-d0065e8940b7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58304650-1c70-4e2d-9ef7-d0065e8940b7-openshift-service-ca\") pod \"58304650-1c70-4e2d-9ef7-d0065e8940b7\" (UID: \"58304650-1c70-4e2d-9ef7-d0065e8940b7\") "
Nov 01 15:04:13.647981 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:13.647876    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58304650-1c70-4e2d-9ef7-d0065e8940b7-kube-api-access-mq9jp\" (UniqueName: \"kubernetes.io/projected/58304650-1c70-4e2d-9ef7-d0065e8940b7-kube-api-access-mq9jp\") pod \"58304650-1c70-4e2d-9ef7-d0065e8940b7\" (UID: \"58304650-1c70-4e2d-9ef7-d0065e8940b7\") "
Nov 01 15:04:13.647981 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:13.647959    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58304650-1c70-4e2d-9ef7-d0065e8940b7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58304650-1c70-4e2d-9ef7-d0065e8940b7-maas-api-tls\") pod \"58304650-1c70-4e2d-9ef7-d0065e8940b7\" (UID: \"58304650-1c70-4e2d-9ef7-d0065e8940b7\") "
Nov 01 15:04:13.649623 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:13.649590    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58304650-1c70-4e2d-9ef7-d0065e8940b7-openshift-service-ca" pod "58304650-1c70-4e2d-9ef7-d0065e8940b7" (UID: "58304650-1c70-4e2d-9ef7-d0065e8940b7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:04:13.665189 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:13.656647    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58304650-1c70-4e2d-9ef7-d0065e8940b7-metrics-certs" pod "58304650-1c70-4e2d-9ef7-d0065e8940b7" (UID: "58304650-1c70-4e2d-9ef7-d0065e8940b7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:04:13.671838 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:13.671448    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58304650-1c70-4e2d-9ef7-d0065e8940b7-kube-api-access-mq9jp" pod "58304650-1c70-4e2d-9ef7-d0065e8940b7" (UID: "58304650-1c70-4e2d-9ef7-d0065e8940b7"). InnerVolumeSpecName "kube-api-access-mq9jp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:04:13.696495 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:13.696422    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58304650-1c70-4e2d-9ef7-d0065e8940b7-maas-api-tls" pod "58304650-1c70-4e2d-9ef7-d0065e8940b7" (UID: "58304650-1c70-4e2d-9ef7-d0065e8940b7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:04:13.715643 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:13.715599    4993 generic.go:356] "Generic (PLEG): container finished" podID="58304650-1c70-4e2d-9ef7-d0065e8940b7" containerID="1dc8f12973464ae1aa7b6e1fa5ac94b6aa590f0f254aeb2a3f008771c2d36112" exitCode=0
Nov 01 15:04:13.715810 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:13.715652    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8d6ea2-5f9d54c9dc-ffcvt" event={"ID":"58304650-1c70-4e2d-9ef7-d0065e8940b7","Type":"ContainerDied","Data":"1dc8f12973464ae1aa7b6e1fa5ac94b6aa590f0f254aeb2a3f008771c2d36112"}
Nov 01 15:04:13.715810 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:13.715705    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8d6ea2-5f9d54c9dc-ffcvt" event={"ID":"58304650-1c70-4e2d-9ef7-d0065e8940b7","Type":"ContainerDied","Data":"d342fce623431b19f7c7c0f211660c2be38f35a876481e5ce6f878fad59ec390"}
Nov 01 15:04:13.715810 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:13.715734    4993 scope.go:122] "RemoveContainer" containerID="1dc8f12973464ae1aa7b6e1fa5ac94b6aa590f0f254aeb2a3f008771c2d36112"
Nov 01 15:04:13.715995 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:13.715845    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8d6ea2-5f9d54c9dc-ffcvt"
Nov 01 15:04:13.732196 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:13.731566    4993 scope.go:122] "RemoveContainer" containerID="1dc8f12973464ae1aa7b6e1fa5ac94b6aa590f0f254aeb2a3f008771c2d36112"
Nov 01 15:04:13.732196 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:04:13.732088    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dc8f12973464ae1aa7b6e1fa5ac94b6aa590f0f254aeb2a3f008771c2d36112\": container with ID starting with 1dc8f12973464ae1aa7b6e1fa5ac94b6aa590f0f254aeb2a3f008771c2d36112 not found: ID does not exist" containerID="1dc8f12973464ae1aa7b6e1fa5ac94b6aa590f0f254aeb2a3f008771c2d36112"
Nov 01 15:04:13.732196 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:13.732130    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dc8f12973464ae1aa7b6e1fa5ac94b6aa590f0f254aeb2a3f008771c2d36112"} err="failed to get container status \"1dc8f12973464ae1aa7b6e1fa5ac94b6aa590f0f254aeb2a3f008771c2d36112\": rpc error: code = NotFound desc = could not find container \"1dc8f12973464ae1aa7b6e1fa5ac94b6aa590f0f254aeb2a3f008771c2d36112\": container with ID starting with 1dc8f12973464ae1aa7b6e1fa5ac94b6aa590f0f254aeb2a3f008771c2d36112 not found: ID does not exist"
Nov 01 15:04:13.749811 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:13.749778    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58304650-1c70-4e2d-9ef7-d0065e8940b7-maas-api-tls\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:04:13.749811 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:13.749811    4993 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58304650-1c70-4e2d-9ef7-d0065e8940b7-metrics-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:04:13.750050 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:13.749830    4993 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58304650-1c70-4e2d-9ef7-d0065e8940b7-openshift-service-ca\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:04:13.750050 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:13.749845    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mq9jp\" (UniqueName: \"kubernetes.io/projected/58304650-1c70-4e2d-9ef7-d0065e8940b7-kube-api-access-mq9jp\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:04:13.888721 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:13.888609    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-8d6ea2-5f9d54c9dc-ffcvt"]
Nov 01 15:04:13.893546 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:13.893508    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-8d6ea2-5f9d54c9dc-ffcvt"]
Nov 01 15:04:14.175958 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:14.175872    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-d06633/auth-test-model-d06633-kserve-685c8fd8f-cwl5h"]
Nov 01 15:04:14.176493 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:14.176471    4993 cpu_manager.go:414] "containerMap: removing container" podUID="58304650-1c70-4e2d-9ef7-d0065e8940b7" containerName="maas-api"
Nov 01 15:04:14.176493 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:14.176492    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58304650-1c70-4e2d-9ef7-d0065e8940b7" containerName="maas-api"
Nov 01 15:04:14.176654 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:14.176537    4993 cpu_manager.go:414] "containerMap: removing container" podUID="e2bf57b9-f35a-46c6-91e4-bbabd4908195" containerName="maas-api"
Nov 01 15:04:14.176654 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:14.176543    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2bf57b9-f35a-46c6-91e4-bbabd4908195" containerName="maas-api"
Nov 01 15:04:14.176654 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:14.176560    4993 cpu_manager.go:414] "containerMap: removing container" podUID="cf03339d-d720-4474-b495-2d149d09d33a" containerName="maas-api"
Nov 01 15:04:14.176654 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:14.176568    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf03339d-d720-4474-b495-2d149d09d33a" containerName="maas-api"
Nov 01 15:04:14.176654 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:14.176655    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2bf57b9-f35a-46c6-91e4-bbabd4908195" containerName="maas-api"
Nov 01 15:04:14.176833 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:14.176665    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf03339d-d720-4474-b495-2d149d09d33a" containerName="maas-api"
Nov 01 15:04:14.176833 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:14.176673    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="58304650-1c70-4e2d-9ef7-d0065e8940b7" containerName="maas-api"
Nov 01 15:04:14.181518 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:14.181497    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-d06633/auth-test-model-d06633-kserve-685c8fd8f-cwl5h"
Nov 01 15:04:14.282422 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:14.282373    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-d06633\"/\"auth-test-model-d06633-kserve-self-signed-certs\""
Nov 01 15:04:14.303251 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:14.285454    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-d06633\"/\"kube-root-ca.crt\""
Nov 01 15:04:14.303251 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:14.285541    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-d06633\"/\"openshift-service-ca.crt\""
Nov 01 15:04:14.303251 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:14.285659    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-d06633\"/\"default-dockercfg-pdd65\""
Nov 01 15:04:14.303251 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:14.301946    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-d06633/auth-test-model-d06633-kserve-685c8fd8f-cwl5h"]
Nov 01 15:04:14.306005 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:14.305975    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzt7g\" (UniqueName: \"kubernetes.io/projected/28b45c9e-a664-452c-8972-5d68d0f09a48-kube-api-access-kzt7g\") pod \"auth-test-model-d06633-kserve-685c8fd8f-cwl5h\" (UID: \"28b45c9e-a664-452c-8972-5d68d0f09a48\") " pod="ai-tenant-e2e-shared-a-w4-d06633/auth-test-model-d06633-kserve-685c8fd8f-cwl5h"
Nov 01 15:04:14.318245 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:14.316621    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28b45c9e-a664-452c-8972-5d68d0f09a48-home\") pod \"auth-test-model-d06633-kserve-685c8fd8f-cwl5h\" (UID: \"28b45c9e-a664-452c-8972-5d68d0f09a48\") " pod="ai-tenant-e2e-shared-a-w4-d06633/auth-test-model-d06633-kserve-685c8fd8f-cwl5h"
Nov 01 15:04:14.318245 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:14.316721    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28b45c9e-a664-452c-8972-5d68d0f09a48-model-cache\") pod \"auth-test-model-d06633-kserve-685c8fd8f-cwl5h\" (UID: \"28b45c9e-a664-452c-8972-5d68d0f09a48\") " pod="ai-tenant-e2e-shared-a-w4-d06633/auth-test-model-d06633-kserve-685c8fd8f-cwl5h"
Nov 01 15:04:14.318245 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:14.316811    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28b45c9e-a664-452c-8972-5d68d0f09a48-tmp-dir\") pod \"auth-test-model-d06633-kserve-685c8fd8f-cwl5h\" (UID: \"28b45c9e-a664-452c-8972-5d68d0f09a48\") " pod="ai-tenant-e2e-shared-a-w4-d06633/auth-test-model-d06633-kserve-685c8fd8f-cwl5h"
Nov 01 15:04:14.318245 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:14.316862    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28b45c9e-a664-452c-8972-5d68d0f09a48-tls-certs\") pod \"auth-test-model-d06633-kserve-685c8fd8f-cwl5h\" (UID: \"28b45c9e-a664-452c-8972-5d68d0f09a48\") " pod="ai-tenant-e2e-shared-a-w4-d06633/auth-test-model-d06633-kserve-685c8fd8f-cwl5h"
Nov 01 15:04:14.318245 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:14.316906    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28b45c9e-a664-452c-8972-5d68d0f09a48-dshm\") pod \"auth-test-model-d06633-kserve-685c8fd8f-cwl5h\" (UID: \"28b45c9e-a664-452c-8972-5d68d0f09a48\") " pod="ai-tenant-e2e-shared-a-w4-d06633/auth-test-model-d06633-kserve-685c8fd8f-cwl5h"
Nov 01 15:04:14.417794 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:14.417756    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28b45c9e-a664-452c-8972-5d68d0f09a48-model-cache\") pod \"auth-test-model-d06633-kserve-685c8fd8f-cwl5h\" (UID: \"28b45c9e-a664-452c-8972-5d68d0f09a48\") " pod="ai-tenant-e2e-shared-a-w4-d06633/auth-test-model-d06633-kserve-685c8fd8f-cwl5h"
Nov 01 15:04:14.418010 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:14.417826    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28b45c9e-a664-452c-8972-5d68d0f09a48-tmp-dir\") pod \"auth-test-model-d06633-kserve-685c8fd8f-cwl5h\" (UID: \"28b45c9e-a664-452c-8972-5d68d0f09a48\") " pod="ai-tenant-e2e-shared-a-w4-d06633/auth-test-model-d06633-kserve-685c8fd8f-cwl5h"
Nov 01 15:04:14.418010 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:14.417862    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28b45c9e-a664-452c-8972-5d68d0f09a48-tls-certs\") pod \"auth-test-model-d06633-kserve-685c8fd8f-cwl5h\" (UID: \"28b45c9e-a664-452c-8972-5d68d0f09a48\") " pod="ai-tenant-e2e-shared-a-w4-d06633/auth-test-model-d06633-kserve-685c8fd8f-cwl5h"
Nov 01 15:04:14.418010 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:14.417887    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28b45c9e-a664-452c-8972-5d68d0f09a48-dshm\") pod \"auth-test-model-d06633-kserve-685c8fd8f-cwl5h\" (UID: \"28b45c9e-a664-452c-8972-5d68d0f09a48\") " pod="ai-tenant-e2e-shared-a-w4-d06633/auth-test-model-d06633-kserve-685c8fd8f-cwl5h"
Nov 01 15:04:14.418010 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:14.417980    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzt7g\" (UniqueName: \"kubernetes.io/projected/28b45c9e-a664-452c-8972-5d68d0f09a48-kube-api-access-kzt7g\") pod \"auth-test-model-d06633-kserve-685c8fd8f-cwl5h\" (UID: \"28b45c9e-a664-452c-8972-5d68d0f09a48\") " pod="ai-tenant-e2e-shared-a-w4-d06633/auth-test-model-d06633-kserve-685c8fd8f-cwl5h"
Nov 01 15:04:14.418010 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:14.418009    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28b45c9e-a664-452c-8972-5d68d0f09a48-home\") pod \"auth-test-model-d06633-kserve-685c8fd8f-cwl5h\" (UID: \"28b45c9e-a664-452c-8972-5d68d0f09a48\") " pod="ai-tenant-e2e-shared-a-w4-d06633/auth-test-model-d06633-kserve-685c8fd8f-cwl5h"
Nov 01 15:04:14.418355 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:14.418245    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28b45c9e-a664-452c-8972-5d68d0f09a48-model-cache\") pod \"auth-test-model-d06633-kserve-685c8fd8f-cwl5h\" (UID: \"28b45c9e-a664-452c-8972-5d68d0f09a48\") " pod="ai-tenant-e2e-shared-a-w4-d06633/auth-test-model-d06633-kserve-685c8fd8f-cwl5h"
Nov 01 15:04:14.418355 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:14.418289    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28b45c9e-a664-452c-8972-5d68d0f09a48-tmp-dir\") pod \"auth-test-model-d06633-kserve-685c8fd8f-cwl5h\" (UID: \"28b45c9e-a664-452c-8972-5d68d0f09a48\") " pod="ai-tenant-e2e-shared-a-w4-d06633/auth-test-model-d06633-kserve-685c8fd8f-cwl5h"
Nov 01 15:04:14.418459 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:14.418372    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28b45c9e-a664-452c-8972-5d68d0f09a48-home\") pod \"auth-test-model-d06633-kserve-685c8fd8f-cwl5h\" (UID: \"28b45c9e-a664-452c-8972-5d68d0f09a48\") " pod="ai-tenant-e2e-shared-a-w4-d06633/auth-test-model-d06633-kserve-685c8fd8f-cwl5h"
Nov 01 15:04:14.420610 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:14.420577    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28b45c9e-a664-452c-8972-5d68d0f09a48-tls-certs\") pod \"auth-test-model-d06633-kserve-685c8fd8f-cwl5h\" (UID: \"28b45c9e-a664-452c-8972-5d68d0f09a48\") " pod="ai-tenant-e2e-shared-a-w4-d06633/auth-test-model-d06633-kserve-685c8fd8f-cwl5h"
Nov 01 15:04:14.420754 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:14.420621    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28b45c9e-a664-452c-8972-5d68d0f09a48-dshm\") pod \"auth-test-model-d06633-kserve-685c8fd8f-cwl5h\" (UID: \"28b45c9e-a664-452c-8972-5d68d0f09a48\") " pod="ai-tenant-e2e-shared-a-w4-d06633/auth-test-model-d06633-kserve-685c8fd8f-cwl5h"
Nov 01 15:04:14.482234 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:14.472692    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzt7g\" (UniqueName: \"kubernetes.io/projected/28b45c9e-a664-452c-8972-5d68d0f09a48-kube-api-access-kzt7g\") pod \"auth-test-model-d06633-kserve-685c8fd8f-cwl5h\" (UID: \"28b45c9e-a664-452c-8972-5d68d0f09a48\") " pod="ai-tenant-e2e-shared-a-w4-d06633/auth-test-model-d06633-kserve-685c8fd8f-cwl5h"
Nov 01 15:04:14.497531 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:14.497502    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-d06633/auth-test-model-d06633-kserve-685c8fd8f-cwl5h"
Nov 01 15:04:14.798486 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:14.798453    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58304650-1c70-4e2d-9ef7-d0065e8940b7" path="/var/lib/kubelet/pods/58304650-1c70-4e2d-9ef7-d0065e8940b7/volumes"
Nov 01 15:04:14.798912 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:14.798897    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf03339d-d720-4474-b495-2d149d09d33a" path="/var/lib/kubelet/pods/cf03339d-d720-4474-b495-2d149d09d33a/volumes"
Nov 01 15:04:14.875148 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:14.875107    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-d06633/auth-test-model-d06633-kserve-685c8fd8f-cwl5h"]
Nov 01 15:04:14.881187 ip-10-0-134-37 kubenswrapper[4993]: W1101 15:04:14.881137    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod28b45c9e_a664_452c_8972_5d68d0f09a48.slice/crio-2dd3f5704ce1d0ed3bc5ace1e191bdf2cb58838cffa449efbb6ba58e32c9241f WatchSource:0}: Error finding container 2dd3f5704ce1d0ed3bc5ace1e191bdf2cb58838cffa449efbb6ba58e32c9241f: Status 404 returned error can't find the container with id 2dd3f5704ce1d0ed3bc5ace1e191bdf2cb58838cffa449efbb6ba58e32c9241f
Nov 01 15:04:15.734772 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:15.734734    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-d06633/auth-test-model-d06633-kserve-685c8fd8f-cwl5h" event={"ID":"28b45c9e-a664-452c-8972-5d68d0f09a48","Type":"ContainerStarted","Data":"b30fd2482643d3041fc30d56ed8251f83a8c842289003d4476553c6cd766ac8e"}
Nov 01 15:04:15.734772 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:15.734782    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-d06633/auth-test-model-d06633-kserve-685c8fd8f-cwl5h" event={"ID":"28b45c9e-a664-452c-8972-5d68d0f09a48","Type":"ContainerStarted","Data":"2dd3f5704ce1d0ed3bc5ace1e191bdf2cb58838cffa449efbb6ba58e32c9241f"}
Nov 01 15:04:15.844235 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:15.844155    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-d06633/auth-test-model-d06633-kserve-685c8fd8f-cwl5h" podStartSLOduration=1.84414187 podStartE2EDuration="1.84414187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:04:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:04:15.843641185 +0000 UTC m=+2843.173503923" watchObservedRunningTime="2026-11-01 15:04:15.84414187 +0000 UTC m=+2843.174004597"
Nov 01 15:04:16.498209 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:16.498163    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-d06633/auth-test-model-d06633-kserve-685c8fd8f-cwl5h"
Nov 01 15:04:24.498358 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:24.498311    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-d06633/auth-test-model-d06633-kserve-685c8fd8f-cwl5h"
Nov 01 15:04:24.512572 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:24.512542    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-d06633/auth-test-model-d06633-kserve-685c8fd8f-cwl5h"
Nov 01 15:04:24.821177 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:24.821092    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-d06633/auth-test-model-d06633-kserve-685c8fd8f-cwl5h"
Nov 01 15:04:27.503981 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:04:27.503943    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:04:28.275352 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:28.275317    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n"]
Nov 01 15:04:28.275600 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:28.275574    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n" podUID="34a09630-08a4-4cf5-a72d-6931a77c3ea7" containerName="istio-proxy" containerID="cri-o://0f30b63febac6fd9293dbe241feb50545820316df7880497d551d6ad37b8b726" gracePeriod=30
Nov 01 15:04:28.325614 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:28.325577    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl"]
Nov 01 15:04:28.325805 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:28.325784    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl" podUID="4e1cf19c-563c-45d0-9a6a-a10fdb869f4d" containerName="istio-proxy" containerID="cri-o://cebabe451d28b38d6560c657aa57ca8aa09065ee1fe3849ba558e83fbc430e11" gracePeriod=30
Nov 01 15:04:33.606306 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:33.606271    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n"
Nov 01 15:04:33.647038 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:33.647008    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl"
Nov 01 15:04:33.728320 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:33.728286    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e1cf19c-563c-45d0-9a6a-a10fdb869f4d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e1cf19c-563c-45d0-9a6a-a10fdb869f4d-workload-certs\") pod \"4e1cf19c-563c-45d0-9a6a-a10fdb869f4d\" (UID: \"4e1cf19c-563c-45d0-9a6a-a10fdb869f4d\") "
Nov 01 15:04:33.728521 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:33.728349    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34a09630-08a4-4cf5-a72d-6931a77c3ea7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34a09630-08a4-4cf5-a72d-6931a77c3ea7-istio-data\") pod \"34a09630-08a4-4cf5-a72d-6931a77c3ea7\" (UID: \"34a09630-08a4-4cf5-a72d-6931a77c3ea7\") "
Nov 01 15:04:33.728521 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:33.728376    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e1cf19c-563c-45d0-9a6a-a10fdb869f4d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e1cf19c-563c-45d0-9a6a-a10fdb869f4d-istio-data\") pod \"4e1cf19c-563c-45d0-9a6a-a10fdb869f4d\" (UID: \"4e1cf19c-563c-45d0-9a6a-a10fdb869f4d\") "
Nov 01 15:04:33.728521 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:33.728429    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34a09630-08a4-4cf5-a72d-6931a77c3ea7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34a09630-08a4-4cf5-a72d-6931a77c3ea7-credential-socket\") pod \"34a09630-08a4-4cf5-a72d-6931a77c3ea7\" (UID: \"34a09630-08a4-4cf5-a72d-6931a77c3ea7\") "
Nov 01 15:04:33.728521 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:33.728459    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e1cf19c-563c-45d0-9a6a-a10fdb869f4d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e1cf19c-563c-45d0-9a6a-a10fdb869f4d-istio-podinfo\") pod \"4e1cf19c-563c-45d0-9a6a-a10fdb869f4d\" (UID: \"4e1cf19c-563c-45d0-9a6a-a10fdb869f4d\") "
Nov 01 15:04:33.728521 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:33.728507    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/34a09630-08a4-4cf5-a72d-6931a77c3ea7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34a09630-08a4-4cf5-a72d-6931a77c3ea7-istio-podinfo\") pod \"34a09630-08a4-4cf5-a72d-6931a77c3ea7\" (UID: \"34a09630-08a4-4cf5-a72d-6931a77c3ea7\") "
Nov 01 15:04:33.728788 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:33.728561    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34a09630-08a4-4cf5-a72d-6931a77c3ea7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34a09630-08a4-4cf5-a72d-6931a77c3ea7-istio-envoy\") pod \"34a09630-08a4-4cf5-a72d-6931a77c3ea7\" (UID: \"34a09630-08a4-4cf5-a72d-6931a77c3ea7\") "
Nov 01 15:04:33.728788 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:33.728637    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e1cf19c-563c-45d0-9a6a-a10fdb869f4d-workload-certs" pod "4e1cf19c-563c-45d0-9a6a-a10fdb869f4d" (UID: "4e1cf19c-563c-45d0-9a6a-a10fdb869f4d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:04:33.728788 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:33.728642    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e1cf19c-563c-45d0-9a6a-a10fdb869f4d-istio-token\" (UniqueName: \"kubernetes.io/projected/4e1cf19c-563c-45d0-9a6a-a10fdb869f4d-istio-token\") pod \"4e1cf19c-563c-45d0-9a6a-a10fdb869f4d\" (UID: \"4e1cf19c-563c-45d0-9a6a-a10fdb869f4d\") "
Nov 01 15:04:33.728788 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:33.728709    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34a09630-08a4-4cf5-a72d-6931a77c3ea7-istio-data" pod "34a09630-08a4-4cf5-a72d-6931a77c3ea7" (UID: "34a09630-08a4-4cf5-a72d-6931a77c3ea7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:04:33.728788 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:33.728730    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e1cf19c-563c-45d0-9a6a-a10fdb869f4d-istio-data" pod "4e1cf19c-563c-45d0-9a6a-a10fdb869f4d" (UID: "4e1cf19c-563c-45d0-9a6a-a10fdb869f4d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:04:33.728788 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:33.728722    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e1cf19c-563c-45d0-9a6a-a10fdb869f4d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e1cf19c-563c-45d0-9a6a-a10fdb869f4d-credential-socket\") pod \"4e1cf19c-563c-45d0-9a6a-a10fdb869f4d\" (UID: \"4e1cf19c-563c-45d0-9a6a-a10fdb869f4d\") "
Nov 01 15:04:33.729106 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:33.728815    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e1cf19c-563c-45d0-9a6a-a10fdb869f4d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e1cf19c-563c-45d0-9a6a-a10fdb869f4d-istio-envoy\") pod \"4e1cf19c-563c-45d0-9a6a-a10fdb869f4d\" (UID: \"4e1cf19c-563c-45d0-9a6a-a10fdb869f4d\") "
Nov 01 15:04:33.729106 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:33.728932    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34a09630-08a4-4cf5-a72d-6931a77c3ea7-credential-socket" pod "34a09630-08a4-4cf5-a72d-6931a77c3ea7" (UID: "34a09630-08a4-4cf5-a72d-6931a77c3ea7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:04:33.729106 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:33.728943    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e1cf19c-563c-45d0-9a6a-a10fdb869f4d-credential-socket" pod "4e1cf19c-563c-45d0-9a6a-a10fdb869f4d" (UID: "4e1cf19c-563c-45d0-9a6a-a10fdb869f4d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:04:33.729340 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:33.729181    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34a09630-08a4-4cf5-a72d-6931a77c3ea7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34a09630-08a4-4cf5-a72d-6931a77c3ea7-workload-certs\") pod \"34a09630-08a4-4cf5-a72d-6931a77c3ea7\" (UID: \"34a09630-08a4-4cf5-a72d-6931a77c3ea7\") "
Nov 01 15:04:33.729340 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:33.729234    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34a09630-08a4-4cf5-a72d-6931a77c3ea7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34a09630-08a4-4cf5-a72d-6931a77c3ea7-workload-socket\") pod \"34a09630-08a4-4cf5-a72d-6931a77c3ea7\" (UID: \"34a09630-08a4-4cf5-a72d-6931a77c3ea7\") "
Nov 01 15:04:33.729340 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:33.729291    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34a09630-08a4-4cf5-a72d-6931a77c3ea7-kube-api-access-zq5lb\" (UniqueName: \"kubernetes.io/projected/34a09630-08a4-4cf5-a72d-6931a77c3ea7-kube-api-access-zq5lb\") pod \"34a09630-08a4-4cf5-a72d-6931a77c3ea7\" (UID: \"34a09630-08a4-4cf5-a72d-6931a77c3ea7\") "
Nov 01 15:04:33.729507 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:33.729407    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34a09630-08a4-4cf5-a72d-6931a77c3ea7-workload-certs" pod "34a09630-08a4-4cf5-a72d-6931a77c3ea7" (UID: "34a09630-08a4-4cf5-a72d-6931a77c3ea7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:04:33.729507 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:33.729434    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e1cf19c-563c-45d0-9a6a-a10fdb869f4d-kube-api-access-cfdb4\" (UniqueName: \"kubernetes.io/projected/4e1cf19c-563c-45d0-9a6a-a10fdb869f4d-kube-api-access-cfdb4\") pod \"4e1cf19c-563c-45d0-9a6a-a10fdb869f4d\" (UID: \"4e1cf19c-563c-45d0-9a6a-a10fdb869f4d\") "
Nov 01 15:04:33.729619 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:33.729520    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34a09630-08a4-4cf5-a72d-6931a77c3ea7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34a09630-08a4-4cf5-a72d-6931a77c3ea7-istiod-ca-cert\") pod \"34a09630-08a4-4cf5-a72d-6931a77c3ea7\" (UID: \"34a09630-08a4-4cf5-a72d-6931a77c3ea7\") "
Nov 01 15:04:33.729619 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:33.729560    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e1cf19c-563c-45d0-9a6a-a10fdb869f4d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e1cf19c-563c-45d0-9a6a-a10fdb869f4d-workload-socket\") pod \"4e1cf19c-563c-45d0-9a6a-a10fdb869f4d\" (UID: \"4e1cf19c-563c-45d0-9a6a-a10fdb869f4d\") "
Nov 01 15:04:33.729619 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:33.729584    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e1cf19c-563c-45d0-9a6a-a10fdb869f4d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e1cf19c-563c-45d0-9a6a-a10fdb869f4d-istiod-ca-cert\") pod \"4e1cf19c-563c-45d0-9a6a-a10fdb869f4d\" (UID: \"4e1cf19c-563c-45d0-9a6a-a10fdb869f4d\") "
Nov 01 15:04:33.729781 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:33.729629    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34a09630-08a4-4cf5-a72d-6931a77c3ea7-istio-token\" (UniqueName: \"kubernetes.io/projected/34a09630-08a4-4cf5-a72d-6931a77c3ea7-istio-token\") pod \"34a09630-08a4-4cf5-a72d-6931a77c3ea7\" (UID: \"34a09630-08a4-4cf5-a72d-6931a77c3ea7\") "
Nov 01 15:04:33.729781 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:33.729656    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34a09630-08a4-4cf5-a72d-6931a77c3ea7-workload-socket" pod "34a09630-08a4-4cf5-a72d-6931a77c3ea7" (UID: "34a09630-08a4-4cf5-a72d-6931a77c3ea7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:04:33.730508 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:33.730123    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e1cf19c-563c-45d0-9a6a-a10fdb869f4d-workload-socket" pod "4e1cf19c-563c-45d0-9a6a-a10fdb869f4d" (UID: "4e1cf19c-563c-45d0-9a6a-a10fdb869f4d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:04:33.730508 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:33.730293    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e1cf19c-563c-45d0-9a6a-a10fdb869f4d-istiod-ca-cert" pod "4e1cf19c-563c-45d0-9a6a-a10fdb869f4d" (UID: "4e1cf19c-563c-45d0-9a6a-a10fdb869f4d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:04:33.730508 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:33.730363    4993 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34a09630-08a4-4cf5-a72d-6931a77c3ea7-workload-socket\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:04:33.730508 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:33.730421    4993 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e1cf19c-563c-45d0-9a6a-a10fdb869f4d-workload-socket\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:04:33.730508 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:33.730440    4993 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e1cf19c-563c-45d0-9a6a-a10fdb869f4d-workload-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:04:33.730508 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:33.730459    4993 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34a09630-08a4-4cf5-a72d-6931a77c3ea7-istio-data\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:04:33.730508 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:33.730461    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34a09630-08a4-4cf5-a72d-6931a77c3ea7-istiod-ca-cert" pod "34a09630-08a4-4cf5-a72d-6931a77c3ea7" (UID: "34a09630-08a4-4cf5-a72d-6931a77c3ea7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:04:33.730508 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:33.730476    4993 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e1cf19c-563c-45d0-9a6a-a10fdb869f4d-istio-data\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:04:33.730508 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:33.730494    4993 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34a09630-08a4-4cf5-a72d-6931a77c3ea7-credential-socket\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:04:33.730508 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:33.730511    4993 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e1cf19c-563c-45d0-9a6a-a10fdb869f4d-credential-socket\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:04:33.731182 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:33.730526    4993 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34a09630-08a4-4cf5-a72d-6931a77c3ea7-workload-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:04:33.731386 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:33.731358    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/34a09630-08a4-4cf5-a72d-6931a77c3ea7-istio-podinfo" pod "34a09630-08a4-4cf5-a72d-6931a77c3ea7" (UID: "34a09630-08a4-4cf5-a72d-6931a77c3ea7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 15:04:33.731716 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:33.731679    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e1cf19c-563c-45d0-9a6a-a10fdb869f4d-istio-podinfo" pod "4e1cf19c-563c-45d0-9a6a-a10fdb869f4d" (UID: "4e1cf19c-563c-45d0-9a6a-a10fdb869f4d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 15:04:33.731830 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:33.731731    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34a09630-08a4-4cf5-a72d-6931a77c3ea7-istio-envoy" pod "34a09630-08a4-4cf5-a72d-6931a77c3ea7" (UID: "34a09630-08a4-4cf5-a72d-6931a77c3ea7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:04:33.732116 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:33.732081    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e1cf19c-563c-45d0-9a6a-a10fdb869f4d-istio-envoy" pod "4e1cf19c-563c-45d0-9a6a-a10fdb869f4d" (UID: "4e1cf19c-563c-45d0-9a6a-a10fdb869f4d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:04:33.732244 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:33.732218    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e1cf19c-563c-45d0-9a6a-a10fdb869f4d-istio-token" pod "4e1cf19c-563c-45d0-9a6a-a10fdb869f4d" (UID: "4e1cf19c-563c-45d0-9a6a-a10fdb869f4d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:04:33.732325 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:33.732239    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e1cf19c-563c-45d0-9a6a-a10fdb869f4d-kube-api-access-cfdb4" pod "4e1cf19c-563c-45d0-9a6a-a10fdb869f4d" (UID: "4e1cf19c-563c-45d0-9a6a-a10fdb869f4d"). InnerVolumeSpecName "kube-api-access-cfdb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:04:33.732566 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:33.732546    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34a09630-08a4-4cf5-a72d-6931a77c3ea7-kube-api-access-zq5lb" pod "34a09630-08a4-4cf5-a72d-6931a77c3ea7" (UID: "34a09630-08a4-4cf5-a72d-6931a77c3ea7"). InnerVolumeSpecName "kube-api-access-zq5lb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:04:33.733377 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:33.733355    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34a09630-08a4-4cf5-a72d-6931a77c3ea7-istio-token" pod "34a09630-08a4-4cf5-a72d-6931a77c3ea7" (UID: "34a09630-08a4-4cf5-a72d-6931a77c3ea7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:04:33.831930 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:33.831839    4993 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e1cf19c-563c-45d0-9a6a-a10fdb869f4d-istiod-ca-cert\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:04:33.831930 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:33.831877    4993 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34a09630-08a4-4cf5-a72d-6931a77c3ea7-istio-token\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:04:33.831930 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:33.831891    4993 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e1cf19c-563c-45d0-9a6a-a10fdb869f4d-istio-podinfo\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:04:33.831930 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:33.831904    4993 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34a09630-08a4-4cf5-a72d-6931a77c3ea7-istio-podinfo\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:04:33.831930 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:33.831917    4993 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34a09630-08a4-4cf5-a72d-6931a77c3ea7-istio-envoy\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:04:33.831930 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:33.831928    4993 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e1cf19c-563c-45d0-9a6a-a10fdb869f4d-istio-token\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:04:33.832530 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:33.831942    4993 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e1cf19c-563c-45d0-9a6a-a10fdb869f4d-istio-envoy\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:04:33.832530 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:33.831957    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zq5lb\" (UniqueName: \"kubernetes.io/projected/34a09630-08a4-4cf5-a72d-6931a77c3ea7-kube-api-access-zq5lb\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:04:33.832530 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:33.831972    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfdb4\" (UniqueName: \"kubernetes.io/projected/4e1cf19c-563c-45d0-9a6a-a10fdb869f4d-kube-api-access-cfdb4\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:04:33.832530 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:33.831985    4993 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34a09630-08a4-4cf5-a72d-6931a77c3ea7-istiod-ca-cert\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:04:33.865992 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:33.865957    4993 generic.go:356] "Generic (PLEG): container finished" podID="34a09630-08a4-4cf5-a72d-6931a77c3ea7" containerID="0f30b63febac6fd9293dbe241feb50545820316df7880497d551d6ad37b8b726" exitCode=0
Nov 01 15:04:33.866204 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:33.866019    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n"
Nov 01 15:04:33.866204 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:33.866041    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n" event={"ID":"34a09630-08a4-4cf5-a72d-6931a77c3ea7","Type":"ContainerDied","Data":"0f30b63febac6fd9293dbe241feb50545820316df7880497d551d6ad37b8b726"}
Nov 01 15:04:33.866204 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:33.866114    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n" event={"ID":"34a09630-08a4-4cf5-a72d-6931a77c3ea7","Type":"ContainerDied","Data":"a33cf881997206a2ede0ec23e97098a22eea3c7c52cf750f0f8f85b61b18863c"}
Nov 01 15:04:33.866204 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:33.866143    4993 scope.go:122] "RemoveContainer" containerID="0f30b63febac6fd9293dbe241feb50545820316df7880497d551d6ad37b8b726"
Nov 01 15:04:33.867721 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:33.867694    4993 generic.go:356] "Generic (PLEG): container finished" podID="4e1cf19c-563c-45d0-9a6a-a10fdb869f4d" containerID="cebabe451d28b38d6560c657aa57ca8aa09065ee1fe3849ba558e83fbc430e11" exitCode=0
Nov 01 15:04:33.867849 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:33.867765    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl"
Nov 01 15:04:33.867849 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:33.867781    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl" event={"ID":"4e1cf19c-563c-45d0-9a6a-a10fdb869f4d","Type":"ContainerDied","Data":"cebabe451d28b38d6560c657aa57ca8aa09065ee1fe3849ba558e83fbc430e11"}
Nov 01 15:04:33.867849 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:33.867826    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl" event={"ID":"4e1cf19c-563c-45d0-9a6a-a10fdb869f4d","Type":"ContainerDied","Data":"d3bc823cbf69364db6976a1b8f50b01be9413edd95073d289bf16fb19d08a28a"}
Nov 01 15:04:33.878135 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:33.878111    4993 scope.go:122] "RemoveContainer" containerID="0f30b63febac6fd9293dbe241feb50545820316df7880497d551d6ad37b8b726"
Nov 01 15:04:33.878446 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:04:33.878427    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f30b63febac6fd9293dbe241feb50545820316df7880497d551d6ad37b8b726\": container with ID starting with 0f30b63febac6fd9293dbe241feb50545820316df7880497d551d6ad37b8b726 not found: ID does not exist" containerID="0f30b63febac6fd9293dbe241feb50545820316df7880497d551d6ad37b8b726"
Nov 01 15:04:33.878526 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:33.878462    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f30b63febac6fd9293dbe241feb50545820316df7880497d551d6ad37b8b726"} err="failed to get container status \"0f30b63febac6fd9293dbe241feb50545820316df7880497d551d6ad37b8b726\": rpc error: code = NotFound desc = could not find container \"0f30b63febac6fd9293dbe241feb50545820316df7880497d551d6ad37b8b726\": container with ID starting with 0f30b63febac6fd9293dbe241feb50545820316df7880497d551d6ad37b8b726 not found: ID does not exist"
Nov 01 15:04:33.878526 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:33.878485    4993 scope.go:122] "RemoveContainer" containerID="cebabe451d28b38d6560c657aa57ca8aa09065ee1fe3849ba558e83fbc430e11"
Nov 01 15:04:33.909128 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:33.899131    4993 scope.go:122] "RemoveContainer" containerID="cebabe451d28b38d6560c657aa57ca8aa09065ee1fe3849ba558e83fbc430e11"
Nov 01 15:04:33.909128 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:04:33.900231    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cebabe451d28b38d6560c657aa57ca8aa09065ee1fe3849ba558e83fbc430e11\": container with ID starting with cebabe451d28b38d6560c657aa57ca8aa09065ee1fe3849ba558e83fbc430e11 not found: ID does not exist" containerID="cebabe451d28b38d6560c657aa57ca8aa09065ee1fe3849ba558e83fbc430e11"
Nov 01 15:04:33.909128 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:33.900277    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cebabe451d28b38d6560c657aa57ca8aa09065ee1fe3849ba558e83fbc430e11"} err="failed to get container status \"cebabe451d28b38d6560c657aa57ca8aa09065ee1fe3849ba558e83fbc430e11\": rpc error: code = NotFound desc = could not find container \"cebabe451d28b38d6560c657aa57ca8aa09065ee1fe3849ba558e83fbc430e11\": container with ID starting with cebabe451d28b38d6560c657aa57ca8aa09065ee1fe3849ba558e83fbc430e11 not found: ID does not exist"
Nov 01 15:04:33.987904 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:33.987872    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n"]
Nov 01 15:04:33.993015 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:33.992986    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-2854n"]
Nov 01 15:04:34.094906 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:34.094817    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl"]
Nov 01 15:04:34.099200 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:34.099151    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-q4ktl"]
Nov 01 15:04:34.796243 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:34.796209    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34a09630-08a4-4cf5-a72d-6931a77c3ea7" path="/var/lib/kubelet/pods/34a09630-08a4-4cf5-a72d-6931a77c3ea7/volumes"
Nov 01 15:04:34.796721 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:34.796706    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e1cf19c-563c-45d0-9a6a-a10fdb869f4d" path="/var/lib/kubelet/pods/4e1cf19c-563c-45d0-9a6a-a10fdb869f4d/volumes"
Nov 01 15:04:44.980608 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:44.980474    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hk2fv"]
Nov 01 15:04:44.985479 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:44.980714    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hk2fv" podUID="4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81" containerName="istio-proxy" containerID="cri-o://3fce53d3d877e579e59648fdae1b5e8072d2da4814fb0331671432d09a5c1805" gracePeriod=30
Nov 01 15:04:50.246920 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:50.246892    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hk2fv"
Nov 01 15:04:50.334212 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:50.334110    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81-kube-api-access-w6vxp\" (UniqueName: \"kubernetes.io/projected/4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81-kube-api-access-w6vxp\") pod \"4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81\" (UID: \"4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81\") "
Nov 01 15:04:50.334212 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:50.334150    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81-istio-data\") pod \"4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81\" (UID: \"4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81\") "
Nov 01 15:04:50.334451 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:50.334223    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81-istio-envoy\") pod \"4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81\" (UID: \"4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81\") "
Nov 01 15:04:50.334451 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:50.334287    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81-workload-certs\") pod \"4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81\" (UID: \"4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81\") "
Nov 01 15:04:50.334451 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:50.334318    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81-istio-podinfo\") pod \"4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81\" (UID: \"4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81\") "
Nov 01 15:04:50.334451 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:50.334383    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81-istiod-ca-cert\") pod \"4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81\" (UID: \"4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81\") "
Nov 01 15:04:50.334451 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:50.334416    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81-credential-socket\") pod \"4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81\" (UID: \"4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81\") "
Nov 01 15:04:50.334724 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:50.334497    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81-istio-token\" (UniqueName: \"kubernetes.io/projected/4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81-istio-token\") pod \"4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81\" (UID: \"4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81\") "
Nov 01 15:04:50.334724 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:50.334542    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81-workload-socket\") pod \"4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81\" (UID: \"4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81\") "
Nov 01 15:04:50.334724 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:50.334550    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81-istio-data" pod "4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81" (UID: "4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:04:50.334724 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:50.334664    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81-workload-certs" pod "4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81" (UID: "4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:04:50.334929 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:50.334768    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81-credential-socket" pod "4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81" (UID: "4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:04:50.334929 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:50.334878    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81-workload-socket" pod "4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81" (UID: "4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:04:50.334929 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:50.334820    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81-istiod-ca-cert" pod "4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81" (UID: "4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:04:50.335203 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:50.335164    4993 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81-istio-data\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:04:50.335203 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:50.335205    4993 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81-workload-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:04:50.335426 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:50.335220    4993 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81-istiod-ca-cert\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:04:50.335426 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:50.335235    4993 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81-credential-socket\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:04:50.335426 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:50.335249    4993 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81-workload-socket\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:04:50.336646 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:50.336617    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81-kube-api-access-w6vxp" pod "4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81" (UID: "4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81"). InnerVolumeSpecName "kube-api-access-w6vxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:04:50.336646 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:50.336635    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81-istio-token" pod "4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81" (UID: "4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:04:50.336843 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:50.336751    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81-istio-envoy" pod "4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81" (UID: "4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:04:50.336843 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:50.336790    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81-istio-podinfo" pod "4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81" (UID: "4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 15:04:50.436830 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:50.436791    4993 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81-istio-envoy\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:04:50.436830 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:50.436824    4993 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81-istio-podinfo\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:04:50.436830 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:50.436834    4993 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81-istio-token\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:04:50.437064 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:50.436844    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6vxp\" (UniqueName: \"kubernetes.io/projected/4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81-kube-api-access-w6vxp\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:04:50.984072 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:50.984038    4993 generic.go:356] "Generic (PLEG): container finished" podID="4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81" containerID="3fce53d3d877e579e59648fdae1b5e8072d2da4814fb0331671432d09a5c1805" exitCode=0
Nov 01 15:04:50.984285 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:50.984105    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hk2fv"
Nov 01 15:04:50.984285 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:50.984118    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hk2fv" event={"ID":"4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81","Type":"ContainerDied","Data":"3fce53d3d877e579e59648fdae1b5e8072d2da4814fb0331671432d09a5c1805"}
Nov 01 15:04:50.984285 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:50.984200    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hk2fv" event={"ID":"4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81","Type":"ContainerDied","Data":"2839545b6d4539dac9acdc2cb19091d9835f7ee93a01b18a0d742322f6032f98"}
Nov 01 15:04:50.984285 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:50.984226    4993 scope.go:122] "RemoveContainer" containerID="3fce53d3d877e579e59648fdae1b5e8072d2da4814fb0331671432d09a5c1805"
Nov 01 15:04:50.994655 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:50.994635    4993 scope.go:122] "RemoveContainer" containerID="3fce53d3d877e579e59648fdae1b5e8072d2da4814fb0331671432d09a5c1805"
Nov 01 15:04:50.994978 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:04:50.994954    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fce53d3d877e579e59648fdae1b5e8072d2da4814fb0331671432d09a5c1805\": container with ID starting with 3fce53d3d877e579e59648fdae1b5e8072d2da4814fb0331671432d09a5c1805 not found: ID does not exist" containerID="3fce53d3d877e579e59648fdae1b5e8072d2da4814fb0331671432d09a5c1805"
Nov 01 15:04:50.995046 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:50.994990    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fce53d3d877e579e59648fdae1b5e8072d2da4814fb0331671432d09a5c1805"} err="failed to get container status \"3fce53d3d877e579e59648fdae1b5e8072d2da4814fb0331671432d09a5c1805\": rpc error: code = NotFound desc = could not find container \"3fce53d3d877e579e59648fdae1b5e8072d2da4814fb0331671432d09a5c1805\": container with ID starting with 3fce53d3d877e579e59648fdae1b5e8072d2da4814fb0331671432d09a5c1805 not found: ID does not exist"
Nov 01 15:04:51.185194 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:51.185080    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hk2fv"]
Nov 01 15:04:51.231161 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:51.231119    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-hk2fv"]
Nov 01 15:04:52.796555 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:52.796519    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81" path="/var/lib/kubelet/pods/4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81/volumes"
Nov 01 15:04:54.814317 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:54.814279    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl"]
Nov 01 15:04:54.814711 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:54.814509    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl" podUID="0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48" containerName="istio-proxy" containerID="cri-o://5cc2aa36c36438f325003aa99c9fa79d127cd36a8f67ee1aa8b9788a5aa34ba9" gracePeriod=30
Nov 01 15:04:57.501099 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:04:57.501053    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:04:59.880277 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:59.880248    4993 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 01 15:04:59.880640 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:04:59.880313    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl" podUID="0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 01 15:05:00.047311 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:00.047270    4993 generic.go:356] "Generic (PLEG): container finished" podID="0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48" containerID="5cc2aa36c36438f325003aa99c9fa79d127cd36a8f67ee1aa8b9788a5aa34ba9" exitCode=0
Nov 01 15:05:00.047469 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:00.047346    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl" event={"ID":"0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48","Type":"ContainerDied","Data":"5cc2aa36c36438f325003aa99c9fa79d127cd36a8f67ee1aa8b9788a5aa34ba9"}
Nov 01 15:05:00.080082 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:00.080014    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl"
Nov 01 15:05:00.151269 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:00.151235    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48-kube-api-access-tmrmz\" (UniqueName: \"kubernetes.io/projected/0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48-kube-api-access-tmrmz\") pod \"0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48\" (UID: \"0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48\") "
Nov 01 15:05:00.151468 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:00.151280    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48-istio-podinfo\") pod \"0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48\" (UID: \"0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48\") "
Nov 01 15:05:00.151468 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:00.151316    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48-istiod-ca-cert\") pod \"0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48\" (UID: \"0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48\") "
Nov 01 15:05:00.151468 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:00.151352    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48-istio-token\" (UniqueName: \"kubernetes.io/projected/0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48-istio-token\") pod \"0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48\" (UID: \"0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48\") "
Nov 01 15:05:00.151468 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:00.151384    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48-workload-socket\") pod \"0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48\" (UID: \"0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48\") "
Nov 01 15:05:00.151695 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:00.151501    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48-istio-data\") pod \"0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48\" (UID: \"0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48\") "
Nov 01 15:05:00.151695 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:00.151526    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48-credential-socket\") pod \"0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48\" (UID: \"0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48\") "
Nov 01 15:05:00.151695 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:00.151689    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48-workload-certs\") pod \"0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48\" (UID: \"0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48\") "
Nov 01 15:05:00.151867 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:00.151716    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48-istio-envoy\") pod \"0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48\" (UID: \"0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48\") "
Nov 01 15:05:00.151867 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:00.151761    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48-istio-data" pod "0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48" (UID: "0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:05:00.151867 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:00.151799    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48-credential-socket" pod "0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48" (UID: "0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:05:00.152024 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:00.151978    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48-workload-certs" pod "0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48" (UID: "0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:05:00.152261 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:00.152235    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48-workload-socket" pod "0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48" (UID: "0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:05:00.152370 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:00.152239    4993 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48-istio-data\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:05:00.152370 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:00.152298    4993 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48-credential-socket\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:05:00.152370 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:00.152309    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48-istiod-ca-cert" pod "0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48" (UID: "0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:05:00.152370 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:00.152316    4993 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48-workload-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:05:00.153836 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:00.153810    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48-kube-api-access-tmrmz" pod "0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48" (UID: "0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48"). InnerVolumeSpecName "kube-api-access-tmrmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:05:00.153836 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:00.153820    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48-istio-podinfo" pod "0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48" (UID: "0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 15:05:00.154013 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:00.153849    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48-istio-token" pod "0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48" (UID: "0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:05:00.154234 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:00.154214    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48-istio-envoy" pod "0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48" (UID: "0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:05:00.253558 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:00.253524    4993 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48-istiod-ca-cert\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:05:00.253558 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:00.253551    4993 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48-istio-token\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:05:00.253558 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:00.253562    4993 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48-workload-socket\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:05:00.253785 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:00.253572    4993 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48-istio-envoy\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:05:00.253785 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:00.253583    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmrmz\" (UniqueName: \"kubernetes.io/projected/0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48-kube-api-access-tmrmz\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:05:00.253785 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:00.253593    4993 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48-istio-podinfo\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:05:01.056108 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:01.056019    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl"
Nov 01 15:05:01.056108 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:01.056056    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl" event={"ID":"0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48","Type":"ContainerDied","Data":"52deff652672642d3fb2a6a47ea34f25b386720653068ddfd64087e404ba97f6"}
Nov 01 15:05:01.056672 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:01.056134    4993 scope.go:122] "RemoveContainer" containerID="5cc2aa36c36438f325003aa99c9fa79d127cd36a8f67ee1aa8b9788a5aa34ba9"
Nov 01 15:05:01.221038 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:01.221003    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl"]
Nov 01 15:05:01.267351 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:01.267315    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-glkzl"]
Nov 01 15:05:02.795099 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:02.795064    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48" path="/var/lib/kubelet/pods/0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48/volumes"
Nov 01 15:05:18.037098 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:18.037061    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-7794bf-7f688777c8-cbhr5"]
Nov 01 15:05:18.037642 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:18.037627    4993 cpu_manager.go:414] "containerMap: removing container" podUID="4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81" containerName="istio-proxy"
Nov 01 15:05:18.037694 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:18.037644    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81" containerName="istio-proxy"
Nov 01 15:05:18.037694 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:18.037661    4993 cpu_manager.go:414] "containerMap: removing container" podUID="34a09630-08a4-4cf5-a72d-6931a77c3ea7" containerName="istio-proxy"
Nov 01 15:05:18.037694 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:18.037668    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34a09630-08a4-4cf5-a72d-6931a77c3ea7" containerName="istio-proxy"
Nov 01 15:05:18.037694 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:18.037678    4993 cpu_manager.go:414] "containerMap: removing container" podUID="4e1cf19c-563c-45d0-9a6a-a10fdb869f4d" containerName="istio-proxy"
Nov 01 15:05:18.037694 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:18.037683    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e1cf19c-563c-45d0-9a6a-a10fdb869f4d" containerName="istio-proxy"
Nov 01 15:05:18.037866 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:18.037698    4993 cpu_manager.go:414] "containerMap: removing container" podUID="0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48" containerName="istio-proxy"
Nov 01 15:05:18.037866 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:18.037702    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48" containerName="istio-proxy"
Nov 01 15:05:18.037866 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:18.037805    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d0df8f2-5aa9-4dd1-b3c0-33844c92cd81" containerName="istio-proxy"
Nov 01 15:05:18.037866 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:18.037815    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="34a09630-08a4-4cf5-a72d-6931a77c3ea7" containerName="istio-proxy"
Nov 01 15:05:18.037866 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:18.037820    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e1cf19c-563c-45d0-9a6a-a10fdb869f4d" containerName="istio-proxy"
Nov 01 15:05:18.037866 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:18.037827    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ea7d60e-70b5-4ae6-bb5e-db66fe29ef48" containerName="istio-proxy"
Nov 01 15:05:18.042631 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:18.042605    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7794bf-7f688777c8-cbhr5"
Nov 01 15:05:18.067668 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:18.067636    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fef68d07-91c6-499b-9405-889a33bd1375-maas-api-tls\") pod \"maas-api-e2e-explicit-7794bf-7f688777c8-cbhr5\" (UID: \"fef68d07-91c6-499b-9405-889a33bd1375\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7794bf-7f688777c8-cbhr5"
Nov 01 15:05:18.067798 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:18.067719    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hsvb\" (UniqueName: \"kubernetes.io/projected/fef68d07-91c6-499b-9405-889a33bd1375-kube-api-access-4hsvb\") pod \"maas-api-e2e-explicit-7794bf-7f688777c8-cbhr5\" (UID: \"fef68d07-91c6-499b-9405-889a33bd1375\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7794bf-7f688777c8-cbhr5"
Nov 01 15:05:18.067798 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:18.067780    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fef68d07-91c6-499b-9405-889a33bd1375-openshift-service-ca\") pod \"maas-api-e2e-explicit-7794bf-7f688777c8-cbhr5\" (UID: \"fef68d07-91c6-499b-9405-889a33bd1375\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7794bf-7f688777c8-cbhr5"
Nov 01 15:05:18.067927 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:18.067913    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fef68d07-91c6-499b-9405-889a33bd1375-metrics-certs\") pod \"maas-api-e2e-explicit-7794bf-7f688777c8-cbhr5\" (UID: \"fef68d07-91c6-499b-9405-889a33bd1375\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7794bf-7f688777c8-cbhr5"
Nov 01 15:05:18.091620 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:18.091567    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-7794bf-serving-cert\""
Nov 01 15:05:18.140276 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:18.140242    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-7794bf-7f688777c8-cbhr5"]
Nov 01 15:05:18.169520 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:18.169476    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fef68d07-91c6-499b-9405-889a33bd1375-maas-api-tls\") pod \"maas-api-e2e-explicit-7794bf-7f688777c8-cbhr5\" (UID: \"fef68d07-91c6-499b-9405-889a33bd1375\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7794bf-7f688777c8-cbhr5"
Nov 01 15:05:18.169718 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:18.169536    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hsvb\" (UniqueName: \"kubernetes.io/projected/fef68d07-91c6-499b-9405-889a33bd1375-kube-api-access-4hsvb\") pod \"maas-api-e2e-explicit-7794bf-7f688777c8-cbhr5\" (UID: \"fef68d07-91c6-499b-9405-889a33bd1375\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7794bf-7f688777c8-cbhr5"
Nov 01 15:05:18.169718 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:18.169606    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fef68d07-91c6-499b-9405-889a33bd1375-openshift-service-ca\") pod \"maas-api-e2e-explicit-7794bf-7f688777c8-cbhr5\" (UID: \"fef68d07-91c6-499b-9405-889a33bd1375\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7794bf-7f688777c8-cbhr5"
Nov 01 15:05:18.169718 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:05:18.169643    4993 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-7794bf-serving-cert: secret "maas-api-e2e-explicit-7794bf-serving-cert" not found
Nov 01 15:05:18.169890 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:05:18.169766    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fef68d07-91c6-499b-9405-889a33bd1375-maas-api-tls podName:fef68d07-91c6-499b-9405-889a33bd1375 nodeName:}" failed. No retries permitted until 2026-11-01 15:05:18.669718147 +0000 UTC m=+2905.999580871 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fef68d07-91c6-499b-9405-889a33bd1375-maas-api-tls") pod "maas-api-e2e-explicit-7794bf-7f688777c8-cbhr5" (UID: "fef68d07-91c6-499b-9405-889a33bd1375") : secret "maas-api-e2e-explicit-7794bf-serving-cert" not found
Nov 01 15:05:18.170018 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:18.169992    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fef68d07-91c6-499b-9405-889a33bd1375-metrics-certs\") pod \"maas-api-e2e-explicit-7794bf-7f688777c8-cbhr5\" (UID: \"fef68d07-91c6-499b-9405-889a33bd1375\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7794bf-7f688777c8-cbhr5"
Nov 01 15:05:18.170471 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:18.170450    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fef68d07-91c6-499b-9405-889a33bd1375-openshift-service-ca\") pod \"maas-api-e2e-explicit-7794bf-7f688777c8-cbhr5\" (UID: \"fef68d07-91c6-499b-9405-889a33bd1375\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7794bf-7f688777c8-cbhr5"
Nov 01 15:05:18.172731 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:18.172699    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fef68d07-91c6-499b-9405-889a33bd1375-metrics-certs\") pod \"maas-api-e2e-explicit-7794bf-7f688777c8-cbhr5\" (UID: \"fef68d07-91c6-499b-9405-889a33bd1375\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7794bf-7f688777c8-cbhr5"
Nov 01 15:05:18.224083 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:18.224050    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hsvb\" (UniqueName: \"kubernetes.io/projected/fef68d07-91c6-499b-9405-889a33bd1375-kube-api-access-4hsvb\") pod \"maas-api-e2e-explicit-7794bf-7f688777c8-cbhr5\" (UID: \"fef68d07-91c6-499b-9405-889a33bd1375\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7794bf-7f688777c8-cbhr5"
Nov 01 15:05:18.676634 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:18.676604    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fef68d07-91c6-499b-9405-889a33bd1375-maas-api-tls\") pod \"maas-api-e2e-explicit-7794bf-7f688777c8-cbhr5\" (UID: \"fef68d07-91c6-499b-9405-889a33bd1375\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7794bf-7f688777c8-cbhr5"
Nov 01 15:05:18.679055 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:18.679030    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fef68d07-91c6-499b-9405-889a33bd1375-maas-api-tls\") pod \"maas-api-e2e-explicit-7794bf-7f688777c8-cbhr5\" (UID: \"fef68d07-91c6-499b-9405-889a33bd1375\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7794bf-7f688777c8-cbhr5"
Nov 01 15:05:18.956269 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:18.956144    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7794bf-7f688777c8-cbhr5"
Nov 01 15:05:19.336814 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:19.336777    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-7794bf-7f688777c8-cbhr5"]
Nov 01 15:05:19.342439 ip-10-0-134-37 kubenswrapper[4993]: W1101 15:05:19.342409    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfef68d07_91c6_499b_9405_889a33bd1375.slice/crio-59bac8fa63a759b32fb1644168ac3cc160472d6f5e3153eb4ed5e0f0bbddcac1 WatchSource:0}: Error finding container 59bac8fa63a759b32fb1644168ac3cc160472d6f5e3153eb4ed5e0f0bbddcac1: Status 404 returned error can't find the container with id 59bac8fa63a759b32fb1644168ac3cc160472d6f5e3153eb4ed5e0f0bbddcac1
Nov 01 15:05:19.344356 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:19.344335    4993 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 15:05:20.194205 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:20.194142    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7794bf-7f688777c8-cbhr5" event={"ID":"fef68d07-91c6-499b-9405-889a33bd1375","Type":"ContainerStarted","Data":"59bac8fa63a759b32fb1644168ac3cc160472d6f5e3153eb4ed5e0f0bbddcac1"}
Nov 01 15:05:21.202248 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:21.202204    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7794bf-7f688777c8-cbhr5" event={"ID":"fef68d07-91c6-499b-9405-889a33bd1375","Type":"ContainerStarted","Data":"f21f0349137bf9af67d1b2a4d42e58ff0b46538547b5ed0eabbf78c1b6e15a63"}
Nov 01 15:05:21.202711 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:21.202280    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7794bf-7f688777c8-cbhr5"
Nov 01 15:05:21.311984 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:21.311925    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7794bf-7f688777c8-cbhr5" podStartSLOduration=3.409724218 podStartE2EDuration="4.311909783s" totalImagesPullingTime="902.185565ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:05:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:05:21.308162609 +0000 UTC m=+2908.638025348" watchObservedRunningTime="2026-11-01 15:05:21.311909783 +0000 UTC m=+2908.641772557"
Nov 01 15:05:27.147493 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:27.147451    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1ac11c-76b994fd57-mhmp5"]
Nov 01 15:05:27.151623 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:27.151598    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1ac11c-76b994fd57-mhmp5"
Nov 01 15:05:27.215256 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:27.215226    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7794bf-7f688777c8-cbhr5"
Nov 01 15:05:27.247988 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:27.247957    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-1ac11c-serving-cert\""
Nov 01 15:05:27.250838 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:27.250816    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1ac11c-76b994fd57-mhmp5"]
Nov 01 15:05:27.285045 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:27.285013    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ad8a332-6847-48a9-9249-415156dff36d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-1ac11c-76b994fd57-mhmp5\" (UID: \"6ad8a332-6847-48a9-9249-415156dff36d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1ac11c-76b994fd57-mhmp5"
Nov 01 15:05:27.285275 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:27.285255    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ad8a332-6847-48a9-9249-415156dff36d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-1ac11c-76b994fd57-mhmp5\" (UID: \"6ad8a332-6847-48a9-9249-415156dff36d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1ac11c-76b994fd57-mhmp5"
Nov 01 15:05:27.285485 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:27.285467    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q72nt\" (UniqueName: \"kubernetes.io/projected/6ad8a332-6847-48a9-9249-415156dff36d-kube-api-access-q72nt\") pod \"maas-api-e2e-shared-a-w3-1ac11c-76b994fd57-mhmp5\" (UID: \"6ad8a332-6847-48a9-9249-415156dff36d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1ac11c-76b994fd57-mhmp5"
Nov 01 15:05:27.285580 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:27.285565    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ad8a332-6847-48a9-9249-415156dff36d-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-1ac11c-76b994fd57-mhmp5\" (UID: \"6ad8a332-6847-48a9-9249-415156dff36d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1ac11c-76b994fd57-mhmp5"
Nov 01 15:05:27.387222 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:27.387178    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ad8a332-6847-48a9-9249-415156dff36d-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-1ac11c-76b994fd57-mhmp5\" (UID: \"6ad8a332-6847-48a9-9249-415156dff36d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1ac11c-76b994fd57-mhmp5"
Nov 01 15:05:27.387422 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:27.387302    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ad8a332-6847-48a9-9249-415156dff36d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-1ac11c-76b994fd57-mhmp5\" (UID: \"6ad8a332-6847-48a9-9249-415156dff36d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1ac11c-76b994fd57-mhmp5"
Nov 01 15:05:27.387422 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:27.387338    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ad8a332-6847-48a9-9249-415156dff36d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-1ac11c-76b994fd57-mhmp5\" (UID: \"6ad8a332-6847-48a9-9249-415156dff36d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1ac11c-76b994fd57-mhmp5"
Nov 01 15:05:27.387555 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:27.387431    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q72nt\" (UniqueName: \"kubernetes.io/projected/6ad8a332-6847-48a9-9249-415156dff36d-kube-api-access-q72nt\") pod \"maas-api-e2e-shared-a-w3-1ac11c-76b994fd57-mhmp5\" (UID: \"6ad8a332-6847-48a9-9249-415156dff36d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1ac11c-76b994fd57-mhmp5"
Nov 01 15:05:27.387555 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:05:27.387508    4993 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1ac11c-serving-cert: secret "maas-api-e2e-shared-a-w3-1ac11c-serving-cert" not found
Nov 01 15:05:27.387659 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:05:27.387599    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ad8a332-6847-48a9-9249-415156dff36d-maas-api-tls podName:6ad8a332-6847-48a9-9249-415156dff36d nodeName:}" failed. No retries permitted until 2026-11-01 15:05:27.887577068 +0000 UTC m=+2915.217439800 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6ad8a332-6847-48a9-9249-415156dff36d-maas-api-tls") pod "maas-api-e2e-shared-a-w3-1ac11c-76b994fd57-mhmp5" (UID: "6ad8a332-6847-48a9-9249-415156dff36d") : secret "maas-api-e2e-shared-a-w3-1ac11c-serving-cert" not found
Nov 01 15:05:27.388056 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:27.388034    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ad8a332-6847-48a9-9249-415156dff36d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-1ac11c-76b994fd57-mhmp5\" (UID: \"6ad8a332-6847-48a9-9249-415156dff36d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1ac11c-76b994fd57-mhmp5"
Nov 01 15:05:27.389829 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:27.389798    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ad8a332-6847-48a9-9249-415156dff36d-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-1ac11c-76b994fd57-mhmp5\" (UID: \"6ad8a332-6847-48a9-9249-415156dff36d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1ac11c-76b994fd57-mhmp5"
Nov 01 15:05:27.443650 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:27.443572    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q72nt\" (UniqueName: \"kubernetes.io/projected/6ad8a332-6847-48a9-9249-415156dff36d-kube-api-access-q72nt\") pod \"maas-api-e2e-shared-a-w3-1ac11c-76b994fd57-mhmp5\" (UID: \"6ad8a332-6847-48a9-9249-415156dff36d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1ac11c-76b994fd57-mhmp5"
Nov 01 15:05:27.507438 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:05:27.507400    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:05:27.893183 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:27.893134    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ad8a332-6847-48a9-9249-415156dff36d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-1ac11c-76b994fd57-mhmp5\" (UID: \"6ad8a332-6847-48a9-9249-415156dff36d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1ac11c-76b994fd57-mhmp5"
Nov 01 15:05:27.895666 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:27.895636    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ad8a332-6847-48a9-9249-415156dff36d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-1ac11c-76b994fd57-mhmp5\" (UID: \"6ad8a332-6847-48a9-9249-415156dff36d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1ac11c-76b994fd57-mhmp5"
Nov 01 15:05:28.065922 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:28.065887    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1ac11c-76b994fd57-mhmp5"
Nov 01 15:05:28.440520 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:28.440486    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1ac11c-76b994fd57-mhmp5"]
Nov 01 15:05:28.445870 ip-10-0-134-37 kubenswrapper[4993]: W1101 15:05:28.445835    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ad8a332_6847_48a9_9249_415156dff36d.slice/crio-09ecb773d90a34938e10313a09eb4ed2369b5a9169cf9bbe2366da16103eee81 WatchSource:0}: Error finding container 09ecb773d90a34938e10313a09eb4ed2369b5a9169cf9bbe2366da16103eee81: Status 404 returned error can't find the container with id 09ecb773d90a34938e10313a09eb4ed2369b5a9169cf9bbe2366da16103eee81
Nov 01 15:05:29.253440 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:29.253384    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1ac11c-76b994fd57-mhmp5" event={"ID":"6ad8a332-6847-48a9-9249-415156dff36d","Type":"ContainerStarted","Data":"09ecb773d90a34938e10313a09eb4ed2369b5a9169cf9bbe2366da16103eee81"}
Nov 01 15:05:30.261819 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:30.261784    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1ac11c-76b994fd57-mhmp5" event={"ID":"6ad8a332-6847-48a9-9249-415156dff36d","Type":"ContainerStarted","Data":"b8d32f611baf4308b14e4141f1815b91b6ac3719c8f40c99c461db83ad8a6210"}
Nov 01 15:05:30.262314 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:30.261956    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1ac11c-76b994fd57-mhmp5"
Nov 01 15:05:30.381569 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:30.381514    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1ac11c-76b994fd57-mhmp5" podStartSLOduration=2.497588989 podStartE2EDuration="3.381500028s" totalImagesPullingTime="883.911039ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:05:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:05:30.377285371 +0000 UTC m=+2917.707148107" watchObservedRunningTime="2026-11-01 15:05:30.381500028 +0000 UTC m=+2917.711362766"
Nov 01 15:05:36.275716 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:05:36.275624    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1ac11c-76b994fd57-mhmp5"
Nov 01 15:05:57.501140 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:05:57.501100    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:06:27.502125 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:06:27.502080    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:06:53.226128 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:06:53.226098    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xmwtp_bb59051d-c2df-4362-a157-00f14f915e79/console-operator/3.log"
Nov 01 15:06:53.226709 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:06:53.226099    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xmwtp_bb59051d-c2df-4362-a157-00f14f915e79/console-operator/3.log"
Nov 01 15:06:57.500789 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:06:57.500752    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:07:27.501937 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:07:27.501855    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:07:46.250558 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:07:46.250522    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8fd363-c4bc9fcf8-pm6vm"]
Nov 01 15:07:46.254576 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:07:46.254552    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8fd363-c4bc9fcf8-pm6vm"
Nov 01 15:07:46.303185 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:07:46.303140    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-8fd363-serving-cert\""
Nov 01 15:07:46.321726 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:07:46.321692    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1896a74d-cf1e-48cc-8ab8-330e11570845-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-8fd363-c4bc9fcf8-pm6vm\" (UID: \"1896a74d-cf1e-48cc-8ab8-330e11570845\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8fd363-c4bc9fcf8-pm6vm"
Nov 01 15:07:46.321920 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:07:46.321814    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1896a74d-cf1e-48cc-8ab8-330e11570845-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-8fd363-c4bc9fcf8-pm6vm\" (UID: \"1896a74d-cf1e-48cc-8ab8-330e11570845\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8fd363-c4bc9fcf8-pm6vm"
Nov 01 15:07:46.321993 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:07:46.321920    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp9w6\" (UniqueName: \"kubernetes.io/projected/1896a74d-cf1e-48cc-8ab8-330e11570845-kube-api-access-rp9w6\") pod \"maas-api-e2e-shared-b-w3-8fd363-c4bc9fcf8-pm6vm\" (UID: \"1896a74d-cf1e-48cc-8ab8-330e11570845\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8fd363-c4bc9fcf8-pm6vm"
Nov 01 15:07:46.321993 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:07:46.321969    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1896a74d-cf1e-48cc-8ab8-330e11570845-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-8fd363-c4bc9fcf8-pm6vm\" (UID: \"1896a74d-cf1e-48cc-8ab8-330e11570845\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8fd363-c4bc9fcf8-pm6vm"
Nov 01 15:07:46.354279 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:07:46.354243    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8fd363-c4bc9fcf8-pm6vm"]
Nov 01 15:07:46.423188 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:07:46.423136    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1896a74d-cf1e-48cc-8ab8-330e11570845-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-8fd363-c4bc9fcf8-pm6vm\" (UID: \"1896a74d-cf1e-48cc-8ab8-330e11570845\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8fd363-c4bc9fcf8-pm6vm"
Nov 01 15:07:46.423375 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:07:46.423293    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1896a74d-cf1e-48cc-8ab8-330e11570845-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-8fd363-c4bc9fcf8-pm6vm\" (UID: \"1896a74d-cf1e-48cc-8ab8-330e11570845\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8fd363-c4bc9fcf8-pm6vm"
Nov 01 15:07:46.423375 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:07:46.423351    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1896a74d-cf1e-48cc-8ab8-330e11570845-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-8fd363-c4bc9fcf8-pm6vm\" (UID: \"1896a74d-cf1e-48cc-8ab8-330e11570845\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8fd363-c4bc9fcf8-pm6vm"
Nov 01 15:07:46.423487 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:07:46.423435    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp9w6\" (UniqueName: \"kubernetes.io/projected/1896a74d-cf1e-48cc-8ab8-330e11570845-kube-api-access-rp9w6\") pod \"maas-api-e2e-shared-b-w3-8fd363-c4bc9fcf8-pm6vm\" (UID: \"1896a74d-cf1e-48cc-8ab8-330e11570845\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8fd363-c4bc9fcf8-pm6vm"
Nov 01 15:07:46.423487 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:07:46.423465    4993 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8fd363-serving-cert: secret "maas-api-e2e-shared-b-w3-8fd363-serving-cert" not found
Nov 01 15:07:46.423577 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:07:46.423558    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1896a74d-cf1e-48cc-8ab8-330e11570845-maas-api-tls podName:1896a74d-cf1e-48cc-8ab8-330e11570845 nodeName:}" failed. No retries permitted until 2026-11-01 15:07:46.923537324 +0000 UTC m=+3054.253400042 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1896a74d-cf1e-48cc-8ab8-330e11570845-maas-api-tls") pod "maas-api-e2e-shared-b-w3-8fd363-c4bc9fcf8-pm6vm" (UID: "1896a74d-cf1e-48cc-8ab8-330e11570845") : secret "maas-api-e2e-shared-b-w3-8fd363-serving-cert" not found
Nov 01 15:07:46.423962 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:07:46.423939    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1896a74d-cf1e-48cc-8ab8-330e11570845-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-8fd363-c4bc9fcf8-pm6vm\" (UID: \"1896a74d-cf1e-48cc-8ab8-330e11570845\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8fd363-c4bc9fcf8-pm6vm"
Nov 01 15:07:46.425603 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:07:46.425583    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1896a74d-cf1e-48cc-8ab8-330e11570845-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-8fd363-c4bc9fcf8-pm6vm\" (UID: \"1896a74d-cf1e-48cc-8ab8-330e11570845\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8fd363-c4bc9fcf8-pm6vm"
Nov 01 15:07:46.478036 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:07:46.478001    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp9w6\" (UniqueName: \"kubernetes.io/projected/1896a74d-cf1e-48cc-8ab8-330e11570845-kube-api-access-rp9w6\") pod \"maas-api-e2e-shared-b-w3-8fd363-c4bc9fcf8-pm6vm\" (UID: \"1896a74d-cf1e-48cc-8ab8-330e11570845\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8fd363-c4bc9fcf8-pm6vm"
Nov 01 15:07:46.929691 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:07:46.929652    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1896a74d-cf1e-48cc-8ab8-330e11570845-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-8fd363-c4bc9fcf8-pm6vm\" (UID: \"1896a74d-cf1e-48cc-8ab8-330e11570845\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8fd363-c4bc9fcf8-pm6vm"
Nov 01 15:07:46.932087 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:07:46.932064    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1896a74d-cf1e-48cc-8ab8-330e11570845-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-8fd363-c4bc9fcf8-pm6vm\" (UID: \"1896a74d-cf1e-48cc-8ab8-330e11570845\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8fd363-c4bc9fcf8-pm6vm"
Nov 01 15:07:47.172406 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:07:47.172368    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8fd363-c4bc9fcf8-pm6vm"
Nov 01 15:07:47.552015 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:07:47.551976    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8fd363-c4bc9fcf8-pm6vm"]
Nov 01 15:07:47.557852 ip-10-0-134-37 kubenswrapper[4993]: W1101 15:07:47.557807    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1896a74d_cf1e_48cc_8ab8_330e11570845.slice/crio-a0a2652c181b58e59686213bb7cc0b73d542e6e5aa614ceef214257e90574819 WatchSource:0}: Error finding container a0a2652c181b58e59686213bb7cc0b73d542e6e5aa614ceef214257e90574819: Status 404 returned error can't find the container with id a0a2652c181b58e59686213bb7cc0b73d542e6e5aa614ceef214257e90574819
Nov 01 15:07:48.189523 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:07:48.189485    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8fd363-c4bc9fcf8-pm6vm" event={"ID":"1896a74d-cf1e-48cc-8ab8-330e11570845","Type":"ContainerStarted","Data":"a0a2652c181b58e59686213bb7cc0b73d542e6e5aa614ceef214257e90574819"}
Nov 01 15:07:49.198587 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:07:49.198535    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8fd363-c4bc9fcf8-pm6vm" event={"ID":"1896a74d-cf1e-48cc-8ab8-330e11570845","Type":"ContainerStarted","Data":"344ab2c54a9fd135fde8842986ad8ac27ca1f2ace7e593ab39d4aaa90c16c054"}
Nov 01 15:07:49.198994 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:07:49.198616    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8fd363-c4bc9fcf8-pm6vm"
Nov 01 15:07:49.305976 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:07:49.305916    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8fd363-c4bc9fcf8-pm6vm" podStartSLOduration=2.268355099 podStartE2EDuration="3.305902228s" totalImagesPullingTime="1.037547129s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:07:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:07:49.305154084 +0000 UTC m=+3056.635016821" watchObservedRunningTime="2026-11-01 15:07:49.305902228 +0000 UTC m=+3056.635764965"
Nov 01 15:07:55.211304 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:07:55.211267    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8fd363-c4bc9fcf8-pm6vm"
Nov 01 15:07:57.501336 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:07:57.501299    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:08:27.508682 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:08:27.508641    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:08:30.520991 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:08:30.520956    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-7794bf-7f688777c8-cbhr5"]
Nov 01 15:08:30.521679 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:08:30.521247    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7794bf-7f688777c8-cbhr5" podUID="fef68d07-91c6-499b-9405-889a33bd1375" containerName="maas-api" containerID="cri-o://f21f0349137bf9af67d1b2a4d42e58ff0b46538547b5ed0eabbf78c1b6e15a63" gracePeriod=30
Nov 01 15:08:30.850270 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:08:30.850230    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7794bf-7f688777c8-cbhr5"
Nov 01 15:08:30.868885 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:08:30.868837    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fef68d07-91c6-499b-9405-889a33bd1375-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fef68d07-91c6-499b-9405-889a33bd1375-metrics-certs\") pod \"fef68d07-91c6-499b-9405-889a33bd1375\" (UID: \"fef68d07-91c6-499b-9405-889a33bd1375\") "
Nov 01 15:08:30.869054 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:08:30.868907    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fef68d07-91c6-499b-9405-889a33bd1375-kube-api-access-4hsvb\" (UniqueName: \"kubernetes.io/projected/fef68d07-91c6-499b-9405-889a33bd1375-kube-api-access-4hsvb\") pod \"fef68d07-91c6-499b-9405-889a33bd1375\" (UID: \"fef68d07-91c6-499b-9405-889a33bd1375\") "
Nov 01 15:08:30.869054 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:08:30.868954    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fef68d07-91c6-499b-9405-889a33bd1375-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fef68d07-91c6-499b-9405-889a33bd1375-maas-api-tls\") pod \"fef68d07-91c6-499b-9405-889a33bd1375\" (UID: \"fef68d07-91c6-499b-9405-889a33bd1375\") "
Nov 01 15:08:30.869220 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:08:30.869145    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fef68d07-91c6-499b-9405-889a33bd1375-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fef68d07-91c6-499b-9405-889a33bd1375-openshift-service-ca\") pod \"fef68d07-91c6-499b-9405-889a33bd1375\" (UID: \"fef68d07-91c6-499b-9405-889a33bd1375\") "
Nov 01 15:08:30.870082 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:08:30.870054    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fef68d07-91c6-499b-9405-889a33bd1375-openshift-service-ca" pod "fef68d07-91c6-499b-9405-889a33bd1375" (UID: "fef68d07-91c6-499b-9405-889a33bd1375"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:08:30.873006 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:08:30.872966    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fef68d07-91c6-499b-9405-889a33bd1375-metrics-certs" pod "fef68d07-91c6-499b-9405-889a33bd1375" (UID: "fef68d07-91c6-499b-9405-889a33bd1375"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:08:30.873543 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:08:30.873507    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fef68d07-91c6-499b-9405-889a33bd1375-maas-api-tls" pod "fef68d07-91c6-499b-9405-889a33bd1375" (UID: "fef68d07-91c6-499b-9405-889a33bd1375"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:08:30.905589 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:08:30.905340    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fef68d07-91c6-499b-9405-889a33bd1375-kube-api-access-4hsvb" pod "fef68d07-91c6-499b-9405-889a33bd1375" (UID: "fef68d07-91c6-499b-9405-889a33bd1375"). InnerVolumeSpecName "kube-api-access-4hsvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:08:30.971017 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:08:30.970984    4993 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fef68d07-91c6-499b-9405-889a33bd1375-openshift-service-ca\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:08:30.971017 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:08:30.971015    4993 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fef68d07-91c6-499b-9405-889a33bd1375-metrics-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:08:30.971017 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:08:30.971024    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hsvb\" (UniqueName: \"kubernetes.io/projected/fef68d07-91c6-499b-9405-889a33bd1375-kube-api-access-4hsvb\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:08:30.971273 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:08:30.971034    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fef68d07-91c6-499b-9405-889a33bd1375-maas-api-tls\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:08:31.475260 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:08:31.475227    4993 generic.go:356] "Generic (PLEG): container finished" podID="fef68d07-91c6-499b-9405-889a33bd1375" containerID="f21f0349137bf9af67d1b2a4d42e58ff0b46538547b5ed0eabbf78c1b6e15a63" exitCode=0
Nov 01 15:08:31.475444 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:08:31.475279    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7794bf-7f688777c8-cbhr5"
Nov 01 15:08:31.475444 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:08:31.475315    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7794bf-7f688777c8-cbhr5" event={"ID":"fef68d07-91c6-499b-9405-889a33bd1375","Type":"ContainerDied","Data":"f21f0349137bf9af67d1b2a4d42e58ff0b46538547b5ed0eabbf78c1b6e15a63"}
Nov 01 15:08:31.475444 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:08:31.475377    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7794bf-7f688777c8-cbhr5" event={"ID":"fef68d07-91c6-499b-9405-889a33bd1375","Type":"ContainerDied","Data":"59bac8fa63a759b32fb1644168ac3cc160472d6f5e3153eb4ed5e0f0bbddcac1"}
Nov 01 15:08:31.475444 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:08:31.475407    4993 scope.go:122] "RemoveContainer" containerID="f21f0349137bf9af67d1b2a4d42e58ff0b46538547b5ed0eabbf78c1b6e15a63"
Nov 01 15:08:31.488495 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:08:31.488465    4993 scope.go:122] "RemoveContainer" containerID="f21f0349137bf9af67d1b2a4d42e58ff0b46538547b5ed0eabbf78c1b6e15a63"
Nov 01 15:08:31.488938 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:08:31.488908    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f21f0349137bf9af67d1b2a4d42e58ff0b46538547b5ed0eabbf78c1b6e15a63\": container with ID starting with f21f0349137bf9af67d1b2a4d42e58ff0b46538547b5ed0eabbf78c1b6e15a63 not found: ID does not exist" containerID="f21f0349137bf9af67d1b2a4d42e58ff0b46538547b5ed0eabbf78c1b6e15a63"
Nov 01 15:08:31.489050 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:08:31.488953    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f21f0349137bf9af67d1b2a4d42e58ff0b46538547b5ed0eabbf78c1b6e15a63"} err="failed to get container status \"f21f0349137bf9af67d1b2a4d42e58ff0b46538547b5ed0eabbf78c1b6e15a63\": rpc error: code = NotFound desc = could not find container \"f21f0349137bf9af67d1b2a4d42e58ff0b46538547b5ed0eabbf78c1b6e15a63\": container with ID starting with f21f0349137bf9af67d1b2a4d42e58ff0b46538547b5ed0eabbf78c1b6e15a63 not found: ID does not exist"
Nov 01 15:08:31.633431 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:08:31.633399    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-7794bf-7f688777c8-cbhr5"]
Nov 01 15:08:31.679642 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:08:31.679604    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-7794bf-7f688777c8-cbhr5"]
Nov 01 15:08:32.796107 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:08:32.796028    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fef68d07-91c6-499b-9405-889a33bd1375" path="/var/lib/kubelet/pods/fef68d07-91c6-499b-9405-889a33bd1375/volumes"
Nov 01 15:08:57.503402 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:08:57.503358    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:09:27.500842 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:09:27.500806    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:09:45.004403 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:09:45.004366    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9"]
Nov 01 15:09:45.006809 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:09:45.004570    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9" podUID="9e600803-b75d-48b9-847b-d236b79cf527" containerName="istio-proxy" containerID="cri-o://e9bb8ae4bd6b6ae513fb8f6b73b3ec284e4067a30a4aef63fb246cd0ad7f1702" gracePeriod=30
Nov 01 15:09:50.280465 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:09:50.280406    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9"
Nov 01 15:09:50.328015 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:09:50.327985    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e600803-b75d-48b9-847b-d236b79cf527-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e600803-b75d-48b9-847b-d236b79cf527-istio-data\") pod \"9e600803-b75d-48b9-847b-d236b79cf527\" (UID: \"9e600803-b75d-48b9-847b-d236b79cf527\") "
Nov 01 15:09:50.328226 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:09:50.328024    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e600803-b75d-48b9-847b-d236b79cf527-istio-token\" (UniqueName: \"kubernetes.io/projected/9e600803-b75d-48b9-847b-d236b79cf527-istio-token\") pod \"9e600803-b75d-48b9-847b-d236b79cf527\" (UID: \"9e600803-b75d-48b9-847b-d236b79cf527\") "
Nov 01 15:09:50.328226 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:09:50.328082    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e600803-b75d-48b9-847b-d236b79cf527-kube-api-access-hzx67\" (UniqueName: \"kubernetes.io/projected/9e600803-b75d-48b9-847b-d236b79cf527-kube-api-access-hzx67\") pod \"9e600803-b75d-48b9-847b-d236b79cf527\" (UID: \"9e600803-b75d-48b9-847b-d236b79cf527\") "
Nov 01 15:09:50.328226 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:09:50.328111    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e600803-b75d-48b9-847b-d236b79cf527-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e600803-b75d-48b9-847b-d236b79cf527-credential-socket\") pod \"9e600803-b75d-48b9-847b-d236b79cf527\" (UID: \"9e600803-b75d-48b9-847b-d236b79cf527\") "
Nov 01 15:09:50.328407 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:09:50.328237    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e600803-b75d-48b9-847b-d236b79cf527-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e600803-b75d-48b9-847b-d236b79cf527-istio-envoy\") pod \"9e600803-b75d-48b9-847b-d236b79cf527\" (UID: \"9e600803-b75d-48b9-847b-d236b79cf527\") "
Nov 01 15:09:50.328407 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:09:50.328299    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e600803-b75d-48b9-847b-d236b79cf527-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e600803-b75d-48b9-847b-d236b79cf527-workload-socket\") pod \"9e600803-b75d-48b9-847b-d236b79cf527\" (UID: \"9e600803-b75d-48b9-847b-d236b79cf527\") "
Nov 01 15:09:50.328407 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:09:50.328321    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e600803-b75d-48b9-847b-d236b79cf527-istio-data" pod "9e600803-b75d-48b9-847b-d236b79cf527" (UID: "9e600803-b75d-48b9-847b-d236b79cf527"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:09:50.328407 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:09:50.328355    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9e600803-b75d-48b9-847b-d236b79cf527-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e600803-b75d-48b9-847b-d236b79cf527-istio-podinfo\") pod \"9e600803-b75d-48b9-847b-d236b79cf527\" (UID: \"9e600803-b75d-48b9-847b-d236b79cf527\") "
Nov 01 15:09:50.328628 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:09:50.328421    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e600803-b75d-48b9-847b-d236b79cf527-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e600803-b75d-48b9-847b-d236b79cf527-workload-certs\") pod \"9e600803-b75d-48b9-847b-d236b79cf527\" (UID: \"9e600803-b75d-48b9-847b-d236b79cf527\") "
Nov 01 15:09:50.328628 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:09:50.328479    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e600803-b75d-48b9-847b-d236b79cf527-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e600803-b75d-48b9-847b-d236b79cf527-istiod-ca-cert\") pod \"9e600803-b75d-48b9-847b-d236b79cf527\" (UID: \"9e600803-b75d-48b9-847b-d236b79cf527\") "
Nov 01 15:09:50.328628 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:09:50.328442    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e600803-b75d-48b9-847b-d236b79cf527-credential-socket" pod "9e600803-b75d-48b9-847b-d236b79cf527" (UID: "9e600803-b75d-48b9-847b-d236b79cf527"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:09:50.328628 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:09:50.328540    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e600803-b75d-48b9-847b-d236b79cf527-workload-socket" pod "9e600803-b75d-48b9-847b-d236b79cf527" (UID: "9e600803-b75d-48b9-847b-d236b79cf527"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:09:50.328842 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:09:50.328694    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e600803-b75d-48b9-847b-d236b79cf527-workload-certs" pod "9e600803-b75d-48b9-847b-d236b79cf527" (UID: "9e600803-b75d-48b9-847b-d236b79cf527"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:09:50.328989 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:09:50.328964    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e600803-b75d-48b9-847b-d236b79cf527-istiod-ca-cert" pod "9e600803-b75d-48b9-847b-d236b79cf527" (UID: "9e600803-b75d-48b9-847b-d236b79cf527"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:09:50.329075 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:09:50.328966    4993 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e600803-b75d-48b9-847b-d236b79cf527-workload-socket\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:09:50.329075 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:09:50.329016    4993 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e600803-b75d-48b9-847b-d236b79cf527-workload-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:09:50.329075 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:09:50.329036    4993 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e600803-b75d-48b9-847b-d236b79cf527-istio-data\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:09:50.329075 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:09:50.329054    4993 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e600803-b75d-48b9-847b-d236b79cf527-credential-socket\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:09:50.330476 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:09:50.330447    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e600803-b75d-48b9-847b-d236b79cf527-kube-api-access-hzx67" pod "9e600803-b75d-48b9-847b-d236b79cf527" (UID: "9e600803-b75d-48b9-847b-d236b79cf527"). InnerVolumeSpecName "kube-api-access-hzx67". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:09:50.330588 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:09:50.330514    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e600803-b75d-48b9-847b-d236b79cf527-istio-podinfo" pod "9e600803-b75d-48b9-847b-d236b79cf527" (UID: "9e600803-b75d-48b9-847b-d236b79cf527"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 15:09:50.330588 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:09:50.330515    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e600803-b75d-48b9-847b-d236b79cf527-istio-token" pod "9e600803-b75d-48b9-847b-d236b79cf527" (UID: "9e600803-b75d-48b9-847b-d236b79cf527"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:09:50.330588 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:09:50.330531    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e600803-b75d-48b9-847b-d236b79cf527-istio-envoy" pod "9e600803-b75d-48b9-847b-d236b79cf527" (UID: "9e600803-b75d-48b9-847b-d236b79cf527"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:09:50.429912 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:09:50.429878    4993 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e600803-b75d-48b9-847b-d236b79cf527-istio-envoy\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:09:50.429912 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:09:50.429910    4993 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e600803-b75d-48b9-847b-d236b79cf527-istio-podinfo\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:09:50.429912 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:09:50.429920    4993 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e600803-b75d-48b9-847b-d236b79cf527-istiod-ca-cert\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:09:50.430204 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:09:50.429929    4993 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e600803-b75d-48b9-847b-d236b79cf527-istio-token\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:09:50.430204 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:09:50.429939    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzx67\" (UniqueName: \"kubernetes.io/projected/9e600803-b75d-48b9-847b-d236b79cf527-kube-api-access-hzx67\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:09:50.976340 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:09:50.976301    4993 generic.go:356] "Generic (PLEG): container finished" podID="9e600803-b75d-48b9-847b-d236b79cf527" containerID="e9bb8ae4bd6b6ae513fb8f6b73b3ec284e4067a30a4aef63fb246cd0ad7f1702" exitCode=0
Nov 01 15:09:50.976512 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:09:50.976348    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9"
Nov 01 15:09:50.976512 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:09:50.976363    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9" event={"ID":"9e600803-b75d-48b9-847b-d236b79cf527","Type":"ContainerDied","Data":"e9bb8ae4bd6b6ae513fb8f6b73b3ec284e4067a30a4aef63fb246cd0ad7f1702"}
Nov 01 15:09:50.976512 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:09:50.976411    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9" event={"ID":"9e600803-b75d-48b9-847b-d236b79cf527","Type":"ContainerDied","Data":"1faf051a50f213d65a87df54bef1764c2d61188197b9cc81b4ad9541c9d6ec60"}
Nov 01 15:09:50.976512 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:09:50.976435    4993 scope.go:122] "RemoveContainer" containerID="e9bb8ae4bd6b6ae513fb8f6b73b3ec284e4067a30a4aef63fb246cd0ad7f1702"
Nov 01 15:09:50.987474 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:09:50.987449    4993 scope.go:122] "RemoveContainer" containerID="e9bb8ae4bd6b6ae513fb8f6b73b3ec284e4067a30a4aef63fb246cd0ad7f1702"
Nov 01 15:09:50.987807 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:09:50.987782    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9bb8ae4bd6b6ae513fb8f6b73b3ec284e4067a30a4aef63fb246cd0ad7f1702\": container with ID starting with e9bb8ae4bd6b6ae513fb8f6b73b3ec284e4067a30a4aef63fb246cd0ad7f1702 not found: ID does not exist" containerID="e9bb8ae4bd6b6ae513fb8f6b73b3ec284e4067a30a4aef63fb246cd0ad7f1702"
Nov 01 15:09:50.987879 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:09:50.987824    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9bb8ae4bd6b6ae513fb8f6b73b3ec284e4067a30a4aef63fb246cd0ad7f1702"} err="failed to get container status \"e9bb8ae4bd6b6ae513fb8f6b73b3ec284e4067a30a4aef63fb246cd0ad7f1702\": rpc error: code = NotFound desc = could not find container \"e9bb8ae4bd6b6ae513fb8f6b73b3ec284e4067a30a4aef63fb246cd0ad7f1702\": container with ID starting with e9bb8ae4bd6b6ae513fb8f6b73b3ec284e4067a30a4aef63fb246cd0ad7f1702 not found: ID does not exist"
Nov 01 15:09:51.133734 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:09:51.133640    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9"]
Nov 01 15:09:51.180866 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:09:51.180823    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-xwqs9"]
Nov 01 15:09:52.796595 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:09:52.796557    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e600803-b75d-48b9-847b-d236b79cf527" path="/var/lib/kubelet/pods/9e600803-b75d-48b9-847b-d236b79cf527/volumes"
Nov 01 15:09:53.167545 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:09:53.167454    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-4e07cb/auth-test-model-4e07cb-kserve-66869df457-8z2kf"]
Nov 01 15:09:57.502427 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:09:57.502388    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:10:08.188865 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:10:08.188759    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-4e07cb/auth-test-model-4e07cb-kserve-66869df457-8z2kf" podUID="ec23a9b1-8adf-4be6-88f9-298452111158" containerName="main" containerID="cri-o://cb1bbc705193b7fbc92258bbb4f7d6564a2df3107f17740170b3faf36c6bb9d9" gracePeriod=45
Nov 01 15:10:08.337707 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:10:08.337661    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-4e07cb/auth-test-model-4e07cb-kserve-66869df457-8z2kf" podUID="ec23a9b1-8adf-4be6-88f9-298452111158" containerName="main" probeResult="failure" output="Get \"https://10.132.0.121:8000/ready\": dial tcp 10.132.0.121:8000: connect: connection refused"
Nov 01 15:10:08.474226 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:10:08.474195    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-4e07cb/auth-test-model-4e07cb-kserve-66869df457-8z2kf"
Nov 01 15:10:08.542361 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:10:08.542328    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec23a9b1-8adf-4be6-88f9-298452111158-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec23a9b1-8adf-4be6-88f9-298452111158-tmp-dir\") pod \"ec23a9b1-8adf-4be6-88f9-298452111158\" (UID: \"ec23a9b1-8adf-4be6-88f9-298452111158\") "
Nov 01 15:10:08.542539 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:10:08.542419    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec23a9b1-8adf-4be6-88f9-298452111158-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec23a9b1-8adf-4be6-88f9-298452111158-dshm\") pod \"ec23a9b1-8adf-4be6-88f9-298452111158\" (UID: \"ec23a9b1-8adf-4be6-88f9-298452111158\") "
Nov 01 15:10:08.542539 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:10:08.542457    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec23a9b1-8adf-4be6-88f9-298452111158-home\" (UniqueName: \"kubernetes.io/empty-dir/ec23a9b1-8adf-4be6-88f9-298452111158-home\") pod \"ec23a9b1-8adf-4be6-88f9-298452111158\" (UID: \"ec23a9b1-8adf-4be6-88f9-298452111158\") "
Nov 01 15:10:08.542539 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:10:08.542498    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec23a9b1-8adf-4be6-88f9-298452111158-tls-certs\" (UniqueName: \"kubernetes.io/secret/ec23a9b1-8adf-4be6-88f9-298452111158-tls-certs\") pod \"ec23a9b1-8adf-4be6-88f9-298452111158\" (UID: \"ec23a9b1-8adf-4be6-88f9-298452111158\") "
Nov 01 15:10:08.542716 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:10:08.542575    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec23a9b1-8adf-4be6-88f9-298452111158-kube-api-access-6fw9s\" (UniqueName: \"kubernetes.io/projected/ec23a9b1-8adf-4be6-88f9-298452111158-kube-api-access-6fw9s\") pod \"ec23a9b1-8adf-4be6-88f9-298452111158\" (UID: \"ec23a9b1-8adf-4be6-88f9-298452111158\") "
Nov 01 15:10:08.542716 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:10:08.542610    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec23a9b1-8adf-4be6-88f9-298452111158-tmp-dir" pod "ec23a9b1-8adf-4be6-88f9-298452111158" (UID: "ec23a9b1-8adf-4be6-88f9-298452111158"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:10:08.542847 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:10:08.542739    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec23a9b1-8adf-4be6-88f9-298452111158-home" pod "ec23a9b1-8adf-4be6-88f9-298452111158" (UID: "ec23a9b1-8adf-4be6-88f9-298452111158"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:10:08.542847 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:10:08.542749    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec23a9b1-8adf-4be6-88f9-298452111158-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec23a9b1-8adf-4be6-88f9-298452111158-model-cache\") pod \"ec23a9b1-8adf-4be6-88f9-298452111158\" (UID: \"ec23a9b1-8adf-4be6-88f9-298452111158\") "
Nov 01 15:10:08.543010 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:10:08.542983    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec23a9b1-8adf-4be6-88f9-298452111158-model-cache" pod "ec23a9b1-8adf-4be6-88f9-298452111158" (UID: "ec23a9b1-8adf-4be6-88f9-298452111158"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:10:08.543216 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:10:08.543192    4993 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec23a9b1-8adf-4be6-88f9-298452111158-tmp-dir\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:10:08.543216 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:10:08.543213    4993 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec23a9b1-8adf-4be6-88f9-298452111158-home\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:10:08.543388 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:10:08.543223    4993 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec23a9b1-8adf-4be6-88f9-298452111158-model-cache\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:10:08.545287 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:10:08.545245    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec23a9b1-8adf-4be6-88f9-298452111158-tls-certs" pod "ec23a9b1-8adf-4be6-88f9-298452111158" (UID: "ec23a9b1-8adf-4be6-88f9-298452111158"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:10:08.545397 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:10:08.545342    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec23a9b1-8adf-4be6-88f9-298452111158-dshm" pod "ec23a9b1-8adf-4be6-88f9-298452111158" (UID: "ec23a9b1-8adf-4be6-88f9-298452111158"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:10:08.545397 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:10:08.545376    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec23a9b1-8adf-4be6-88f9-298452111158-kube-api-access-6fw9s" pod "ec23a9b1-8adf-4be6-88f9-298452111158" (UID: "ec23a9b1-8adf-4be6-88f9-298452111158"). InnerVolumeSpecName "kube-api-access-6fw9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:10:08.644149 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:10:08.644108    4993 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec23a9b1-8adf-4be6-88f9-298452111158-dshm\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:10:08.644149 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:10:08.644147    4993 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec23a9b1-8adf-4be6-88f9-298452111158-tls-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:10:08.644403 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:10:08.644177    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fw9s\" (UniqueName: \"kubernetes.io/projected/ec23a9b1-8adf-4be6-88f9-298452111158-kube-api-access-6fw9s\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:10:09.096440 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:10:09.096344    4993 generic.go:356] "Generic (PLEG): container finished" podID="ec23a9b1-8adf-4be6-88f9-298452111158" containerID="cb1bbc705193b7fbc92258bbb4f7d6564a2df3107f17740170b3faf36c6bb9d9" exitCode=0
Nov 01 15:10:09.096440 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:10:09.096393    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-4e07cb/auth-test-model-4e07cb-kserve-66869df457-8z2kf"
Nov 01 15:10:09.096440 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:10:09.096396    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-4e07cb/auth-test-model-4e07cb-kserve-66869df457-8z2kf" event={"ID":"ec23a9b1-8adf-4be6-88f9-298452111158","Type":"ContainerDied","Data":"cb1bbc705193b7fbc92258bbb4f7d6564a2df3107f17740170b3faf36c6bb9d9"}
Nov 01 15:10:09.096440 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:10:09.096433    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-4e07cb/auth-test-model-4e07cb-kserve-66869df457-8z2kf" event={"ID":"ec23a9b1-8adf-4be6-88f9-298452111158","Type":"ContainerDied","Data":"f83fb4bcf316cb0a2c137b6b4812b93a1266bc56f37353c6772d8010f3f2e782"}
Nov 01 15:10:09.096733 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:10:09.096455    4993 scope.go:122] "RemoveContainer" containerID="cb1bbc705193b7fbc92258bbb4f7d6564a2df3107f17740170b3faf36c6bb9d9"
Nov 01 15:10:09.106003 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:10:09.105981    4993 scope.go:122] "RemoveContainer" containerID="cb1bbc705193b7fbc92258bbb4f7d6564a2df3107f17740170b3faf36c6bb9d9"
Nov 01 15:10:09.106294 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:10:09.106270    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb1bbc705193b7fbc92258bbb4f7d6564a2df3107f17740170b3faf36c6bb9d9\": container with ID starting with cb1bbc705193b7fbc92258bbb4f7d6564a2df3107f17740170b3faf36c6bb9d9 not found: ID does not exist" containerID="cb1bbc705193b7fbc92258bbb4f7d6564a2df3107f17740170b3faf36c6bb9d9"
Nov 01 15:10:09.106383 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:10:09.106301    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb1bbc705193b7fbc92258bbb4f7d6564a2df3107f17740170b3faf36c6bb9d9"} err="failed to get container status \"cb1bbc705193b7fbc92258bbb4f7d6564a2df3107f17740170b3faf36c6bb9d9\": rpc error: code = NotFound desc = could not find container \"cb1bbc705193b7fbc92258bbb4f7d6564a2df3107f17740170b3faf36c6bb9d9\": container with ID starting with cb1bbc705193b7fbc92258bbb4f7d6564a2df3107f17740170b3faf36c6bb9d9 not found: ID does not exist"
Nov 01 15:10:09.258908 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:10:09.258877    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-4e07cb/auth-test-model-4e07cb-kserve-66869df457-8z2kf"]
Nov 01 15:10:09.309049 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:10:09.309010    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-4e07cb/auth-test-model-4e07cb-kserve-66869df457-8z2kf"]
Nov 01 15:10:10.807001 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:10:10.806968    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec23a9b1-8adf-4be6-88f9-298452111158" path="/var/lib/kubelet/pods/ec23a9b1-8adf-4be6-88f9-298452111158/volumes"
Nov 01 15:10:27.499787 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:10:27.499752    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:10:49.030858 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:10:49.030816    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-d06633/auth-test-model-d06633-kserve-685c8fd8f-cwl5h"]
Nov 01 15:10:57.505073 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:10:57.505032    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:11:04.051082 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:11:04.051025    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-d06633/auth-test-model-d06633-kserve-685c8fd8f-cwl5h" podUID="28b45c9e-a664-452c-8972-5d68d0f09a48" containerName="main" containerID="cri-o://b30fd2482643d3041fc30d56ed8251f83a8c842289003d4476553c6cd766ac8e" gracePeriod=45
Nov 01 15:11:04.316439 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:11:04.316369    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-d06633/auth-test-model-d06633-kserve-685c8fd8f-cwl5h"
Nov 01 15:11:04.408931 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:11:04.408888    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28b45c9e-a664-452c-8972-5d68d0f09a48-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28b45c9e-a664-452c-8972-5d68d0f09a48-tmp-dir\") pod \"28b45c9e-a664-452c-8972-5d68d0f09a48\" (UID: \"28b45c9e-a664-452c-8972-5d68d0f09a48\") "
Nov 01 15:11:04.409109 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:11:04.409011    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28b45c9e-a664-452c-8972-5d68d0f09a48-kube-api-access-kzt7g\" (UniqueName: \"kubernetes.io/projected/28b45c9e-a664-452c-8972-5d68d0f09a48-kube-api-access-kzt7g\") pod \"28b45c9e-a664-452c-8972-5d68d0f09a48\" (UID: \"28b45c9e-a664-452c-8972-5d68d0f09a48\") "
Nov 01 15:11:04.409109 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:11:04.409104    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28b45c9e-a664-452c-8972-5d68d0f09a48-dshm\" (UniqueName: \"kubernetes.io/empty-dir/28b45c9e-a664-452c-8972-5d68d0f09a48-dshm\") pod \"28b45c9e-a664-452c-8972-5d68d0f09a48\" (UID: \"28b45c9e-a664-452c-8972-5d68d0f09a48\") "
Nov 01 15:11:04.409236 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:11:04.409211    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28b45c9e-a664-452c-8972-5d68d0f09a48-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28b45c9e-a664-452c-8972-5d68d0f09a48-model-cache\") pod \"28b45c9e-a664-452c-8972-5d68d0f09a48\" (UID: \"28b45c9e-a664-452c-8972-5d68d0f09a48\") "
Nov 01 15:11:04.409287 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:11:04.409250    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28b45c9e-a664-452c-8972-5d68d0f09a48-tls-certs\" (UniqueName: \"kubernetes.io/secret/28b45c9e-a664-452c-8972-5d68d0f09a48-tls-certs\") pod \"28b45c9e-a664-452c-8972-5d68d0f09a48\" (UID: \"28b45c9e-a664-452c-8972-5d68d0f09a48\") "
Nov 01 15:11:04.409287 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:11:04.409249    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28b45c9e-a664-452c-8972-5d68d0f09a48-tmp-dir" pod "28b45c9e-a664-452c-8972-5d68d0f09a48" (UID: "28b45c9e-a664-452c-8972-5d68d0f09a48"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:11:04.409287 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:11:04.409279    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28b45c9e-a664-452c-8972-5d68d0f09a48-home\" (UniqueName: \"kubernetes.io/empty-dir/28b45c9e-a664-452c-8972-5d68d0f09a48-home\") pod \"28b45c9e-a664-452c-8972-5d68d0f09a48\" (UID: \"28b45c9e-a664-452c-8972-5d68d0f09a48\") "
Nov 01 15:11:04.409584 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:11:04.409547    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28b45c9e-a664-452c-8972-5d68d0f09a48-model-cache" pod "28b45c9e-a664-452c-8972-5d68d0f09a48" (UID: "28b45c9e-a664-452c-8972-5d68d0f09a48"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:11:04.409584 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:11:04.409570    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28b45c9e-a664-452c-8972-5d68d0f09a48-home" pod "28b45c9e-a664-452c-8972-5d68d0f09a48" (UID: "28b45c9e-a664-452c-8972-5d68d0f09a48"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:11:04.409742 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:11:04.409631    4993 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28b45c9e-a664-452c-8972-5d68d0f09a48-tmp-dir\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:11:04.411447 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:11:04.411409    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28b45c9e-a664-452c-8972-5d68d0f09a48-tls-certs" pod "28b45c9e-a664-452c-8972-5d68d0f09a48" (UID: "28b45c9e-a664-452c-8972-5d68d0f09a48"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:11:04.411565 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:11:04.411427    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28b45c9e-a664-452c-8972-5d68d0f09a48-dshm" pod "28b45c9e-a664-452c-8972-5d68d0f09a48" (UID: "28b45c9e-a664-452c-8972-5d68d0f09a48"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:11:04.411565 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:11:04.411522    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28b45c9e-a664-452c-8972-5d68d0f09a48-kube-api-access-kzt7g" pod "28b45c9e-a664-452c-8972-5d68d0f09a48" (UID: "28b45c9e-a664-452c-8972-5d68d0f09a48"). InnerVolumeSpecName "kube-api-access-kzt7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:11:04.457436 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:11:04.457400    4993 generic.go:356] "Generic (PLEG): container finished" podID="28b45c9e-a664-452c-8972-5d68d0f09a48" containerID="b30fd2482643d3041fc30d56ed8251f83a8c842289003d4476553c6cd766ac8e" exitCode=0
Nov 01 15:11:04.457600 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:11:04.457477    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-d06633/auth-test-model-d06633-kserve-685c8fd8f-cwl5h"
Nov 01 15:11:04.457600 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:11:04.457495    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-d06633/auth-test-model-d06633-kserve-685c8fd8f-cwl5h" event={"ID":"28b45c9e-a664-452c-8972-5d68d0f09a48","Type":"ContainerDied","Data":"b30fd2482643d3041fc30d56ed8251f83a8c842289003d4476553c6cd766ac8e"}
Nov 01 15:11:04.457600 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:11:04.457554    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-d06633/auth-test-model-d06633-kserve-685c8fd8f-cwl5h" event={"ID":"28b45c9e-a664-452c-8972-5d68d0f09a48","Type":"ContainerDied","Data":"2dd3f5704ce1d0ed3bc5ace1e191bdf2cb58838cffa449efbb6ba58e32c9241f"}
Nov 01 15:11:04.457600 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:11:04.457578    4993 scope.go:122] "RemoveContainer" containerID="b30fd2482643d3041fc30d56ed8251f83a8c842289003d4476553c6cd766ac8e"
Nov 01 15:11:04.469631 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:11:04.469592    4993 scope.go:122] "RemoveContainer" containerID="b30fd2482643d3041fc30d56ed8251f83a8c842289003d4476553c6cd766ac8e"
Nov 01 15:11:04.469909 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:11:04.469882    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b30fd2482643d3041fc30d56ed8251f83a8c842289003d4476553c6cd766ac8e\": container with ID starting with b30fd2482643d3041fc30d56ed8251f83a8c842289003d4476553c6cd766ac8e not found: ID does not exist" containerID="b30fd2482643d3041fc30d56ed8251f83a8c842289003d4476553c6cd766ac8e"
Nov 01 15:11:04.469972 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:11:04.469920    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b30fd2482643d3041fc30d56ed8251f83a8c842289003d4476553c6cd766ac8e"} err="failed to get container status \"b30fd2482643d3041fc30d56ed8251f83a8c842289003d4476553c6cd766ac8e\": rpc error: code = NotFound desc = could not find container \"b30fd2482643d3041fc30d56ed8251f83a8c842289003d4476553c6cd766ac8e\": container with ID starting with b30fd2482643d3041fc30d56ed8251f83a8c842289003d4476553c6cd766ac8e not found: ID does not exist"
Nov 01 15:11:04.510585 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:11:04.510548    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzt7g\" (UniqueName: \"kubernetes.io/projected/28b45c9e-a664-452c-8972-5d68d0f09a48-kube-api-access-kzt7g\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:11:04.510585 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:11:04.510585    4993 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28b45c9e-a664-452c-8972-5d68d0f09a48-dshm\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:11:04.510585 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:11:04.510596    4993 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28b45c9e-a664-452c-8972-5d68d0f09a48-model-cache\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:11:04.510788 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:11:04.510613    4993 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28b45c9e-a664-452c-8972-5d68d0f09a48-tls-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:11:04.510788 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:11:04.510623    4993 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28b45c9e-a664-452c-8972-5d68d0f09a48-home\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:11:04.610970 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:11:04.610875    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-d06633/auth-test-model-d06633-kserve-685c8fd8f-cwl5h"]
Nov 01 15:11:04.614869 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:11:04.614831    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-d06633/auth-test-model-d06633-kserve-685c8fd8f-cwl5h"]
Nov 01 15:11:04.795394 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:11:04.795362    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28b45c9e-a664-452c-8972-5d68d0f09a48" path="/var/lib/kubelet/pods/28b45c9e-a664-452c-8972-5d68d0f09a48/volumes"
Nov 01 15:11:27.501082 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:11:27.501036    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:11:37.814530 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:11:37.814452    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea1620-559674b8f9-64pfg"]
Nov 01 15:11:37.815196 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:11:37.815162    4993 cpu_manager.go:414] "containerMap: removing container" podUID="28b45c9e-a664-452c-8972-5d68d0f09a48" containerName="main"
Nov 01 15:11:37.815250 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:11:37.815197    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28b45c9e-a664-452c-8972-5d68d0f09a48" containerName="main"
Nov 01 15:11:37.815250 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:11:37.815226    4993 cpu_manager.go:414] "containerMap: removing container" podUID="9e600803-b75d-48b9-847b-d236b79cf527" containerName="istio-proxy"
Nov 01 15:11:37.815250 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:11:37.815234    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e600803-b75d-48b9-847b-d236b79cf527" containerName="istio-proxy"
Nov 01 15:11:37.815250 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:11:37.815245    4993 cpu_manager.go:414] "containerMap: removing container" podUID="ec23a9b1-8adf-4be6-88f9-298452111158" containerName="main"
Nov 01 15:11:37.815380 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:11:37.815253    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec23a9b1-8adf-4be6-88f9-298452111158" containerName="main"
Nov 01 15:11:37.815380 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:11:37.815298    4993 cpu_manager.go:414] "containerMap: removing container" podUID="fef68d07-91c6-499b-9405-889a33bd1375" containerName="maas-api"
Nov 01 15:11:37.815380 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:11:37.815306    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fef68d07-91c6-499b-9405-889a33bd1375" containerName="maas-api"
Nov 01 15:11:37.815469 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:11:37.815424    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="fef68d07-91c6-499b-9405-889a33bd1375" containerName="maas-api"
Nov 01 15:11:37.815469 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:11:37.815437    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="28b45c9e-a664-452c-8972-5d68d0f09a48" containerName="main"
Nov 01 15:11:37.815469 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:11:37.815450    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e600803-b75d-48b9-847b-d236b79cf527" containerName="istio-proxy"
Nov 01 15:11:37.815469 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:11:37.815460    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec23a9b1-8adf-4be6-88f9-298452111158" containerName="main"
Nov 01 15:11:37.819378 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:11:37.819357    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea1620-559674b8f9-64pfg"
Nov 01 15:11:37.897659 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:11:37.897600    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a1719a8-89fb-44bb-be61-d8c9423e32d3-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-ea1620-559674b8f9-64pfg\" (UID: \"8a1719a8-89fb-44bb-be61-d8c9423e32d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea1620-559674b8f9-64pfg"
Nov 01 15:11:37.897859 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:11:37.897757    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a1719a8-89fb-44bb-be61-d8c9423e32d3-maas-api-tls\") pod \"maas-api-e2e-extmod-a-ea1620-559674b8f9-64pfg\" (UID: \"8a1719a8-89fb-44bb-be61-d8c9423e32d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea1620-559674b8f9-64pfg"
Nov 01 15:11:37.897859 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:11:37.897852    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a1719a8-89fb-44bb-be61-d8c9423e32d3-metrics-certs\") pod \"maas-api-e2e-extmod-a-ea1620-559674b8f9-64pfg\" (UID: \"8a1719a8-89fb-44bb-be61-d8c9423e32d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea1620-559674b8f9-64pfg"
Nov 01 15:11:37.898002 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:11:37.897981    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbxnd\" (UniqueName: \"kubernetes.io/projected/8a1719a8-89fb-44bb-be61-d8c9423e32d3-kube-api-access-lbxnd\") pod \"maas-api-e2e-extmod-a-ea1620-559674b8f9-64pfg\" (UID: \"8a1719a8-89fb-44bb-be61-d8c9423e32d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea1620-559674b8f9-64pfg"
Nov 01 15:11:37.914864 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:11:37.914835    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-ea1620-serving-cert\""
Nov 01 15:11:37.917590 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:11:37.917569    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea1620-559674b8f9-64pfg"]
Nov 01 15:11:37.999347 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:11:37.999300    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a1719a8-89fb-44bb-be61-d8c9423e32d3-metrics-certs\") pod \"maas-api-e2e-extmod-a-ea1620-559674b8f9-64pfg\" (UID: \"8a1719a8-89fb-44bb-be61-d8c9423e32d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea1620-559674b8f9-64pfg"
Nov 01 15:11:37.999347 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:11:37.999347    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbxnd\" (UniqueName: \"kubernetes.io/projected/8a1719a8-89fb-44bb-be61-d8c9423e32d3-kube-api-access-lbxnd\") pod \"maas-api-e2e-extmod-a-ea1620-559674b8f9-64pfg\" (UID: \"8a1719a8-89fb-44bb-be61-d8c9423e32d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea1620-559674b8f9-64pfg"
Nov 01 15:11:37.999606 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:11:37.999483    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a1719a8-89fb-44bb-be61-d8c9423e32d3-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-ea1620-559674b8f9-64pfg\" (UID: \"8a1719a8-89fb-44bb-be61-d8c9423e32d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea1620-559674b8f9-64pfg"
Nov 01 15:11:37.999606 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:11:37.999539    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a1719a8-89fb-44bb-be61-d8c9423e32d3-maas-api-tls\") pod \"maas-api-e2e-extmod-a-ea1620-559674b8f9-64pfg\" (UID: \"8a1719a8-89fb-44bb-be61-d8c9423e32d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea1620-559674b8f9-64pfg"
Nov 01 15:11:37.999743 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:11:37.999684    4993 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea1620-serving-cert: secret "maas-api-e2e-extmod-a-ea1620-serving-cert" not found
Nov 01 15:11:37.999816 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:11:37.999752    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a1719a8-89fb-44bb-be61-d8c9423e32d3-maas-api-tls podName:8a1719a8-89fb-44bb-be61-d8c9423e32d3 nodeName:}" failed. No retries permitted until 2026-11-01 15:11:38.499731471 +0000 UTC m=+3285.829594203 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8a1719a8-89fb-44bb-be61-d8c9423e32d3-maas-api-tls") pod "maas-api-e2e-extmod-a-ea1620-559674b8f9-64pfg" (UID: "8a1719a8-89fb-44bb-be61-d8c9423e32d3") : secret "maas-api-e2e-extmod-a-ea1620-serving-cert" not found
Nov 01 15:11:38.000484 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:11:38.000443    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a1719a8-89fb-44bb-be61-d8c9423e32d3-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-ea1620-559674b8f9-64pfg\" (UID: \"8a1719a8-89fb-44bb-be61-d8c9423e32d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea1620-559674b8f9-64pfg"
Nov 01 15:11:38.002309 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:11:38.002285    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a1719a8-89fb-44bb-be61-d8c9423e32d3-metrics-certs\") pod \"maas-api-e2e-extmod-a-ea1620-559674b8f9-64pfg\" (UID: \"8a1719a8-89fb-44bb-be61-d8c9423e32d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea1620-559674b8f9-64pfg"
Nov 01 15:11:38.059036 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:11:38.059002    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbxnd\" (UniqueName: \"kubernetes.io/projected/8a1719a8-89fb-44bb-be61-d8c9423e32d3-kube-api-access-lbxnd\") pod \"maas-api-e2e-extmod-a-ea1620-559674b8f9-64pfg\" (UID: \"8a1719a8-89fb-44bb-be61-d8c9423e32d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea1620-559674b8f9-64pfg"
Nov 01 15:11:38.505178 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:11:38.505129    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a1719a8-89fb-44bb-be61-d8c9423e32d3-maas-api-tls\") pod \"maas-api-e2e-extmod-a-ea1620-559674b8f9-64pfg\" (UID: \"8a1719a8-89fb-44bb-be61-d8c9423e32d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea1620-559674b8f9-64pfg"
Nov 01 15:11:38.507725 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:11:38.507690    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a1719a8-89fb-44bb-be61-d8c9423e32d3-maas-api-tls\") pod \"maas-api-e2e-extmod-a-ea1620-559674b8f9-64pfg\" (UID: \"8a1719a8-89fb-44bb-be61-d8c9423e32d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea1620-559674b8f9-64pfg"
Nov 01 15:11:38.732859 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:11:38.732823    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea1620-559674b8f9-64pfg"
Nov 01 15:11:39.113560 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:11:39.113469    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea1620-559674b8f9-64pfg"]
Nov 01 15:11:39.119846 ip-10-0-134-37 kubenswrapper[4993]: W1101 15:11:39.119817    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a1719a8_89fb_44bb_be61_d8c9423e32d3.slice/crio-f8fdd6aa366a877b03824f2fe4c24746a536f535629b1c8c2bb4d557a96ad065 WatchSource:0}: Error finding container f8fdd6aa366a877b03824f2fe4c24746a536f535629b1c8c2bb4d557a96ad065: Status 404 returned error can't find the container with id f8fdd6aa366a877b03824f2fe4c24746a536f535629b1c8c2bb4d557a96ad065
Nov 01 15:11:39.121653 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:11:39.121629    4993 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 15:11:39.694409 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:11:39.694365    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea1620-559674b8f9-64pfg" event={"ID":"8a1719a8-89fb-44bb-be61-d8c9423e32d3","Type":"ContainerStarted","Data":"f8fdd6aa366a877b03824f2fe4c24746a536f535629b1c8c2bb4d557a96ad065"}
Nov 01 15:11:40.704071 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:11:40.704022    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea1620-559674b8f9-64pfg" event={"ID":"8a1719a8-89fb-44bb-be61-d8c9423e32d3","Type":"ContainerStarted","Data":"3fd48622d9c6aee573733a36edf75f8e60895b65870bd153df9201456340fc19"}
Nov 01 15:11:40.704510 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:11:40.704123    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea1620-559674b8f9-64pfg"
Nov 01 15:11:40.813896 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:11:40.813840    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea1620-559674b8f9-64pfg" podStartSLOduration=2.970515732 podStartE2EDuration="3.813825866s" totalImagesPullingTime="843.310134ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:11:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:11:40.811502206 +0000 UTC m=+3288.141364944" watchObservedRunningTime="2026-11-01 15:11:40.813825866 +0000 UTC m=+3288.143688625"
Nov 01 15:11:46.720290 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:11:46.720255    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea1620-559674b8f9-64pfg"
Nov 01 15:11:53.266917 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:11:53.266877    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xmwtp_bb59051d-c2df-4362-a157-00f14f915e79/console-operator/3.log"
Nov 01 15:11:53.268855 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:11:53.268830    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xmwtp_bb59051d-c2df-4362-a157-00f14f915e79/console-operator/3.log"
Nov 01 15:11:57.501018 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:11:57.500969    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:12:27.500207 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:12:27.500141    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:12:45.759366 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:12:45.759331    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-926972-547bc47bff-cg427"]
Nov 01 15:12:45.763372 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:12:45.763351    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-926972-547bc47bff-cg427"
Nov 01 15:12:45.861067 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:12:45.861030    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-926972-serving-cert\""
Nov 01 15:12:45.863928 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:12:45.863900    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-926972-547bc47bff-cg427"]
Nov 01 15:12:45.877185 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:12:45.875399    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27mz5\" (UniqueName: \"kubernetes.io/projected/7240e446-e298-491a-b31f-9f8ded98d74f-kube-api-access-27mz5\") pod \"maas-api-e2e-extmod-a-926972-547bc47bff-cg427\" (UID: \"7240e446-e298-491a-b31f-9f8ded98d74f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-926972-547bc47bff-cg427"
Nov 01 15:12:45.877185 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:12:45.875458    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7240e446-e298-491a-b31f-9f8ded98d74f-metrics-certs\") pod \"maas-api-e2e-extmod-a-926972-547bc47bff-cg427\" (UID: \"7240e446-e298-491a-b31f-9f8ded98d74f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-926972-547bc47bff-cg427"
Nov 01 15:12:45.877185 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:12:45.875974    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7240e446-e298-491a-b31f-9f8ded98d74f-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-926972-547bc47bff-cg427\" (UID: \"7240e446-e298-491a-b31f-9f8ded98d74f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-926972-547bc47bff-cg427"
Nov 01 15:12:45.877185 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:12:45.876153    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7240e446-e298-491a-b31f-9f8ded98d74f-maas-api-tls\") pod \"maas-api-e2e-extmod-a-926972-547bc47bff-cg427\" (UID: \"7240e446-e298-491a-b31f-9f8ded98d74f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-926972-547bc47bff-cg427"
Nov 01 15:12:45.977940 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:12:45.977898    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27mz5\" (UniqueName: \"kubernetes.io/projected/7240e446-e298-491a-b31f-9f8ded98d74f-kube-api-access-27mz5\") pod \"maas-api-e2e-extmod-a-926972-547bc47bff-cg427\" (UID: \"7240e446-e298-491a-b31f-9f8ded98d74f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-926972-547bc47bff-cg427"
Nov 01 15:12:45.978138 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:12:45.977957    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7240e446-e298-491a-b31f-9f8ded98d74f-metrics-certs\") pod \"maas-api-e2e-extmod-a-926972-547bc47bff-cg427\" (UID: \"7240e446-e298-491a-b31f-9f8ded98d74f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-926972-547bc47bff-cg427"
Nov 01 15:12:45.978138 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:12:45.978026    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7240e446-e298-491a-b31f-9f8ded98d74f-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-926972-547bc47bff-cg427\" (UID: \"7240e446-e298-491a-b31f-9f8ded98d74f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-926972-547bc47bff-cg427"
Nov 01 15:12:45.978138 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:12:45.978078    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7240e446-e298-491a-b31f-9f8ded98d74f-maas-api-tls\") pod \"maas-api-e2e-extmod-a-926972-547bc47bff-cg427\" (UID: \"7240e446-e298-491a-b31f-9f8ded98d74f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-926972-547bc47bff-cg427"
Nov 01 15:12:45.978328 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:12:45.978247    4993 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-926972-serving-cert: secret "maas-api-e2e-extmod-a-926972-serving-cert" not found
Nov 01 15:12:45.978328 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:12:45.978318    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7240e446-e298-491a-b31f-9f8ded98d74f-maas-api-tls podName:7240e446-e298-491a-b31f-9f8ded98d74f nodeName:}" failed. No retries permitted until 2026-11-01 15:12:46.478301232 +0000 UTC m=+3353.808163948 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7240e446-e298-491a-b31f-9f8ded98d74f-maas-api-tls") pod "maas-api-e2e-extmod-a-926972-547bc47bff-cg427" (UID: "7240e446-e298-491a-b31f-9f8ded98d74f") : secret "maas-api-e2e-extmod-a-926972-serving-cert" not found
Nov 01 15:12:45.979122 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:12:45.979100    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7240e446-e298-491a-b31f-9f8ded98d74f-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-926972-547bc47bff-cg427\" (UID: \"7240e446-e298-491a-b31f-9f8ded98d74f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-926972-547bc47bff-cg427"
Nov 01 15:12:45.980951 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:12:45.980921    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7240e446-e298-491a-b31f-9f8ded98d74f-metrics-certs\") pod \"maas-api-e2e-extmod-a-926972-547bc47bff-cg427\" (UID: \"7240e446-e298-491a-b31f-9f8ded98d74f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-926972-547bc47bff-cg427"
Nov 01 15:12:46.034690 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:12:46.034611    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27mz5\" (UniqueName: \"kubernetes.io/projected/7240e446-e298-491a-b31f-9f8ded98d74f-kube-api-access-27mz5\") pod \"maas-api-e2e-extmod-a-926972-547bc47bff-cg427\" (UID: \"7240e446-e298-491a-b31f-9f8ded98d74f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-926972-547bc47bff-cg427"
Nov 01 15:12:46.486122 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:12:46.486084    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7240e446-e298-491a-b31f-9f8ded98d74f-maas-api-tls\") pod \"maas-api-e2e-extmod-a-926972-547bc47bff-cg427\" (UID: \"7240e446-e298-491a-b31f-9f8ded98d74f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-926972-547bc47bff-cg427"
Nov 01 15:12:46.488753 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:12:46.488718    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7240e446-e298-491a-b31f-9f8ded98d74f-maas-api-tls\") pod \"maas-api-e2e-extmod-a-926972-547bc47bff-cg427\" (UID: \"7240e446-e298-491a-b31f-9f8ded98d74f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-926972-547bc47bff-cg427"
Nov 01 15:12:46.678069 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:12:46.678035    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-926972-547bc47bff-cg427"
Nov 01 15:12:47.059288 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:12:47.059203    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-926972-547bc47bff-cg427"]
Nov 01 15:12:47.064939 ip-10-0-134-37 kubenswrapper[4993]: W1101 15:12:47.064910    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7240e446_e298_491a_b31f_9f8ded98d74f.slice/crio-23f9a616e46b7f246c5b6774764315b68b75ad8f8b8fdef28bb5788060420301 WatchSource:0}: Error finding container 23f9a616e46b7f246c5b6774764315b68b75ad8f8b8fdef28bb5788060420301: Status 404 returned error can't find the container with id 23f9a616e46b7f246c5b6774764315b68b75ad8f8b8fdef28bb5788060420301
Nov 01 15:12:47.116188 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:12:47.116139    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-926972-547bc47bff-cg427" event={"ID":"7240e446-e298-491a-b31f-9f8ded98d74f","Type":"ContainerStarted","Data":"23f9a616e46b7f246c5b6774764315b68b75ad8f8b8fdef28bb5788060420301"}
Nov 01 15:12:48.123717 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:12:48.123680    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-926972-547bc47bff-cg427" event={"ID":"7240e446-e298-491a-b31f-9f8ded98d74f","Type":"ContainerStarted","Data":"b77c1077b86a1d2b6eb7a2aab1eb05da3dbeed237526286f3ab8752cc84cc0b8"}
Nov 01 15:12:48.124134 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:12:48.123763    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-926972-547bc47bff-cg427"
Nov 01 15:12:48.244505 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:12:48.244445    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-926972-547bc47bff-cg427" podStartSLOduration=2.340188928 podStartE2EDuration="3.244432058s" totalImagesPullingTime="904.24313ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:12:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:12:48.24414102 +0000 UTC m=+3355.574003757" watchObservedRunningTime="2026-11-01 15:12:48.244432058 +0000 UTC m=+3355.574294794"
Nov 01 15:12:54.135214 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:12:54.135162    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-926972-547bc47bff-cg427"
Nov 01 15:12:57.499825 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:12:57.499786    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:13:27.501961 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:13:27.501877    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:13:57.502357 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:13:57.502321    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:13:58.927849 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:13:58.927811    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq"]
Nov 01 15:13:58.928479 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:13:58.928034    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq" podUID="bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e" containerName="istio-proxy" containerID="cri-o://65a5002b2d0ed05bad4537af8054c47a895fe65569c006c25f46c2fdaf7926e1" gracePeriod=30
Nov 01 15:13:59.411119 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:13:59.411084    4993 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 15:13:59.411333 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:13:59.411148    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq" podUID="bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 15:14:02.884639 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:02.884596    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-15ad1e-7d6f47cdd6-klmqs"]
Nov 01 15:14:02.888862 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:02.888841    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-15ad1e-7d6f47cdd6-klmqs"
Nov 01 15:14:02.977961 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:02.977923    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bb8f2\" (UniqueName: \"kubernetes.io/projected/6a9774c0-61a8-41a4-8f4c-93bc06a03e94-kube-api-access-bb8f2\") pod \"maas-api-e2e-extmod-b-15ad1e-7d6f47cdd6-klmqs\" (UID: \"6a9774c0-61a8-41a4-8f4c-93bc06a03e94\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-15ad1e-7d6f47cdd6-klmqs"
Nov 01 15:14:02.978162 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:02.977977    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a9774c0-61a8-41a4-8f4c-93bc06a03e94-metrics-certs\") pod \"maas-api-e2e-extmod-b-15ad1e-7d6f47cdd6-klmqs\" (UID: \"6a9774c0-61a8-41a4-8f4c-93bc06a03e94\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-15ad1e-7d6f47cdd6-klmqs"
Nov 01 15:14:02.978162 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:02.978006    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a9774c0-61a8-41a4-8f4c-93bc06a03e94-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-15ad1e-7d6f47cdd6-klmqs\" (UID: \"6a9774c0-61a8-41a4-8f4c-93bc06a03e94\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-15ad1e-7d6f47cdd6-klmqs"
Nov 01 15:14:02.978262 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:02.978226    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a9774c0-61a8-41a4-8f4c-93bc06a03e94-maas-api-tls\") pod \"maas-api-e2e-extmod-b-15ad1e-7d6f47cdd6-klmqs\" (UID: \"6a9774c0-61a8-41a4-8f4c-93bc06a03e94\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-15ad1e-7d6f47cdd6-klmqs"
Nov 01 15:14:02.985475 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:02.985440    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-15ad1e-serving-cert\""
Nov 01 15:14:02.988572 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:02.988548    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-15ad1e-7d6f47cdd6-klmqs"]
Nov 01 15:14:03.079113 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:03.079074    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a9774c0-61a8-41a4-8f4c-93bc06a03e94-maas-api-tls\") pod \"maas-api-e2e-extmod-b-15ad1e-7d6f47cdd6-klmqs\" (UID: \"6a9774c0-61a8-41a4-8f4c-93bc06a03e94\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-15ad1e-7d6f47cdd6-klmqs"
Nov 01 15:14:03.079323 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:03.079176    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb8f2\" (UniqueName: \"kubernetes.io/projected/6a9774c0-61a8-41a4-8f4c-93bc06a03e94-kube-api-access-bb8f2\") pod \"maas-api-e2e-extmod-b-15ad1e-7d6f47cdd6-klmqs\" (UID: \"6a9774c0-61a8-41a4-8f4c-93bc06a03e94\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-15ad1e-7d6f47cdd6-klmqs"
Nov 01 15:14:03.079323 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:03.079208    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a9774c0-61a8-41a4-8f4c-93bc06a03e94-metrics-certs\") pod \"maas-api-e2e-extmod-b-15ad1e-7d6f47cdd6-klmqs\" (UID: \"6a9774c0-61a8-41a4-8f4c-93bc06a03e94\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-15ad1e-7d6f47cdd6-klmqs"
Nov 01 15:14:03.079323 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:03.079226    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a9774c0-61a8-41a4-8f4c-93bc06a03e94-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-15ad1e-7d6f47cdd6-klmqs\" (UID: \"6a9774c0-61a8-41a4-8f4c-93bc06a03e94\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-15ad1e-7d6f47cdd6-klmqs"
Nov 01 15:14:03.079323 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:14:03.079230    4993 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-15ad1e-serving-cert: secret "maas-api-e2e-extmod-b-15ad1e-serving-cert" not found
Nov 01 15:14:03.079323 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:14:03.079303    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a9774c0-61a8-41a4-8f4c-93bc06a03e94-maas-api-tls podName:6a9774c0-61a8-41a4-8f4c-93bc06a03e94 nodeName:}" failed. No retries permitted until 2026-11-01 15:14:03.579284911 +0000 UTC m=+3430.909147626 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6a9774c0-61a8-41a4-8f4c-93bc06a03e94-maas-api-tls") pod "maas-api-e2e-extmod-b-15ad1e-7d6f47cdd6-klmqs" (UID: "6a9774c0-61a8-41a4-8f4c-93bc06a03e94") : secret "maas-api-e2e-extmod-b-15ad1e-serving-cert" not found
Nov 01 15:14:03.079984 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:03.079962    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a9774c0-61a8-41a4-8f4c-93bc06a03e94-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-15ad1e-7d6f47cdd6-klmqs\" (UID: \"6a9774c0-61a8-41a4-8f4c-93bc06a03e94\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-15ad1e-7d6f47cdd6-klmqs"
Nov 01 15:14:03.081848 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:03.081820    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a9774c0-61a8-41a4-8f4c-93bc06a03e94-metrics-certs\") pod \"maas-api-e2e-extmod-b-15ad1e-7d6f47cdd6-klmqs\" (UID: \"6a9774c0-61a8-41a4-8f4c-93bc06a03e94\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-15ad1e-7d6f47cdd6-klmqs"
Nov 01 15:14:03.132948 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:03.132914    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb8f2\" (UniqueName: \"kubernetes.io/projected/6a9774c0-61a8-41a4-8f4c-93bc06a03e94-kube-api-access-bb8f2\") pod \"maas-api-e2e-extmod-b-15ad1e-7d6f47cdd6-klmqs\" (UID: \"6a9774c0-61a8-41a4-8f4c-93bc06a03e94\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-15ad1e-7d6f47cdd6-klmqs"
Nov 01 15:14:03.584744 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:03.584706    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a9774c0-61a8-41a4-8f4c-93bc06a03e94-maas-api-tls\") pod \"maas-api-e2e-extmod-b-15ad1e-7d6f47cdd6-klmqs\" (UID: \"6a9774c0-61a8-41a4-8f4c-93bc06a03e94\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-15ad1e-7d6f47cdd6-klmqs"
Nov 01 15:14:03.587206 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:03.587179    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a9774c0-61a8-41a4-8f4c-93bc06a03e94-maas-api-tls\") pod \"maas-api-e2e-extmod-b-15ad1e-7d6f47cdd6-klmqs\" (UID: \"6a9774c0-61a8-41a4-8f4c-93bc06a03e94\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-15ad1e-7d6f47cdd6-klmqs"
Nov 01 15:14:03.803762 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:03.803721    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-15ad1e-7d6f47cdd6-klmqs"
Nov 01 15:14:04.184206 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:04.184145    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-15ad1e-7d6f47cdd6-klmqs"]
Nov 01 15:14:04.189427 ip-10-0-134-37 kubenswrapper[4993]: W1101 15:14:04.189360    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a9774c0_61a8_41a4_8f4c_93bc06a03e94.slice/crio-dd92da8677911eaf7e8ed81ba257b92a9d002e864c44b9be9054bbc63c1972fc WatchSource:0}: Error finding container dd92da8677911eaf7e8ed81ba257b92a9d002e864c44b9be9054bbc63c1972fc: Status 404 returned error can't find the container with id dd92da8677911eaf7e8ed81ba257b92a9d002e864c44b9be9054bbc63c1972fc
Nov 01 15:14:04.210357 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:04.210333    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq"
Nov 01 15:14:04.392485 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:04.392447    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e-workload-socket\") pod \"bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e\" (UID: \"bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e\") "
Nov 01 15:14:04.392667 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:04.392532    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e-istio-data\") pod \"bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e\" (UID: \"bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e\") "
Nov 01 15:14:04.392667 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:04.392570    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e-kube-api-access-7mk2z\" (UniqueName: \"kubernetes.io/projected/bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e-kube-api-access-7mk2z\") pod \"bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e\" (UID: \"bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e\") "
Nov 01 15:14:04.392667 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:04.392595    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e-istio-podinfo\") pod \"bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e\" (UID: \"bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e\") "
Nov 01 15:14:04.392667 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:04.392618    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e-istio-token\" (UniqueName: \"kubernetes.io/projected/bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e-istio-token\") pod \"bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e\" (UID: \"bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e\") "
Nov 01 15:14:04.392905 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:04.392694    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e-istio-envoy\") pod \"bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e\" (UID: \"bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e\") "
Nov 01 15:14:04.392905 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:04.392776    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e-credential-socket\") pod \"bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e\" (UID: \"bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e\") "
Nov 01 15:14:04.392905 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:04.392799    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e-workload-socket" pod "bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e" (UID: "bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:14:04.392905 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:04.392823    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e-istio-data" pod "bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e" (UID: "bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:14:04.392905 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:04.392859    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e-istiod-ca-cert\") pod \"bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e\" (UID: \"bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e\") "
Nov 01 15:14:04.393188 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:04.393064    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e-workload-certs\") pod \"bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e\" (UID: \"bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e\") "
Nov 01 15:14:04.393257 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:04.393201    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e-credential-socket" pod "bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e" (UID: "bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:14:04.393487 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:04.393447    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e-workload-certs" pod "bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e" (UID: "bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:14:04.393592 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:04.393509    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e-istiod-ca-cert" pod "bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e" (UID: "bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:14:04.393661 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:04.393609    4993 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e-credential-socket\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:14:04.393661 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:04.393622    4993 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e-istiod-ca-cert\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:14:04.393661 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:04.393631    4993 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e-workload-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:14:04.393661 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:04.393640    4993 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e-workload-socket\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:14:04.393661 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:04.393652    4993 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e-istio-data\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:14:04.395092 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:04.395068    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e-istio-podinfo" pod "bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e" (UID: "bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 15:14:04.395232 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:04.395155    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e-kube-api-access-7mk2z" pod "bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e" (UID: "bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e"). InnerVolumeSpecName "kube-api-access-7mk2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:14:04.395442 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:04.395414    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e-istio-token" pod "bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e" (UID: "bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:14:04.395485 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:04.395448    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e-istio-envoy" pod "bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e" (UID: "bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:14:04.494608 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:04.494564    4993 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e-istio-podinfo\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:14:04.494608 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:04.494604    4993 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e-istio-token\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:14:04.494608 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:04.494614    4993 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e-istio-envoy\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:14:04.494853 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:04.494624    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mk2z\" (UniqueName: \"kubernetes.io/projected/bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e-kube-api-access-7mk2z\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:14:04.611263 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:04.611227    4993 generic.go:356] "Generic (PLEG): container finished" podID="bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e" containerID="65a5002b2d0ed05bad4537af8054c47a895fe65569c006c25f46c2fdaf7926e1" exitCode=0
Nov 01 15:14:04.611470 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:04.611266    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq" event={"ID":"bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e","Type":"ContainerDied","Data":"65a5002b2d0ed05bad4537af8054c47a895fe65569c006c25f46c2fdaf7926e1"}
Nov 01 15:14:04.611470 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:04.611327    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq" event={"ID":"bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e","Type":"ContainerDied","Data":"29f89407d84fbd487c792a2b39b375397fccab82d64954e01df480aaa70d9228"}
Nov 01 15:14:04.611470 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:04.611351    4993 scope.go:122] "RemoveContainer" containerID="65a5002b2d0ed05bad4537af8054c47a895fe65569c006c25f46c2fdaf7926e1"
Nov 01 15:14:04.611470 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:04.611279    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq"
Nov 01 15:14:04.612816 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:04.612792    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-15ad1e-7d6f47cdd6-klmqs" event={"ID":"6a9774c0-61a8-41a4-8f4c-93bc06a03e94","Type":"ContainerStarted","Data":"dd92da8677911eaf7e8ed81ba257b92a9d002e864c44b9be9054bbc63c1972fc"}
Nov 01 15:14:04.622982 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:04.622965    4993 scope.go:122] "RemoveContainer" containerID="65a5002b2d0ed05bad4537af8054c47a895fe65569c006c25f46c2fdaf7926e1"
Nov 01 15:14:04.623321 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:14:04.623302    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65a5002b2d0ed05bad4537af8054c47a895fe65569c006c25f46c2fdaf7926e1\": container with ID starting with 65a5002b2d0ed05bad4537af8054c47a895fe65569c006c25f46c2fdaf7926e1 not found: ID does not exist" containerID="65a5002b2d0ed05bad4537af8054c47a895fe65569c006c25f46c2fdaf7926e1"
Nov 01 15:14:04.623382 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:04.623332    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65a5002b2d0ed05bad4537af8054c47a895fe65569c006c25f46c2fdaf7926e1"} err="failed to get container status \"65a5002b2d0ed05bad4537af8054c47a895fe65569c006c25f46c2fdaf7926e1\": rpc error: code = NotFound desc = could not find container \"65a5002b2d0ed05bad4537af8054c47a895fe65569c006c25f46c2fdaf7926e1\": container with ID starting with 65a5002b2d0ed05bad4537af8054c47a895fe65569c006c25f46c2fdaf7926e1 not found: ID does not exist"
Nov 01 15:14:04.767264 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:04.767163    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq"]
Nov 01 15:14:04.813084 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:04.813052    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-82flq"]
Nov 01 15:14:05.621590 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:05.621543    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-15ad1e-7d6f47cdd6-klmqs" event={"ID":"6a9774c0-61a8-41a4-8f4c-93bc06a03e94","Type":"ContainerStarted","Data":"d9fcc5b7c5739242ca4fcfe7074329ce08cdd3c1e772e986e7e5dfc8d399b190"}
Nov 01 15:14:05.622035 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:05.621752    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-15ad1e-7d6f47cdd6-klmqs"
Nov 01 15:14:05.733308 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:05.733247    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-15ad1e-7d6f47cdd6-klmqs" podStartSLOduration=2.8139974309999998 podStartE2EDuration="3.733231425s" totalImagesPullingTime="919.233994ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:14:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:14:05.730277767 +0000 UTC m=+3433.060140505" watchObservedRunningTime="2026-11-01 15:14:05.733231425 +0000 UTC m=+3433.063094161"
Nov 01 15:14:06.797326 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:06.797287    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e" path="/var/lib/kubelet/pods/bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e/volumes"
Nov 01 15:14:11.632905 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:11.632868    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-15ad1e-7d6f47cdd6-klmqs"
Nov 01 15:14:21.344867 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:21.344827    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-e92165-7f6b6f494f-ht5kl"]
Nov 01 15:14:21.345614 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:21.345595    4993 cpu_manager.go:414] "containerMap: removing container" podUID="bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e" containerName="istio-proxy"
Nov 01 15:14:21.345665 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:21.345616    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e" containerName="istio-proxy"
Nov 01 15:14:21.345723 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:21.345714    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc9b2c86-eb1c-4ac3-8270-7b2a4da4946e" containerName="istio-proxy"
Nov 01 15:14:21.350306 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:21.350285    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e92165-7f6b6f494f-ht5kl"
Nov 01 15:14:21.404545 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:21.404507    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-e92165-7f6b6f494f-ht5kl"]
Nov 01 15:14:21.488852 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:21.488816    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-e92165-serving-cert\""
Nov 01 15:14:21.492367 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:21.491655    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7334c698-a18f-4c7a-87e0-5c915c7939d0-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-e92165-7f6b6f494f-ht5kl\" (UID: \"7334c698-a18f-4c7a-87e0-5c915c7939d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e92165-7f6b6f494f-ht5kl"
Nov 01 15:14:21.492642 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:21.492624    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7334c698-a18f-4c7a-87e0-5c915c7939d0-metrics-certs\") pod \"maas-api-e2e-extmod-b-e92165-7f6b6f494f-ht5kl\" (UID: \"7334c698-a18f-4c7a-87e0-5c915c7939d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e92165-7f6b6f494f-ht5kl"
Nov 01 15:14:21.492809 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:21.492783    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7334c698-a18f-4c7a-87e0-5c915c7939d0-maas-api-tls\") pod \"maas-api-e2e-extmod-b-e92165-7f6b6f494f-ht5kl\" (UID: \"7334c698-a18f-4c7a-87e0-5c915c7939d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e92165-7f6b6f494f-ht5kl"
Nov 01 15:14:21.493046 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:21.493028    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trnb8\" (UniqueName: \"kubernetes.io/projected/7334c698-a18f-4c7a-87e0-5c915c7939d0-kube-api-access-trnb8\") pod \"maas-api-e2e-extmod-b-e92165-7f6b6f494f-ht5kl\" (UID: \"7334c698-a18f-4c7a-87e0-5c915c7939d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e92165-7f6b6f494f-ht5kl"
Nov 01 15:14:21.594645 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:21.594602    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7334c698-a18f-4c7a-87e0-5c915c7939d0-metrics-certs\") pod \"maas-api-e2e-extmod-b-e92165-7f6b6f494f-ht5kl\" (UID: \"7334c698-a18f-4c7a-87e0-5c915c7939d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e92165-7f6b6f494f-ht5kl"
Nov 01 15:14:21.594976 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:21.594904    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7334c698-a18f-4c7a-87e0-5c915c7939d0-maas-api-tls\") pod \"maas-api-e2e-extmod-b-e92165-7f6b6f494f-ht5kl\" (UID: \"7334c698-a18f-4c7a-87e0-5c915c7939d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e92165-7f6b6f494f-ht5kl"
Nov 01 15:14:21.595105 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:21.595024    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trnb8\" (UniqueName: \"kubernetes.io/projected/7334c698-a18f-4c7a-87e0-5c915c7939d0-kube-api-access-trnb8\") pod \"maas-api-e2e-extmod-b-e92165-7f6b6f494f-ht5kl\" (UID: \"7334c698-a18f-4c7a-87e0-5c915c7939d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e92165-7f6b6f494f-ht5kl"
Nov 01 15:14:21.595105 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:14:21.595050    4993 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-e92165-serving-cert: secret "maas-api-e2e-extmod-b-e92165-serving-cert" not found
Nov 01 15:14:21.595219 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:14:21.595131    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7334c698-a18f-4c7a-87e0-5c915c7939d0-maas-api-tls podName:7334c698-a18f-4c7a-87e0-5c915c7939d0 nodeName:}" failed. No retries permitted until 2026-11-01 15:14:22.095109216 +0000 UTC m=+3449.424971951 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7334c698-a18f-4c7a-87e0-5c915c7939d0-maas-api-tls") pod "maas-api-e2e-extmod-b-e92165-7f6b6f494f-ht5kl" (UID: "7334c698-a18f-4c7a-87e0-5c915c7939d0") : secret "maas-api-e2e-extmod-b-e92165-serving-cert" not found
Nov 01 15:14:21.595219 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:21.595188    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7334c698-a18f-4c7a-87e0-5c915c7939d0-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-e92165-7f6b6f494f-ht5kl\" (UID: \"7334c698-a18f-4c7a-87e0-5c915c7939d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e92165-7f6b6f494f-ht5kl"
Nov 01 15:14:21.595991 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:21.595967    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7334c698-a18f-4c7a-87e0-5c915c7939d0-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-e92165-7f6b6f494f-ht5kl\" (UID: \"7334c698-a18f-4c7a-87e0-5c915c7939d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e92165-7f6b6f494f-ht5kl"
Nov 01 15:14:21.598320 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:21.598296    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7334c698-a18f-4c7a-87e0-5c915c7939d0-metrics-certs\") pod \"maas-api-e2e-extmod-b-e92165-7f6b6f494f-ht5kl\" (UID: \"7334c698-a18f-4c7a-87e0-5c915c7939d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e92165-7f6b6f494f-ht5kl"
Nov 01 15:14:21.649635 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:21.649598    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trnb8\" (UniqueName: \"kubernetes.io/projected/7334c698-a18f-4c7a-87e0-5c915c7939d0-kube-api-access-trnb8\") pod \"maas-api-e2e-extmod-b-e92165-7f6b6f494f-ht5kl\" (UID: \"7334c698-a18f-4c7a-87e0-5c915c7939d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e92165-7f6b6f494f-ht5kl"
Nov 01 15:14:22.102041 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:22.102001    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7334c698-a18f-4c7a-87e0-5c915c7939d0-maas-api-tls\") pod \"maas-api-e2e-extmod-b-e92165-7f6b6f494f-ht5kl\" (UID: \"7334c698-a18f-4c7a-87e0-5c915c7939d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e92165-7f6b6f494f-ht5kl"
Nov 01 15:14:22.104637 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:22.104602    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7334c698-a18f-4c7a-87e0-5c915c7939d0-maas-api-tls\") pod \"maas-api-e2e-extmod-b-e92165-7f6b6f494f-ht5kl\" (UID: \"7334c698-a18f-4c7a-87e0-5c915c7939d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e92165-7f6b6f494f-ht5kl"
Nov 01 15:14:22.266428 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:22.266116    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e92165-7f6b6f494f-ht5kl"
Nov 01 15:14:22.651125 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:22.651082    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-e92165-7f6b6f494f-ht5kl"]
Nov 01 15:14:22.656004 ip-10-0-134-37 kubenswrapper[4993]: W1101 15:14:22.655970    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7334c698_a18f_4c7a_87e0_5c915c7939d0.slice/crio-1f0ec4e6784120afb049063d3faa96efad2f51f871a5f4d0d6d43d522b0fb961 WatchSource:0}: Error finding container 1f0ec4e6784120afb049063d3faa96efad2f51f871a5f4d0d6d43d522b0fb961: Status 404 returned error can't find the container with id 1f0ec4e6784120afb049063d3faa96efad2f51f871a5f4d0d6d43d522b0fb961
Nov 01 15:14:22.731002 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:22.730961    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e92165-7f6b6f494f-ht5kl" event={"ID":"7334c698-a18f-4c7a-87e0-5c915c7939d0","Type":"ContainerStarted","Data":"1f0ec4e6784120afb049063d3faa96efad2f51f871a5f4d0d6d43d522b0fb961"}
Nov 01 15:14:23.739881 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:23.739833    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e92165-7f6b6f494f-ht5kl" event={"ID":"7334c698-a18f-4c7a-87e0-5c915c7939d0","Type":"ContainerStarted","Data":"563569785b2c8d5adbc6ba4cc0de78952dfc7c60f3af4f56c169565255cc358f"}
Nov 01 15:14:23.740362 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:23.739926    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e92165-7f6b6f494f-ht5kl"
Nov 01 15:14:23.852482 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:23.852379    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e92165-7f6b6f494f-ht5kl" podStartSLOduration=1.97649067 podStartE2EDuration="2.852363172s" totalImagesPullingTime="875.872502ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:14:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:14:23.849894067 +0000 UTC m=+3451.179756804" watchObservedRunningTime="2026-11-01 15:14:23.852363172 +0000 UTC m=+3451.182225909"
Nov 01 15:14:27.500059 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:14:27.500023    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:14:29.751047 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:29.751014    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e92165-7f6b6f494f-ht5kl"
Nov 01 15:14:44.900101 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:44.900067    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-c89271-559779d48c-jv2zw"]
Nov 01 15:14:44.904201 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:44.904161    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c89271-559779d48c-jv2zw"
Nov 01 15:14:44.955125 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:44.955090    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-c89271-serving-cert\""
Nov 01 15:14:45.004229 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:45.004188    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-c89271-559779d48c-jv2zw"]
Nov 01 15:14:45.074327 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:45.074208    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j7b9\" (UniqueName: \"kubernetes.io/projected/1abb5edc-e03b-45bb-85c6-c14bdf7f71ac-kube-api-access-7j7b9\") pod \"maas-api-e2e-extmod-b-c89271-559779d48c-jv2zw\" (UID: \"1abb5edc-e03b-45bb-85c6-c14bdf7f71ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c89271-559779d48c-jv2zw"
Nov 01 15:14:45.079384 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:45.074328    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1abb5edc-e03b-45bb-85c6-c14bdf7f71ac-maas-api-tls\") pod \"maas-api-e2e-extmod-b-c89271-559779d48c-jv2zw\" (UID: \"1abb5edc-e03b-45bb-85c6-c14bdf7f71ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c89271-559779d48c-jv2zw"
Nov 01 15:14:45.079384 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:45.074438    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1abb5edc-e03b-45bb-85c6-c14bdf7f71ac-metrics-certs\") pod \"maas-api-e2e-extmod-b-c89271-559779d48c-jv2zw\" (UID: \"1abb5edc-e03b-45bb-85c6-c14bdf7f71ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c89271-559779d48c-jv2zw"
Nov 01 15:14:45.079384 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:45.074489    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1abb5edc-e03b-45bb-85c6-c14bdf7f71ac-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-c89271-559779d48c-jv2zw\" (UID: \"1abb5edc-e03b-45bb-85c6-c14bdf7f71ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c89271-559779d48c-jv2zw"
Nov 01 15:14:45.176025 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:45.175932    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1abb5edc-e03b-45bb-85c6-c14bdf7f71ac-metrics-certs\") pod \"maas-api-e2e-extmod-b-c89271-559779d48c-jv2zw\" (UID: \"1abb5edc-e03b-45bb-85c6-c14bdf7f71ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c89271-559779d48c-jv2zw"
Nov 01 15:14:45.176025 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:45.176016    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1abb5edc-e03b-45bb-85c6-c14bdf7f71ac-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-c89271-559779d48c-jv2zw\" (UID: \"1abb5edc-e03b-45bb-85c6-c14bdf7f71ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c89271-559779d48c-jv2zw"
Nov 01 15:14:45.176257 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:45.176043    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j7b9\" (UniqueName: \"kubernetes.io/projected/1abb5edc-e03b-45bb-85c6-c14bdf7f71ac-kube-api-access-7j7b9\") pod \"maas-api-e2e-extmod-b-c89271-559779d48c-jv2zw\" (UID: \"1abb5edc-e03b-45bb-85c6-c14bdf7f71ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c89271-559779d48c-jv2zw"
Nov 01 15:14:45.176257 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:45.176123    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1abb5edc-e03b-45bb-85c6-c14bdf7f71ac-maas-api-tls\") pod \"maas-api-e2e-extmod-b-c89271-559779d48c-jv2zw\" (UID: \"1abb5edc-e03b-45bb-85c6-c14bdf7f71ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c89271-559779d48c-jv2zw"
Nov 01 15:14:45.176807 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:45.176779    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1abb5edc-e03b-45bb-85c6-c14bdf7f71ac-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-c89271-559779d48c-jv2zw\" (UID: \"1abb5edc-e03b-45bb-85c6-c14bdf7f71ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c89271-559779d48c-jv2zw"
Nov 01 15:14:45.178489 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:45.178469    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1abb5edc-e03b-45bb-85c6-c14bdf7f71ac-maas-api-tls\") pod \"maas-api-e2e-extmod-b-c89271-559779d48c-jv2zw\" (UID: \"1abb5edc-e03b-45bb-85c6-c14bdf7f71ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c89271-559779d48c-jv2zw"
Nov 01 15:14:45.178551 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:45.178516    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1abb5edc-e03b-45bb-85c6-c14bdf7f71ac-metrics-certs\") pod \"maas-api-e2e-extmod-b-c89271-559779d48c-jv2zw\" (UID: \"1abb5edc-e03b-45bb-85c6-c14bdf7f71ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c89271-559779d48c-jv2zw"
Nov 01 15:14:45.229443 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:45.229401    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j7b9\" (UniqueName: \"kubernetes.io/projected/1abb5edc-e03b-45bb-85c6-c14bdf7f71ac-kube-api-access-7j7b9\") pod \"maas-api-e2e-extmod-b-c89271-559779d48c-jv2zw\" (UID: \"1abb5edc-e03b-45bb-85c6-c14bdf7f71ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c89271-559779d48c-jv2zw"
Nov 01 15:14:45.519315 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:45.519278    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c89271-559779d48c-jv2zw"
Nov 01 15:14:45.903498 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:45.903405    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-c89271-559779d48c-jv2zw"]
Nov 01 15:14:45.908406 ip-10-0-134-37 kubenswrapper[4993]: W1101 15:14:45.908370    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1abb5edc_e03b_45bb_85c6_c14bdf7f71ac.slice/crio-a4da7bc40f124306877e5e9ad97a2fc9ae44406219f99c5800adb0a952aab679 WatchSource:0}: Error finding container a4da7bc40f124306877e5e9ad97a2fc9ae44406219f99c5800adb0a952aab679: Status 404 returned error can't find the container with id a4da7bc40f124306877e5e9ad97a2fc9ae44406219f99c5800adb0a952aab679
Nov 01 15:14:46.891579 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:46.891543    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c89271-559779d48c-jv2zw" event={"ID":"1abb5edc-e03b-45bb-85c6-c14bdf7f71ac","Type":"ContainerStarted","Data":"cea91cc823765731cb6cec81cd6d7e410b2f0a4c2c9a59cab0584758dfbdb438"}
Nov 01 15:14:46.891727 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:46.891607    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c89271-559779d48c-jv2zw" event={"ID":"1abb5edc-e03b-45bb-85c6-c14bdf7f71ac","Type":"ContainerStarted","Data":"a4da7bc40f124306877e5e9ad97a2fc9ae44406219f99c5800adb0a952aab679"}
Nov 01 15:14:46.892079 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:46.892055    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c89271-559779d48c-jv2zw"
Nov 01 15:14:47.031755 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:47.031703    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c89271-559779d48c-jv2zw" podStartSLOduration=2.14978689 podStartE2EDuration="3.031688857s" totalImagesPullingTime="881.901967ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:14:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:14:47.028300973 +0000 UTC m=+3474.358163715" watchObservedRunningTime="2026-11-01 15:14:47.031688857 +0000 UTC m=+3474.361551594"
Nov 01 15:14:52.906009 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:14:52.905980    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c89271-559779d48c-jv2zw"
Nov 01 15:14:57.499746 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:14:57.499703    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:15:00.237794 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:15:00.237755    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-926972-29892435-vlm74"]
Nov 01 15:15:00.242435 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:15:00.242410    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-926972-29892435-vlm74"
Nov 01 15:15:00.320683 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:15:00.320648    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\""
Nov 01 15:15:00.349700 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:15:00.349668    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvk8n\" (UniqueName: \"kubernetes.io/projected/c35e44f6-5a00-4c5e-9eb3-7edc9cfbb937-kube-api-access-qvk8n\") pod \"maas-api-key-cleanup-e2e-extmod-a-926972-29892435-vlm74\" (UID: \"c35e44f6-5a00-4c5e-9eb3-7edc9cfbb937\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-926972-29892435-vlm74"
Nov 01 15:15:00.349858 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:15:00.349725    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c35e44f6-5a00-4c5e-9eb3-7edc9cfbb937-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-926972-29892435-vlm74\" (UID: \"c35e44f6-5a00-4c5e-9eb3-7edc9cfbb937\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-926972-29892435-vlm74"
Nov 01 15:15:00.372226 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:15:00.372184    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-926972-29892435-vlm74"]
Nov 01 15:15:00.451491 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:15:00.451445    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvk8n\" (UniqueName: \"kubernetes.io/projected/c35e44f6-5a00-4c5e-9eb3-7edc9cfbb937-kube-api-access-qvk8n\") pod \"maas-api-key-cleanup-e2e-extmod-a-926972-29892435-vlm74\" (UID: \"c35e44f6-5a00-4c5e-9eb3-7edc9cfbb937\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-926972-29892435-vlm74"
Nov 01 15:15:00.451811 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:15:00.451775    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c35e44f6-5a00-4c5e-9eb3-7edc9cfbb937-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-926972-29892435-vlm74\" (UID: \"c35e44f6-5a00-4c5e-9eb3-7edc9cfbb937\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-926972-29892435-vlm74"
Nov 01 15:15:00.452768 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:15:00.452740    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c35e44f6-5a00-4c5e-9eb3-7edc9cfbb937-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-926972-29892435-vlm74\" (UID: \"c35e44f6-5a00-4c5e-9eb3-7edc9cfbb937\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-926972-29892435-vlm74"
Nov 01 15:15:00.504709 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:15:00.504631    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvk8n\" (UniqueName: \"kubernetes.io/projected/c35e44f6-5a00-4c5e-9eb3-7edc9cfbb937-kube-api-access-qvk8n\") pod \"maas-api-key-cleanup-e2e-extmod-a-926972-29892435-vlm74\" (UID: \"c35e44f6-5a00-4c5e-9eb3-7edc9cfbb937\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-926972-29892435-vlm74"
Nov 01 15:15:00.555672 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:15:00.555632    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-926972-29892435-vlm74"
Nov 01 15:15:00.929736 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:15:00.929636    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-926972-29892435-vlm74"]
Nov 01 15:15:00.935285 ip-10-0-134-37 kubenswrapper[4993]: W1101 15:15:00.935256    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc35e44f6_5a00_4c5e_9eb3_7edc9cfbb937.slice/crio-63f4e0c0af506a105a5f75c9af9215de61143812b88b2e9f04f98d11d2d63985 WatchSource:0}: Error finding container 63f4e0c0af506a105a5f75c9af9215de61143812b88b2e9f04f98d11d2d63985: Status 404 returned error can't find the container with id 63f4e0c0af506a105a5f75c9af9215de61143812b88b2e9f04f98d11d2d63985
Nov 01 15:15:00.980143 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:15:00.980104    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-926972-29892435-vlm74" event={"ID":"c35e44f6-5a00-4c5e-9eb3-7edc9cfbb937","Type":"ContainerStarted","Data":"63f4e0c0af506a105a5f75c9af9215de61143812b88b2e9f04f98d11d2d63985"}
Nov 01 15:15:01.988185 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:15:01.988130    4993 generic.go:356] "Generic (PLEG): container finished" podID="c35e44f6-5a00-4c5e-9eb3-7edc9cfbb937" containerID="9e0ba8e19588f74cc4f5fd87c6214eef3355bfadabea92ddcd5bde97b280bc7d" exitCode=60
Nov 01 15:15:01.988635 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:15:01.988219    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-926972-29892435-vlm74" event={"ID":"c35e44f6-5a00-4c5e-9eb3-7edc9cfbb937","Type":"ContainerDied","Data":"9e0ba8e19588f74cc4f5fd87c6214eef3355bfadabea92ddcd5bde97b280bc7d"}
Nov 01 15:15:01.988635 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:15:01.988568    4993 scope.go:122] "RemoveContainer" containerID="9e0ba8e19588f74cc4f5fd87c6214eef3355bfadabea92ddcd5bde97b280bc7d"
Nov 01 15:15:02.995673 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:15:02.995635    4993 generic.go:356] "Generic (PLEG): container finished" podID="c35e44f6-5a00-4c5e-9eb3-7edc9cfbb937" containerID="ea0b44e2ce685b4b812c439ecc957179708faca3fd28d598876b2a8c36f7a009" exitCode=60
Nov 01 15:15:02.996156 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:15:02.995712    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-926972-29892435-vlm74" event={"ID":"c35e44f6-5a00-4c5e-9eb3-7edc9cfbb937","Type":"ContainerDied","Data":"ea0b44e2ce685b4b812c439ecc957179708faca3fd28d598876b2a8c36f7a009"}
Nov 01 15:15:02.996156 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:15:02.995764    4993 scope.go:122] "RemoveContainer" containerID="9e0ba8e19588f74cc4f5fd87c6214eef3355bfadabea92ddcd5bde97b280bc7d"
Nov 01 15:15:02.996156 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:15:02.996076    4993 scope.go:122] "RemoveContainer" containerID="ea0b44e2ce685b4b812c439ecc957179708faca3fd28d598876b2a8c36f7a009"
Nov 01 15:15:02.996456 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:15:02.996428    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-926972-29892435-vlm74_odh-ai-gateway-infra(c35e44f6-5a00-4c5e-9eb3-7edc9cfbb937)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-926972-29892435-vlm74" podUID="c35e44f6-5a00-4c5e-9eb3-7edc9cfbb937"
Nov 01 15:15:04.003559 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:15:04.003533    4993 scope.go:122] "RemoveContainer" containerID="ea0b44e2ce685b4b812c439ecc957179708faca3fd28d598876b2a8c36f7a009"
Nov 01 15:15:04.003950 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:15:04.003779    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-926972-29892435-vlm74_odh-ai-gateway-infra(c35e44f6-5a00-4c5e-9eb3-7edc9cfbb937)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-926972-29892435-vlm74" podUID="c35e44f6-5a00-4c5e-9eb3-7edc9cfbb937"
Nov 01 15:15:14.788064 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:15:14.788032    4993 scope.go:122] "RemoveContainer" containerID="ea0b44e2ce685b4b812c439ecc957179708faca3fd28d598876b2a8c36f7a009"
Nov 01 15:15:15.076212 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:15:15.076153    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-926972-29892435-vlm74" event={"ID":"c35e44f6-5a00-4c5e-9eb3-7edc9cfbb937","Type":"ContainerStarted","Data":"bce86ec57dd85651d01f04045899de00373afa6616caaad4b6639f3c84f7f728"}
Nov 01 15:15:15.185409 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:15:15.185346    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-926972-29892435-vlm74" podStartSLOduration=15.185328582 podStartE2EDuration="15.185328582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:15:15.182368888 +0000 UTC m=+3502.512231628" watchObservedRunningTime="2026-11-01 15:15:15.185328582 +0000 UTC m=+3502.515191320"
Nov 01 15:15:16.085910 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:15:16.085868    4993 generic.go:356] "Generic (PLEG): container finished" podID="c35e44f6-5a00-4c5e-9eb3-7edc9cfbb937" containerID="bce86ec57dd85651d01f04045899de00373afa6616caaad4b6639f3c84f7f728" exitCode=60
Nov 01 15:15:16.085910 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:15:16.085895    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-926972-29892435-vlm74" event={"ID":"c35e44f6-5a00-4c5e-9eb3-7edc9cfbb937","Type":"ContainerDied","Data":"bce86ec57dd85651d01f04045899de00373afa6616caaad4b6639f3c84f7f728"}
Nov 01 15:15:16.086471 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:15:16.085971    4993 scope.go:122] "RemoveContainer" containerID="ea0b44e2ce685b4b812c439ecc957179708faca3fd28d598876b2a8c36f7a009"
Nov 01 15:15:16.086471 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:15:16.086370    4993 scope.go:122] "RemoveContainer" containerID="bce86ec57dd85651d01f04045899de00373afa6616caaad4b6639f3c84f7f728"
Nov 01 15:15:16.086720 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:15:16.086694    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-926972-29892435-vlm74_odh-ai-gateway-infra(c35e44f6-5a00-4c5e-9eb3-7edc9cfbb937)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-926972-29892435-vlm74" podUID="c35e44f6-5a00-4c5e-9eb3-7edc9cfbb937"
Nov 01 15:15:16.192839 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:15:16.192806    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-926972-29892435-vlm74"]
Nov 01 15:15:17.251248 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:15:17.251221    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-926972-29892435-vlm74"
Nov 01 15:15:17.362465 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:15:17.362391    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c35e44f6-5a00-4c5e-9eb3-7edc9cfbb937-kube-api-access-qvk8n\" (UniqueName: \"kubernetes.io/projected/c35e44f6-5a00-4c5e-9eb3-7edc9cfbb937-kube-api-access-qvk8n\") pod \"c35e44f6-5a00-4c5e-9eb3-7edc9cfbb937\" (UID: \"c35e44f6-5a00-4c5e-9eb3-7edc9cfbb937\") "
Nov 01 15:15:17.362465 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:15:17.362450    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c35e44f6-5a00-4c5e-9eb3-7edc9cfbb937-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c35e44f6-5a00-4c5e-9eb3-7edc9cfbb937-maas-api-service-ca\") pod \"c35e44f6-5a00-4c5e-9eb3-7edc9cfbb937\" (UID: \"c35e44f6-5a00-4c5e-9eb3-7edc9cfbb937\") "
Nov 01 15:15:17.362844 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:15:17.362822    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c35e44f6-5a00-4c5e-9eb3-7edc9cfbb937-maas-api-service-ca" pod "c35e44f6-5a00-4c5e-9eb3-7edc9cfbb937" (UID: "c35e44f6-5a00-4c5e-9eb3-7edc9cfbb937"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:15:17.366337 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:15:17.366308    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c35e44f6-5a00-4c5e-9eb3-7edc9cfbb937-kube-api-access-qvk8n" pod "c35e44f6-5a00-4c5e-9eb3-7edc9cfbb937" (UID: "c35e44f6-5a00-4c5e-9eb3-7edc9cfbb937"). InnerVolumeSpecName "kube-api-access-qvk8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:15:17.464290 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:15:17.464254    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvk8n\" (UniqueName: \"kubernetes.io/projected/c35e44f6-5a00-4c5e-9eb3-7edc9cfbb937-kube-api-access-qvk8n\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:15:17.464290 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:15:17.464286    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c35e44f6-5a00-4c5e-9eb3-7edc9cfbb937-maas-api-service-ca\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:15:18.104467 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:15:18.104436    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-926972-29892435-vlm74"
Nov 01 15:15:18.104653 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:15:18.104434    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-926972-29892435-vlm74" event={"ID":"c35e44f6-5a00-4c5e-9eb3-7edc9cfbb937","Type":"ContainerDied","Data":"63f4e0c0af506a105a5f75c9af9215de61143812b88b2e9f04f98d11d2d63985"}
Nov 01 15:15:18.104653 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:15:18.104571    4993 scope.go:122] "RemoveContainer" containerID="bce86ec57dd85651d01f04045899de00373afa6616caaad4b6639f3c84f7f728"
Nov 01 15:15:18.259254 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:15:18.259212    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-926972-29892435-vlm74"]
Nov 01 15:15:18.305055 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:15:18.305020    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-926972-29892435-vlm74"]
Nov 01 15:15:18.796521 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:15:18.796487    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c35e44f6-5a00-4c5e-9eb3-7edc9cfbb937" path="/var/lib/kubelet/pods/c35e44f6-5a00-4c5e-9eb3-7edc9cfbb937/volumes"
Nov 01 15:15:27.501899 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:15:27.501862    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:15:57.500313 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:15:57.500276    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:16:27.501879 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:16:27.501837    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:16:53.308420 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:16:53.308392    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xmwtp_bb59051d-c2df-4362-a157-00f14f915e79/console-operator/3.log"
Nov 01 15:16:53.311909 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:16:53.311885    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xmwtp_bb59051d-c2df-4362-a157-00f14f915e79/console-operator/3.log"
Nov 01 15:16:57.501025 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:16:57.500985    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:17:27.499278 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:17:27.499240    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:17:45.167431 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:17:45.167398    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea1620-559674b8f9-64pfg"]
Nov 01 15:17:45.167898 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:17:45.167623    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea1620-559674b8f9-64pfg" podUID="8a1719a8-89fb-44bb-be61-d8c9423e32d3" containerName="maas-api" containerID="cri-o://3fd48622d9c6aee573733a36edf75f8e60895b65870bd153df9201456340fc19" gracePeriod=30
Nov 01 15:17:45.457902 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:17:45.457835    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea1620-559674b8f9-64pfg"
Nov 01 15:17:45.484458 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:17:45.484422    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a1719a8-89fb-44bb-be61-d8c9423e32d3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a1719a8-89fb-44bb-be61-d8c9423e32d3-openshift-service-ca\") pod \"8a1719a8-89fb-44bb-be61-d8c9423e32d3\" (UID: \"8a1719a8-89fb-44bb-be61-d8c9423e32d3\") "
Nov 01 15:17:45.484633 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:17:45.484580    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a1719a8-89fb-44bb-be61-d8c9423e32d3-kube-api-access-lbxnd\" (UniqueName: \"kubernetes.io/projected/8a1719a8-89fb-44bb-be61-d8c9423e32d3-kube-api-access-lbxnd\") pod \"8a1719a8-89fb-44bb-be61-d8c9423e32d3\" (UID: \"8a1719a8-89fb-44bb-be61-d8c9423e32d3\") "
Nov 01 15:17:45.484633 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:17:45.484614    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a1719a8-89fb-44bb-be61-d8c9423e32d3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a1719a8-89fb-44bb-be61-d8c9423e32d3-maas-api-tls\") pod \"8a1719a8-89fb-44bb-be61-d8c9423e32d3\" (UID: \"8a1719a8-89fb-44bb-be61-d8c9423e32d3\") "
Nov 01 15:17:45.484761 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:17:45.484672    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a1719a8-89fb-44bb-be61-d8c9423e32d3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a1719a8-89fb-44bb-be61-d8c9423e32d3-metrics-certs\") pod \"8a1719a8-89fb-44bb-be61-d8c9423e32d3\" (UID: \"8a1719a8-89fb-44bb-be61-d8c9423e32d3\") "
Nov 01 15:17:45.484871 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:17:45.484840    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a1719a8-89fb-44bb-be61-d8c9423e32d3-openshift-service-ca" pod "8a1719a8-89fb-44bb-be61-d8c9423e32d3" (UID: "8a1719a8-89fb-44bb-be61-d8c9423e32d3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:17:45.485233 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:17:45.485208    4993 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a1719a8-89fb-44bb-be61-d8c9423e32d3-openshift-service-ca\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:17:45.485233 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:17:45.486899    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a1719a8-89fb-44bb-be61-d8c9423e32d3-maas-api-tls" pod "8a1719a8-89fb-44bb-be61-d8c9423e32d3" (UID: "8a1719a8-89fb-44bb-be61-d8c9423e32d3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:17:45.487240 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:17:45.487186    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a1719a8-89fb-44bb-be61-d8c9423e32d3-kube-api-access-lbxnd" pod "8a1719a8-89fb-44bb-be61-d8c9423e32d3" (UID: "8a1719a8-89fb-44bb-be61-d8c9423e32d3"). InnerVolumeSpecName "kube-api-access-lbxnd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:17:45.487671 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:17:45.487640    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a1719a8-89fb-44bb-be61-d8c9423e32d3-metrics-certs" pod "8a1719a8-89fb-44bb-be61-d8c9423e32d3" (UID: "8a1719a8-89fb-44bb-be61-d8c9423e32d3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:17:45.586790 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:17:45.586751    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbxnd\" (UniqueName: \"kubernetes.io/projected/8a1719a8-89fb-44bb-be61-d8c9423e32d3-kube-api-access-lbxnd\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:17:45.586790 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:17:45.586785    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a1719a8-89fb-44bb-be61-d8c9423e32d3-maas-api-tls\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:17:45.586790 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:17:45.586796    4993 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a1719a8-89fb-44bb-be61-d8c9423e32d3-metrics-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:17:46.049994 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:17:46.049959    4993 generic.go:356] "Generic (PLEG): container finished" podID="8a1719a8-89fb-44bb-be61-d8c9423e32d3" containerID="3fd48622d9c6aee573733a36edf75f8e60895b65870bd153df9201456340fc19" exitCode=0
Nov 01 15:17:46.050203 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:17:46.050000    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea1620-559674b8f9-64pfg" event={"ID":"8a1719a8-89fb-44bb-be61-d8c9423e32d3","Type":"ContainerDied","Data":"3fd48622d9c6aee573733a36edf75f8e60895b65870bd153df9201456340fc19"}
Nov 01 15:17:46.050203 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:17:46.050003    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea1620-559674b8f9-64pfg"
Nov 01 15:17:46.050203 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:17:46.050045    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea1620-559674b8f9-64pfg" event={"ID":"8a1719a8-89fb-44bb-be61-d8c9423e32d3","Type":"ContainerDied","Data":"f8fdd6aa366a877b03824f2fe4c24746a536f535629b1c8c2bb4d557a96ad065"}
Nov 01 15:17:46.050203 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:17:46.050068    4993 scope.go:122] "RemoveContainer" containerID="3fd48622d9c6aee573733a36edf75f8e60895b65870bd153df9201456340fc19"
Nov 01 15:17:46.060912 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:17:46.060892    4993 scope.go:122] "RemoveContainer" containerID="3fd48622d9c6aee573733a36edf75f8e60895b65870bd153df9201456340fc19"
Nov 01 15:17:46.061193 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:17:46.061148    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fd48622d9c6aee573733a36edf75f8e60895b65870bd153df9201456340fc19\": container with ID starting with 3fd48622d9c6aee573733a36edf75f8e60895b65870bd153df9201456340fc19 not found: ID does not exist" containerID="3fd48622d9c6aee573733a36edf75f8e60895b65870bd153df9201456340fc19"
Nov 01 15:17:46.061343 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:17:46.061319    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fd48622d9c6aee573733a36edf75f8e60895b65870bd153df9201456340fc19"} err="failed to get container status \"3fd48622d9c6aee573733a36edf75f8e60895b65870bd153df9201456340fc19\": rpc error: code = NotFound desc = could not find container \"3fd48622d9c6aee573733a36edf75f8e60895b65870bd153df9201456340fc19\": container with ID starting with 3fd48622d9c6aee573733a36edf75f8e60895b65870bd153df9201456340fc19 not found: ID does not exist"
Nov 01 15:17:46.202382 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:17:46.202347    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea1620-559674b8f9-64pfg"]
Nov 01 15:17:46.249769 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:17:46.249725    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ea1620-559674b8f9-64pfg"]
Nov 01 15:17:46.795421 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:17:46.795388    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a1719a8-89fb-44bb-be61-d8c9423e32d3" path="/var/lib/kubelet/pods/8a1719a8-89fb-44bb-be61-d8c9423e32d3/volumes"
Nov 01 15:17:57.502932 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:17:57.502894    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:18:01.061469 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:18:01.061426    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-c89271-559779d48c-jv2zw"]
Nov 01 15:18:01.062106 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:18:01.061680    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c89271-559779d48c-jv2zw" podUID="1abb5edc-e03b-45bb-85c6-c14bdf7f71ac" containerName="maas-api" containerID="cri-o://cea91cc823765731cb6cec81cd6d7e410b2f0a4c2c9a59cab0584758dfbdb438" gracePeriod=30
Nov 01 15:18:01.381536 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:18:01.381503    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c89271-559779d48c-jv2zw"
Nov 01 15:18:01.469136 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:18:01.469098    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1abb5edc-e03b-45bb-85c6-c14bdf7f71ac-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1abb5edc-e03b-45bb-85c6-c14bdf7f71ac-openshift-service-ca\") pod \"1abb5edc-e03b-45bb-85c6-c14bdf7f71ac\" (UID: \"1abb5edc-e03b-45bb-85c6-c14bdf7f71ac\") "
Nov 01 15:18:01.469563 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:18:01.469540    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1abb5edc-e03b-45bb-85c6-c14bdf7f71ac-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1abb5edc-e03b-45bb-85c6-c14bdf7f71ac-maas-api-tls\") pod \"1abb5edc-e03b-45bb-85c6-c14bdf7f71ac\" (UID: \"1abb5edc-e03b-45bb-85c6-c14bdf7f71ac\") "
Nov 01 15:18:01.470136 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:18:01.469749    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1abb5edc-e03b-45bb-85c6-c14bdf7f71ac-openshift-service-ca" pod "1abb5edc-e03b-45bb-85c6-c14bdf7f71ac" (UID: "1abb5edc-e03b-45bb-85c6-c14bdf7f71ac"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:18:01.470388 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:18:01.470373    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1abb5edc-e03b-45bb-85c6-c14bdf7f71ac-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1abb5edc-e03b-45bb-85c6-c14bdf7f71ac-metrics-certs\") pod \"1abb5edc-e03b-45bb-85c6-c14bdf7f71ac\" (UID: \"1abb5edc-e03b-45bb-85c6-c14bdf7f71ac\") "
Nov 01 15:18:01.470843 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:18:01.470825    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1abb5edc-e03b-45bb-85c6-c14bdf7f71ac-kube-api-access-7j7b9\" (UniqueName: \"kubernetes.io/projected/1abb5edc-e03b-45bb-85c6-c14bdf7f71ac-kube-api-access-7j7b9\") pod \"1abb5edc-e03b-45bb-85c6-c14bdf7f71ac\" (UID: \"1abb5edc-e03b-45bb-85c6-c14bdf7f71ac\") "
Nov 01 15:18:01.471799 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:18:01.471778    4993 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1abb5edc-e03b-45bb-85c6-c14bdf7f71ac-openshift-service-ca\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:18:01.484915 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:18:01.484857    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1abb5edc-e03b-45bb-85c6-c14bdf7f71ac-maas-api-tls" pod "1abb5edc-e03b-45bb-85c6-c14bdf7f71ac" (UID: "1abb5edc-e03b-45bb-85c6-c14bdf7f71ac"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:18:01.484915 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:18:01.484898    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1abb5edc-e03b-45bb-85c6-c14bdf7f71ac-kube-api-access-7j7b9" pod "1abb5edc-e03b-45bb-85c6-c14bdf7f71ac" (UID: "1abb5edc-e03b-45bb-85c6-c14bdf7f71ac"). InnerVolumeSpecName "kube-api-access-7j7b9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:18:01.485585 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:18:01.485550    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1abb5edc-e03b-45bb-85c6-c14bdf7f71ac-metrics-certs" pod "1abb5edc-e03b-45bb-85c6-c14bdf7f71ac" (UID: "1abb5edc-e03b-45bb-85c6-c14bdf7f71ac"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:18:01.573346 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:18:01.573310    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1abb5edc-e03b-45bb-85c6-c14bdf7f71ac-maas-api-tls\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:18:01.573346 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:18:01.573341    4993 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1abb5edc-e03b-45bb-85c6-c14bdf7f71ac-metrics-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:18:01.573346 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:18:01.573354    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7j7b9\" (UniqueName: \"kubernetes.io/projected/1abb5edc-e03b-45bb-85c6-c14bdf7f71ac-kube-api-access-7j7b9\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:18:02.156284 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:18:02.156249    4993 generic.go:356] "Generic (PLEG): container finished" podID="1abb5edc-e03b-45bb-85c6-c14bdf7f71ac" containerID="cea91cc823765731cb6cec81cd6d7e410b2f0a4c2c9a59cab0584758dfbdb438" exitCode=0
Nov 01 15:18:02.156816 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:18:02.156308    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c89271-559779d48c-jv2zw"
Nov 01 15:18:02.156816 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:18:02.156340    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c89271-559779d48c-jv2zw" event={"ID":"1abb5edc-e03b-45bb-85c6-c14bdf7f71ac","Type":"ContainerDied","Data":"cea91cc823765731cb6cec81cd6d7e410b2f0a4c2c9a59cab0584758dfbdb438"}
Nov 01 15:18:02.156816 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:18:02.156394    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c89271-559779d48c-jv2zw" event={"ID":"1abb5edc-e03b-45bb-85c6-c14bdf7f71ac","Type":"ContainerDied","Data":"a4da7bc40f124306877e5e9ad97a2fc9ae44406219f99c5800adb0a952aab679"}
Nov 01 15:18:02.156816 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:18:02.156416    4993 scope.go:122] "RemoveContainer" containerID="cea91cc823765731cb6cec81cd6d7e410b2f0a4c2c9a59cab0584758dfbdb438"
Nov 01 15:18:02.168458 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:18:02.168437    4993 scope.go:122] "RemoveContainer" containerID="cea91cc823765731cb6cec81cd6d7e410b2f0a4c2c9a59cab0584758dfbdb438"
Nov 01 15:18:02.168787 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:18:02.168761    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cea91cc823765731cb6cec81cd6d7e410b2f0a4c2c9a59cab0584758dfbdb438\": container with ID starting with cea91cc823765731cb6cec81cd6d7e410b2f0a4c2c9a59cab0584758dfbdb438 not found: ID does not exist" containerID="cea91cc823765731cb6cec81cd6d7e410b2f0a4c2c9a59cab0584758dfbdb438"
Nov 01 15:18:02.168876 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:18:02.168800    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cea91cc823765731cb6cec81cd6d7e410b2f0a4c2c9a59cab0584758dfbdb438"} err="failed to get container status \"cea91cc823765731cb6cec81cd6d7e410b2f0a4c2c9a59cab0584758dfbdb438\": rpc error: code = NotFound desc = could not find container \"cea91cc823765731cb6cec81cd6d7e410b2f0a4c2c9a59cab0584758dfbdb438\": container with ID starting with cea91cc823765731cb6cec81cd6d7e410b2f0a4c2c9a59cab0584758dfbdb438 not found: ID does not exist"
Nov 01 15:18:02.309614 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:18:02.309582    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-c89271-559779d48c-jv2zw"]
Nov 01 15:18:02.312250 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:18:02.312221    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-c89271-559779d48c-jv2zw"]
Nov 01 15:18:02.796498 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:18:02.796465    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1abb5edc-e03b-45bb-85c6-c14bdf7f71ac" path="/var/lib/kubelet/pods/1abb5edc-e03b-45bb-85c6-c14bdf7f71ac/volumes"
Nov 01 15:18:27.501347 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:18:27.501305    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:18:47.459749 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:18:47.459712    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-926972-547bc47bff-cg427"]
Nov 01 15:18:47.460256 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:18:47.459982    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-926972-547bc47bff-cg427" podUID="7240e446-e298-491a-b31f-9f8ded98d74f" containerName="maas-api" containerID="cri-o://b77c1077b86a1d2b6eb7a2aab1eb05da3dbeed237526286f3ab8752cc84cc0b8" gracePeriod=30
Nov 01 15:18:47.751095 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:18:47.751062    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-926972-547bc47bff-cg427"
Nov 01 15:18:47.811649 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:18:47.811613    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7240e446-e298-491a-b31f-9f8ded98d74f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7240e446-e298-491a-b31f-9f8ded98d74f-openshift-service-ca\") pod \"7240e446-e298-491a-b31f-9f8ded98d74f\" (UID: \"7240e446-e298-491a-b31f-9f8ded98d74f\") "
Nov 01 15:18:47.811847 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:18:47.811712    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7240e446-e298-491a-b31f-9f8ded98d74f-kube-api-access-27mz5\" (UniqueName: \"kubernetes.io/projected/7240e446-e298-491a-b31f-9f8ded98d74f-kube-api-access-27mz5\") pod \"7240e446-e298-491a-b31f-9f8ded98d74f\" (UID: \"7240e446-e298-491a-b31f-9f8ded98d74f\") "
Nov 01 15:18:47.811847 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:18:47.811754    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7240e446-e298-491a-b31f-9f8ded98d74f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7240e446-e298-491a-b31f-9f8ded98d74f-metrics-certs\") pod \"7240e446-e298-491a-b31f-9f8ded98d74f\" (UID: \"7240e446-e298-491a-b31f-9f8ded98d74f\") "
Nov 01 15:18:47.811847 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:18:47.811781    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7240e446-e298-491a-b31f-9f8ded98d74f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7240e446-e298-491a-b31f-9f8ded98d74f-maas-api-tls\") pod \"7240e446-e298-491a-b31f-9f8ded98d74f\" (UID: \"7240e446-e298-491a-b31f-9f8ded98d74f\") "
Nov 01 15:18:47.812053 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:18:47.811988    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7240e446-e298-491a-b31f-9f8ded98d74f-openshift-service-ca" pod "7240e446-e298-491a-b31f-9f8ded98d74f" (UID: "7240e446-e298-491a-b31f-9f8ded98d74f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:18:47.812303 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:18:47.812276    4993 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7240e446-e298-491a-b31f-9f8ded98d74f-openshift-service-ca\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:18:47.814066 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:18:47.814030    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7240e446-e298-491a-b31f-9f8ded98d74f-kube-api-access-27mz5" pod "7240e446-e298-491a-b31f-9f8ded98d74f" (UID: "7240e446-e298-491a-b31f-9f8ded98d74f"). InnerVolumeSpecName "kube-api-access-27mz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:18:47.814220 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:18:47.814059    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7240e446-e298-491a-b31f-9f8ded98d74f-metrics-certs" pod "7240e446-e298-491a-b31f-9f8ded98d74f" (UID: "7240e446-e298-491a-b31f-9f8ded98d74f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:18:47.814300 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:18:47.814249    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7240e446-e298-491a-b31f-9f8ded98d74f-maas-api-tls" pod "7240e446-e298-491a-b31f-9f8ded98d74f" (UID: "7240e446-e298-491a-b31f-9f8ded98d74f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:18:47.913998 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:18:47.913562    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27mz5\" (UniqueName: \"kubernetes.io/projected/7240e446-e298-491a-b31f-9f8ded98d74f-kube-api-access-27mz5\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:18:47.913998 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:18:47.913595    4993 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7240e446-e298-491a-b31f-9f8ded98d74f-metrics-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:18:47.913998 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:18:47.913613    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7240e446-e298-491a-b31f-9f8ded98d74f-maas-api-tls\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:18:48.450845 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:18:48.450810    4993 generic.go:356] "Generic (PLEG): container finished" podID="7240e446-e298-491a-b31f-9f8ded98d74f" containerID="b77c1077b86a1d2b6eb7a2aab1eb05da3dbeed237526286f3ab8752cc84cc0b8" exitCode=0
Nov 01 15:18:48.451028 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:18:48.450867    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-926972-547bc47bff-cg427"
Nov 01 15:18:48.451028 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:18:48.450886    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-926972-547bc47bff-cg427" event={"ID":"7240e446-e298-491a-b31f-9f8ded98d74f","Type":"ContainerDied","Data":"b77c1077b86a1d2b6eb7a2aab1eb05da3dbeed237526286f3ab8752cc84cc0b8"}
Nov 01 15:18:48.451028 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:18:48.450945    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-926972-547bc47bff-cg427" event={"ID":"7240e446-e298-491a-b31f-9f8ded98d74f","Type":"ContainerDied","Data":"23f9a616e46b7f246c5b6774764315b68b75ad8f8b8fdef28bb5788060420301"}
Nov 01 15:18:48.451028 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:18:48.450974    4993 scope.go:122] "RemoveContainer" containerID="b77c1077b86a1d2b6eb7a2aab1eb05da3dbeed237526286f3ab8752cc84cc0b8"
Nov 01 15:18:48.461766 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:18:48.461609    4993 scope.go:122] "RemoveContainer" containerID="b77c1077b86a1d2b6eb7a2aab1eb05da3dbeed237526286f3ab8752cc84cc0b8"
Nov 01 15:18:48.461982 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:18:48.461902    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b77c1077b86a1d2b6eb7a2aab1eb05da3dbeed237526286f3ab8752cc84cc0b8\": container with ID starting with b77c1077b86a1d2b6eb7a2aab1eb05da3dbeed237526286f3ab8752cc84cc0b8 not found: ID does not exist" containerID="b77c1077b86a1d2b6eb7a2aab1eb05da3dbeed237526286f3ab8752cc84cc0b8"
Nov 01 15:18:48.461982 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:18:48.461933    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b77c1077b86a1d2b6eb7a2aab1eb05da3dbeed237526286f3ab8752cc84cc0b8"} err="failed to get container status \"b77c1077b86a1d2b6eb7a2aab1eb05da3dbeed237526286f3ab8752cc84cc0b8\": rpc error: code = NotFound desc = could not find container \"b77c1077b86a1d2b6eb7a2aab1eb05da3dbeed237526286f3ab8752cc84cc0b8\": container with ID starting with b77c1077b86a1d2b6eb7a2aab1eb05da3dbeed237526286f3ab8752cc84cc0b8 not found: ID does not exist"
Nov 01 15:18:48.603028 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:18:48.602994    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-926972-547bc47bff-cg427"]
Nov 01 15:18:48.605568 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:18:48.605543    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-926972-547bc47bff-cg427"]
Nov 01 15:18:48.795420 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:18:48.795390    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7240e446-e298-491a-b31f-9f8ded98d74f" path="/var/lib/kubelet/pods/7240e446-e298-491a-b31f-9f8ded98d74f/volumes"
Nov 01 15:18:53.141510 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:18:53.141471    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-e92165-7f6b6f494f-ht5kl"]
Nov 01 15:18:53.142032 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:18:53.141713    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e92165-7f6b6f494f-ht5kl" podUID="7334c698-a18f-4c7a-87e0-5c915c7939d0" containerName="maas-api" containerID="cri-o://563569785b2c8d5adbc6ba4cc0de78952dfc7c60f3af4f56c169565255cc358f" gracePeriod=30
Nov 01 15:18:53.441887 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:18:53.441856    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e92165-7f6b6f494f-ht5kl"
Nov 01 15:18:53.484701 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:18:53.484579    4993 generic.go:356] "Generic (PLEG): container finished" podID="7334c698-a18f-4c7a-87e0-5c915c7939d0" containerID="563569785b2c8d5adbc6ba4cc0de78952dfc7c60f3af4f56c169565255cc358f" exitCode=0
Nov 01 15:18:53.484701 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:18:53.484613    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e92165-7f6b6f494f-ht5kl" event={"ID":"7334c698-a18f-4c7a-87e0-5c915c7939d0","Type":"ContainerDied","Data":"563569785b2c8d5adbc6ba4cc0de78952dfc7c60f3af4f56c169565255cc358f"}
Nov 01 15:18:53.484701 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:18:53.484640    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e92165-7f6b6f494f-ht5kl"
Nov 01 15:18:53.484701 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:18:53.484682    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e92165-7f6b6f494f-ht5kl" event={"ID":"7334c698-a18f-4c7a-87e0-5c915c7939d0","Type":"ContainerDied","Data":"1f0ec4e6784120afb049063d3faa96efad2f51f871a5f4d0d6d43d522b0fb961"}
Nov 01 15:18:53.485010 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:18:53.484714    4993 scope.go:122] "RemoveContainer" containerID="563569785b2c8d5adbc6ba4cc0de78952dfc7c60f3af4f56c169565255cc358f"
Nov 01 15:18:53.496621 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:18:53.496583    4993 scope.go:122] "RemoveContainer" containerID="563569785b2c8d5adbc6ba4cc0de78952dfc7c60f3af4f56c169565255cc358f"
Nov 01 15:18:53.497010 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:18:53.496978    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"563569785b2c8d5adbc6ba4cc0de78952dfc7c60f3af4f56c169565255cc358f\": container with ID starting with 563569785b2c8d5adbc6ba4cc0de78952dfc7c60f3af4f56c169565255cc358f not found: ID does not exist" containerID="563569785b2c8d5adbc6ba4cc0de78952dfc7c60f3af4f56c169565255cc358f"
Nov 01 15:18:53.497111 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:18:53.497023    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"563569785b2c8d5adbc6ba4cc0de78952dfc7c60f3af4f56c169565255cc358f"} err="failed to get container status \"563569785b2c8d5adbc6ba4cc0de78952dfc7c60f3af4f56c169565255cc358f\": rpc error: code = NotFound desc = could not find container \"563569785b2c8d5adbc6ba4cc0de78952dfc7c60f3af4f56c169565255cc358f\": container with ID starting with 563569785b2c8d5adbc6ba4cc0de78952dfc7c60f3af4f56c169565255cc358f not found: ID does not exist"
Nov 01 15:18:53.576122 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:18:53.576073    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7334c698-a18f-4c7a-87e0-5c915c7939d0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7334c698-a18f-4c7a-87e0-5c915c7939d0-openshift-service-ca\") pod \"7334c698-a18f-4c7a-87e0-5c915c7939d0\" (UID: \"7334c698-a18f-4c7a-87e0-5c915c7939d0\") "
Nov 01 15:18:53.576458 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:18:53.576436    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7334c698-a18f-4c7a-87e0-5c915c7939d0-kube-api-access-trnb8\" (UniqueName: \"kubernetes.io/projected/7334c698-a18f-4c7a-87e0-5c915c7939d0-kube-api-access-trnb8\") pod \"7334c698-a18f-4c7a-87e0-5c915c7939d0\" (UID: \"7334c698-a18f-4c7a-87e0-5c915c7939d0\") "
Nov 01 15:18:53.576973 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:18:53.576639    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7334c698-a18f-4c7a-87e0-5c915c7939d0-openshift-service-ca" pod "7334c698-a18f-4c7a-87e0-5c915c7939d0" (UID: "7334c698-a18f-4c7a-87e0-5c915c7939d0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:18:53.577256 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:18:53.577235    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7334c698-a18f-4c7a-87e0-5c915c7939d0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7334c698-a18f-4c7a-87e0-5c915c7939d0-maas-api-tls\") pod \"7334c698-a18f-4c7a-87e0-5c915c7939d0\" (UID: \"7334c698-a18f-4c7a-87e0-5c915c7939d0\") "
Nov 01 15:18:53.577451 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:18:53.577434    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7334c698-a18f-4c7a-87e0-5c915c7939d0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7334c698-a18f-4c7a-87e0-5c915c7939d0-metrics-certs\") pod \"7334c698-a18f-4c7a-87e0-5c915c7939d0\" (UID: \"7334c698-a18f-4c7a-87e0-5c915c7939d0\") "
Nov 01 15:18:53.578017 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:18:53.577994    4993 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7334c698-a18f-4c7a-87e0-5c915c7939d0-openshift-service-ca\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:18:53.581221 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:18:53.580430    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7334c698-a18f-4c7a-87e0-5c915c7939d0-maas-api-tls" pod "7334c698-a18f-4c7a-87e0-5c915c7939d0" (UID: "7334c698-a18f-4c7a-87e0-5c915c7939d0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:18:53.581221 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:18:53.580558    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7334c698-a18f-4c7a-87e0-5c915c7939d0-kube-api-access-trnb8" pod "7334c698-a18f-4c7a-87e0-5c915c7939d0" (UID: "7334c698-a18f-4c7a-87e0-5c915c7939d0"). InnerVolumeSpecName "kube-api-access-trnb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:18:53.581658 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:18:53.581635    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7334c698-a18f-4c7a-87e0-5c915c7939d0-metrics-certs" pod "7334c698-a18f-4c7a-87e0-5c915c7939d0" (UID: "7334c698-a18f-4c7a-87e0-5c915c7939d0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:18:53.679459 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:18:53.679358    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trnb8\" (UniqueName: \"kubernetes.io/projected/7334c698-a18f-4c7a-87e0-5c915c7939d0-kube-api-access-trnb8\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:18:53.679459 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:18:53.679398    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7334c698-a18f-4c7a-87e0-5c915c7939d0-maas-api-tls\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:18:53.679459 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:18:53.679413    4993 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7334c698-a18f-4c7a-87e0-5c915c7939d0-metrics-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:18:53.897674 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:18:53.897640    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-e92165-7f6b6f494f-ht5kl"]
Nov 01 15:18:53.901151 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:18:53.901126    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-e92165-7f6b6f494f-ht5kl"]
Nov 01 15:18:54.795889 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:18:54.795853    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7334c698-a18f-4c7a-87e0-5c915c7939d0" path="/var/lib/kubelet/pods/7334c698-a18f-4c7a-87e0-5c915c7939d0/volumes"
Nov 01 15:18:57.500313 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:18:57.500277    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:19:20.639373 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:20.639280    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jqv2m"]
Nov 01 15:19:20.640008 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:20.639985    4993 cpu_manager.go:414] "containerMap: removing container" podUID="c35e44f6-5a00-4c5e-9eb3-7edc9cfbb937" containerName="cleanup"
Nov 01 15:19:20.640102 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:20.640009    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c35e44f6-5a00-4c5e-9eb3-7edc9cfbb937" containerName="cleanup"
Nov 01 15:19:20.640102 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:20.640042    4993 cpu_manager.go:414] "containerMap: removing container" podUID="1abb5edc-e03b-45bb-85c6-c14bdf7f71ac" containerName="maas-api"
Nov 01 15:19:20.640102 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:20.640050    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1abb5edc-e03b-45bb-85c6-c14bdf7f71ac" containerName="maas-api"
Nov 01 15:19:20.640102 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:20.640064    4993 cpu_manager.go:414] "containerMap: removing container" podUID="c35e44f6-5a00-4c5e-9eb3-7edc9cfbb937" containerName="cleanup"
Nov 01 15:19:20.640102 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:20.640071    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c35e44f6-5a00-4c5e-9eb3-7edc9cfbb937" containerName="cleanup"
Nov 01 15:19:20.640102 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:20.640097    4993 cpu_manager.go:414] "containerMap: removing container" podUID="8a1719a8-89fb-44bb-be61-d8c9423e32d3" containerName="maas-api"
Nov 01 15:19:20.640444 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:20.640105    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a1719a8-89fb-44bb-be61-d8c9423e32d3" containerName="maas-api"
Nov 01 15:19:20.640444 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:20.640118    4993 cpu_manager.go:414] "containerMap: removing container" podUID="7334c698-a18f-4c7a-87e0-5c915c7939d0" containerName="maas-api"
Nov 01 15:19:20.640444 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:20.640126    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7334c698-a18f-4c7a-87e0-5c915c7939d0" containerName="maas-api"
Nov 01 15:19:20.640444 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:20.640138    4993 cpu_manager.go:414] "containerMap: removing container" podUID="c35e44f6-5a00-4c5e-9eb3-7edc9cfbb937" containerName="cleanup"
Nov 01 15:19:20.640444 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:20.640145    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c35e44f6-5a00-4c5e-9eb3-7edc9cfbb937" containerName="cleanup"
Nov 01 15:19:20.640444 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:20.640184    4993 cpu_manager.go:414] "containerMap: removing container" podUID="7240e446-e298-491a-b31f-9f8ded98d74f" containerName="maas-api"
Nov 01 15:19:20.640444 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:20.640191    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7240e446-e298-491a-b31f-9f8ded98d74f" containerName="maas-api"
Nov 01 15:19:20.640444 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:20.640326    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="7240e446-e298-491a-b31f-9f8ded98d74f" containerName="maas-api"
Nov 01 15:19:20.640444 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:20.640344    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="c35e44f6-5a00-4c5e-9eb3-7edc9cfbb937" containerName="cleanup"
Nov 01 15:19:20.640444 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:20.640362    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="1abb5edc-e03b-45bb-85c6-c14bdf7f71ac" containerName="maas-api"
Nov 01 15:19:20.640444 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:20.640372    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="c35e44f6-5a00-4c5e-9eb3-7edc9cfbb937" containerName="cleanup"
Nov 01 15:19:20.640444 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:20.640386    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a1719a8-89fb-44bb-be61-d8c9423e32d3" containerName="maas-api"
Nov 01 15:19:20.640444 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:20.640396    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="7334c698-a18f-4c7a-87e0-5c915c7939d0" containerName="maas-api"
Nov 01 15:19:20.644351 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:20.644328    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jqv2m"
Nov 01 15:19:20.800256 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:20.800213    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d18c69f-6d32-40d7-b857-d0d686235013-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jqv2m\" (UID: \"8d18c69f-6d32-40d7-b857-d0d686235013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jqv2m"
Nov 01 15:19:20.800449 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:20.800265    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d18c69f-6d32-40d7-b857-d0d686235013-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jqv2m\" (UID: \"8d18c69f-6d32-40d7-b857-d0d686235013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jqv2m"
Nov 01 15:19:20.800449 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:20.800336    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d18c69f-6d32-40d7-b857-d0d686235013-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jqv2m\" (UID: \"8d18c69f-6d32-40d7-b857-d0d686235013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jqv2m"
Nov 01 15:19:20.800449 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:20.800398    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d18c69f-6d32-40d7-b857-d0d686235013-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jqv2m\" (UID: \"8d18c69f-6d32-40d7-b857-d0d686235013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jqv2m"
Nov 01 15:19:20.800652 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:20.800513    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbz5b\" (UniqueName: \"kubernetes.io/projected/8d18c69f-6d32-40d7-b857-d0d686235013-kube-api-access-jbz5b\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jqv2m\" (UID: \"8d18c69f-6d32-40d7-b857-d0d686235013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jqv2m"
Nov 01 15:19:20.800652 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:20.800546    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d18c69f-6d32-40d7-b857-d0d686235013-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jqv2m\" (UID: \"8d18c69f-6d32-40d7-b857-d0d686235013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jqv2m"
Nov 01 15:19:20.800773 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:20.800692    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d18c69f-6d32-40d7-b857-d0d686235013-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jqv2m\" (UID: \"8d18c69f-6d32-40d7-b857-d0d686235013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jqv2m"
Nov 01 15:19:20.800773 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:20.800753    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d18c69f-6d32-40d7-b857-d0d686235013-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jqv2m\" (UID: \"8d18c69f-6d32-40d7-b857-d0d686235013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jqv2m"
Nov 01 15:19:20.800881 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:20.800813    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d18c69f-6d32-40d7-b857-d0d686235013-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jqv2m\" (UID: \"8d18c69f-6d32-40d7-b857-d0d686235013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jqv2m"
Nov 01 15:19:20.835615 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:20.835574    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jqv2m"]
Nov 01 15:19:20.901806 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:20.901711    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d18c69f-6d32-40d7-b857-d0d686235013-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jqv2m\" (UID: \"8d18c69f-6d32-40d7-b857-d0d686235013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jqv2m"
Nov 01 15:19:20.901806 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:20.901778    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d18c69f-6d32-40d7-b857-d0d686235013-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jqv2m\" (UID: \"8d18c69f-6d32-40d7-b857-d0d686235013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jqv2m"
Nov 01 15:19:20.901806 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:20.901805    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbz5b\" (UniqueName: \"kubernetes.io/projected/8d18c69f-6d32-40d7-b857-d0d686235013-kube-api-access-jbz5b\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jqv2m\" (UID: \"8d18c69f-6d32-40d7-b857-d0d686235013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jqv2m"
Nov 01 15:19:20.902051 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:20.901828    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d18c69f-6d32-40d7-b857-d0d686235013-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jqv2m\" (UID: \"8d18c69f-6d32-40d7-b857-d0d686235013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jqv2m"
Nov 01 15:19:20.902051 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:20.901938    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d18c69f-6d32-40d7-b857-d0d686235013-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jqv2m\" (UID: \"8d18c69f-6d32-40d7-b857-d0d686235013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jqv2m"
Nov 01 15:19:20.902051 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:20.901977    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d18c69f-6d32-40d7-b857-d0d686235013-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jqv2m\" (UID: \"8d18c69f-6d32-40d7-b857-d0d686235013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jqv2m"
Nov 01 15:19:20.902051 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:20.902035    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d18c69f-6d32-40d7-b857-d0d686235013-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jqv2m\" (UID: \"8d18c69f-6d32-40d7-b857-d0d686235013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jqv2m"
Nov 01 15:19:20.902284 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:20.902104    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d18c69f-6d32-40d7-b857-d0d686235013-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jqv2m\" (UID: \"8d18c69f-6d32-40d7-b857-d0d686235013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jqv2m"
Nov 01 15:19:20.902284 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:20.902127    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d18c69f-6d32-40d7-b857-d0d686235013-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jqv2m\" (UID: \"8d18c69f-6d32-40d7-b857-d0d686235013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jqv2m"
Nov 01 15:19:20.902461 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:20.902429    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d18c69f-6d32-40d7-b857-d0d686235013-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jqv2m\" (UID: \"8d18c69f-6d32-40d7-b857-d0d686235013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jqv2m"
Nov 01 15:19:20.902523 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:20.902489    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d18c69f-6d32-40d7-b857-d0d686235013-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jqv2m\" (UID: \"8d18c69f-6d32-40d7-b857-d0d686235013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jqv2m"
Nov 01 15:19:20.902657 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:20.902630    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d18c69f-6d32-40d7-b857-d0d686235013-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jqv2m\" (UID: \"8d18c69f-6d32-40d7-b857-d0d686235013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jqv2m"
Nov 01 15:19:20.902840 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:20.902689    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d18c69f-6d32-40d7-b857-d0d686235013-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jqv2m\" (UID: \"8d18c69f-6d32-40d7-b857-d0d686235013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jqv2m"
Nov 01 15:19:20.902840 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:20.902836    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d18c69f-6d32-40d7-b857-d0d686235013-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jqv2m\" (UID: \"8d18c69f-6d32-40d7-b857-d0d686235013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jqv2m"
Nov 01 15:19:20.904556 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:20.904532    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d18c69f-6d32-40d7-b857-d0d686235013-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jqv2m\" (UID: \"8d18c69f-6d32-40d7-b857-d0d686235013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jqv2m"
Nov 01 15:19:20.904646 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:20.904622    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d18c69f-6d32-40d7-b857-d0d686235013-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jqv2m\" (UID: \"8d18c69f-6d32-40d7-b857-d0d686235013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jqv2m"
Nov 01 15:19:20.955081 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:20.955046    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d18c69f-6d32-40d7-b857-d0d686235013-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jqv2m\" (UID: \"8d18c69f-6d32-40d7-b857-d0d686235013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jqv2m"
Nov 01 15:19:20.955217 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:20.955085    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbz5b\" (UniqueName: \"kubernetes.io/projected/8d18c69f-6d32-40d7-b857-d0d686235013-kube-api-access-jbz5b\") pod \"maas-default-gateway-openshift-default-7f5667f86c-jqv2m\" (UID: \"8d18c69f-6d32-40d7-b857-d0d686235013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jqv2m"
Nov 01 15:19:20.960253 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:20.960232    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jqv2m"
Nov 01 15:19:21.406044 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:21.406007    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jqv2m"]
Nov 01 15:19:21.411018 ip-10-0-134-37 kubenswrapper[4993]: W1101 15:19:21.410989    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d18c69f_6d32_40d7_b857_d0d686235013.slice/crio-0d9b9d79fbbd55e537e3d4b9cf86ad128093269058bfb571e59a3ec3751e2c3c WatchSource:0}: Error finding container 0d9b9d79fbbd55e537e3d4b9cf86ad128093269058bfb571e59a3ec3751e2c3c: Status 404 returned error can't find the container with id 0d9b9d79fbbd55e537e3d4b9cf86ad128093269058bfb571e59a3ec3751e2c3c
Nov 01 15:19:21.413572 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:21.413538    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 15:19:21.413696 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:21.413623    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 15:19:21.413696 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:21.413669    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 15:19:21.671296 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:21.671203    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jqv2m" event={"ID":"8d18c69f-6d32-40d7-b857-d0d686235013","Type":"ContainerStarted","Data":"a800ea4265bea44c90a5e6b419b173fad026aac68c05ad9a9237d682eb74022d"}
Nov 01 15:19:21.671296 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:21.671261    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jqv2m" event={"ID":"8d18c69f-6d32-40d7-b857-d0d686235013","Type":"ContainerStarted","Data":"0d9b9d79fbbd55e537e3d4b9cf86ad128093269058bfb571e59a3ec3751e2c3c"}
Nov 01 15:19:21.785094 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:21.785031    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jqv2m" podStartSLOduration=1.784999096 podStartE2EDuration="1.784999096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:19:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:19:21.784626515 +0000 UTC m=+3749.114489253" watchObservedRunningTime="2026-11-01 15:19:21.784999096 +0000 UTC m=+3749.114861834"
Nov 01 15:19:21.960864 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:21.960785    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jqv2m"
Nov 01 15:19:22.977311 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:22.977269    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-jqv2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body=
Nov 01 15:19:22.977797 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:22.977346    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jqv2m" podUID="8d18c69f-6d32-40d7-b857-d0d686235013" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused"
Nov 01 15:19:23.961609 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:23.961564    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-jqv2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body=
Nov 01 15:19:23.961909 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:23.961878    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jqv2m" podUID="8d18c69f-6d32-40d7-b857-d0d686235013" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused"
Nov 01 15:19:24.975238 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:24.974609    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-jqv2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body=
Nov 01 15:19:24.975238 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:24.974687    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jqv2m" podUID="8d18c69f-6d32-40d7-b857-d0d686235013" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused"
Nov 01 15:19:25.970579 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:25.970535    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-jqv2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body=
Nov 01 15:19:25.970904 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:25.970870    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jqv2m" podUID="8d18c69f-6d32-40d7-b857-d0d686235013" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused"
Nov 01 15:19:26.964612 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:26.964572    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-jqv2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body=
Nov 01 15:19:26.965663 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:26.965627    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jqv2m" podUID="8d18c69f-6d32-40d7-b857-d0d686235013" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused"
Nov 01 15:19:27.507327 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:19:27.507295    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:19:27.961372 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:27.961277    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-jqv2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body=
Nov 01 15:19:27.961533 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:27.961364    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jqv2m" podUID="8d18c69f-6d32-40d7-b857-d0d686235013" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused"
Nov 01 15:19:28.961362 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:28.961321    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-jqv2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body=
Nov 01 15:19:28.961776 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:28.961392    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jqv2m" podUID="8d18c69f-6d32-40d7-b857-d0d686235013" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused"
Nov 01 15:19:30.191488 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:30.191444    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jqv2m"
Nov 01 15:19:30.192137 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:30.192108    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jqv2m"
Nov 01 15:19:30.196064 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:30.196019    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jqv2m"
Nov 01 15:19:35.735620 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:35.735582    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-scdm5"]
Nov 01 15:19:35.736250 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:35.736232    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="c35e44f6-5a00-4c5e-9eb3-7edc9cfbb937" containerName="cleanup"
Nov 01 15:19:35.739708 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:35.739689    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-scdm5"
Nov 01 15:19:35.795986 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:35.795950    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06108f6d-156c-47cc-b45f-3c421be59030-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-scdm5\" (UID: \"06108f6d-156c-47cc-b45f-3c421be59030\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-scdm5"
Nov 01 15:19:35.796183 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:35.796013    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06108f6d-156c-47cc-b45f-3c421be59030-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-scdm5\" (UID: \"06108f6d-156c-47cc-b45f-3c421be59030\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-scdm5"
Nov 01 15:19:35.796183 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:35.796078    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06108f6d-156c-47cc-b45f-3c421be59030-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-scdm5\" (UID: \"06108f6d-156c-47cc-b45f-3c421be59030\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-scdm5"
Nov 01 15:19:35.796183 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:35.796111    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06108f6d-156c-47cc-b45f-3c421be59030-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-scdm5\" (UID: \"06108f6d-156c-47cc-b45f-3c421be59030\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-scdm5"
Nov 01 15:19:35.796353 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:35.796191    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06108f6d-156c-47cc-b45f-3c421be59030-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-scdm5\" (UID: \"06108f6d-156c-47cc-b45f-3c421be59030\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-scdm5"
Nov 01 15:19:35.796414 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:35.796346    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06108f6d-156c-47cc-b45f-3c421be59030-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-scdm5\" (UID: \"06108f6d-156c-47cc-b45f-3c421be59030\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-scdm5"
Nov 01 15:19:35.796414 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:35.796402    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06108f6d-156c-47cc-b45f-3c421be59030-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-scdm5\" (UID: \"06108f6d-156c-47cc-b45f-3c421be59030\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-scdm5"
Nov 01 15:19:35.796509 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:35.796454    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmbrp\" (UniqueName: \"kubernetes.io/projected/06108f6d-156c-47cc-b45f-3c421be59030-kube-api-access-cmbrp\") pod \"maas-default-gateway-openshift-default-7f5667f86c-scdm5\" (UID: \"06108f6d-156c-47cc-b45f-3c421be59030\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-scdm5"
Nov 01 15:19:35.796556 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:35.796505    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06108f6d-156c-47cc-b45f-3c421be59030-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-scdm5\" (UID: \"06108f6d-156c-47cc-b45f-3c421be59030\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-scdm5"
Nov 01 15:19:35.839726 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:35.839689    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-scdm5"]
Nov 01 15:19:35.897865 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:35.897818    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06108f6d-156c-47cc-b45f-3c421be59030-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-scdm5\" (UID: \"06108f6d-156c-47cc-b45f-3c421be59030\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-scdm5"
Nov 01 15:19:35.898071 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:35.897882    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06108f6d-156c-47cc-b45f-3c421be59030-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-scdm5\" (UID: \"06108f6d-156c-47cc-b45f-3c421be59030\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-scdm5"
Nov 01 15:19:35.898071 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:35.897935    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06108f6d-156c-47cc-b45f-3c421be59030-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-scdm5\" (UID: \"06108f6d-156c-47cc-b45f-3c421be59030\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-scdm5"
Nov 01 15:19:35.898071 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:35.897999    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06108f6d-156c-47cc-b45f-3c421be59030-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-scdm5\" (UID: \"06108f6d-156c-47cc-b45f-3c421be59030\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-scdm5"
Nov 01 15:19:35.898277 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:35.898149    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06108f6d-156c-47cc-b45f-3c421be59030-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-scdm5\" (UID: \"06108f6d-156c-47cc-b45f-3c421be59030\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-scdm5"
Nov 01 15:19:35.898277 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:35.898225    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06108f6d-156c-47cc-b45f-3c421be59030-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-scdm5\" (UID: \"06108f6d-156c-47cc-b45f-3c421be59030\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-scdm5"
Nov 01 15:19:35.898277 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:35.898270    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmbrp\" (UniqueName: \"kubernetes.io/projected/06108f6d-156c-47cc-b45f-3c421be59030-kube-api-access-cmbrp\") pod \"maas-default-gateway-openshift-default-7f5667f86c-scdm5\" (UID: \"06108f6d-156c-47cc-b45f-3c421be59030\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-scdm5"
Nov 01 15:19:35.898443 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:35.898312    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06108f6d-156c-47cc-b45f-3c421be59030-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-scdm5\" (UID: \"06108f6d-156c-47cc-b45f-3c421be59030\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-scdm5"
Nov 01 15:19:35.898443 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:35.898338    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06108f6d-156c-47cc-b45f-3c421be59030-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-scdm5\" (UID: \"06108f6d-156c-47cc-b45f-3c421be59030\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-scdm5"
Nov 01 15:19:35.898618 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:35.898580    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06108f6d-156c-47cc-b45f-3c421be59030-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-scdm5\" (UID: \"06108f6d-156c-47cc-b45f-3c421be59030\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-scdm5"
Nov 01 15:19:35.898773 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:35.898712    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06108f6d-156c-47cc-b45f-3c421be59030-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-scdm5\" (UID: \"06108f6d-156c-47cc-b45f-3c421be59030\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-scdm5"
Nov 01 15:19:35.898773 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:35.898756    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06108f6d-156c-47cc-b45f-3c421be59030-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-scdm5\" (UID: \"06108f6d-156c-47cc-b45f-3c421be59030\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-scdm5"
Nov 01 15:19:35.899024 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:35.899004    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06108f6d-156c-47cc-b45f-3c421be59030-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-scdm5\" (UID: \"06108f6d-156c-47cc-b45f-3c421be59030\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-scdm5"
Nov 01 15:19:35.899157 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:35.899127    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06108f6d-156c-47cc-b45f-3c421be59030-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-scdm5\" (UID: \"06108f6d-156c-47cc-b45f-3c421be59030\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-scdm5"
Nov 01 15:19:35.900683 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:35.900657    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06108f6d-156c-47cc-b45f-3c421be59030-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-scdm5\" (UID: \"06108f6d-156c-47cc-b45f-3c421be59030\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-scdm5"
Nov 01 15:19:35.901391 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:35.901369    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06108f6d-156c-47cc-b45f-3c421be59030-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-scdm5\" (UID: \"06108f6d-156c-47cc-b45f-3c421be59030\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-scdm5"
Nov 01 15:19:35.951813 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:35.951770    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06108f6d-156c-47cc-b45f-3c421be59030-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-scdm5\" (UID: \"06108f6d-156c-47cc-b45f-3c421be59030\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-scdm5"
Nov 01 15:19:35.952023 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:35.951999    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmbrp\" (UniqueName: \"kubernetes.io/projected/06108f6d-156c-47cc-b45f-3c421be59030-kube-api-access-cmbrp\") pod \"maas-default-gateway-openshift-default-7f5667f86c-scdm5\" (UID: \"06108f6d-156c-47cc-b45f-3c421be59030\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-scdm5"
Nov 01 15:19:36.053653 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:36.053555    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-scdm5"
Nov 01 15:19:36.480964 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:36.480877    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-scdm5"]
Nov 01 15:19:36.487289 ip-10-0-134-37 kubenswrapper[4993]: W1101 15:19:36.487254    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06108f6d_156c_47cc_b45f_3c421be59030.slice/crio-43a24eb8c8795fc2676abd46cdfb3bae5289a891ddaec40fe576990a311ca44b WatchSource:0}: Error finding container 43a24eb8c8795fc2676abd46cdfb3bae5289a891ddaec40fe576990a311ca44b: Status 404 returned error can't find the container with id 43a24eb8c8795fc2676abd46cdfb3bae5289a891ddaec40fe576990a311ca44b
Nov 01 15:19:36.490103 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:36.490064    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 15:19:36.490245 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:36.490134    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 15:19:36.490245 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:36.490164    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 15:19:36.775380 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:36.775339    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-scdm5" event={"ID":"06108f6d-156c-47cc-b45f-3c421be59030","Type":"ContainerStarted","Data":"533dbef1755d193b2354d9f4659ddfffde69613ceed7dce0304838b6894d7a57"}
Nov 01 15:19:36.775808 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:36.775408    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-scdm5" event={"ID":"06108f6d-156c-47cc-b45f-3c421be59030","Type":"ContainerStarted","Data":"43a24eb8c8795fc2676abd46cdfb3bae5289a891ddaec40fe576990a311ca44b"}
Nov 01 15:19:36.889929 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:36.889858    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-scdm5" podStartSLOduration=1.8898401630000001 podStartE2EDuration="1.889840163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:19:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:19:36.886832951 +0000 UTC m=+3764.216695690" watchObservedRunningTime="2026-11-01 15:19:36.889840163 +0000 UTC m=+3764.219702900"
Nov 01 15:19:37.054614 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:37.054510    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-scdm5"
Nov 01 15:19:37.056148 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:37.056115    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-scdm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Nov 01 15:19:37.056272 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:37.056202    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-scdm5" podUID="06108f6d-156c-47cc-b45f-3c421be59030" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Nov 01 15:19:38.058567 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:38.058526    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-scdm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Nov 01 15:19:38.059189 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:38.059128    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-scdm5" podUID="06108f6d-156c-47cc-b45f-3c421be59030" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Nov 01 15:19:39.060194 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:39.055478    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-scdm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Nov 01 15:19:39.060194 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:39.055538    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-scdm5" podUID="06108f6d-156c-47cc-b45f-3c421be59030" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Nov 01 15:19:40.055599 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:40.055557    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-scdm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Nov 01 15:19:40.055936 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:40.055903    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-scdm5" podUID="06108f6d-156c-47cc-b45f-3c421be59030" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Nov 01 15:19:41.058609 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:41.058557    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-scdm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Nov 01 15:19:41.059226 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:41.059188    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-scdm5" podUID="06108f6d-156c-47cc-b45f-3c421be59030" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Nov 01 15:19:42.059446 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:42.059407    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-scdm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Nov 01 15:19:42.060069 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:42.060036    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-scdm5" podUID="06108f6d-156c-47cc-b45f-3c421be59030" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Nov 01 15:19:43.054909 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:43.054844    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-scdm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Nov 01 15:19:43.054909 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:43.054906    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-scdm5" podUID="06108f6d-156c-47cc-b45f-3c421be59030" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Nov 01 15:19:44.053939 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:44.053900    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-scdm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Nov 01 15:19:44.054366 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:44.053962    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-scdm5" podUID="06108f6d-156c-47cc-b45f-3c421be59030" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Nov 01 15:19:45.091659 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:45.091541    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-scdm5"
Nov 01 15:19:45.092969 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:45.092932    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-scdm5"
Nov 01 15:19:45.093666 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:19:45.093641    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-scdm5"
Nov 01 15:19:57.501244 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:19:57.501209    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:20:03.508642 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:20:03.508600    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-15ad1e-7d6f47cdd6-klmqs"]
Nov 01 15:20:03.509152 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:20:03.508813    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-15ad1e-7d6f47cdd6-klmqs" podUID="6a9774c0-61a8-41a4-8f4c-93bc06a03e94" containerName="maas-api" containerID="cri-o://d9fcc5b7c5739242ca4fcfe7074329ce08cdd3c1e772e986e7e5dfc8d399b190" gracePeriod=30
Nov 01 15:20:03.805295 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:20:03.805265    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-15ad1e-7d6f47cdd6-klmqs"
Nov 01 15:20:03.873253 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:20:03.873212    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a9774c0-61a8-41a4-8f4c-93bc06a03e94-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a9774c0-61a8-41a4-8f4c-93bc06a03e94-metrics-certs\") pod \"6a9774c0-61a8-41a4-8f4c-93bc06a03e94\" (UID: \"6a9774c0-61a8-41a4-8f4c-93bc06a03e94\") "
Nov 01 15:20:03.873476 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:20:03.873294    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a9774c0-61a8-41a4-8f4c-93bc06a03e94-kube-api-access-bb8f2\" (UniqueName: \"kubernetes.io/projected/6a9774c0-61a8-41a4-8f4c-93bc06a03e94-kube-api-access-bb8f2\") pod \"6a9774c0-61a8-41a4-8f4c-93bc06a03e94\" (UID: \"6a9774c0-61a8-41a4-8f4c-93bc06a03e94\") "
Nov 01 15:20:03.873476 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:20:03.873321    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a9774c0-61a8-41a4-8f4c-93bc06a03e94-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a9774c0-61a8-41a4-8f4c-93bc06a03e94-openshift-service-ca\") pod \"6a9774c0-61a8-41a4-8f4c-93bc06a03e94\" (UID: \"6a9774c0-61a8-41a4-8f4c-93bc06a03e94\") "
Nov 01 15:20:03.873476 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:20:03.873364    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a9774c0-61a8-41a4-8f4c-93bc06a03e94-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a9774c0-61a8-41a4-8f4c-93bc06a03e94-maas-api-tls\") pod \"6a9774c0-61a8-41a4-8f4c-93bc06a03e94\" (UID: \"6a9774c0-61a8-41a4-8f4c-93bc06a03e94\") "
Nov 01 15:20:03.873825 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:20:03.873789    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a9774c0-61a8-41a4-8f4c-93bc06a03e94-openshift-service-ca" pod "6a9774c0-61a8-41a4-8f4c-93bc06a03e94" (UID: "6a9774c0-61a8-41a4-8f4c-93bc06a03e94"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:20:03.873965 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:20:03.873944    4993 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a9774c0-61a8-41a4-8f4c-93bc06a03e94-openshift-service-ca\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:20:03.875570 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:20:03.875543    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a9774c0-61a8-41a4-8f4c-93bc06a03e94-kube-api-access-bb8f2" pod "6a9774c0-61a8-41a4-8f4c-93bc06a03e94" (UID: "6a9774c0-61a8-41a4-8f4c-93bc06a03e94"). InnerVolumeSpecName "kube-api-access-bb8f2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:20:03.875709 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:20:03.875673    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a9774c0-61a8-41a4-8f4c-93bc06a03e94-maas-api-tls" pod "6a9774c0-61a8-41a4-8f4c-93bc06a03e94" (UID: "6a9774c0-61a8-41a4-8f4c-93bc06a03e94"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:20:03.876010 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:20:03.875993    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a9774c0-61a8-41a4-8f4c-93bc06a03e94-metrics-certs" pod "6a9774c0-61a8-41a4-8f4c-93bc06a03e94" (UID: "6a9774c0-61a8-41a4-8f4c-93bc06a03e94"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:20:03.975407 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:20:03.975364    4993 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a9774c0-61a8-41a4-8f4c-93bc06a03e94-metrics-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:20:03.975407 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:20:03.975397    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bb8f2\" (UniqueName: \"kubernetes.io/projected/6a9774c0-61a8-41a4-8f4c-93bc06a03e94-kube-api-access-bb8f2\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:20:03.975407 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:20:03.975408    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a9774c0-61a8-41a4-8f4c-93bc06a03e94-maas-api-tls\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:20:03.982235 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:20:03.982204    4993 generic.go:356] "Generic (PLEG): container finished" podID="6a9774c0-61a8-41a4-8f4c-93bc06a03e94" containerID="d9fcc5b7c5739242ca4fcfe7074329ce08cdd3c1e772e986e7e5dfc8d399b190" exitCode=0
Nov 01 15:20:03.982419 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:20:03.982279    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-15ad1e-7d6f47cdd6-klmqs" event={"ID":"6a9774c0-61a8-41a4-8f4c-93bc06a03e94","Type":"ContainerDied","Data":"d9fcc5b7c5739242ca4fcfe7074329ce08cdd3c1e772e986e7e5dfc8d399b190"}
Nov 01 15:20:03.982419 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:20:03.982309    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-15ad1e-7d6f47cdd6-klmqs"
Nov 01 15:20:03.982419 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:20:03.982325    4993 scope.go:122] "RemoveContainer" containerID="d9fcc5b7c5739242ca4fcfe7074329ce08cdd3c1e772e986e7e5dfc8d399b190"
Nov 01 15:20:03.982581 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:20:03.982316    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-15ad1e-7d6f47cdd6-klmqs" event={"ID":"6a9774c0-61a8-41a4-8f4c-93bc06a03e94","Type":"ContainerDied","Data":"dd92da8677911eaf7e8ed81ba257b92a9d002e864c44b9be9054bbc63c1972fc"}
Nov 01 15:20:03.995063 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:20:03.995040    4993 scope.go:122] "RemoveContainer" containerID="d9fcc5b7c5739242ca4fcfe7074329ce08cdd3c1e772e986e7e5dfc8d399b190"
Nov 01 15:20:03.995348 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:20:03.995325    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9fcc5b7c5739242ca4fcfe7074329ce08cdd3c1e772e986e7e5dfc8d399b190\": container with ID starting with d9fcc5b7c5739242ca4fcfe7074329ce08cdd3c1e772e986e7e5dfc8d399b190 not found: ID does not exist" containerID="d9fcc5b7c5739242ca4fcfe7074329ce08cdd3c1e772e986e7e5dfc8d399b190"
Nov 01 15:20:03.995408 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:20:03.995357    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9fcc5b7c5739242ca4fcfe7074329ce08cdd3c1e772e986e7e5dfc8d399b190"} err="failed to get container status \"d9fcc5b7c5739242ca4fcfe7074329ce08cdd3c1e772e986e7e5dfc8d399b190\": rpc error: code = NotFound desc = could not find container \"d9fcc5b7c5739242ca4fcfe7074329ce08cdd3c1e772e986e7e5dfc8d399b190\": container with ID starting with d9fcc5b7c5739242ca4fcfe7074329ce08cdd3c1e772e986e7e5dfc8d399b190 not found: ID does not exist"
Nov 01 15:20:04.150214 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:20:04.150115    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-15ad1e-7d6f47cdd6-klmqs"]
Nov 01 15:20:04.162194 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:20:04.162150    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-15ad1e-7d6f47cdd6-klmqs"]
Nov 01 15:20:04.795795 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:20:04.795758    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a9774c0-61a8-41a4-8f4c-93bc06a03e94" path="/var/lib/kubelet/pods/6a9774c0-61a8-41a4-8f4c-93bc06a03e94/volumes"
Nov 01 15:20:27.501074 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:20:27.501039    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:20:39.336744 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:20:39.334382    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-af2dd2-865d4789d4-2cvq5"]
Nov 01 15:20:39.338599 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:20:39.338568    4993 cpu_manager.go:414] "containerMap: removing container" podUID="6a9774c0-61a8-41a4-8f4c-93bc06a03e94" containerName="maas-api"
Nov 01 15:20:39.338768 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:20:39.338753    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a9774c0-61a8-41a4-8f4c-93bc06a03e94" containerName="maas-api"
Nov 01 15:20:39.339039 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:20:39.339027    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a9774c0-61a8-41a4-8f4c-93bc06a03e94" containerName="maas-api"
Nov 01 15:20:39.343227 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:20:39.343205    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-af2dd2-865d4789d4-2cvq5"
Nov 01 15:20:39.381221 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:20:39.377943    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4808fe31-322b-4ae7-bf3b-f46024a38aa2-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-af2dd2-865d4789d4-2cvq5\" (UID: \"4808fe31-322b-4ae7-bf3b-f46024a38aa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-af2dd2-865d4789d4-2cvq5"
Nov 01 15:20:39.381221 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:20:39.377996    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4808fe31-322b-4ae7-bf3b-f46024a38aa2-maas-api-tls\") pod \"maas-api-e2e-extmod-b-af2dd2-865d4789d4-2cvq5\" (UID: \"4808fe31-322b-4ae7-bf3b-f46024a38aa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-af2dd2-865d4789d4-2cvq5"
Nov 01 15:20:39.381221 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:20:39.378050    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4808fe31-322b-4ae7-bf3b-f46024a38aa2-metrics-certs\") pod \"maas-api-e2e-extmod-b-af2dd2-865d4789d4-2cvq5\" (UID: \"4808fe31-322b-4ae7-bf3b-f46024a38aa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-af2dd2-865d4789d4-2cvq5"
Nov 01 15:20:39.381221 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:20:39.378230    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wqjv\" (UniqueName: \"kubernetes.io/projected/4808fe31-322b-4ae7-bf3b-f46024a38aa2-kube-api-access-5wqjv\") pod \"maas-api-e2e-extmod-b-af2dd2-865d4789d4-2cvq5\" (UID: \"4808fe31-322b-4ae7-bf3b-f46024a38aa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-af2dd2-865d4789d4-2cvq5"
Nov 01 15:20:39.446620 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:20:39.446578    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-af2dd2-serving-cert\""
Nov 01 15:20:39.448963 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:20:39.448939    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-af2dd2-865d4789d4-2cvq5"]
Nov 01 15:20:39.511189 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:20:39.505934    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wqjv\" (UniqueName: \"kubernetes.io/projected/4808fe31-322b-4ae7-bf3b-f46024a38aa2-kube-api-access-5wqjv\") pod \"maas-api-e2e-extmod-b-af2dd2-865d4789d4-2cvq5\" (UID: \"4808fe31-322b-4ae7-bf3b-f46024a38aa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-af2dd2-865d4789d4-2cvq5"
Nov 01 15:20:39.511189 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:20:39.506017    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4808fe31-322b-4ae7-bf3b-f46024a38aa2-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-af2dd2-865d4789d4-2cvq5\" (UID: \"4808fe31-322b-4ae7-bf3b-f46024a38aa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-af2dd2-865d4789d4-2cvq5"
Nov 01 15:20:39.511189 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:20:39.506055    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4808fe31-322b-4ae7-bf3b-f46024a38aa2-maas-api-tls\") pod \"maas-api-e2e-extmod-b-af2dd2-865d4789d4-2cvq5\" (UID: \"4808fe31-322b-4ae7-bf3b-f46024a38aa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-af2dd2-865d4789d4-2cvq5"
Nov 01 15:20:39.527229 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:20:39.512514    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4808fe31-322b-4ae7-bf3b-f46024a38aa2-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-af2dd2-865d4789d4-2cvq5\" (UID: \"4808fe31-322b-4ae7-bf3b-f46024a38aa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-af2dd2-865d4789d4-2cvq5"
Nov 01 15:20:39.529208 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:20:39.527535    4993 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-af2dd2-serving-cert: secret "maas-api-e2e-extmod-b-af2dd2-serving-cert" not found
Nov 01 15:20:39.529911 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:20:39.529896    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4808fe31-322b-4ae7-bf3b-f46024a38aa2-maas-api-tls podName:4808fe31-322b-4ae7-bf3b-f46024a38aa2 nodeName:}" failed. No retries permitted until 2026-11-01 15:20:40.02987049 +0000 UTC m=+3827.359733225 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4808fe31-322b-4ae7-bf3b-f46024a38aa2-maas-api-tls") pod "maas-api-e2e-extmod-b-af2dd2-865d4789d4-2cvq5" (UID: "4808fe31-322b-4ae7-bf3b-f46024a38aa2") : secret "maas-api-e2e-extmod-b-af2dd2-serving-cert" not found
Nov 01 15:20:39.543208 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:20:39.506137    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4808fe31-322b-4ae7-bf3b-f46024a38aa2-metrics-certs\") pod \"maas-api-e2e-extmod-b-af2dd2-865d4789d4-2cvq5\" (UID: \"4808fe31-322b-4ae7-bf3b-f46024a38aa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-af2dd2-865d4789d4-2cvq5"
Nov 01 15:20:39.545311 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:20:39.543850    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4808fe31-322b-4ae7-bf3b-f46024a38aa2-metrics-certs\") pod \"maas-api-e2e-extmod-b-af2dd2-865d4789d4-2cvq5\" (UID: \"4808fe31-322b-4ae7-bf3b-f46024a38aa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-af2dd2-865d4789d4-2cvq5"
Nov 01 15:20:39.593236 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:20:39.583023    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wqjv\" (UniqueName: \"kubernetes.io/projected/4808fe31-322b-4ae7-bf3b-f46024a38aa2-kube-api-access-5wqjv\") pod \"maas-api-e2e-extmod-b-af2dd2-865d4789d4-2cvq5\" (UID: \"4808fe31-322b-4ae7-bf3b-f46024a38aa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-af2dd2-865d4789d4-2cvq5"
Nov 01 15:20:40.046023 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:20:40.045988    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4808fe31-322b-4ae7-bf3b-f46024a38aa2-maas-api-tls\") pod \"maas-api-e2e-extmod-b-af2dd2-865d4789d4-2cvq5\" (UID: \"4808fe31-322b-4ae7-bf3b-f46024a38aa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-af2dd2-865d4789d4-2cvq5"
Nov 01 15:20:40.048575 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:20:40.048542    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4808fe31-322b-4ae7-bf3b-f46024a38aa2-maas-api-tls\") pod \"maas-api-e2e-extmod-b-af2dd2-865d4789d4-2cvq5\" (UID: \"4808fe31-322b-4ae7-bf3b-f46024a38aa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-af2dd2-865d4789d4-2cvq5"
Nov 01 15:20:40.048749 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:20:40.048730    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-af2dd2-865d4789d4-2cvq5"
Nov 01 15:20:40.785544 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:20:40.784721    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-af2dd2-865d4789d4-2cvq5"]
Nov 01 15:20:40.839323 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:20:40.838844    4993 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 15:20:41.400287 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:20:41.400241    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-af2dd2-865d4789d4-2cvq5" event={"ID":"4808fe31-322b-4ae7-bf3b-f46024a38aa2","Type":"ContainerStarted","Data":"a7cdabd4c74df9f0a7c7eeac48182f3e1e8172a4fab21c3e3f40c34fc154af78"}
Nov 01 15:20:42.409384 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:20:42.409344    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-af2dd2-865d4789d4-2cvq5" event={"ID":"4808fe31-322b-4ae7-bf3b-f46024a38aa2","Type":"ContainerStarted","Data":"c527f0cc6281fdb509985094a6c3b0cb2b801190c2b3eff2fc13c795336bff2d"}
Nov 01 15:20:42.409881 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:20:42.409524    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-af2dd2-865d4789d4-2cvq5"
Nov 01 15:20:42.517386 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:20:42.517324    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-af2dd2-865d4789d4-2cvq5" podStartSLOduration=2.436334009 podStartE2EDuration="3.517306969s" totalImagesPullingTime="1.08097296s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:20:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:20:42.516312687 +0000 UTC m=+3829.846175433" watchObservedRunningTime="2026-11-01 15:20:42.517306969 +0000 UTC m=+3829.847169707"
Nov 01 15:20:48.422178 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:20:48.422146    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-af2dd2-865d4789d4-2cvq5"
Nov 01 15:20:57.499636 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:20:57.499590    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:21:27.500641 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:21:27.500593    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:21:37.847657 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:21:37.847616    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fc8c64-7676f5f848-8mrgd"]
Nov 01 15:21:37.851540 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:21:37.851520    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fc8c64-7676f5f848-8mrgd"
Nov 01 15:21:37.947575 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:21:37.947528    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-fc8c64-serving-cert\""
Nov 01 15:21:37.954423 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:21:37.954387    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fc8c64-7676f5f848-8mrgd"]
Nov 01 15:21:37.957729 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:21:37.957689    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92088a5b-f048-40be-9425-25c069453656-maas-api-tls\") pod \"maas-api-e2e-inf-a-fc8c64-7676f5f848-8mrgd\" (UID: \"92088a5b-f048-40be-9425-25c069453656\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fc8c64-7676f5f848-8mrgd"
Nov 01 15:21:37.957931 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:21:37.957907    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92088a5b-f048-40be-9425-25c069453656-openshift-service-ca\") pod \"maas-api-e2e-inf-a-fc8c64-7676f5f848-8mrgd\" (UID: \"92088a5b-f048-40be-9425-25c069453656\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fc8c64-7676f5f848-8mrgd"
Nov 01 15:21:37.958062 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:21:37.958049    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92088a5b-f048-40be-9425-25c069453656-metrics-certs\") pod \"maas-api-e2e-inf-a-fc8c64-7676f5f848-8mrgd\" (UID: \"92088a5b-f048-40be-9425-25c069453656\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fc8c64-7676f5f848-8mrgd"
Nov 01 15:21:37.958110 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:21:37.958091    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtcxm\" (UniqueName: \"kubernetes.io/projected/92088a5b-f048-40be-9425-25c069453656-kube-api-access-qtcxm\") pod \"maas-api-e2e-inf-a-fc8c64-7676f5f848-8mrgd\" (UID: \"92088a5b-f048-40be-9425-25c069453656\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fc8c64-7676f5f848-8mrgd"
Nov 01 15:21:38.059537 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:21:38.059493    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92088a5b-f048-40be-9425-25c069453656-metrics-certs\") pod \"maas-api-e2e-inf-a-fc8c64-7676f5f848-8mrgd\" (UID: \"92088a5b-f048-40be-9425-25c069453656\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fc8c64-7676f5f848-8mrgd"
Nov 01 15:21:38.059720 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:21:38.059562    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtcxm\" (UniqueName: \"kubernetes.io/projected/92088a5b-f048-40be-9425-25c069453656-kube-api-access-qtcxm\") pod \"maas-api-e2e-inf-a-fc8c64-7676f5f848-8mrgd\" (UID: \"92088a5b-f048-40be-9425-25c069453656\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fc8c64-7676f5f848-8mrgd"
Nov 01 15:21:38.059720 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:21:38.059637    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92088a5b-f048-40be-9425-25c069453656-maas-api-tls\") pod \"maas-api-e2e-inf-a-fc8c64-7676f5f848-8mrgd\" (UID: \"92088a5b-f048-40be-9425-25c069453656\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fc8c64-7676f5f848-8mrgd"
Nov 01 15:21:38.059720 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:21:38.059708    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92088a5b-f048-40be-9425-25c069453656-openshift-service-ca\") pod \"maas-api-e2e-inf-a-fc8c64-7676f5f848-8mrgd\" (UID: \"92088a5b-f048-40be-9425-25c069453656\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fc8c64-7676f5f848-8mrgd"
Nov 01 15:21:38.059881 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:21:38.059724    4993 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-fc8c64-serving-cert: secret "maas-api-e2e-inf-a-fc8c64-serving-cert" not found
Nov 01 15:21:38.059881 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:21:38.059832    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92088a5b-f048-40be-9425-25c069453656-maas-api-tls podName:92088a5b-f048-40be-9425-25c069453656 nodeName:}" failed. No retries permitted until 2026-11-01 15:21:38.559810762 +0000 UTC m=+3885.889673493 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/92088a5b-f048-40be-9425-25c069453656-maas-api-tls") pod "maas-api-e2e-inf-a-fc8c64-7676f5f848-8mrgd" (UID: "92088a5b-f048-40be-9425-25c069453656") : secret "maas-api-e2e-inf-a-fc8c64-serving-cert" not found
Nov 01 15:21:38.060419 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:21:38.060394    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92088a5b-f048-40be-9425-25c069453656-openshift-service-ca\") pod \"maas-api-e2e-inf-a-fc8c64-7676f5f848-8mrgd\" (UID: \"92088a5b-f048-40be-9425-25c069453656\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fc8c64-7676f5f848-8mrgd"
Nov 01 15:21:38.062048 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:21:38.062024    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92088a5b-f048-40be-9425-25c069453656-metrics-certs\") pod \"maas-api-e2e-inf-a-fc8c64-7676f5f848-8mrgd\" (UID: \"92088a5b-f048-40be-9425-25c069453656\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fc8c64-7676f5f848-8mrgd"
Nov 01 15:21:38.117491 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:21:38.117415    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtcxm\" (UniqueName: \"kubernetes.io/projected/92088a5b-f048-40be-9425-25c069453656-kube-api-access-qtcxm\") pod \"maas-api-e2e-inf-a-fc8c64-7676f5f848-8mrgd\" (UID: \"92088a5b-f048-40be-9425-25c069453656\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fc8c64-7676f5f848-8mrgd"
Nov 01 15:21:38.565518 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:21:38.565483    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92088a5b-f048-40be-9425-25c069453656-maas-api-tls\") pod \"maas-api-e2e-inf-a-fc8c64-7676f5f848-8mrgd\" (UID: \"92088a5b-f048-40be-9425-25c069453656\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fc8c64-7676f5f848-8mrgd"
Nov 01 15:21:38.568087 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:21:38.568055    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92088a5b-f048-40be-9425-25c069453656-maas-api-tls\") pod \"maas-api-e2e-inf-a-fc8c64-7676f5f848-8mrgd\" (UID: \"92088a5b-f048-40be-9425-25c069453656\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fc8c64-7676f5f848-8mrgd"
Nov 01 15:21:38.767462 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:21:38.767426    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fc8c64-7676f5f848-8mrgd"
Nov 01 15:21:39.140125 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:21:39.140040    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fc8c64-7676f5f848-8mrgd"]
Nov 01 15:21:39.145972 ip-10-0-134-37 kubenswrapper[4993]: W1101 15:21:39.145947    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92088a5b_f048_40be_9425_25c069453656.slice/crio-acef18ca3da1ce26b4fa033dca25e81ff2185b5139fb72b5aeb4e7ab8d9f9a33 WatchSource:0}: Error finding container acef18ca3da1ce26b4fa033dca25e81ff2185b5139fb72b5aeb4e7ab8d9f9a33: Status 404 returned error can't find the container with id acef18ca3da1ce26b4fa033dca25e81ff2185b5139fb72b5aeb4e7ab8d9f9a33
Nov 01 15:21:39.795410 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:21:39.795358    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fc8c64-7676f5f848-8mrgd" event={"ID":"92088a5b-f048-40be-9425-25c069453656","Type":"ContainerStarted","Data":"acef18ca3da1ce26b4fa033dca25e81ff2185b5139fb72b5aeb4e7ab8d9f9a33"}
Nov 01 15:21:40.809014 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:21:40.808962    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fc8c64-7676f5f848-8mrgd" event={"ID":"92088a5b-f048-40be-9425-25c069453656","Type":"ContainerStarted","Data":"58d1ab95d86677f008a5360cef0c26f821ef80b83d033266b96d5cd0cb922c1d"}
Nov 01 15:21:40.809548 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:21:40.809050    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fc8c64-7676f5f848-8mrgd"
Nov 01 15:21:40.919027 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:21:40.918976    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fc8c64-7676f5f848-8mrgd" podStartSLOduration=3.06556973 podStartE2EDuration="3.918963218s" totalImagesPullingTime="853.393488ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:21:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:21:40.915770828 +0000 UTC m=+3888.245633567" watchObservedRunningTime="2026-11-01 15:21:40.918963218 +0000 UTC m=+3888.248825955"
Nov 01 15:21:46.820275 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:21:46.820243    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fc8c64-7676f5f848-8mrgd"
Nov 01 15:21:53.349234 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:21:53.349204    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xmwtp_bb59051d-c2df-4362-a157-00f14f915e79/console-operator/3.log"
Nov 01 15:21:53.356434 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:21:53.356405    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xmwtp_bb59051d-c2df-4362-a157-00f14f915e79/console-operator/3.log"
Nov 01 15:21:55.729803 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:21:55.729768    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3d5a5b-69d698df5d-jxs6q"]
Nov 01 15:21:55.733900 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:21:55.733883    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d5a5b-69d698df5d-jxs6q"
Nov 01 15:21:55.788650 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:21:55.788612    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3d5a5b-69d698df5d-jxs6q"]
Nov 01 15:21:55.808123 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:21:55.808080    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-3d5a5b-serving-cert\""
Nov 01 15:21:55.859231 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:21:55.859160    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99cda18e-b745-4634-a2b3-a524343c0085-metrics-certs\") pod \"maas-api-e2e-inf-a-3d5a5b-69d698df5d-jxs6q\" (UID: \"99cda18e-b745-4634-a2b3-a524343c0085\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d5a5b-69d698df5d-jxs6q"
Nov 01 15:21:55.859429 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:21:55.859266    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8r9k\" (UniqueName: \"kubernetes.io/projected/99cda18e-b745-4634-a2b3-a524343c0085-kube-api-access-c8r9k\") pod \"maas-api-e2e-inf-a-3d5a5b-69d698df5d-jxs6q\" (UID: \"99cda18e-b745-4634-a2b3-a524343c0085\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d5a5b-69d698df5d-jxs6q"
Nov 01 15:21:55.859429 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:21:55.859318    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99cda18e-b745-4634-a2b3-a524343c0085-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3d5a5b-69d698df5d-jxs6q\" (UID: \"99cda18e-b745-4634-a2b3-a524343c0085\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d5a5b-69d698df5d-jxs6q"
Nov 01 15:21:55.859553 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:21:55.859490    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99cda18e-b745-4634-a2b3-a524343c0085-maas-api-tls\") pod \"maas-api-e2e-inf-a-3d5a5b-69d698df5d-jxs6q\" (UID: \"99cda18e-b745-4634-a2b3-a524343c0085\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d5a5b-69d698df5d-jxs6q"
Nov 01 15:21:55.960775 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:21:55.960741    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99cda18e-b745-4634-a2b3-a524343c0085-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3d5a5b-69d698df5d-jxs6q\" (UID: \"99cda18e-b745-4634-a2b3-a524343c0085\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d5a5b-69d698df5d-jxs6q"
Nov 01 15:21:55.960983 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:21:55.960909    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99cda18e-b745-4634-a2b3-a524343c0085-maas-api-tls\") pod \"maas-api-e2e-inf-a-3d5a5b-69d698df5d-jxs6q\" (UID: \"99cda18e-b745-4634-a2b3-a524343c0085\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d5a5b-69d698df5d-jxs6q"
Nov 01 15:21:55.960983 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:21:55.960961    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99cda18e-b745-4634-a2b3-a524343c0085-metrics-certs\") pod \"maas-api-e2e-inf-a-3d5a5b-69d698df5d-jxs6q\" (UID: \"99cda18e-b745-4634-a2b3-a524343c0085\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d5a5b-69d698df5d-jxs6q"
Nov 01 15:21:55.961156 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:21:55.961010    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8r9k\" (UniqueName: \"kubernetes.io/projected/99cda18e-b745-4634-a2b3-a524343c0085-kube-api-access-c8r9k\") pod \"maas-api-e2e-inf-a-3d5a5b-69d698df5d-jxs6q\" (UID: \"99cda18e-b745-4634-a2b3-a524343c0085\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d5a5b-69d698df5d-jxs6q"
Nov 01 15:21:55.961567 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:21:55.961384    4993 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-3d5a5b-serving-cert: secret "maas-api-e2e-inf-a-3d5a5b-serving-cert" not found
Nov 01 15:21:55.961567 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:21:55.961453    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99cda18e-b745-4634-a2b3-a524343c0085-maas-api-tls podName:99cda18e-b745-4634-a2b3-a524343c0085 nodeName:}" failed. No retries permitted until 2026-11-01 15:21:56.461434359 +0000 UTC m=+3903.791297093 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/99cda18e-b745-4634-a2b3-a524343c0085-maas-api-tls") pod "maas-api-e2e-inf-a-3d5a5b-69d698df5d-jxs6q" (UID: "99cda18e-b745-4634-a2b3-a524343c0085") : secret "maas-api-e2e-inf-a-3d5a5b-serving-cert" not found
Nov 01 15:21:55.961987 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:21:55.961962    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99cda18e-b745-4634-a2b3-a524343c0085-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3d5a5b-69d698df5d-jxs6q\" (UID: \"99cda18e-b745-4634-a2b3-a524343c0085\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d5a5b-69d698df5d-jxs6q"
Nov 01 15:21:55.963854 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:21:55.963827    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99cda18e-b745-4634-a2b3-a524343c0085-metrics-certs\") pod \"maas-api-e2e-inf-a-3d5a5b-69d698df5d-jxs6q\" (UID: \"99cda18e-b745-4634-a2b3-a524343c0085\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d5a5b-69d698df5d-jxs6q"
Nov 01 15:21:56.015419 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:21:56.015383    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8r9k\" (UniqueName: \"kubernetes.io/projected/99cda18e-b745-4634-a2b3-a524343c0085-kube-api-access-c8r9k\") pod \"maas-api-e2e-inf-a-3d5a5b-69d698df5d-jxs6q\" (UID: \"99cda18e-b745-4634-a2b3-a524343c0085\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d5a5b-69d698df5d-jxs6q"
Nov 01 15:21:56.468635 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:21:56.468524    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99cda18e-b745-4634-a2b3-a524343c0085-maas-api-tls\") pod \"maas-api-e2e-inf-a-3d5a5b-69d698df5d-jxs6q\" (UID: \"99cda18e-b745-4634-a2b3-a524343c0085\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d5a5b-69d698df5d-jxs6q"
Nov 01 15:21:56.471229 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:21:56.471163    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99cda18e-b745-4634-a2b3-a524343c0085-maas-api-tls\") pod \"maas-api-e2e-inf-a-3d5a5b-69d698df5d-jxs6q\" (UID: \"99cda18e-b745-4634-a2b3-a524343c0085\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d5a5b-69d698df5d-jxs6q"
Nov 01 15:21:56.647314 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:21:56.647279    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d5a5b-69d698df5d-jxs6q"
Nov 01 15:21:57.034254 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:21:57.034213    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3d5a5b-69d698df5d-jxs6q"]
Nov 01 15:21:57.039412 ip-10-0-134-37 kubenswrapper[4993]: W1101 15:21:57.039383    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99cda18e_b745_4634_a2b3_a524343c0085.slice/crio-857cbbd6984af6ed48824753bc5b8857da0d3ce2ffe56c652667758f4f2d2aa8 WatchSource:0}: Error finding container 857cbbd6984af6ed48824753bc5b8857da0d3ce2ffe56c652667758f4f2d2aa8: Status 404 returned error can't find the container with id 857cbbd6984af6ed48824753bc5b8857da0d3ce2ffe56c652667758f4f2d2aa8
Nov 01 15:21:57.501318 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:21:57.501284    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:21:57.922499 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:21:57.922396    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d5a5b-69d698df5d-jxs6q" event={"ID":"99cda18e-b745-4634-a2b3-a524343c0085","Type":"ContainerStarted","Data":"857cbbd6984af6ed48824753bc5b8857da0d3ce2ffe56c652667758f4f2d2aa8"}
Nov 01 15:21:58.932824 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:21:58.932785    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d5a5b-69d698df5d-jxs6q" event={"ID":"99cda18e-b745-4634-a2b3-a524343c0085","Type":"ContainerStarted","Data":"3835a0a7d8aa147315a44ff9370d0b0af1f37a145404b7cb4ccb980f062a70d7"}
Nov 01 15:21:58.933396 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:21:58.932870    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d5a5b-69d698df5d-jxs6q"
Nov 01 15:21:59.042280 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:21:59.042215    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d5a5b-69d698df5d-jxs6q" podStartSLOduration=2.978335478 podStartE2EDuration="4.042198552s" totalImagesPullingTime="1.063863074s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:21:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:21:59.040216511 +0000 UTC m=+3906.370079276" watchObservedRunningTime="2026-11-01 15:21:59.042198552 +0000 UTC m=+3906.372061288"
Nov 01 15:22:04.945576 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:22:04.945497    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d5a5b-69d698df5d-jxs6q"
Nov 01 15:22:27.500064 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:22:27.500027    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:22:43.872191 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:22:43.864300    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-4pmtn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 15:22:43.872191 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:22:43.864395    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4pmtn" podUID="306b0405-fb73-435a-ae73-42a90cd2d2f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 15:22:57.499943 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:22:57.499907    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:23:27.501293 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:23:27.501253    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:23:51.789700 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:23:51.789662    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-775095-7448b7c55c-kvffx"]
Nov 01 15:23:51.794081 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:23:51.794056    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-775095-7448b7c55c-kvffx"
Nov 01 15:23:51.873803 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:23:51.873768    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b92fac92-7c11-49c1-839b-84598e2532c6-metrics-certs\") pod \"maas-api-e2e-inf-a-775095-7448b7c55c-kvffx\" (UID: \"b92fac92-7c11-49c1-839b-84598e2532c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-775095-7448b7c55c-kvffx"
Nov 01 15:23:51.873969 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:23:51.873831    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq629\" (UniqueName: \"kubernetes.io/projected/b92fac92-7c11-49c1-839b-84598e2532c6-kube-api-access-gq629\") pod \"maas-api-e2e-inf-a-775095-7448b7c55c-kvffx\" (UID: \"b92fac92-7c11-49c1-839b-84598e2532c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-775095-7448b7c55c-kvffx"
Nov 01 15:23:51.874045 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:23:51.873959    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b92fac92-7c11-49c1-839b-84598e2532c6-maas-api-tls\") pod \"maas-api-e2e-inf-a-775095-7448b7c55c-kvffx\" (UID: \"b92fac92-7c11-49c1-839b-84598e2532c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-775095-7448b7c55c-kvffx"
Nov 01 15:23:51.874109 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:23:51.874056    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b92fac92-7c11-49c1-839b-84598e2532c6-openshift-service-ca\") pod \"maas-api-e2e-inf-a-775095-7448b7c55c-kvffx\" (UID: \"b92fac92-7c11-49c1-839b-84598e2532c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-775095-7448b7c55c-kvffx"
Nov 01 15:23:51.893004 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:23:51.892976    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-775095-serving-cert\""
Nov 01 15:23:51.896264 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:23:51.896237    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-775095-7448b7c55c-kvffx"]
Nov 01 15:23:51.975299 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:23:51.975268    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b92fac92-7c11-49c1-839b-84598e2532c6-metrics-certs\") pod \"maas-api-e2e-inf-a-775095-7448b7c55c-kvffx\" (UID: \"b92fac92-7c11-49c1-839b-84598e2532c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-775095-7448b7c55c-kvffx"
Nov 01 15:23:51.975491 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:23:51.975335    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq629\" (UniqueName: \"kubernetes.io/projected/b92fac92-7c11-49c1-839b-84598e2532c6-kube-api-access-gq629\") pod \"maas-api-e2e-inf-a-775095-7448b7c55c-kvffx\" (UID: \"b92fac92-7c11-49c1-839b-84598e2532c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-775095-7448b7c55c-kvffx"
Nov 01 15:23:51.975491 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:23:51.975378    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b92fac92-7c11-49c1-839b-84598e2532c6-maas-api-tls\") pod \"maas-api-e2e-inf-a-775095-7448b7c55c-kvffx\" (UID: \"b92fac92-7c11-49c1-839b-84598e2532c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-775095-7448b7c55c-kvffx"
Nov 01 15:23:51.975491 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:23:51.975402    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b92fac92-7c11-49c1-839b-84598e2532c6-openshift-service-ca\") pod \"maas-api-e2e-inf-a-775095-7448b7c55c-kvffx\" (UID: \"b92fac92-7c11-49c1-839b-84598e2532c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-775095-7448b7c55c-kvffx"
Nov 01 15:23:51.975666 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:23:51.975508    4993 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-775095-serving-cert: secret "maas-api-e2e-inf-a-775095-serving-cert" not found
Nov 01 15:23:51.975666 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:23:51.975608    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b92fac92-7c11-49c1-839b-84598e2532c6-maas-api-tls podName:b92fac92-7c11-49c1-839b-84598e2532c6 nodeName:}" failed. No retries permitted until 2026-11-01 15:23:52.475589103 +0000 UTC m=+4019.805451837 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b92fac92-7c11-49c1-839b-84598e2532c6-maas-api-tls") pod "maas-api-e2e-inf-a-775095-7448b7c55c-kvffx" (UID: "b92fac92-7c11-49c1-839b-84598e2532c6") : secret "maas-api-e2e-inf-a-775095-serving-cert" not found
Nov 01 15:23:51.976162 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:23:51.976141    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b92fac92-7c11-49c1-839b-84598e2532c6-openshift-service-ca\") pod \"maas-api-e2e-inf-a-775095-7448b7c55c-kvffx\" (UID: \"b92fac92-7c11-49c1-839b-84598e2532c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-775095-7448b7c55c-kvffx"
Nov 01 15:23:51.977738 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:23:51.977717    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b92fac92-7c11-49c1-839b-84598e2532c6-metrics-certs\") pod \"maas-api-e2e-inf-a-775095-7448b7c55c-kvffx\" (UID: \"b92fac92-7c11-49c1-839b-84598e2532c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-775095-7448b7c55c-kvffx"
Nov 01 15:23:52.031505 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:23:52.031474    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq629\" (UniqueName: \"kubernetes.io/projected/b92fac92-7c11-49c1-839b-84598e2532c6-kube-api-access-gq629\") pod \"maas-api-e2e-inf-a-775095-7448b7c55c-kvffx\" (UID: \"b92fac92-7c11-49c1-839b-84598e2532c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-775095-7448b7c55c-kvffx"
Nov 01 15:23:52.481434 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:23:52.481398    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b92fac92-7c11-49c1-839b-84598e2532c6-maas-api-tls\") pod \"maas-api-e2e-inf-a-775095-7448b7c55c-kvffx\" (UID: \"b92fac92-7c11-49c1-839b-84598e2532c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-775095-7448b7c55c-kvffx"
Nov 01 15:23:52.483883 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:23:52.483857    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b92fac92-7c11-49c1-839b-84598e2532c6-maas-api-tls\") pod \"maas-api-e2e-inf-a-775095-7448b7c55c-kvffx\" (UID: \"b92fac92-7c11-49c1-839b-84598e2532c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-775095-7448b7c55c-kvffx"
Nov 01 15:23:52.708100 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:23:52.708065    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-775095-7448b7c55c-kvffx"
Nov 01 15:23:53.085005 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:23:53.084926    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-775095-7448b7c55c-kvffx"]
Nov 01 15:23:53.088848 ip-10-0-134-37 kubenswrapper[4993]: W1101 15:23:53.088814    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb92fac92_7c11_49c1_839b_84598e2532c6.slice/crio-e07b03e802f5862144b10bce1fddc00923018b85b60793ec37fc8848892f6a92 WatchSource:0}: Error finding container e07b03e802f5862144b10bce1fddc00923018b85b60793ec37fc8848892f6a92: Status 404 returned error can't find the container with id e07b03e802f5862144b10bce1fddc00923018b85b60793ec37fc8848892f6a92
Nov 01 15:23:53.700706 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:23:53.700655    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-775095-7448b7c55c-kvffx" event={"ID":"b92fac92-7c11-49c1-839b-84598e2532c6","Type":"ContainerStarted","Data":"e07b03e802f5862144b10bce1fddc00923018b85b60793ec37fc8848892f6a92"}
Nov 01 15:23:54.708819 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:23:54.708780    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-775095-7448b7c55c-kvffx" event={"ID":"b92fac92-7c11-49c1-839b-84598e2532c6","Type":"ContainerStarted","Data":"9540751fbed22baaefd6a12aa570d901f030e839b5021d33fbf2fee04cca2d5d"}
Nov 01 15:23:54.709358 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:23:54.708911    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-775095-7448b7c55c-kvffx"
Nov 01 15:23:54.818149 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:23:54.818092    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-775095-7448b7c55c-kvffx" podStartSLOduration=2.913528362 podStartE2EDuration="3.818079511s" totalImagesPullingTime="904.551149ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:23:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:23:54.816430893 +0000 UTC m=+4022.146293631" watchObservedRunningTime="2026-11-01 15:23:54.818079511 +0000 UTC m=+4022.147942294"
Nov 01 15:23:57.503248 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:23:57.503214    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:24:00.722705 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:24:00.722667    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-775095-7448b7c55c-kvffx"
Nov 01 15:24:09.577839 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:24:09.577800    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5feb76-7d5f658777-cb5kl"]
Nov 01 15:24:09.582875 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:24:09.582852    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5feb76-7d5f658777-cb5kl"
Nov 01 15:24:09.631806 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:24:09.631775    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-5feb76-serving-cert\""
Nov 01 15:24:09.680716 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:24:09.680675    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5feb76-7d5f658777-cb5kl"]
Nov 01 15:24:09.681281 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:24:09.681255    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad273a86-1e46-4cc2-a4b9-20a42c57e688-metrics-certs\") pod \"maas-api-e2e-inf-a-5feb76-7d5f658777-cb5kl\" (UID: \"ad273a86-1e46-4cc2-a4b9-20a42c57e688\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5feb76-7d5f658777-cb5kl"
Nov 01 15:24:09.681394 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:24:09.681295    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad273a86-1e46-4cc2-a4b9-20a42c57e688-openshift-service-ca\") pod \"maas-api-e2e-inf-a-5feb76-7d5f658777-cb5kl\" (UID: \"ad273a86-1e46-4cc2-a4b9-20a42c57e688\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5feb76-7d5f658777-cb5kl"
Nov 01 15:24:09.681484 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:24:09.681461    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtzwk\" (UniqueName: \"kubernetes.io/projected/ad273a86-1e46-4cc2-a4b9-20a42c57e688-kube-api-access-dtzwk\") pod \"maas-api-e2e-inf-a-5feb76-7d5f658777-cb5kl\" (UID: \"ad273a86-1e46-4cc2-a4b9-20a42c57e688\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5feb76-7d5f658777-cb5kl"
Nov 01 15:24:09.681542 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:24:09.681520    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad273a86-1e46-4cc2-a4b9-20a42c57e688-maas-api-tls\") pod \"maas-api-e2e-inf-a-5feb76-7d5f658777-cb5kl\" (UID: \"ad273a86-1e46-4cc2-a4b9-20a42c57e688\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5feb76-7d5f658777-cb5kl"
Nov 01 15:24:09.783410 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:24:09.783368    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad273a86-1e46-4cc2-a4b9-20a42c57e688-metrics-certs\") pod \"maas-api-e2e-inf-a-5feb76-7d5f658777-cb5kl\" (UID: \"ad273a86-1e46-4cc2-a4b9-20a42c57e688\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5feb76-7d5f658777-cb5kl"
Nov 01 15:24:09.783629 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:24:09.783425    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad273a86-1e46-4cc2-a4b9-20a42c57e688-openshift-service-ca\") pod \"maas-api-e2e-inf-a-5feb76-7d5f658777-cb5kl\" (UID: \"ad273a86-1e46-4cc2-a4b9-20a42c57e688\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5feb76-7d5f658777-cb5kl"
Nov 01 15:24:09.783629 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:24:09.783540    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtzwk\" (UniqueName: \"kubernetes.io/projected/ad273a86-1e46-4cc2-a4b9-20a42c57e688-kube-api-access-dtzwk\") pod \"maas-api-e2e-inf-a-5feb76-7d5f658777-cb5kl\" (UID: \"ad273a86-1e46-4cc2-a4b9-20a42c57e688\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5feb76-7d5f658777-cb5kl"
Nov 01 15:24:09.783629 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:24:09.783601    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad273a86-1e46-4cc2-a4b9-20a42c57e688-maas-api-tls\") pod \"maas-api-e2e-inf-a-5feb76-7d5f658777-cb5kl\" (UID: \"ad273a86-1e46-4cc2-a4b9-20a42c57e688\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5feb76-7d5f658777-cb5kl"
Nov 01 15:24:09.783804 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:24:09.783694    4993 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-5feb76-serving-cert: secret "maas-api-e2e-inf-a-5feb76-serving-cert" not found
Nov 01 15:24:09.783804 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:24:09.783777    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad273a86-1e46-4cc2-a4b9-20a42c57e688-maas-api-tls podName:ad273a86-1e46-4cc2-a4b9-20a42c57e688 nodeName:}" failed. No retries permitted until 2026-11-01 15:24:10.283755983 +0000 UTC m=+4037.613618712 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ad273a86-1e46-4cc2-a4b9-20a42c57e688-maas-api-tls") pod "maas-api-e2e-inf-a-5feb76-7d5f658777-cb5kl" (UID: "ad273a86-1e46-4cc2-a4b9-20a42c57e688") : secret "maas-api-e2e-inf-a-5feb76-serving-cert" not found
Nov 01 15:24:09.784465 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:24:09.784442    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad273a86-1e46-4cc2-a4b9-20a42c57e688-openshift-service-ca\") pod \"maas-api-e2e-inf-a-5feb76-7d5f658777-cb5kl\" (UID: \"ad273a86-1e46-4cc2-a4b9-20a42c57e688\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5feb76-7d5f658777-cb5kl"
Nov 01 15:24:09.786402 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:24:09.786380    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad273a86-1e46-4cc2-a4b9-20a42c57e688-metrics-certs\") pod \"maas-api-e2e-inf-a-5feb76-7d5f658777-cb5kl\" (UID: \"ad273a86-1e46-4cc2-a4b9-20a42c57e688\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5feb76-7d5f658777-cb5kl"
Nov 01 15:24:09.837454 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:24:09.837375    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtzwk\" (UniqueName: \"kubernetes.io/projected/ad273a86-1e46-4cc2-a4b9-20a42c57e688-kube-api-access-dtzwk\") pod \"maas-api-e2e-inf-a-5feb76-7d5f658777-cb5kl\" (UID: \"ad273a86-1e46-4cc2-a4b9-20a42c57e688\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5feb76-7d5f658777-cb5kl"
Nov 01 15:24:10.290336 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:24:10.290288    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad273a86-1e46-4cc2-a4b9-20a42c57e688-maas-api-tls\") pod \"maas-api-e2e-inf-a-5feb76-7d5f658777-cb5kl\" (UID: \"ad273a86-1e46-4cc2-a4b9-20a42c57e688\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5feb76-7d5f658777-cb5kl"
Nov 01 15:24:10.292886 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:24:10.292853    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad273a86-1e46-4cc2-a4b9-20a42c57e688-maas-api-tls\") pod \"maas-api-e2e-inf-a-5feb76-7d5f658777-cb5kl\" (UID: \"ad273a86-1e46-4cc2-a4b9-20a42c57e688\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5feb76-7d5f658777-cb5kl"
Nov 01 15:24:10.497132 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:24:10.497092    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5feb76-7d5f658777-cb5kl"
Nov 01 15:24:10.867413 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:24:10.867319    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5feb76-7d5f658777-cb5kl"]
Nov 01 15:24:10.873338 ip-10-0-134-37 kubenswrapper[4993]: W1101 15:24:10.873304    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad273a86_1e46_4cc2_a4b9_20a42c57e688.slice/crio-a8f3a836a21a272c8aaf1bab2e9a9e54c816b70301e4f8b7a117c97bbc261973 WatchSource:0}: Error finding container a8f3a836a21a272c8aaf1bab2e9a9e54c816b70301e4f8b7a117c97bbc261973: Status 404 returned error can't find the container with id a8f3a836a21a272c8aaf1bab2e9a9e54c816b70301e4f8b7a117c97bbc261973
Nov 01 15:24:11.818240 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:24:11.818202    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5feb76-7d5f658777-cb5kl" event={"ID":"ad273a86-1e46-4cc2-a4b9-20a42c57e688","Type":"ContainerStarted","Data":"a8f3a836a21a272c8aaf1bab2e9a9e54c816b70301e4f8b7a117c97bbc261973"}
Nov 01 15:24:12.826358 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:24:12.826316    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5feb76-7d5f658777-cb5kl" event={"ID":"ad273a86-1e46-4cc2-a4b9-20a42c57e688","Type":"ContainerStarted","Data":"6174be6f3fb7733761ba2a3f969f64b908566e39b97c534ddf408e5e7379b415"}
Nov 01 15:24:12.826891 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:24:12.826510    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5feb76-7d5f658777-cb5kl"
Nov 01 15:24:12.936268 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:24:12.936211    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5feb76-7d5f658777-cb5kl" podStartSLOduration=2.975679847 podStartE2EDuration="3.936195652s" totalImagesPullingTime="960.515805ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:24:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:24:12.933281026 +0000 UTC m=+4040.263143764" watchObservedRunningTime="2026-11-01 15:24:12.936195652 +0000 UTC m=+4040.266058389"
Nov 01 15:24:18.839493 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:24:18.839460    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5feb76-7d5f658777-cb5kl"
Nov 01 15:24:27.500925 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:24:27.500886    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:24:45.456568 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:24:45.456534    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-89e5ef-68ddd79477-br8sg"]
Nov 01 15:24:45.460514 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:24:45.460497    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89e5ef-68ddd79477-br8sg"
Nov 01 15:24:45.556528 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:24:45.556493    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-89e5ef-serving-cert\""
Nov 01 15:24:45.559710 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:24:45.559678    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-89e5ef-68ddd79477-br8sg"]
Nov 01 15:24:45.598125 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:24:45.598063    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17dcaed4-caa8-4a09-84bb-f445a8dc3a2f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-89e5ef-68ddd79477-br8sg\" (UID: \"17dcaed4-caa8-4a09-84bb-f445a8dc3a2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89e5ef-68ddd79477-br8sg"
Nov 01 15:24:45.598301 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:24:45.598240    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17dcaed4-caa8-4a09-84bb-f445a8dc3a2f-maas-api-tls\") pod \"maas-api-e2e-inf-b-89e5ef-68ddd79477-br8sg\" (UID: \"17dcaed4-caa8-4a09-84bb-f445a8dc3a2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89e5ef-68ddd79477-br8sg"
Nov 01 15:24:45.598301 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:24:45.598276    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sm99x\" (UniqueName: \"kubernetes.io/projected/17dcaed4-caa8-4a09-84bb-f445a8dc3a2f-kube-api-access-sm99x\") pod \"maas-api-e2e-inf-b-89e5ef-68ddd79477-br8sg\" (UID: \"17dcaed4-caa8-4a09-84bb-f445a8dc3a2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89e5ef-68ddd79477-br8sg"
Nov 01 15:24:45.598473 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:24:45.598452    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17dcaed4-caa8-4a09-84bb-f445a8dc3a2f-metrics-certs\") pod \"maas-api-e2e-inf-b-89e5ef-68ddd79477-br8sg\" (UID: \"17dcaed4-caa8-4a09-84bb-f445a8dc3a2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89e5ef-68ddd79477-br8sg"
Nov 01 15:24:45.699500 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:24:45.699462    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17dcaed4-caa8-4a09-84bb-f445a8dc3a2f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-89e5ef-68ddd79477-br8sg\" (UID: \"17dcaed4-caa8-4a09-84bb-f445a8dc3a2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89e5ef-68ddd79477-br8sg"
Nov 01 15:24:45.699688 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:24:45.699523    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17dcaed4-caa8-4a09-84bb-f445a8dc3a2f-maas-api-tls\") pod \"maas-api-e2e-inf-b-89e5ef-68ddd79477-br8sg\" (UID: \"17dcaed4-caa8-4a09-84bb-f445a8dc3a2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89e5ef-68ddd79477-br8sg"
Nov 01 15:24:45.699688 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:24:45.699542    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm99x\" (UniqueName: \"kubernetes.io/projected/17dcaed4-caa8-4a09-84bb-f445a8dc3a2f-kube-api-access-sm99x\") pod \"maas-api-e2e-inf-b-89e5ef-68ddd79477-br8sg\" (UID: \"17dcaed4-caa8-4a09-84bb-f445a8dc3a2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89e5ef-68ddd79477-br8sg"
Nov 01 15:24:45.699688 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:24:45.699606    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17dcaed4-caa8-4a09-84bb-f445a8dc3a2f-metrics-certs\") pod \"maas-api-e2e-inf-b-89e5ef-68ddd79477-br8sg\" (UID: \"17dcaed4-caa8-4a09-84bb-f445a8dc3a2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89e5ef-68ddd79477-br8sg"
Nov 01 15:24:45.700239 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:24:45.700208    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17dcaed4-caa8-4a09-84bb-f445a8dc3a2f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-89e5ef-68ddd79477-br8sg\" (UID: \"17dcaed4-caa8-4a09-84bb-f445a8dc3a2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89e5ef-68ddd79477-br8sg"
Nov 01 15:24:45.702054 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:24:45.702028    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17dcaed4-caa8-4a09-84bb-f445a8dc3a2f-metrics-certs\") pod \"maas-api-e2e-inf-b-89e5ef-68ddd79477-br8sg\" (UID: \"17dcaed4-caa8-4a09-84bb-f445a8dc3a2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89e5ef-68ddd79477-br8sg"
Nov 01 15:24:45.702205 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:24:45.702158    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17dcaed4-caa8-4a09-84bb-f445a8dc3a2f-maas-api-tls\") pod \"maas-api-e2e-inf-b-89e5ef-68ddd79477-br8sg\" (UID: \"17dcaed4-caa8-4a09-84bb-f445a8dc3a2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89e5ef-68ddd79477-br8sg"
Nov 01 15:24:45.752501 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:24:45.752465    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sm99x\" (UniqueName: \"kubernetes.io/projected/17dcaed4-caa8-4a09-84bb-f445a8dc3a2f-kube-api-access-sm99x\") pod \"maas-api-e2e-inf-b-89e5ef-68ddd79477-br8sg\" (UID: \"17dcaed4-caa8-4a09-84bb-f445a8dc3a2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89e5ef-68ddd79477-br8sg"
Nov 01 15:24:45.773789 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:24:45.773754    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89e5ef-68ddd79477-br8sg"
Nov 01 15:24:46.147253 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:24:46.147162    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-89e5ef-68ddd79477-br8sg"]
Nov 01 15:24:46.150822 ip-10-0-134-37 kubenswrapper[4993]: W1101 15:24:46.150792    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17dcaed4_caa8_4a09_84bb_f445a8dc3a2f.slice/crio-3a71dd49e035acb49ff2cbfebd12c7114ae9e124b4b2feece6cf6db98c8ba4fb WatchSource:0}: Error finding container 3a71dd49e035acb49ff2cbfebd12c7114ae9e124b4b2feece6cf6db98c8ba4fb: Status 404 returned error can't find the container with id 3a71dd49e035acb49ff2cbfebd12c7114ae9e124b4b2feece6cf6db98c8ba4fb
Nov 01 15:24:47.060406 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:24:47.060377    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89e5ef-68ddd79477-br8sg" event={"ID":"17dcaed4-caa8-4a09-84bb-f445a8dc3a2f","Type":"ContainerStarted","Data":"3a71dd49e035acb49ff2cbfebd12c7114ae9e124b4b2feece6cf6db98c8ba4fb"}
Nov 01 15:24:48.068221 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:24:48.068182    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89e5ef-68ddd79477-br8sg" event={"ID":"17dcaed4-caa8-4a09-84bb-f445a8dc3a2f","Type":"ContainerStarted","Data":"b2d0767bb6734a22bc0d3ce6639e3ca0d85eae045ffdc05f78f06c0e15c18417"}
Nov 01 15:24:48.068653 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:24:48.068321    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89e5ef-68ddd79477-br8sg"
Nov 01 15:24:48.177409 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:24:48.177354    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89e5ef-68ddd79477-br8sg" podStartSLOduration=2.3020006459999998 podStartE2EDuration="3.17732705s" totalImagesPullingTime="875.326404ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:24:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:24:48.175726168 +0000 UTC m=+4075.505588927" watchObservedRunningTime="2026-11-01 15:24:48.17732705 +0000 UTC m=+4075.507189786"
Nov 01 15:24:54.080220 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:24:54.080163    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89e5ef-68ddd79477-br8sg"
Nov 01 15:24:57.500901 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:24:57.500858    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:25:27.501098 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:25:27.501007    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:25:57.501412 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:25:57.501370    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:26:27.503326 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:26:27.503283    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:26:53.395702 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:26:53.395670    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xmwtp_bb59051d-c2df-4362-a157-00f14f915e79/console-operator/3.log"
Nov 01 15:26:53.405438 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:26:53.405416    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xmwtp_bb59051d-c2df-4362-a157-00f14f915e79/console-operator/3.log"
Nov 01 15:26:57.500038 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:26:57.499997    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:27:27.500732 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:27:27.500690    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:27:40.335243 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:27:40.335208    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fc8c64-7676f5f848-8mrgd"]
Nov 01 15:27:40.335759 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:27:40.335409    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fc8c64-7676f5f848-8mrgd" podUID="92088a5b-f048-40be-9425-25c069453656" containerName="maas-api" containerID="cri-o://58d1ab95d86677f008a5360cef0c26f821ef80b83d033266b96d5cd0cb922c1d" gracePeriod=30
Nov 01 15:27:40.638244 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:27:40.638212    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fc8c64-7676f5f848-8mrgd"
Nov 01 15:27:40.726639 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:27:40.726591    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92088a5b-f048-40be-9425-25c069453656-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92088a5b-f048-40be-9425-25c069453656-maas-api-tls\") pod \"92088a5b-f048-40be-9425-25c069453656\" (UID: \"92088a5b-f048-40be-9425-25c069453656\") "
Nov 01 15:27:40.726851 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:27:40.726696    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92088a5b-f048-40be-9425-25c069453656-metrics-certs\" (UniqueName: \"kubernetes.io/secret/92088a5b-f048-40be-9425-25c069453656-metrics-certs\") pod \"92088a5b-f048-40be-9425-25c069453656\" (UID: \"92088a5b-f048-40be-9425-25c069453656\") "
Nov 01 15:27:40.726851 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:27:40.726806    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92088a5b-f048-40be-9425-25c069453656-kube-api-access-qtcxm\" (UniqueName: \"kubernetes.io/projected/92088a5b-f048-40be-9425-25c069453656-kube-api-access-qtcxm\") pod \"92088a5b-f048-40be-9425-25c069453656\" (UID: \"92088a5b-f048-40be-9425-25c069453656\") "
Nov 01 15:27:40.726967 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:27:40.726917    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92088a5b-f048-40be-9425-25c069453656-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92088a5b-f048-40be-9425-25c069453656-openshift-service-ca\") pod \"92088a5b-f048-40be-9425-25c069453656\" (UID: \"92088a5b-f048-40be-9425-25c069453656\") "
Nov 01 15:27:40.727759 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:27:40.727724    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92088a5b-f048-40be-9425-25c069453656-openshift-service-ca" pod "92088a5b-f048-40be-9425-25c069453656" (UID: "92088a5b-f048-40be-9425-25c069453656"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:27:40.732203 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:27:40.730490    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92088a5b-f048-40be-9425-25c069453656-metrics-certs" pod "92088a5b-f048-40be-9425-25c069453656" (UID: "92088a5b-f048-40be-9425-25c069453656"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:27:40.732203 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:27:40.730578    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92088a5b-f048-40be-9425-25c069453656-maas-api-tls" pod "92088a5b-f048-40be-9425-25c069453656" (UID: "92088a5b-f048-40be-9425-25c069453656"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:27:40.732681 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:27:40.732625    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92088a5b-f048-40be-9425-25c069453656-kube-api-access-qtcxm" pod "92088a5b-f048-40be-9425-25c069453656" (UID: "92088a5b-f048-40be-9425-25c069453656"). InnerVolumeSpecName "kube-api-access-qtcxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:27:40.829082 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:27:40.829041    4993 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92088a5b-f048-40be-9425-25c069453656-openshift-service-ca\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:27:40.829082 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:27:40.829076    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92088a5b-f048-40be-9425-25c069453656-maas-api-tls\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:27:40.829082 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:27:40.829086    4993 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92088a5b-f048-40be-9425-25c069453656-metrics-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:27:40.829337 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:27:40.829096    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtcxm\" (UniqueName: \"kubernetes.io/projected/92088a5b-f048-40be-9425-25c069453656-kube-api-access-qtcxm\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:27:41.208480 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:27:41.208445    4993 generic.go:356] "Generic (PLEG): container finished" podID="92088a5b-f048-40be-9425-25c069453656" containerID="58d1ab95d86677f008a5360cef0c26f821ef80b83d033266b96d5cd0cb922c1d" exitCode=0
Nov 01 15:27:41.208682 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:27:41.208496    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fc8c64-7676f5f848-8mrgd"
Nov 01 15:27:41.208682 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:27:41.208524    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fc8c64-7676f5f848-8mrgd" event={"ID":"92088a5b-f048-40be-9425-25c069453656","Type":"ContainerDied","Data":"58d1ab95d86677f008a5360cef0c26f821ef80b83d033266b96d5cd0cb922c1d"}
Nov 01 15:27:41.208682 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:27:41.208580    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fc8c64-7676f5f848-8mrgd" event={"ID":"92088a5b-f048-40be-9425-25c069453656","Type":"ContainerDied","Data":"acef18ca3da1ce26b4fa033dca25e81ff2185b5139fb72b5aeb4e7ab8d9f9a33"}
Nov 01 15:27:41.208682 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:27:41.208602    4993 scope.go:122] "RemoveContainer" containerID="58d1ab95d86677f008a5360cef0c26f821ef80b83d033266b96d5cd0cb922c1d"
Nov 01 15:27:41.218876 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:27:41.218856    4993 scope.go:122] "RemoveContainer" containerID="58d1ab95d86677f008a5360cef0c26f821ef80b83d033266b96d5cd0cb922c1d"
Nov 01 15:27:41.219151 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:27:41.219132    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58d1ab95d86677f008a5360cef0c26f821ef80b83d033266b96d5cd0cb922c1d\": container with ID starting with 58d1ab95d86677f008a5360cef0c26f821ef80b83d033266b96d5cd0cb922c1d not found: ID does not exist" containerID="58d1ab95d86677f008a5360cef0c26f821ef80b83d033266b96d5cd0cb922c1d"
Nov 01 15:27:41.219238 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:27:41.219159    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58d1ab95d86677f008a5360cef0c26f821ef80b83d033266b96d5cd0cb922c1d"} err="failed to get container status \"58d1ab95d86677f008a5360cef0c26f821ef80b83d033266b96d5cd0cb922c1d\": rpc error: code = NotFound desc = could not find container \"58d1ab95d86677f008a5360cef0c26f821ef80b83d033266b96d5cd0cb922c1d\": container with ID starting with 58d1ab95d86677f008a5360cef0c26f821ef80b83d033266b96d5cd0cb922c1d not found: ID does not exist"
Nov 01 15:27:41.361910 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:27:41.361876    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fc8c64-7676f5f848-8mrgd"]
Nov 01 15:27:41.365193 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:27:41.365155    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fc8c64-7676f5f848-8mrgd"]
Nov 01 15:27:42.797067 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:27:42.797034    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92088a5b-f048-40be-9425-25c069453656" path="/var/lib/kubelet/pods/92088a5b-f048-40be-9425-25c069453656/volumes"
Nov 01 15:27:57.504713 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:27:57.504673    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:28:00.886401 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:28:00.886356    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3d5a5b-69d698df5d-jxs6q"]
Nov 01 15:28:00.886953 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:28:00.886816    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d5a5b-69d698df5d-jxs6q" podUID="99cda18e-b745-4634-a2b3-a524343c0085" containerName="maas-api" containerID="cri-o://3835a0a7d8aa147315a44ff9370d0b0af1f37a145404b7cb4ccb980f062a70d7" gracePeriod=30
Nov 01 15:28:01.218983 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:28:01.218955    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d5a5b-69d698df5d-jxs6q"
Nov 01 15:28:01.358556 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:28:01.355458    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99cda18e-b745-4634-a2b3-a524343c0085-kube-api-access-c8r9k\" (UniqueName: \"kubernetes.io/projected/99cda18e-b745-4634-a2b3-a524343c0085-kube-api-access-c8r9k\") pod \"99cda18e-b745-4634-a2b3-a524343c0085\" (UID: \"99cda18e-b745-4634-a2b3-a524343c0085\") "
Nov 01 15:28:01.358556 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:28:01.355715    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99cda18e-b745-4634-a2b3-a524343c0085-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99cda18e-b745-4634-a2b3-a524343c0085-maas-api-tls\") pod \"99cda18e-b745-4634-a2b3-a524343c0085\" (UID: \"99cda18e-b745-4634-a2b3-a524343c0085\") "
Nov 01 15:28:01.358556 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:28:01.355759    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99cda18e-b745-4634-a2b3-a524343c0085-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99cda18e-b745-4634-a2b3-a524343c0085-openshift-service-ca\") pod \"99cda18e-b745-4634-a2b3-a524343c0085\" (UID: \"99cda18e-b745-4634-a2b3-a524343c0085\") "
Nov 01 15:28:01.358556 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:28:01.355839    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99cda18e-b745-4634-a2b3-a524343c0085-metrics-certs\" (UniqueName: \"kubernetes.io/secret/99cda18e-b745-4634-a2b3-a524343c0085-metrics-certs\") pod \"99cda18e-b745-4634-a2b3-a524343c0085\" (UID: \"99cda18e-b745-4634-a2b3-a524343c0085\") "
Nov 01 15:28:01.360920 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:28:01.360028    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99cda18e-b745-4634-a2b3-a524343c0085-openshift-service-ca" pod "99cda18e-b745-4634-a2b3-a524343c0085" (UID: "99cda18e-b745-4634-a2b3-a524343c0085"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:28:01.372433 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:28:01.372393    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99cda18e-b745-4634-a2b3-a524343c0085-maas-api-tls" pod "99cda18e-b745-4634-a2b3-a524343c0085" (UID: "99cda18e-b745-4634-a2b3-a524343c0085"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:28:01.388211 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:28:01.386926    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99cda18e-b745-4634-a2b3-a524343c0085-metrics-certs" pod "99cda18e-b745-4634-a2b3-a524343c0085" (UID: "99cda18e-b745-4634-a2b3-a524343c0085"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:28:01.394645 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:28:01.393766    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99cda18e-b745-4634-a2b3-a524343c0085-kube-api-access-c8r9k" pod "99cda18e-b745-4634-a2b3-a524343c0085" (UID: "99cda18e-b745-4634-a2b3-a524343c0085"). InnerVolumeSpecName "kube-api-access-c8r9k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:28:01.401487 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:28:01.401450    4993 generic.go:356] "Generic (PLEG): container finished" podID="99cda18e-b745-4634-a2b3-a524343c0085" containerID="3835a0a7d8aa147315a44ff9370d0b0af1f37a145404b7cb4ccb980f062a70d7" exitCode=0
Nov 01 15:28:01.401737 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:28:01.401501    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d5a5b-69d698df5d-jxs6q"
Nov 01 15:28:01.401737 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:28:01.401677    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d5a5b-69d698df5d-jxs6q" event={"ID":"99cda18e-b745-4634-a2b3-a524343c0085","Type":"ContainerDied","Data":"3835a0a7d8aa147315a44ff9370d0b0af1f37a145404b7cb4ccb980f062a70d7"}
Nov 01 15:28:01.401737 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:28:01.401728    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d5a5b-69d698df5d-jxs6q" event={"ID":"99cda18e-b745-4634-a2b3-a524343c0085","Type":"ContainerDied","Data":"857cbbd6984af6ed48824753bc5b8857da0d3ce2ffe56c652667758f4f2d2aa8"}
Nov 01 15:28:01.402034 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:28:01.401756    4993 scope.go:122] "RemoveContainer" containerID="3835a0a7d8aa147315a44ff9370d0b0af1f37a145404b7cb4ccb980f062a70d7"
Nov 01 15:28:01.415323 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:28:01.415302    4993 scope.go:122] "RemoveContainer" containerID="3835a0a7d8aa147315a44ff9370d0b0af1f37a145404b7cb4ccb980f062a70d7"
Nov 01 15:28:01.415703 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:28:01.415674    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3835a0a7d8aa147315a44ff9370d0b0af1f37a145404b7cb4ccb980f062a70d7\": container with ID starting with 3835a0a7d8aa147315a44ff9370d0b0af1f37a145404b7cb4ccb980f062a70d7 not found: ID does not exist" containerID="3835a0a7d8aa147315a44ff9370d0b0af1f37a145404b7cb4ccb980f062a70d7"
Nov 01 15:28:01.415827 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:28:01.415714    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3835a0a7d8aa147315a44ff9370d0b0af1f37a145404b7cb4ccb980f062a70d7"} err="failed to get container status \"3835a0a7d8aa147315a44ff9370d0b0af1f37a145404b7cb4ccb980f062a70d7\": rpc error: code = NotFound desc = could not find container \"3835a0a7d8aa147315a44ff9370d0b0af1f37a145404b7cb4ccb980f062a70d7\": container with ID starting with 3835a0a7d8aa147315a44ff9370d0b0af1f37a145404b7cb4ccb980f062a70d7 not found: ID does not exist"
Nov 01 15:28:01.457957 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:28:01.457863    4993 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99cda18e-b745-4634-a2b3-a524343c0085-metrics-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:28:01.457957 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:28:01.457899    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8r9k\" (UniqueName: \"kubernetes.io/projected/99cda18e-b745-4634-a2b3-a524343c0085-kube-api-access-c8r9k\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:28:01.457957 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:28:01.457914    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99cda18e-b745-4634-a2b3-a524343c0085-maas-api-tls\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:28:01.457957 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:28:01.457923    4993 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99cda18e-b745-4634-a2b3-a524343c0085-openshift-service-ca\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:28:01.556241 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:28:01.556198    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3d5a5b-69d698df5d-jxs6q"]
Nov 01 15:28:01.560539 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:28:01.560504    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3d5a5b-69d698df5d-jxs6q"]
Nov 01 15:28:02.796632 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:28:02.796552    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99cda18e-b745-4634-a2b3-a524343c0085" path="/var/lib/kubelet/pods/99cda18e-b745-4634-a2b3-a524343c0085/volumes"
Nov 01 15:28:27.501108 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:28:27.501060    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:28:34.805865 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:28:34.805828    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-775095-7448b7c55c-kvffx"]
Nov 01 15:28:34.806407 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:28:34.806036    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-775095-7448b7c55c-kvffx" podUID="b92fac92-7c11-49c1-839b-84598e2532c6" containerName="maas-api" containerID="cri-o://9540751fbed22baaefd6a12aa570d901f030e839b5021d33fbf2fee04cca2d5d" gracePeriod=30
Nov 01 15:28:35.126668 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:28:35.126635    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-775095-7448b7c55c-kvffx"
Nov 01 15:28:35.261502 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:28:35.261460    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b92fac92-7c11-49c1-839b-84598e2532c6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b92fac92-7c11-49c1-839b-84598e2532c6-maas-api-tls\") pod \"b92fac92-7c11-49c1-839b-84598e2532c6\" (UID: \"b92fac92-7c11-49c1-839b-84598e2532c6\") "
Nov 01 15:28:35.261699 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:28:35.261520    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b92fac92-7c11-49c1-839b-84598e2532c6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b92fac92-7c11-49c1-839b-84598e2532c6-openshift-service-ca\") pod \"b92fac92-7c11-49c1-839b-84598e2532c6\" (UID: \"b92fac92-7c11-49c1-839b-84598e2532c6\") "
Nov 01 15:28:35.261699 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:28:35.261632    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b92fac92-7c11-49c1-839b-84598e2532c6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b92fac92-7c11-49c1-839b-84598e2532c6-metrics-certs\") pod \"b92fac92-7c11-49c1-839b-84598e2532c6\" (UID: \"b92fac92-7c11-49c1-839b-84598e2532c6\") "
Nov 01 15:28:35.261699 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:28:35.261678    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b92fac92-7c11-49c1-839b-84598e2532c6-kube-api-access-gq629\" (UniqueName: \"kubernetes.io/projected/b92fac92-7c11-49c1-839b-84598e2532c6-kube-api-access-gq629\") pod \"b92fac92-7c11-49c1-839b-84598e2532c6\" (UID: \"b92fac92-7c11-49c1-839b-84598e2532c6\") "
Nov 01 15:28:35.262555 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:28:35.262519    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b92fac92-7c11-49c1-839b-84598e2532c6-openshift-service-ca" pod "b92fac92-7c11-49c1-839b-84598e2532c6" (UID: "b92fac92-7c11-49c1-839b-84598e2532c6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:28:35.264973 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:28:35.264947    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b92fac92-7c11-49c1-839b-84598e2532c6-metrics-certs" pod "b92fac92-7c11-49c1-839b-84598e2532c6" (UID: "b92fac92-7c11-49c1-839b-84598e2532c6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:28:35.267158 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:28:35.267131    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b92fac92-7c11-49c1-839b-84598e2532c6-kube-api-access-gq629" pod "b92fac92-7c11-49c1-839b-84598e2532c6" (UID: "b92fac92-7c11-49c1-839b-84598e2532c6"). InnerVolumeSpecName "kube-api-access-gq629". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:28:35.270786 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:28:35.270761    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b92fac92-7c11-49c1-839b-84598e2532c6-maas-api-tls" pod "b92fac92-7c11-49c1-839b-84598e2532c6" (UID: "b92fac92-7c11-49c1-839b-84598e2532c6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:28:35.363671 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:28:35.363589    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b92fac92-7c11-49c1-839b-84598e2532c6-maas-api-tls\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:28:35.363671 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:28:35.363619    4993 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b92fac92-7c11-49c1-839b-84598e2532c6-openshift-service-ca\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:28:35.363671 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:28:35.363633    4993 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b92fac92-7c11-49c1-839b-84598e2532c6-metrics-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:28:35.363671 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:28:35.363649    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gq629\" (UniqueName: \"kubernetes.io/projected/b92fac92-7c11-49c1-839b-84598e2532c6-kube-api-access-gq629\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:28:35.618945 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:28:35.618867    4993 generic.go:356] "Generic (PLEG): container finished" podID="b92fac92-7c11-49c1-839b-84598e2532c6" containerID="9540751fbed22baaefd6a12aa570d901f030e839b5021d33fbf2fee04cca2d5d" exitCode=0
Nov 01 15:28:35.618945 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:28:35.618913    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-775095-7448b7c55c-kvffx"
Nov 01 15:28:35.619141 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:28:35.618954    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-775095-7448b7c55c-kvffx" event={"ID":"b92fac92-7c11-49c1-839b-84598e2532c6","Type":"ContainerDied","Data":"9540751fbed22baaefd6a12aa570d901f030e839b5021d33fbf2fee04cca2d5d"}
Nov 01 15:28:35.619141 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:28:35.619011    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-775095-7448b7c55c-kvffx" event={"ID":"b92fac92-7c11-49c1-839b-84598e2532c6","Type":"ContainerDied","Data":"e07b03e802f5862144b10bce1fddc00923018b85b60793ec37fc8848892f6a92"}
Nov 01 15:28:35.619141 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:28:35.619037    4993 scope.go:122] "RemoveContainer" containerID="9540751fbed22baaefd6a12aa570d901f030e839b5021d33fbf2fee04cca2d5d"
Nov 01 15:28:35.632213 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:28:35.632190    4993 scope.go:122] "RemoveContainer" containerID="9540751fbed22baaefd6a12aa570d901f030e839b5021d33fbf2fee04cca2d5d"
Nov 01 15:28:35.632507 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:28:35.632485    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9540751fbed22baaefd6a12aa570d901f030e839b5021d33fbf2fee04cca2d5d\": container with ID starting with 9540751fbed22baaefd6a12aa570d901f030e839b5021d33fbf2fee04cca2d5d not found: ID does not exist" containerID="9540751fbed22baaefd6a12aa570d901f030e839b5021d33fbf2fee04cca2d5d"
Nov 01 15:28:35.632557 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:28:35.632517    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9540751fbed22baaefd6a12aa570d901f030e839b5021d33fbf2fee04cca2d5d"} err="failed to get container status \"9540751fbed22baaefd6a12aa570d901f030e839b5021d33fbf2fee04cca2d5d\": rpc error: code = NotFound desc = could not find container \"9540751fbed22baaefd6a12aa570d901f030e839b5021d33fbf2fee04cca2d5d\": container with ID starting with 9540751fbed22baaefd6a12aa570d901f030e839b5021d33fbf2fee04cca2d5d not found: ID does not exist"
Nov 01 15:28:35.772801 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:28:35.772751    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-775095-7448b7c55c-kvffx"]
Nov 01 15:28:35.818813 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:28:35.818780    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-775095-7448b7c55c-kvffx"]
Nov 01 15:28:36.796274 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:28:36.796238    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b92fac92-7c11-49c1-839b-84598e2532c6" path="/var/lib/kubelet/pods/b92fac92-7c11-49c1-839b-84598e2532c6/volumes"
Nov 01 15:28:49.668547 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:28:49.661926    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5feb76-7d5f658777-cb5kl"]
Nov 01 15:28:49.668547 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:28:49.662282    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5feb76-7d5f658777-cb5kl" podUID="ad273a86-1e46-4cc2-a4b9-20a42c57e688" containerName="maas-api" containerID="cri-o://6174be6f3fb7733761ba2a3f969f64b908566e39b97c534ddf408e5e7379b415" gracePeriod=30
Nov 01 15:28:49.976415 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:28:49.976383    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5feb76-7d5f658777-cb5kl"
Nov 01 15:28:50.058862 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:28:50.057628    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad273a86-1e46-4cc2-a4b9-20a42c57e688-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad273a86-1e46-4cc2-a4b9-20a42c57e688-openshift-service-ca\") pod \"ad273a86-1e46-4cc2-a4b9-20a42c57e688\" (UID: \"ad273a86-1e46-4cc2-a4b9-20a42c57e688\") "
Nov 01 15:28:50.058862 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:28:50.057758    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad273a86-1e46-4cc2-a4b9-20a42c57e688-kube-api-access-dtzwk\" (UniqueName: \"kubernetes.io/projected/ad273a86-1e46-4cc2-a4b9-20a42c57e688-kube-api-access-dtzwk\") pod \"ad273a86-1e46-4cc2-a4b9-20a42c57e688\" (UID: \"ad273a86-1e46-4cc2-a4b9-20a42c57e688\") "
Nov 01 15:28:50.058862 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:28:50.057832    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad273a86-1e46-4cc2-a4b9-20a42c57e688-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad273a86-1e46-4cc2-a4b9-20a42c57e688-metrics-certs\") pod \"ad273a86-1e46-4cc2-a4b9-20a42c57e688\" (UID: \"ad273a86-1e46-4cc2-a4b9-20a42c57e688\") "
Nov 01 15:28:50.058862 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:28:50.057971    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad273a86-1e46-4cc2-a4b9-20a42c57e688-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad273a86-1e46-4cc2-a4b9-20a42c57e688-maas-api-tls\") pod \"ad273a86-1e46-4cc2-a4b9-20a42c57e688\" (UID: \"ad273a86-1e46-4cc2-a4b9-20a42c57e688\") "
Nov 01 15:28:50.060039 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:28:50.060007    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad273a86-1e46-4cc2-a4b9-20a42c57e688-openshift-service-ca" pod "ad273a86-1e46-4cc2-a4b9-20a42c57e688" (UID: "ad273a86-1e46-4cc2-a4b9-20a42c57e688"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:28:50.070570 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:28:50.070510    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad273a86-1e46-4cc2-a4b9-20a42c57e688-metrics-certs" pod "ad273a86-1e46-4cc2-a4b9-20a42c57e688" (UID: "ad273a86-1e46-4cc2-a4b9-20a42c57e688"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:28:50.071899 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:28:50.071858    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad273a86-1e46-4cc2-a4b9-20a42c57e688-maas-api-tls" pod "ad273a86-1e46-4cc2-a4b9-20a42c57e688" (UID: "ad273a86-1e46-4cc2-a4b9-20a42c57e688"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:28:50.073839 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:28:50.073802    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad273a86-1e46-4cc2-a4b9-20a42c57e688-kube-api-access-dtzwk" pod "ad273a86-1e46-4cc2-a4b9-20a42c57e688" (UID: "ad273a86-1e46-4cc2-a4b9-20a42c57e688"). InnerVolumeSpecName "kube-api-access-dtzwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:28:50.160030 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:28:50.159991    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad273a86-1e46-4cc2-a4b9-20a42c57e688-maas-api-tls\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:28:50.160030 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:28:50.160031    4993 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad273a86-1e46-4cc2-a4b9-20a42c57e688-openshift-service-ca\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:28:50.160267 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:28:50.160046    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtzwk\" (UniqueName: \"kubernetes.io/projected/ad273a86-1e46-4cc2-a4b9-20a42c57e688-kube-api-access-dtzwk\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:28:50.160267 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:28:50.160062    4993 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad273a86-1e46-4cc2-a4b9-20a42c57e688-metrics-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:28:50.718802 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:28:50.718765    4993 generic.go:356] "Generic (PLEG): container finished" podID="ad273a86-1e46-4cc2-a4b9-20a42c57e688" containerID="6174be6f3fb7733761ba2a3f969f64b908566e39b97c534ddf408e5e7379b415" exitCode=0
Nov 01 15:28:50.719255 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:28:50.718813    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5feb76-7d5f658777-cb5kl"
Nov 01 15:28:50.719255 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:28:50.718833    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5feb76-7d5f658777-cb5kl" event={"ID":"ad273a86-1e46-4cc2-a4b9-20a42c57e688","Type":"ContainerDied","Data":"6174be6f3fb7733761ba2a3f969f64b908566e39b97c534ddf408e5e7379b415"}
Nov 01 15:28:50.719255 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:28:50.718881    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5feb76-7d5f658777-cb5kl" event={"ID":"ad273a86-1e46-4cc2-a4b9-20a42c57e688","Type":"ContainerDied","Data":"a8f3a836a21a272c8aaf1bab2e9a9e54c816b70301e4f8b7a117c97bbc261973"}
Nov 01 15:28:50.719255 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:28:50.718903    4993 scope.go:122] "RemoveContainer" containerID="6174be6f3fb7733761ba2a3f969f64b908566e39b97c534ddf408e5e7379b415"
Nov 01 15:28:50.729263 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:28:50.729243    4993 scope.go:122] "RemoveContainer" containerID="6174be6f3fb7733761ba2a3f969f64b908566e39b97c534ddf408e5e7379b415"
Nov 01 15:28:50.729817 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:28:50.729786    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6174be6f3fb7733761ba2a3f969f64b908566e39b97c534ddf408e5e7379b415\": container with ID starting with 6174be6f3fb7733761ba2a3f969f64b908566e39b97c534ddf408e5e7379b415 not found: ID does not exist" containerID="6174be6f3fb7733761ba2a3f969f64b908566e39b97c534ddf408e5e7379b415"
Nov 01 15:28:50.729859 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:28:50.729831    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6174be6f3fb7733761ba2a3f969f64b908566e39b97c534ddf408e5e7379b415"} err="failed to get container status \"6174be6f3fb7733761ba2a3f969f64b908566e39b97c534ddf408e5e7379b415\": rpc error: code = NotFound desc = could not find container \"6174be6f3fb7733761ba2a3f969f64b908566e39b97c534ddf408e5e7379b415\": container with ID starting with 6174be6f3fb7733761ba2a3f969f64b908566e39b97c534ddf408e5e7379b415 not found: ID does not exist"
Nov 01 15:28:50.890936 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:28:50.890899    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5feb76-7d5f658777-cb5kl"]
Nov 01 15:28:50.937019 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:28:50.936980    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5feb76-7d5f658777-cb5kl"]
Nov 01 15:28:52.796973 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:28:52.796935    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad273a86-1e46-4cc2-a4b9-20a42c57e688" path="/var/lib/kubelet/pods/ad273a86-1e46-4cc2-a4b9-20a42c57e688/volumes"
Nov 01 15:28:57.501487 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:28:57.501448    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:29:18.403911 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:29:18.403874    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-89e5ef-68ddd79477-br8sg"]
Nov 01 15:29:18.404542 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:29:18.404076    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89e5ef-68ddd79477-br8sg" podUID="17dcaed4-caa8-4a09-84bb-f445a8dc3a2f" containerName="maas-api" containerID="cri-o://b2d0767bb6734a22bc0d3ce6639e3ca0d85eae045ffdc05f78f06c0e15c18417" gracePeriod=30
Nov 01 15:29:18.731112 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:29:18.731079    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89e5ef-68ddd79477-br8sg"
Nov 01 15:29:18.837398 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:29:18.837254    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17dcaed4-caa8-4a09-84bb-f445a8dc3a2f-kube-api-access-sm99x\" (UniqueName: \"kubernetes.io/projected/17dcaed4-caa8-4a09-84bb-f445a8dc3a2f-kube-api-access-sm99x\") pod \"17dcaed4-caa8-4a09-84bb-f445a8dc3a2f\" (UID: \"17dcaed4-caa8-4a09-84bb-f445a8dc3a2f\") "
Nov 01 15:29:18.837604 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:29:18.837440    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17dcaed4-caa8-4a09-84bb-f445a8dc3a2f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17dcaed4-caa8-4a09-84bb-f445a8dc3a2f-openshift-service-ca\") pod \"17dcaed4-caa8-4a09-84bb-f445a8dc3a2f\" (UID: \"17dcaed4-caa8-4a09-84bb-f445a8dc3a2f\") "
Nov 01 15:29:18.837604 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:29:18.837517    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17dcaed4-caa8-4a09-84bb-f445a8dc3a2f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/17dcaed4-caa8-4a09-84bb-f445a8dc3a2f-metrics-certs\") pod \"17dcaed4-caa8-4a09-84bb-f445a8dc3a2f\" (UID: \"17dcaed4-caa8-4a09-84bb-f445a8dc3a2f\") "
Nov 01 15:29:18.837731 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:29:18.837607    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17dcaed4-caa8-4a09-84bb-f445a8dc3a2f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17dcaed4-caa8-4a09-84bb-f445a8dc3a2f-maas-api-tls\") pod \"17dcaed4-caa8-4a09-84bb-f445a8dc3a2f\" (UID: \"17dcaed4-caa8-4a09-84bb-f445a8dc3a2f\") "
Nov 01 15:29:18.838218 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:29:18.838185    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17dcaed4-caa8-4a09-84bb-f445a8dc3a2f-openshift-service-ca" pod "17dcaed4-caa8-4a09-84bb-f445a8dc3a2f" (UID: "17dcaed4-caa8-4a09-84bb-f445a8dc3a2f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:29:18.839149 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:29:18.839126    4993 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17dcaed4-caa8-4a09-84bb-f445a8dc3a2f-openshift-service-ca\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:29:18.840254 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:29:18.840226    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17dcaed4-caa8-4a09-84bb-f445a8dc3a2f-kube-api-access-sm99x" pod "17dcaed4-caa8-4a09-84bb-f445a8dc3a2f" (UID: "17dcaed4-caa8-4a09-84bb-f445a8dc3a2f"). InnerVolumeSpecName "kube-api-access-sm99x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:29:18.842312 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:29:18.842287    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17dcaed4-caa8-4a09-84bb-f445a8dc3a2f-metrics-certs" pod "17dcaed4-caa8-4a09-84bb-f445a8dc3a2f" (UID: "17dcaed4-caa8-4a09-84bb-f445a8dc3a2f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:29:18.842959 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:29:18.842936    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17dcaed4-caa8-4a09-84bb-f445a8dc3a2f-maas-api-tls" pod "17dcaed4-caa8-4a09-84bb-f445a8dc3a2f" (UID: "17dcaed4-caa8-4a09-84bb-f445a8dc3a2f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:29:18.919634 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:29:18.919593    4993 generic.go:356] "Generic (PLEG): container finished" podID="17dcaed4-caa8-4a09-84bb-f445a8dc3a2f" containerID="b2d0767bb6734a22bc0d3ce6639e3ca0d85eae045ffdc05f78f06c0e15c18417" exitCode=0
Nov 01 15:29:18.919808 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:29:18.919654    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89e5ef-68ddd79477-br8sg"
Nov 01 15:29:18.919808 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:29:18.919675    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89e5ef-68ddd79477-br8sg" event={"ID":"17dcaed4-caa8-4a09-84bb-f445a8dc3a2f","Type":"ContainerDied","Data":"b2d0767bb6734a22bc0d3ce6639e3ca0d85eae045ffdc05f78f06c0e15c18417"}
Nov 01 15:29:18.919808 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:29:18.919728    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89e5ef-68ddd79477-br8sg" event={"ID":"17dcaed4-caa8-4a09-84bb-f445a8dc3a2f","Type":"ContainerDied","Data":"3a71dd49e035acb49ff2cbfebd12c7114ae9e124b4b2feece6cf6db98c8ba4fb"}
Nov 01 15:29:18.919808 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:29:18.919750    4993 scope.go:122] "RemoveContainer" containerID="b2d0767bb6734a22bc0d3ce6639e3ca0d85eae045ffdc05f78f06c0e15c18417"
Nov 01 15:29:18.931225 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:29:18.931205    4993 scope.go:122] "RemoveContainer" containerID="b2d0767bb6734a22bc0d3ce6639e3ca0d85eae045ffdc05f78f06c0e15c18417"
Nov 01 15:29:18.931527 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:29:18.931504    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2d0767bb6734a22bc0d3ce6639e3ca0d85eae045ffdc05f78f06c0e15c18417\": container with ID starting with b2d0767bb6734a22bc0d3ce6639e3ca0d85eae045ffdc05f78f06c0e15c18417 not found: ID does not exist" containerID="b2d0767bb6734a22bc0d3ce6639e3ca0d85eae045ffdc05f78f06c0e15c18417"
Nov 01 15:29:18.931595 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:29:18.931539    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2d0767bb6734a22bc0d3ce6639e3ca0d85eae045ffdc05f78f06c0e15c18417"} err="failed to get container status \"b2d0767bb6734a22bc0d3ce6639e3ca0d85eae045ffdc05f78f06c0e15c18417\": rpc error: code = NotFound desc = could not find container \"b2d0767bb6734a22bc0d3ce6639e3ca0d85eae045ffdc05f78f06c0e15c18417\": container with ID starting with b2d0767bb6734a22bc0d3ce6639e3ca0d85eae045ffdc05f78f06c0e15c18417 not found: ID does not exist"
Nov 01 15:29:18.940006 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:29:18.939982    4993 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17dcaed4-caa8-4a09-84bb-f445a8dc3a2f-metrics-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:29:18.940106 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:29:18.940007    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17dcaed4-caa8-4a09-84bb-f445a8dc3a2f-maas-api-tls\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:29:18.940106 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:29:18.940020    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sm99x\" (UniqueName: \"kubernetes.io/projected/17dcaed4-caa8-4a09-84bb-f445a8dc3a2f-kube-api-access-sm99x\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:29:19.073214 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:29:19.073143    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-89e5ef-68ddd79477-br8sg"]
Nov 01 15:29:19.075849 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:29:19.075819    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-89e5ef-68ddd79477-br8sg"]
Nov 01 15:29:20.795217 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:29:20.795163    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17dcaed4-caa8-4a09-84bb-f445a8dc3a2f" path="/var/lib/kubelet/pods/17dcaed4-caa8-4a09-84bb-f445a8dc3a2f/volumes"
Nov 01 15:29:24.526546 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:29:24.526508    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-96165a-6cccf67998-zjh5q"]
Nov 01 15:29:24.527073 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:29:24.527057    4993 cpu_manager.go:414] "containerMap: removing container" podUID="17dcaed4-caa8-4a09-84bb-f445a8dc3a2f" containerName="maas-api"
Nov 01 15:29:24.527119 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:29:24.527075    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17dcaed4-caa8-4a09-84bb-f445a8dc3a2f" containerName="maas-api"
Nov 01 15:29:24.527119 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:29:24.527097    4993 cpu_manager.go:414] "containerMap: removing container" podUID="99cda18e-b745-4634-a2b3-a524343c0085" containerName="maas-api"
Nov 01 15:29:24.527119 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:29:24.527102    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99cda18e-b745-4634-a2b3-a524343c0085" containerName="maas-api"
Nov 01 15:29:24.527119 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:29:24.527112    4993 cpu_manager.go:414] "containerMap: removing container" podUID="92088a5b-f048-40be-9425-25c069453656" containerName="maas-api"
Nov 01 15:29:24.527119 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:29:24.527117    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92088a5b-f048-40be-9425-25c069453656" containerName="maas-api"
Nov 01 15:29:24.527314 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:29:24.527145    4993 cpu_manager.go:414] "containerMap: removing container" podUID="ad273a86-1e46-4cc2-a4b9-20a42c57e688" containerName="maas-api"
Nov 01 15:29:24.527314 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:29:24.527150    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad273a86-1e46-4cc2-a4b9-20a42c57e688" containerName="maas-api"
Nov 01 15:29:24.527314 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:29:24.527160    4993 cpu_manager.go:414] "containerMap: removing container" podUID="b92fac92-7c11-49c1-839b-84598e2532c6" containerName="maas-api"
Nov 01 15:29:24.527314 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:29:24.527189    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b92fac92-7c11-49c1-839b-84598e2532c6" containerName="maas-api"
Nov 01 15:29:24.527314 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:29:24.527279    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad273a86-1e46-4cc2-a4b9-20a42c57e688" containerName="maas-api"
Nov 01 15:29:24.527314 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:29:24.527288    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="b92fac92-7c11-49c1-839b-84598e2532c6" containerName="maas-api"
Nov 01 15:29:24.527314 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:29:24.527295    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="17dcaed4-caa8-4a09-84bb-f445a8dc3a2f" containerName="maas-api"
Nov 01 15:29:24.527314 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:29:24.527304    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="99cda18e-b745-4634-a2b3-a524343c0085" containerName="maas-api"
Nov 01 15:29:24.527314 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:29:24.527311    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="92088a5b-f048-40be-9425-25c069453656" containerName="maas-api"
Nov 01 15:29:24.531996 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:29:24.531972    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-96165a-6cccf67998-zjh5q"
Nov 01 15:29:24.581277 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:29:24.581220    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-96165a-serving-cert\""
Nov 01 15:29:24.613612 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:29:24.613579    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhwmc\" (UniqueName: \"kubernetes.io/projected/8cebd6df-9dc4-47f7-a67d-9b1f21551d8b-kube-api-access-zhwmc\") pod \"maas-api-e2e-inf-a-96165a-6cccf67998-zjh5q\" (UID: \"8cebd6df-9dc4-47f7-a67d-9b1f21551d8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-96165a-6cccf67998-zjh5q"
Nov 01 15:29:24.613773 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:29:24.613699    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cebd6df-9dc4-47f7-a67d-9b1f21551d8b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-96165a-6cccf67998-zjh5q\" (UID: \"8cebd6df-9dc4-47f7-a67d-9b1f21551d8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-96165a-6cccf67998-zjh5q"
Nov 01 15:29:24.613773 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:29:24.613752    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cebd6df-9dc4-47f7-a67d-9b1f21551d8b-maas-api-tls\") pod \"maas-api-e2e-inf-a-96165a-6cccf67998-zjh5q\" (UID: \"8cebd6df-9dc4-47f7-a67d-9b1f21551d8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-96165a-6cccf67998-zjh5q"
Nov 01 15:29:24.613773 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:29:24.613769    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cebd6df-9dc4-47f7-a67d-9b1f21551d8b-metrics-certs\") pod \"maas-api-e2e-inf-a-96165a-6cccf67998-zjh5q\" (UID: \"8cebd6df-9dc4-47f7-a67d-9b1f21551d8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-96165a-6cccf67998-zjh5q"
Nov 01 15:29:24.630304 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:29:24.630261    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-96165a-6cccf67998-zjh5q"]
Nov 01 15:29:24.714604 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:29:24.714569    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cebd6df-9dc4-47f7-a67d-9b1f21551d8b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-96165a-6cccf67998-zjh5q\" (UID: \"8cebd6df-9dc4-47f7-a67d-9b1f21551d8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-96165a-6cccf67998-zjh5q"
Nov 01 15:29:24.714808 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:29:24.714632    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cebd6df-9dc4-47f7-a67d-9b1f21551d8b-maas-api-tls\") pod \"maas-api-e2e-inf-a-96165a-6cccf67998-zjh5q\" (UID: \"8cebd6df-9dc4-47f7-a67d-9b1f21551d8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-96165a-6cccf67998-zjh5q"
Nov 01 15:29:24.714808 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:29:24.714659    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cebd6df-9dc4-47f7-a67d-9b1f21551d8b-metrics-certs\") pod \"maas-api-e2e-inf-a-96165a-6cccf67998-zjh5q\" (UID: \"8cebd6df-9dc4-47f7-a67d-9b1f21551d8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-96165a-6cccf67998-zjh5q"
Nov 01 15:29:24.714926 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:29:24.714805    4993 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-96165a-serving-cert: secret "maas-api-e2e-inf-a-96165a-serving-cert" not found
Nov 01 15:29:24.714926 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:29:24.714840    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhwmc\" (UniqueName: \"kubernetes.io/projected/8cebd6df-9dc4-47f7-a67d-9b1f21551d8b-kube-api-access-zhwmc\") pod \"maas-api-e2e-inf-a-96165a-6cccf67998-zjh5q\" (UID: \"8cebd6df-9dc4-47f7-a67d-9b1f21551d8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-96165a-6cccf67998-zjh5q"
Nov 01 15:29:24.714926 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:29:24.714883    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cebd6df-9dc4-47f7-a67d-9b1f21551d8b-maas-api-tls podName:8cebd6df-9dc4-47f7-a67d-9b1f21551d8b nodeName:}" failed. No retries permitted until 2026-11-01 15:29:25.214860094 +0000 UTC m=+4352.544722810 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8cebd6df-9dc4-47f7-a67d-9b1f21551d8b-maas-api-tls") pod "maas-api-e2e-inf-a-96165a-6cccf67998-zjh5q" (UID: "8cebd6df-9dc4-47f7-a67d-9b1f21551d8b") : secret "maas-api-e2e-inf-a-96165a-serving-cert" not found
Nov 01 15:29:24.715404 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:29:24.715384    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cebd6df-9dc4-47f7-a67d-9b1f21551d8b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-96165a-6cccf67998-zjh5q\" (UID: \"8cebd6df-9dc4-47f7-a67d-9b1f21551d8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-96165a-6cccf67998-zjh5q"
Nov 01 15:29:24.717290 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:29:24.717270    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cebd6df-9dc4-47f7-a67d-9b1f21551d8b-metrics-certs\") pod \"maas-api-e2e-inf-a-96165a-6cccf67998-zjh5q\" (UID: \"8cebd6df-9dc4-47f7-a67d-9b1f21551d8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-96165a-6cccf67998-zjh5q"
Nov 01 15:29:24.770804 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:29:24.770772    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhwmc\" (UniqueName: \"kubernetes.io/projected/8cebd6df-9dc4-47f7-a67d-9b1f21551d8b-kube-api-access-zhwmc\") pod \"maas-api-e2e-inf-a-96165a-6cccf67998-zjh5q\" (UID: \"8cebd6df-9dc4-47f7-a67d-9b1f21551d8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-96165a-6cccf67998-zjh5q"
Nov 01 15:29:25.222594 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:29:25.222548    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cebd6df-9dc4-47f7-a67d-9b1f21551d8b-maas-api-tls\") pod \"maas-api-e2e-inf-a-96165a-6cccf67998-zjh5q\" (UID: \"8cebd6df-9dc4-47f7-a67d-9b1f21551d8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-96165a-6cccf67998-zjh5q"
Nov 01 15:29:25.225053 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:29:25.225023    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cebd6df-9dc4-47f7-a67d-9b1f21551d8b-maas-api-tls\") pod \"maas-api-e2e-inf-a-96165a-6cccf67998-zjh5q\" (UID: \"8cebd6df-9dc4-47f7-a67d-9b1f21551d8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-96165a-6cccf67998-zjh5q"
Nov 01 15:29:25.445214 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:29:25.445176    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-96165a-6cccf67998-zjh5q"
Nov 01 15:29:25.819426 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:29:25.819345    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-96165a-6cccf67998-zjh5q"]
Nov 01 15:29:25.824455 ip-10-0-134-37 kubenswrapper[4993]: W1101 15:29:25.824424    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cebd6df_9dc4_47f7_a67d_9b1f21551d8b.slice/crio-b90ffa8b25a475a5e11c6111eca6f8e505470eed8ee625defe9049e8e0f79228 WatchSource:0}: Error finding container b90ffa8b25a475a5e11c6111eca6f8e505470eed8ee625defe9049e8e0f79228: Status 404 returned error can't find the container with id b90ffa8b25a475a5e11c6111eca6f8e505470eed8ee625defe9049e8e0f79228
Nov 01 15:29:25.825940 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:29:25.825921    4993 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 15:29:25.981318 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:29:25.981273    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-96165a-6cccf67998-zjh5q" event={"ID":"8cebd6df-9dc4-47f7-a67d-9b1f21551d8b","Type":"ContainerStarted","Data":"b90ffa8b25a475a5e11c6111eca6f8e505470eed8ee625defe9049e8e0f79228"}
Nov 01 15:29:26.989783 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:29:26.989732    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-96165a-6cccf67998-zjh5q" event={"ID":"8cebd6df-9dc4-47f7-a67d-9b1f21551d8b","Type":"ContainerStarted","Data":"2608f1421a8e50446b78a0a028588a8aa4c580832c23bd90dc26d2085cd3315b"}
Nov 01 15:29:26.990298 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:29:26.989826    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-96165a-6cccf67998-zjh5q"
Nov 01 15:29:27.113089 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:29:27.112980    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-96165a-6cccf67998-zjh5q" podStartSLOduration=2.105914545 podStartE2EDuration="3.112963494s" totalImagesPullingTime="1.007048949s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:29:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:29:27.112896142 +0000 UTC m=+4354.442758880" watchObservedRunningTime="2026-11-01 15:29:27.112963494 +0000 UTC m=+4354.442826232"
Nov 01 15:29:27.511822 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:29:27.511782    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:29:33.003127 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:29:33.003042    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-96165a-6cccf67998-zjh5q"
Nov 01 15:29:57.504267 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:29:57.504228    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:30:00.227312 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:30:00.227274    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892450-qnhpb"]
Nov 01 15:30:00.232932 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:30:00.232904    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892450-qnhpb"
Nov 01 15:30:00.288030 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:30:00.287990    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjqrz\" (UniqueName: \"kubernetes.io/projected/023123dd-3dea-4004-ab5a-5ee3f9665ce7-kube-api-access-tjqrz\") pod \"maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892450-qnhpb\" (UID: \"023123dd-3dea-4004-ab5a-5ee3f9665ce7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892450-qnhpb"
Nov 01 15:30:00.288250 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:30:00.288220    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/023123dd-3dea-4004-ab5a-5ee3f9665ce7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892450-qnhpb\" (UID: \"023123dd-3dea-4004-ab5a-5ee3f9665ce7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892450-qnhpb"
Nov 01 15:30:00.326753 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:30:00.326717    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\""
Nov 01 15:30:00.389566 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:30:00.389489    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjqrz\" (UniqueName: \"kubernetes.io/projected/023123dd-3dea-4004-ab5a-5ee3f9665ce7-kube-api-access-tjqrz\") pod \"maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892450-qnhpb\" (UID: \"023123dd-3dea-4004-ab5a-5ee3f9665ce7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892450-qnhpb"
Nov 01 15:30:00.389876 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:30:00.389671    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/023123dd-3dea-4004-ab5a-5ee3f9665ce7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892450-qnhpb\" (UID: \"023123dd-3dea-4004-ab5a-5ee3f9665ce7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892450-qnhpb"
Nov 01 15:30:00.390700 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:30:00.390670    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/023123dd-3dea-4004-ab5a-5ee3f9665ce7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892450-qnhpb\" (UID: \"023123dd-3dea-4004-ab5a-5ee3f9665ce7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892450-qnhpb"
Nov 01 15:30:00.422648 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:30:00.422602    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892450-qnhpb"]
Nov 01 15:30:00.444358 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:30:00.444323    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjqrz\" (UniqueName: \"kubernetes.io/projected/023123dd-3dea-4004-ab5a-5ee3f9665ce7-kube-api-access-tjqrz\") pod \"maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892450-qnhpb\" (UID: \"023123dd-3dea-4004-ab5a-5ee3f9665ce7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892450-qnhpb"
Nov 01 15:30:00.545509 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:30:00.545473    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892450-qnhpb"
Nov 01 15:30:00.918707 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:30:00.918629    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892450-qnhpb"]
Nov 01 15:30:00.925033 ip-10-0-134-37 kubenswrapper[4993]: W1101 15:30:00.924998    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod023123dd_3dea_4004_ab5a_5ee3f9665ce7.slice/crio-56585c5f4aad58520f8fae5d43472328688ac33352cf0d082d5ced2b338a6839 WatchSource:0}: Error finding container 56585c5f4aad58520f8fae5d43472328688ac33352cf0d082d5ced2b338a6839: Status 404 returned error can't find the container with id 56585c5f4aad58520f8fae5d43472328688ac33352cf0d082d5ced2b338a6839
Nov 01 15:30:01.240484 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:30:01.240449    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892450-qnhpb" event={"ID":"023123dd-3dea-4004-ab5a-5ee3f9665ce7","Type":"ContainerStarted","Data":"56585c5f4aad58520f8fae5d43472328688ac33352cf0d082d5ced2b338a6839"}
Nov 01 15:30:02.249022 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:30:02.248991    4993 generic.go:356] "Generic (PLEG): container finished" podID="023123dd-3dea-4004-ab5a-5ee3f9665ce7" containerID="13459e2a8ab8c869b0e8c8e6e923ff362c0081299f5e38ca11f09f133e6aab62" exitCode=60
Nov 01 15:30:02.249453 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:30:02.249082    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892450-qnhpb" event={"ID":"023123dd-3dea-4004-ab5a-5ee3f9665ce7","Type":"ContainerDied","Data":"13459e2a8ab8c869b0e8c8e6e923ff362c0081299f5e38ca11f09f133e6aab62"}
Nov 01 15:30:02.249453 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:30:02.249327    4993 scope.go:122] "RemoveContainer" containerID="13459e2a8ab8c869b0e8c8e6e923ff362c0081299f5e38ca11f09f133e6aab62"
Nov 01 15:30:03.271582 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:30:03.271546    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892450-qnhpb" event={"ID":"023123dd-3dea-4004-ab5a-5ee3f9665ce7","Type":"ContainerStarted","Data":"7d5cc4d25b7e7f6d9433042ef0a21dedf8f5353e6abcd45160fcde1253446415"}
Nov 01 15:30:03.272012 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:30:03.271926    4993 scope.go:122] "RemoveContainer" containerID="7d5cc4d25b7e7f6d9433042ef0a21dedf8f5353e6abcd45160fcde1253446415"
Nov 01 15:30:03.272323 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:30:03.272250    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892450-qnhpb_odh-ai-gateway-infra(023123dd-3dea-4004-ab5a-5ee3f9665ce7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892450-qnhpb" podUID="023123dd-3dea-4004-ab5a-5ee3f9665ce7"
Nov 01 15:30:04.280105 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:30:04.280070    4993 generic.go:356] "Generic (PLEG): container finished" podID="023123dd-3dea-4004-ab5a-5ee3f9665ce7" containerID="7d5cc4d25b7e7f6d9433042ef0a21dedf8f5353e6abcd45160fcde1253446415" exitCode=60
Nov 01 15:30:04.280600 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:30:04.280153    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892450-qnhpb" event={"ID":"023123dd-3dea-4004-ab5a-5ee3f9665ce7","Type":"ContainerDied","Data":"7d5cc4d25b7e7f6d9433042ef0a21dedf8f5353e6abcd45160fcde1253446415"}
Nov 01 15:30:04.280600 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:30:04.280240    4993 scope.go:122] "RemoveContainer" containerID="13459e2a8ab8c869b0e8c8e6e923ff362c0081299f5e38ca11f09f133e6aab62"
Nov 01 15:30:04.280600 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:30:04.280522    4993 scope.go:122] "RemoveContainer" containerID="7d5cc4d25b7e7f6d9433042ef0a21dedf8f5353e6abcd45160fcde1253446415"
Nov 01 15:30:04.280817 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:30:04.280794    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892450-qnhpb_odh-ai-gateway-infra(023123dd-3dea-4004-ab5a-5ee3f9665ce7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892450-qnhpb" podUID="023123dd-3dea-4004-ab5a-5ee3f9665ce7"
Nov 01 15:30:05.288546 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:30:05.288516    4993 scope.go:122] "RemoveContainer" containerID="7d5cc4d25b7e7f6d9433042ef0a21dedf8f5353e6abcd45160fcde1253446415"
Nov 01 15:30:05.288960 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:30:05.288807    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892450-qnhpb_odh-ai-gateway-infra(023123dd-3dea-4004-ab5a-5ee3f9665ce7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892450-qnhpb" podUID="023123dd-3dea-4004-ab5a-5ee3f9665ce7"
Nov 01 15:30:15.793193 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:30:15.791496    4993 scope.go:122] "RemoveContainer" containerID="7d5cc4d25b7e7f6d9433042ef0a21dedf8f5353e6abcd45160fcde1253446415"
Nov 01 15:30:17.402022 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:30:17.401926    4993 generic.go:356] "Generic (PLEG): container finished" podID="023123dd-3dea-4004-ab5a-5ee3f9665ce7" containerID="7b43bf07947540b1d8780d7a6307b6b089ed87ca7c52926a4d88b993b0eda6bf" exitCode=60
Nov 01 15:30:17.402502 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:30:17.402009    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892450-qnhpb" event={"ID":"023123dd-3dea-4004-ab5a-5ee3f9665ce7","Type":"ContainerDied","Data":"7b43bf07947540b1d8780d7a6307b6b089ed87ca7c52926a4d88b993b0eda6bf"}
Nov 01 15:30:17.402502 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:30:17.402089    4993 scope.go:122] "RemoveContainer" containerID="7d5cc4d25b7e7f6d9433042ef0a21dedf8f5353e6abcd45160fcde1253446415"
Nov 01 15:30:17.402601 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:30:17.402521    4993 scope.go:122] "RemoveContainer" containerID="7b43bf07947540b1d8780d7a6307b6b089ed87ca7c52926a4d88b993b0eda6bf"
Nov 01 15:30:17.402914 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:30:17.402869    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892450-qnhpb_odh-ai-gateway-infra(023123dd-3dea-4004-ab5a-5ee3f9665ce7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892450-qnhpb" podUID="023123dd-3dea-4004-ab5a-5ee3f9665ce7"
Nov 01 15:30:18.521846 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:30:18.521802    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892450-qnhpb"]
Nov 01 15:30:18.670838 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:30:18.670811    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892450-qnhpb"
Nov 01 15:30:18.835896 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:30:18.835818    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/023123dd-3dea-4004-ab5a-5ee3f9665ce7-kube-api-access-tjqrz\" (UniqueName: \"kubernetes.io/projected/023123dd-3dea-4004-ab5a-5ee3f9665ce7-kube-api-access-tjqrz\") pod \"023123dd-3dea-4004-ab5a-5ee3f9665ce7\" (UID: \"023123dd-3dea-4004-ab5a-5ee3f9665ce7\") "
Nov 01 15:30:18.836043 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:30:18.835968    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/023123dd-3dea-4004-ab5a-5ee3f9665ce7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/023123dd-3dea-4004-ab5a-5ee3f9665ce7-maas-api-service-ca\") pod \"023123dd-3dea-4004-ab5a-5ee3f9665ce7\" (UID: \"023123dd-3dea-4004-ab5a-5ee3f9665ce7\") "
Nov 01 15:30:18.836382 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:30:18.836353    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/023123dd-3dea-4004-ab5a-5ee3f9665ce7-maas-api-service-ca" pod "023123dd-3dea-4004-ab5a-5ee3f9665ce7" (UID: "023123dd-3dea-4004-ab5a-5ee3f9665ce7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:30:18.838010 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:30:18.837988    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/023123dd-3dea-4004-ab5a-5ee3f9665ce7-kube-api-access-tjqrz" pod "023123dd-3dea-4004-ab5a-5ee3f9665ce7" (UID: "023123dd-3dea-4004-ab5a-5ee3f9665ce7"). InnerVolumeSpecName "kube-api-access-tjqrz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:30:18.937400 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:30:18.937366    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/023123dd-3dea-4004-ab5a-5ee3f9665ce7-maas-api-service-ca\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:30:18.937400 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:30:18.937396    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjqrz\" (UniqueName: \"kubernetes.io/projected/023123dd-3dea-4004-ab5a-5ee3f9665ce7-kube-api-access-tjqrz\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:30:19.425689 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:30:19.425648    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892450-qnhpb"
Nov 01 15:30:19.425898 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:30:19.425642    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892450-qnhpb" event={"ID":"023123dd-3dea-4004-ab5a-5ee3f9665ce7","Type":"ContainerDied","Data":"56585c5f4aad58520f8fae5d43472328688ac33352cf0d082d5ced2b338a6839"}
Nov 01 15:30:19.425898 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:30:19.425802    4993 scope.go:122] "RemoveContainer" containerID="7b43bf07947540b1d8780d7a6307b6b089ed87ca7c52926a4d88b993b0eda6bf"
Nov 01 15:30:19.578281 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:30:19.578237    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892450-qnhpb"]
Nov 01 15:30:19.624557 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:30:19.624518    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892450-qnhpb"]
Nov 01 15:30:20.795711 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:30:20.795665    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="023123dd-3dea-4004-ab5a-5ee3f9665ce7" path="/var/lib/kubelet/pods/023123dd-3dea-4004-ab5a-5ee3f9665ce7/volumes"
Nov 01 15:30:27.503693 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:30:27.503652    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:30:57.500876 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:30:57.500820    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:31:27.501605 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:31:27.501523    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:31:53.436302 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:31:53.436267    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xmwtp_bb59051d-c2df-4362-a157-00f14f915e79/console-operator/3.log"
Nov 01 15:31:53.447077 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:31:53.447051    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xmwtp_bb59051d-c2df-4362-a157-00f14f915e79/console-operator/3.log"
Nov 01 15:31:57.504895 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:31:57.504855    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:32:07.082976 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:32:07.082938    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3f45c3-6fb65f9b8c-6gx8x"]
Nov 01 15:32:07.083794 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:32:07.083703    4993 cpu_manager.go:414] "containerMap: removing container" podUID="023123dd-3dea-4004-ab5a-5ee3f9665ce7" containerName="cleanup"
Nov 01 15:32:07.083794 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:32:07.083725    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="023123dd-3dea-4004-ab5a-5ee3f9665ce7" containerName="cleanup"
Nov 01 15:32:07.083794 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:32:07.083789    4993 cpu_manager.go:414] "containerMap: removing container" podUID="023123dd-3dea-4004-ab5a-5ee3f9665ce7" containerName="cleanup"
Nov 01 15:32:07.083988 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:32:07.083799    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="023123dd-3dea-4004-ab5a-5ee3f9665ce7" containerName="cleanup"
Nov 01 15:32:07.083988 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:32:07.083957    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="023123dd-3dea-4004-ab5a-5ee3f9665ce7" containerName="cleanup"
Nov 01 15:32:07.083988 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:32:07.083972    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="023123dd-3dea-4004-ab5a-5ee3f9665ce7" containerName="cleanup"
Nov 01 15:32:07.087527 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:32:07.087497    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3f45c3-6fb65f9b8c-6gx8x"
Nov 01 15:32:07.137507 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:32:07.137477    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-3f45c3-serving-cert\""
Nov 01 15:32:07.142231 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:32:07.142197    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3f45c3-6fb65f9b8c-6gx8x"]
Nov 01 15:32:07.171493 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:32:07.171454    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2108b10-03af-450f-b750-e56bab9faff2-maas-api-tls\") pod \"maas-api-e2e-inf-b-3f45c3-6fb65f9b8c-6gx8x\" (UID: \"e2108b10-03af-450f-b750-e56bab9faff2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3f45c3-6fb65f9b8c-6gx8x"
Nov 01 15:32:07.171679 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:32:07.171635    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4828\" (UniqueName: \"kubernetes.io/projected/e2108b10-03af-450f-b750-e56bab9faff2-kube-api-access-h4828\") pod \"maas-api-e2e-inf-b-3f45c3-6fb65f9b8c-6gx8x\" (UID: \"e2108b10-03af-450f-b750-e56bab9faff2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3f45c3-6fb65f9b8c-6gx8x"
Nov 01 15:32:07.171738 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:32:07.171701    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2108b10-03af-450f-b750-e56bab9faff2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-3f45c3-6fb65f9b8c-6gx8x\" (UID: \"e2108b10-03af-450f-b750-e56bab9faff2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3f45c3-6fb65f9b8c-6gx8x"
Nov 01 15:32:07.171831 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:32:07.171811    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2108b10-03af-450f-b750-e56bab9faff2-metrics-certs\") pod \"maas-api-e2e-inf-b-3f45c3-6fb65f9b8c-6gx8x\" (UID: \"e2108b10-03af-450f-b750-e56bab9faff2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3f45c3-6fb65f9b8c-6gx8x"
Nov 01 15:32:07.273249 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:32:07.273197    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2108b10-03af-450f-b750-e56bab9faff2-maas-api-tls\") pod \"maas-api-e2e-inf-b-3f45c3-6fb65f9b8c-6gx8x\" (UID: \"e2108b10-03af-450f-b750-e56bab9faff2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3f45c3-6fb65f9b8c-6gx8x"
Nov 01 15:32:07.273454 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:32:07.273355    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4828\" (UniqueName: \"kubernetes.io/projected/e2108b10-03af-450f-b750-e56bab9faff2-kube-api-access-h4828\") pod \"maas-api-e2e-inf-b-3f45c3-6fb65f9b8c-6gx8x\" (UID: \"e2108b10-03af-450f-b750-e56bab9faff2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3f45c3-6fb65f9b8c-6gx8x"
Nov 01 15:32:07.273454 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:32:07.273413    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2108b10-03af-450f-b750-e56bab9faff2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-3f45c3-6fb65f9b8c-6gx8x\" (UID: \"e2108b10-03af-450f-b750-e56bab9faff2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3f45c3-6fb65f9b8c-6gx8x"
Nov 01 15:32:07.273569 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:32:07.273490    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2108b10-03af-450f-b750-e56bab9faff2-metrics-certs\") pod \"maas-api-e2e-inf-b-3f45c3-6fb65f9b8c-6gx8x\" (UID: \"e2108b10-03af-450f-b750-e56bab9faff2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3f45c3-6fb65f9b8c-6gx8x"
Nov 01 15:32:07.273908 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:32:07.273706    4993 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-3f45c3-serving-cert: secret "maas-api-e2e-inf-b-3f45c3-serving-cert" not found
Nov 01 15:32:07.273908 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:32:07.273803    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2108b10-03af-450f-b750-e56bab9faff2-maas-api-tls podName:e2108b10-03af-450f-b750-e56bab9faff2 nodeName:}" failed. No retries permitted until 2026-11-01 15:32:07.773781126 +0000 UTC m=+4515.103643845 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e2108b10-03af-450f-b750-e56bab9faff2-maas-api-tls") pod "maas-api-e2e-inf-b-3f45c3-6fb65f9b8c-6gx8x" (UID: "e2108b10-03af-450f-b750-e56bab9faff2") : secret "maas-api-e2e-inf-b-3f45c3-serving-cert" not found
Nov 01 15:32:07.274520 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:32:07.274496    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2108b10-03af-450f-b750-e56bab9faff2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-3f45c3-6fb65f9b8c-6gx8x\" (UID: \"e2108b10-03af-450f-b750-e56bab9faff2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3f45c3-6fb65f9b8c-6gx8x"
Nov 01 15:32:07.276341 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:32:07.276318    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2108b10-03af-450f-b750-e56bab9faff2-metrics-certs\") pod \"maas-api-e2e-inf-b-3f45c3-6fb65f9b8c-6gx8x\" (UID: \"e2108b10-03af-450f-b750-e56bab9faff2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3f45c3-6fb65f9b8c-6gx8x"
Nov 01 15:32:07.327602 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:32:07.327569    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4828\" (UniqueName: \"kubernetes.io/projected/e2108b10-03af-450f-b750-e56bab9faff2-kube-api-access-h4828\") pod \"maas-api-e2e-inf-b-3f45c3-6fb65f9b8c-6gx8x\" (UID: \"e2108b10-03af-450f-b750-e56bab9faff2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3f45c3-6fb65f9b8c-6gx8x"
Nov 01 15:32:07.782486 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:32:07.782446    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2108b10-03af-450f-b750-e56bab9faff2-maas-api-tls\") pod \"maas-api-e2e-inf-b-3f45c3-6fb65f9b8c-6gx8x\" (UID: \"e2108b10-03af-450f-b750-e56bab9faff2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3f45c3-6fb65f9b8c-6gx8x"
Nov 01 15:32:07.784908 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:32:07.784878    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2108b10-03af-450f-b750-e56bab9faff2-maas-api-tls\") pod \"maas-api-e2e-inf-b-3f45c3-6fb65f9b8c-6gx8x\" (UID: \"e2108b10-03af-450f-b750-e56bab9faff2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3f45c3-6fb65f9b8c-6gx8x"
Nov 01 15:32:08.001310 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:32:08.001279    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3f45c3-6fb65f9b8c-6gx8x"
Nov 01 15:32:08.372235 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:32:08.372134    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3f45c3-6fb65f9b8c-6gx8x"]
Nov 01 15:32:08.377898 ip-10-0-134-37 kubenswrapper[4993]: W1101 15:32:08.377868    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2108b10_03af_450f_b750_e56bab9faff2.slice/crio-65c04f7841ae6bf40d52816e7c8088201b9d8bfd4123c56872fa2813a286b530 WatchSource:0}: Error finding container 65c04f7841ae6bf40d52816e7c8088201b9d8bfd4123c56872fa2813a286b530: Status 404 returned error can't find the container with id 65c04f7841ae6bf40d52816e7c8088201b9d8bfd4123c56872fa2813a286b530
Nov 01 15:32:09.162850 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:32:09.162809    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3f45c3-6fb65f9b8c-6gx8x" event={"ID":"e2108b10-03af-450f-b750-e56bab9faff2","Type":"ContainerStarted","Data":"65c04f7841ae6bf40d52816e7c8088201b9d8bfd4123c56872fa2813a286b530"}
Nov 01 15:32:10.172217 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:32:10.172151    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3f45c3-6fb65f9b8c-6gx8x" event={"ID":"e2108b10-03af-450f-b750-e56bab9faff2","Type":"ContainerStarted","Data":"5b4d7d6057d656f65cc690a3fd7c99cf7bd1dfd54c50562b0ed4701940125ba3"}
Nov 01 15:32:10.172698 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:32:10.172397    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3f45c3-6fb65f9b8c-6gx8x"
Nov 01 15:32:10.280954 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:32:10.280896    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3f45c3-6fb65f9b8c-6gx8x" podStartSLOduration=2.323210097 podStartE2EDuration="3.280882056s" totalImagesPullingTime="957.671959ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:32:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:32:10.278478968 +0000 UTC m=+4517.608341716" watchObservedRunningTime="2026-11-01 15:32:10.280882056 +0000 UTC m=+4517.610744793"
Nov 01 15:32:16.049416 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:32:16.049376    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-600cc7-6d484f766f-qcf6m"]
Nov 01 15:32:16.050251 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:32:16.050225    4993 cpu_manager.go:414] "containerMap: removing container" podUID="023123dd-3dea-4004-ab5a-5ee3f9665ce7" containerName="cleanup"
Nov 01 15:32:16.050251 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:32:16.050251    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="023123dd-3dea-4004-ab5a-5ee3f9665ce7" containerName="cleanup"
Nov 01 15:32:16.050479 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:32:16.050464    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="023123dd-3dea-4004-ab5a-5ee3f9665ce7" containerName="cleanup"
Nov 01 15:32:16.055842 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:32:16.055819    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-600cc7-6d484f766f-qcf6m"
Nov 01 15:32:16.083242 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:32:16.083202    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bbc081b-ce18-4d0b-81ec-2bbf576f63f6-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-600cc7-6d484f766f-qcf6m\" (UID: \"8bbc081b-ce18-4d0b-81ec-2bbf576f63f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-600cc7-6d484f766f-qcf6m"
Nov 01 15:32:16.083405 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:32:16.083311    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bbc081b-ce18-4d0b-81ec-2bbf576f63f6-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-600cc7-6d484f766f-qcf6m\" (UID: \"8bbc081b-ce18-4d0b-81ec-2bbf576f63f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-600cc7-6d484f766f-qcf6m"
Nov 01 15:32:16.083405 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:32:16.083356    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bbc081b-ce18-4d0b-81ec-2bbf576f63f6-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-600cc7-6d484f766f-qcf6m\" (UID: \"8bbc081b-ce18-4d0b-81ec-2bbf576f63f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-600cc7-6d484f766f-qcf6m"
Nov 01 15:32:16.083543 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:32:16.083526    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx85z\" (UniqueName: \"kubernetes.io/projected/8bbc081b-ce18-4d0b-81ec-2bbf576f63f6-kube-api-access-rx85z\") pod \"maas-api-e2e-shared-a-w0-600cc7-6d484f766f-qcf6m\" (UID: \"8bbc081b-ce18-4d0b-81ec-2bbf576f63f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-600cc7-6d484f766f-qcf6m"
Nov 01 15:32:16.107979 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:32:16.107944    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-600cc7-6d484f766f-qcf6m"]
Nov 01 15:32:16.149757 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:32:16.149719    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-600cc7-serving-cert\""
Nov 01 15:32:16.185669 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:32:16.185639    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3f45c3-6fb65f9b8c-6gx8x"
Nov 01 15:32:16.186622 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:32:16.186584    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bbc081b-ce18-4d0b-81ec-2bbf576f63f6-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-600cc7-6d484f766f-qcf6m\" (UID: \"8bbc081b-ce18-4d0b-81ec-2bbf576f63f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-600cc7-6d484f766f-qcf6m"
Nov 01 15:32:16.186757 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:32:16.186647    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bbc081b-ce18-4d0b-81ec-2bbf576f63f6-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-600cc7-6d484f766f-qcf6m\" (UID: \"8bbc081b-ce18-4d0b-81ec-2bbf576f63f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-600cc7-6d484f766f-qcf6m"
Nov 01 15:32:16.186839 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:32:16.186754    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx85z\" (UniqueName: \"kubernetes.io/projected/8bbc081b-ce18-4d0b-81ec-2bbf576f63f6-kube-api-access-rx85z\") pod \"maas-api-e2e-shared-a-w0-600cc7-6d484f766f-qcf6m\" (UID: \"8bbc081b-ce18-4d0b-81ec-2bbf576f63f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-600cc7-6d484f766f-qcf6m"
Nov 01 15:32:16.186925 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:32:16.186909    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bbc081b-ce18-4d0b-81ec-2bbf576f63f6-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-600cc7-6d484f766f-qcf6m\" (UID: \"8bbc081b-ce18-4d0b-81ec-2bbf576f63f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-600cc7-6d484f766f-qcf6m"
Nov 01 15:32:16.187897 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:32:16.187875    4993 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-600cc7-serving-cert: secret "maas-api-e2e-shared-a-w0-600cc7-serving-cert" not found
Nov 01 15:32:16.188147 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:32:16.188132    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bbc081b-ce18-4d0b-81ec-2bbf576f63f6-maas-api-tls podName:8bbc081b-ce18-4d0b-81ec-2bbf576f63f6 nodeName:}" failed. No retries permitted until 2026-11-01 15:32:16.688093324 +0000 UTC m=+4524.017956056 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8bbc081b-ce18-4d0b-81ec-2bbf576f63f6-maas-api-tls") pod "maas-api-e2e-shared-a-w0-600cc7-6d484f766f-qcf6m" (UID: "8bbc081b-ce18-4d0b-81ec-2bbf576f63f6") : secret "maas-api-e2e-shared-a-w0-600cc7-serving-cert" not found
Nov 01 15:32:16.189459 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:32:16.189435    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bbc081b-ce18-4d0b-81ec-2bbf576f63f6-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-600cc7-6d484f766f-qcf6m\" (UID: \"8bbc081b-ce18-4d0b-81ec-2bbf576f63f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-600cc7-6d484f766f-qcf6m"
Nov 01 15:32:16.202962 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:32:16.202934    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bbc081b-ce18-4d0b-81ec-2bbf576f63f6-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-600cc7-6d484f766f-qcf6m\" (UID: \"8bbc081b-ce18-4d0b-81ec-2bbf576f63f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-600cc7-6d484f766f-qcf6m"
Nov 01 15:32:16.241292 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:32:16.241257    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx85z\" (UniqueName: \"kubernetes.io/projected/8bbc081b-ce18-4d0b-81ec-2bbf576f63f6-kube-api-access-rx85z\") pod \"maas-api-e2e-shared-a-w0-600cc7-6d484f766f-qcf6m\" (UID: \"8bbc081b-ce18-4d0b-81ec-2bbf576f63f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-600cc7-6d484f766f-qcf6m"
Nov 01 15:32:16.697219 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:32:16.697178    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bbc081b-ce18-4d0b-81ec-2bbf576f63f6-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-600cc7-6d484f766f-qcf6m\" (UID: \"8bbc081b-ce18-4d0b-81ec-2bbf576f63f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-600cc7-6d484f766f-qcf6m"
Nov 01 15:32:16.699897 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:32:16.699864    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bbc081b-ce18-4d0b-81ec-2bbf576f63f6-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-600cc7-6d484f766f-qcf6m\" (UID: \"8bbc081b-ce18-4d0b-81ec-2bbf576f63f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-600cc7-6d484f766f-qcf6m"
Nov 01 15:32:16.970102 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:32:16.970016    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-600cc7-6d484f766f-qcf6m"
Nov 01 15:32:17.353978 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:32:17.353941    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-600cc7-6d484f766f-qcf6m"]
Nov 01 15:32:17.359888 ip-10-0-134-37 kubenswrapper[4993]: W1101 15:32:17.359857    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bbc081b_ce18_4d0b_81ec_2bbf576f63f6.slice/crio-753a7eae7bf7d7ec4207e7dc98bd21dd8a2fc4116cb638c05a38fb5c53a9ed4c WatchSource:0}: Error finding container 753a7eae7bf7d7ec4207e7dc98bd21dd8a2fc4116cb638c05a38fb5c53a9ed4c: Status 404 returned error can't find the container with id 753a7eae7bf7d7ec4207e7dc98bd21dd8a2fc4116cb638c05a38fb5c53a9ed4c
Nov 01 15:32:18.225427 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:32:18.225391    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-600cc7-6d484f766f-qcf6m" event={"ID":"8bbc081b-ce18-4d0b-81ec-2bbf576f63f6","Type":"ContainerStarted","Data":"753a7eae7bf7d7ec4207e7dc98bd21dd8a2fc4116cb638c05a38fb5c53a9ed4c"}
Nov 01 15:32:19.232844 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:32:19.232802    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-600cc7-6d484f766f-qcf6m" event={"ID":"8bbc081b-ce18-4d0b-81ec-2bbf576f63f6","Type":"ContainerStarted","Data":"71fef5b419223bb1b90ef1c50964f7cbc3fdf5c57203218454ac3ee25678ee91"}
Nov 01 15:32:19.233335 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:32:19.232922    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-600cc7-6d484f766f-qcf6m"
Nov 01 15:32:19.341190 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:32:19.341100    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-600cc7-6d484f766f-qcf6m" podStartSLOduration=2.48611292 podStartE2EDuration="3.341085562s" totalImagesPullingTime="854.972642ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:32:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:32:19.338502523 +0000 UTC m=+4526.668365262" watchObservedRunningTime="2026-11-01 15:32:19.341085562 +0000 UTC m=+4526.670948299"
Nov 01 15:32:25.245438 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:32:25.245407    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-600cc7-6d484f766f-qcf6m"
Nov 01 15:32:27.500089 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:32:27.500055    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:32:57.503758 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:32:57.503709    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:33:14.245658 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:33:14.245621    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-4e07cb/sub-test-model-4e07cb-kserve-6647c6c947-dlfnb"]
Nov 01 15:33:14.249842 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:33:14.249814    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-4e07cb/sub-test-model-4e07cb-kserve-6647c6c947-dlfnb"
Nov 01 15:33:14.287971 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:33:14.287925    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f69qn\" (UniqueName: \"kubernetes.io/projected/6f4a1651-eba0-4c6c-a15f-0192f3167fb9-kube-api-access-f69qn\") pod \"sub-test-model-4e07cb-kserve-6647c6c947-dlfnb\" (UID: \"6f4a1651-eba0-4c6c-a15f-0192f3167fb9\") " pod="ai-tenant-e2e-shared-a-w2-4e07cb/sub-test-model-4e07cb-kserve-6647c6c947-dlfnb"
Nov 01 15:33:14.288150 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:33:14.288032    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f4a1651-eba0-4c6c-a15f-0192f3167fb9-tmp-dir\") pod \"sub-test-model-4e07cb-kserve-6647c6c947-dlfnb\" (UID: \"6f4a1651-eba0-4c6c-a15f-0192f3167fb9\") " pod="ai-tenant-e2e-shared-a-w2-4e07cb/sub-test-model-4e07cb-kserve-6647c6c947-dlfnb"
Nov 01 15:33:14.288150 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:33:14.288082    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f4a1651-eba0-4c6c-a15f-0192f3167fb9-model-cache\") pod \"sub-test-model-4e07cb-kserve-6647c6c947-dlfnb\" (UID: \"6f4a1651-eba0-4c6c-a15f-0192f3167fb9\") " pod="ai-tenant-e2e-shared-a-w2-4e07cb/sub-test-model-4e07cb-kserve-6647c6c947-dlfnb"
Nov 01 15:33:14.288359 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:33:14.288287    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f4a1651-eba0-4c6c-a15f-0192f3167fb9-home\") pod \"sub-test-model-4e07cb-kserve-6647c6c947-dlfnb\" (UID: \"6f4a1651-eba0-4c6c-a15f-0192f3167fb9\") " pod="ai-tenant-e2e-shared-a-w2-4e07cb/sub-test-model-4e07cb-kserve-6647c6c947-dlfnb"
Nov 01 15:33:14.288453 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:33:14.288399    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f4a1651-eba0-4c6c-a15f-0192f3167fb9-tls-certs\") pod \"sub-test-model-4e07cb-kserve-6647c6c947-dlfnb\" (UID: \"6f4a1651-eba0-4c6c-a15f-0192f3167fb9\") " pod="ai-tenant-e2e-shared-a-w2-4e07cb/sub-test-model-4e07cb-kserve-6647c6c947-dlfnb"
Nov 01 15:33:14.288513 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:33:14.288451    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f4a1651-eba0-4c6c-a15f-0192f3167fb9-dshm\") pod \"sub-test-model-4e07cb-kserve-6647c6c947-dlfnb\" (UID: \"6f4a1651-eba0-4c6c-a15f-0192f3167fb9\") " pod="ai-tenant-e2e-shared-a-w2-4e07cb/sub-test-model-4e07cb-kserve-6647c6c947-dlfnb"
Nov 01 15:33:14.299842 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:33:14.299814    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-4e07cb\"/\"openshift-service-ca.crt\""
Nov 01 15:33:14.346290 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:33:14.346141    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-4e07cb\"/\"default-dockercfg-vsng9\""
Nov 01 15:33:14.346290 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:33:14.346181    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-4e07cb\"/\"sub-test-model-4e07cb-kserve-self-signed-certs\""
Nov 01 15:33:14.346290 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:33:14.346244    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-4e07cb\"/\"kube-root-ca.crt\""
Nov 01 15:33:14.349557 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:33:14.349531    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-4e07cb/sub-test-model-4e07cb-kserve-6647c6c947-dlfnb"]
Nov 01 15:33:14.389455 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:33:14.389412    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f69qn\" (UniqueName: \"kubernetes.io/projected/6f4a1651-eba0-4c6c-a15f-0192f3167fb9-kube-api-access-f69qn\") pod \"sub-test-model-4e07cb-kserve-6647c6c947-dlfnb\" (UID: \"6f4a1651-eba0-4c6c-a15f-0192f3167fb9\") " pod="ai-tenant-e2e-shared-a-w2-4e07cb/sub-test-model-4e07cb-kserve-6647c6c947-dlfnb"
Nov 01 15:33:14.389647 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:33:14.389516    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f4a1651-eba0-4c6c-a15f-0192f3167fb9-tmp-dir\") pod \"sub-test-model-4e07cb-kserve-6647c6c947-dlfnb\" (UID: \"6f4a1651-eba0-4c6c-a15f-0192f3167fb9\") " pod="ai-tenant-e2e-shared-a-w2-4e07cb/sub-test-model-4e07cb-kserve-6647c6c947-dlfnb"
Nov 01 15:33:14.389647 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:33:14.389552    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f4a1651-eba0-4c6c-a15f-0192f3167fb9-model-cache\") pod \"sub-test-model-4e07cb-kserve-6647c6c947-dlfnb\" (UID: \"6f4a1651-eba0-4c6c-a15f-0192f3167fb9\") " pod="ai-tenant-e2e-shared-a-w2-4e07cb/sub-test-model-4e07cb-kserve-6647c6c947-dlfnb"
Nov 01 15:33:14.389647 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:33:14.389629    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f4a1651-eba0-4c6c-a15f-0192f3167fb9-home\") pod \"sub-test-model-4e07cb-kserve-6647c6c947-dlfnb\" (UID: \"6f4a1651-eba0-4c6c-a15f-0192f3167fb9\") " pod="ai-tenant-e2e-shared-a-w2-4e07cb/sub-test-model-4e07cb-kserve-6647c6c947-dlfnb"
Nov 01 15:33:14.389818 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:33:14.389667    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f4a1651-eba0-4c6c-a15f-0192f3167fb9-tls-certs\") pod \"sub-test-model-4e07cb-kserve-6647c6c947-dlfnb\" (UID: \"6f4a1651-eba0-4c6c-a15f-0192f3167fb9\") " pod="ai-tenant-e2e-shared-a-w2-4e07cb/sub-test-model-4e07cb-kserve-6647c6c947-dlfnb"
Nov 01 15:33:14.389818 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:33:14.389721    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f4a1651-eba0-4c6c-a15f-0192f3167fb9-dshm\") pod \"sub-test-model-4e07cb-kserve-6647c6c947-dlfnb\" (UID: \"6f4a1651-eba0-4c6c-a15f-0192f3167fb9\") " pod="ai-tenant-e2e-shared-a-w2-4e07cb/sub-test-model-4e07cb-kserve-6647c6c947-dlfnb"
Nov 01 15:33:14.390034 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:33:14.390006    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f4a1651-eba0-4c6c-a15f-0192f3167fb9-tmp-dir\") pod \"sub-test-model-4e07cb-kserve-6647c6c947-dlfnb\" (UID: \"6f4a1651-eba0-4c6c-a15f-0192f3167fb9\") " pod="ai-tenant-e2e-shared-a-w2-4e07cb/sub-test-model-4e07cb-kserve-6647c6c947-dlfnb"
Nov 01 15:33:14.390138 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:33:14.390120    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f4a1651-eba0-4c6c-a15f-0192f3167fb9-model-cache\") pod \"sub-test-model-4e07cb-kserve-6647c6c947-dlfnb\" (UID: \"6f4a1651-eba0-4c6c-a15f-0192f3167fb9\") " pod="ai-tenant-e2e-shared-a-w2-4e07cb/sub-test-model-4e07cb-kserve-6647c6c947-dlfnb"
Nov 01 15:33:14.390301 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:33:14.390278    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f4a1651-eba0-4c6c-a15f-0192f3167fb9-home\") pod \"sub-test-model-4e07cb-kserve-6647c6c947-dlfnb\" (UID: \"6f4a1651-eba0-4c6c-a15f-0192f3167fb9\") " pod="ai-tenant-e2e-shared-a-w2-4e07cb/sub-test-model-4e07cb-kserve-6647c6c947-dlfnb"
Nov 01 15:33:14.392266 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:33:14.392235    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f4a1651-eba0-4c6c-a15f-0192f3167fb9-dshm\") pod \"sub-test-model-4e07cb-kserve-6647c6c947-dlfnb\" (UID: \"6f4a1651-eba0-4c6c-a15f-0192f3167fb9\") " pod="ai-tenant-e2e-shared-a-w2-4e07cb/sub-test-model-4e07cb-kserve-6647c6c947-dlfnb"
Nov 01 15:33:14.392404 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:33:14.392386    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f4a1651-eba0-4c6c-a15f-0192f3167fb9-tls-certs\") pod \"sub-test-model-4e07cb-kserve-6647c6c947-dlfnb\" (UID: \"6f4a1651-eba0-4c6c-a15f-0192f3167fb9\") " pod="ai-tenant-e2e-shared-a-w2-4e07cb/sub-test-model-4e07cb-kserve-6647c6c947-dlfnb"
Nov 01 15:33:14.443154 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:33:14.443122    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f69qn\" (UniqueName: \"kubernetes.io/projected/6f4a1651-eba0-4c6c-a15f-0192f3167fb9-kube-api-access-f69qn\") pod \"sub-test-model-4e07cb-kserve-6647c6c947-dlfnb\" (UID: \"6f4a1651-eba0-4c6c-a15f-0192f3167fb9\") " pod="ai-tenant-e2e-shared-a-w2-4e07cb/sub-test-model-4e07cb-kserve-6647c6c947-dlfnb"
Nov 01 15:33:14.562809 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:33:14.562714    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-4e07cb/sub-test-model-4e07cb-kserve-6647c6c947-dlfnb"
Nov 01 15:33:14.945421 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:33:14.945331    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-4e07cb/sub-test-model-4e07cb-kserve-6647c6c947-dlfnb"]
Nov 01 15:33:14.951412 ip-10-0-134-37 kubenswrapper[4993]: W1101 15:33:14.951379    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6f4a1651_eba0_4c6c_a15f_0192f3167fb9.slice/crio-08aca74ff0ae583f4f8f1d48cbe54937962210c18f866ff619fd9e7302141b64 WatchSource:0}: Error finding container 08aca74ff0ae583f4f8f1d48cbe54937962210c18f866ff619fd9e7302141b64: Status 404 returned error can't find the container with id 08aca74ff0ae583f4f8f1d48cbe54937962210c18f866ff619fd9e7302141b64
Nov 01 15:33:15.592672 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:33:15.592638    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-4e07cb/sub-test-model-4e07cb-kserve-6647c6c947-dlfnb" event={"ID":"6f4a1651-eba0-4c6c-a15f-0192f3167fb9","Type":"ContainerStarted","Data":"cc1bbb9b91a899df3a586e0276ebb14c20c2462ba7e52338d034bda7d19c8979"}
Nov 01 15:33:15.593092 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:33:15.592680    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-4e07cb/sub-test-model-4e07cb-kserve-6647c6c947-dlfnb" event={"ID":"6f4a1651-eba0-4c6c-a15f-0192f3167fb9","Type":"ContainerStarted","Data":"08aca74ff0ae583f4f8f1d48cbe54937962210c18f866ff619fd9e7302141b64"}
Nov 01 15:33:15.700978 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:33:15.700924    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-4e07cb/sub-test-model-4e07cb-kserve-6647c6c947-dlfnb" podStartSLOduration=1.700907984 podStartE2EDuration="1.700907984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:33:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:33:15.698486495 +0000 UTC m=+4583.028349233" watchObservedRunningTime="2026-11-01 15:33:15.700907984 +0000 UTC m=+4583.030770722"
Nov 01 15:33:16.563804 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:33:16.563763    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-4e07cb/sub-test-model-4e07cb-kserve-6647c6c947-dlfnb"
Nov 01 15:33:24.563747 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:33:24.563709    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-4e07cb/sub-test-model-4e07cb-kserve-6647c6c947-dlfnb"
Nov 01 15:33:24.577655 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:33:24.577620    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-4e07cb/sub-test-model-4e07cb-kserve-6647c6c947-dlfnb"
Nov 01 15:33:24.670631 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:33:24.670590    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-4e07cb/sub-test-model-4e07cb-kserve-6647c6c947-dlfnb"
Nov 01 15:33:27.501072 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:33:27.501036    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:33:57.500348 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:33:57.500305    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:34:21.636674 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:34:21.636586    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6a0fa146-c48bbd4f7-dh577"]
Nov 01 15:34:21.641096 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:34:21.641069    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a0fa146-c48bbd4f7-dh577"
Nov 01 15:34:21.690549 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:34:21.690487    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-6a0fa146-serving-cert\""
Nov 01 15:34:21.718101 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:34:21.718050    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/852aa9ab-4e1d-4798-9d0c-676d3ee14abe-openshift-service-ca\") pod \"maas-api-e2e-mt-6a0fa146-c48bbd4f7-dh577\" (UID: \"852aa9ab-4e1d-4798-9d0c-676d3ee14abe\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a0fa146-c48bbd4f7-dh577"
Nov 01 15:34:21.718293 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:34:21.718274    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6czd2\" (UniqueName: \"kubernetes.io/projected/852aa9ab-4e1d-4798-9d0c-676d3ee14abe-kube-api-access-6czd2\") pod \"maas-api-e2e-mt-6a0fa146-c48bbd4f7-dh577\" (UID: \"852aa9ab-4e1d-4798-9d0c-676d3ee14abe\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a0fa146-c48bbd4f7-dh577"
Nov 01 15:34:21.718465 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:34:21.718429    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/852aa9ab-4e1d-4798-9d0c-676d3ee14abe-maas-api-tls\") pod \"maas-api-e2e-mt-6a0fa146-c48bbd4f7-dh577\" (UID: \"852aa9ab-4e1d-4798-9d0c-676d3ee14abe\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a0fa146-c48bbd4f7-dh577"
Nov 01 15:34:21.718602 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:34:21.718541    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/852aa9ab-4e1d-4798-9d0c-676d3ee14abe-metrics-certs\") pod \"maas-api-e2e-mt-6a0fa146-c48bbd4f7-dh577\" (UID: \"852aa9ab-4e1d-4798-9d0c-676d3ee14abe\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a0fa146-c48bbd4f7-dh577"
Nov 01 15:34:21.739461 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:34:21.739421    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6a0fa146-c48bbd4f7-dh577"]
Nov 01 15:34:21.820158 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:34:21.820126    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6czd2\" (UniqueName: \"kubernetes.io/projected/852aa9ab-4e1d-4798-9d0c-676d3ee14abe-kube-api-access-6czd2\") pod \"maas-api-e2e-mt-6a0fa146-c48bbd4f7-dh577\" (UID: \"852aa9ab-4e1d-4798-9d0c-676d3ee14abe\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a0fa146-c48bbd4f7-dh577"
Nov 01 15:34:21.820348 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:34:21.820248    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/852aa9ab-4e1d-4798-9d0c-676d3ee14abe-maas-api-tls\") pod \"maas-api-e2e-mt-6a0fa146-c48bbd4f7-dh577\" (UID: \"852aa9ab-4e1d-4798-9d0c-676d3ee14abe\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a0fa146-c48bbd4f7-dh577"
Nov 01 15:34:21.820348 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:34:21.820316    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/852aa9ab-4e1d-4798-9d0c-676d3ee14abe-metrics-certs\") pod \"maas-api-e2e-mt-6a0fa146-c48bbd4f7-dh577\" (UID: \"852aa9ab-4e1d-4798-9d0c-676d3ee14abe\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a0fa146-c48bbd4f7-dh577"
Nov 01 15:34:21.820443 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:34:21.820363    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/852aa9ab-4e1d-4798-9d0c-676d3ee14abe-openshift-service-ca\") pod \"maas-api-e2e-mt-6a0fa146-c48bbd4f7-dh577\" (UID: \"852aa9ab-4e1d-4798-9d0c-676d3ee14abe\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a0fa146-c48bbd4f7-dh577"
Nov 01 15:34:21.820443 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:34:21.820413    4993 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-6a0fa146-serving-cert: secret "maas-api-e2e-mt-6a0fa146-serving-cert" not found
Nov 01 15:34:21.820529 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:34:21.820485    4993 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/852aa9ab-4e1d-4798-9d0c-676d3ee14abe-maas-api-tls podName:852aa9ab-4e1d-4798-9d0c-676d3ee14abe nodeName:}" failed. No retries permitted until 2026-11-01 15:34:22.320463245 +0000 UTC m=+4649.650325965 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/852aa9ab-4e1d-4798-9d0c-676d3ee14abe-maas-api-tls") pod "maas-api-e2e-mt-6a0fa146-c48bbd4f7-dh577" (UID: "852aa9ab-4e1d-4798-9d0c-676d3ee14abe") : secret "maas-api-e2e-mt-6a0fa146-serving-cert" not found
Nov 01 15:34:21.821022 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:34:21.821000    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/852aa9ab-4e1d-4798-9d0c-676d3ee14abe-openshift-service-ca\") pod \"maas-api-e2e-mt-6a0fa146-c48bbd4f7-dh577\" (UID: \"852aa9ab-4e1d-4798-9d0c-676d3ee14abe\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a0fa146-c48bbd4f7-dh577"
Nov 01 15:34:21.822791 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:34:21.822763    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/852aa9ab-4e1d-4798-9d0c-676d3ee14abe-metrics-certs\") pod \"maas-api-e2e-mt-6a0fa146-c48bbd4f7-dh577\" (UID: \"852aa9ab-4e1d-4798-9d0c-676d3ee14abe\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a0fa146-c48bbd4f7-dh577"
Nov 01 15:34:21.874143 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:34:21.874101    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6czd2\" (UniqueName: \"kubernetes.io/projected/852aa9ab-4e1d-4798-9d0c-676d3ee14abe-kube-api-access-6czd2\") pod \"maas-api-e2e-mt-6a0fa146-c48bbd4f7-dh577\" (UID: \"852aa9ab-4e1d-4798-9d0c-676d3ee14abe\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a0fa146-c48bbd4f7-dh577"
Nov 01 15:34:22.327747 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:34:22.327708    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/852aa9ab-4e1d-4798-9d0c-676d3ee14abe-maas-api-tls\") pod \"maas-api-e2e-mt-6a0fa146-c48bbd4f7-dh577\" (UID: \"852aa9ab-4e1d-4798-9d0c-676d3ee14abe\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a0fa146-c48bbd4f7-dh577"
Nov 01 15:34:22.330237 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:34:22.330206    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/852aa9ab-4e1d-4798-9d0c-676d3ee14abe-maas-api-tls\") pod \"maas-api-e2e-mt-6a0fa146-c48bbd4f7-dh577\" (UID: \"852aa9ab-4e1d-4798-9d0c-676d3ee14abe\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a0fa146-c48bbd4f7-dh577"
Nov 01 15:34:22.556294 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:34:22.556254    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a0fa146-c48bbd4f7-dh577"
Nov 01 15:34:22.933283 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:34:22.933193    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6a0fa146-c48bbd4f7-dh577"]
Nov 01 15:34:22.944005 ip-10-0-134-37 kubenswrapper[4993]: W1101 15:34:22.943969    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod852aa9ab_4e1d_4798_9d0c_676d3ee14abe.slice/crio-f7c83836a0c9f7b9901684bee2c5ff09244e6185d2f44a1b47897de71559974d WatchSource:0}: Error finding container f7c83836a0c9f7b9901684bee2c5ff09244e6185d2f44a1b47897de71559974d: Status 404 returned error can't find the container with id f7c83836a0c9f7b9901684bee2c5ff09244e6185d2f44a1b47897de71559974d
Nov 01 15:34:23.041694 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:34:23.041656    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a0fa146-c48bbd4f7-dh577" event={"ID":"852aa9ab-4e1d-4798-9d0c-676d3ee14abe","Type":"ContainerStarted","Data":"f7c83836a0c9f7b9901684bee2c5ff09244e6185d2f44a1b47897de71559974d"}
Nov 01 15:34:24.049897 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:34:24.049853    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a0fa146-c48bbd4f7-dh577" event={"ID":"852aa9ab-4e1d-4798-9d0c-676d3ee14abe","Type":"ContainerStarted","Data":"777042721ace705b95c586005dcd3c04bd8157f1a1b9b997557d5ba81c6f5de1"}
Nov 01 15:34:24.050322 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:34:24.049944    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a0fa146-c48bbd4f7-dh577"
Nov 01 15:34:24.156588 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:34:24.156483    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a0fa146-c48bbd4f7-dh577" podStartSLOduration=2.247338932 podStartE2EDuration="3.156469711s" totalImagesPullingTime="909.130779ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:34:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:34:24.155199665 +0000 UTC m=+4651.485062399" watchObservedRunningTime="2026-11-01 15:34:24.156469711 +0000 UTC m=+4651.486332447"
Nov 01 15:34:27.502700 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:34:27.502658    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:34:30.061781 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:34:30.061748    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a0fa146-c48bbd4f7-dh577"
Nov 01 15:34:57.500454 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:34:57.500419    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:35:27.502522 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:35:27.502485    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:35:38.955047 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:35:38.954956    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-8fd363/rate-test-model-8fd363-kserve-77db559f77-qtxzx"]
Nov 01 15:35:38.959645 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:35:38.959620    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-8fd363/rate-test-model-8fd363-kserve-77db559f77-qtxzx"
Nov 01 15:35:39.054991 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:35:39.054958    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-8fd363\"/\"openshift-service-ca.crt\""
Nov 01 15:35:39.055204 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:35:39.054991    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-8fd363\"/\"default-dockercfg-tbqbg\""
Nov 01 15:35:39.055204 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:35:39.055044    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-8fd363\"/\"kube-root-ca.crt\""
Nov 01 15:35:39.055335 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:35:39.055244    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-8fd363\"/\"rate-test-model-8fd363-kserve-self-signed-certs\""
Nov 01 15:35:39.058520 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:35:39.058496    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8fd363/rate-test-model-8fd363-kserve-77db559f77-qtxzx"]
Nov 01 15:35:39.071733 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:35:39.071713    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f43a7e63-3095-4ad7-935a-bb46a0af45b9-tmp-dir\") pod \"rate-test-model-8fd363-kserve-77db559f77-qtxzx\" (UID: \"f43a7e63-3095-4ad7-935a-bb46a0af45b9\") " pod="ai-tenant-e2e-shared-b-w3-8fd363/rate-test-model-8fd363-kserve-77db559f77-qtxzx"
Nov 01 15:35:39.071853 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:35:39.071748    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr66c\" (UniqueName: \"kubernetes.io/projected/f43a7e63-3095-4ad7-935a-bb46a0af45b9-kube-api-access-lr66c\") pod \"rate-test-model-8fd363-kserve-77db559f77-qtxzx\" (UID: \"f43a7e63-3095-4ad7-935a-bb46a0af45b9\") " pod="ai-tenant-e2e-shared-b-w3-8fd363/rate-test-model-8fd363-kserve-77db559f77-qtxzx"
Nov 01 15:35:39.071930 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:35:39.071914    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f43a7e63-3095-4ad7-935a-bb46a0af45b9-home\") pod \"rate-test-model-8fd363-kserve-77db559f77-qtxzx\" (UID: \"f43a7e63-3095-4ad7-935a-bb46a0af45b9\") " pod="ai-tenant-e2e-shared-b-w3-8fd363/rate-test-model-8fd363-kserve-77db559f77-qtxzx"
Nov 01 15:35:39.071966 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:35:39.071941    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f43a7e63-3095-4ad7-935a-bb46a0af45b9-dshm\") pod \"rate-test-model-8fd363-kserve-77db559f77-qtxzx\" (UID: \"f43a7e63-3095-4ad7-935a-bb46a0af45b9\") " pod="ai-tenant-e2e-shared-b-w3-8fd363/rate-test-model-8fd363-kserve-77db559f77-qtxzx"
Nov 01 15:35:39.072045 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:35:39.072028    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f43a7e63-3095-4ad7-935a-bb46a0af45b9-model-cache\") pod \"rate-test-model-8fd363-kserve-77db559f77-qtxzx\" (UID: \"f43a7e63-3095-4ad7-935a-bb46a0af45b9\") " pod="ai-tenant-e2e-shared-b-w3-8fd363/rate-test-model-8fd363-kserve-77db559f77-qtxzx"
Nov 01 15:35:39.072080 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:35:39.072065    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f43a7e63-3095-4ad7-935a-bb46a0af45b9-tls-certs\") pod \"rate-test-model-8fd363-kserve-77db559f77-qtxzx\" (UID: \"f43a7e63-3095-4ad7-935a-bb46a0af45b9\") " pod="ai-tenant-e2e-shared-b-w3-8fd363/rate-test-model-8fd363-kserve-77db559f77-qtxzx"
Nov 01 15:35:39.172766 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:35:39.172718    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f43a7e63-3095-4ad7-935a-bb46a0af45b9-model-cache\") pod \"rate-test-model-8fd363-kserve-77db559f77-qtxzx\" (UID: \"f43a7e63-3095-4ad7-935a-bb46a0af45b9\") " pod="ai-tenant-e2e-shared-b-w3-8fd363/rate-test-model-8fd363-kserve-77db559f77-qtxzx"
Nov 01 15:35:39.172972 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:35:39.172775    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f43a7e63-3095-4ad7-935a-bb46a0af45b9-tls-certs\") pod \"rate-test-model-8fd363-kserve-77db559f77-qtxzx\" (UID: \"f43a7e63-3095-4ad7-935a-bb46a0af45b9\") " pod="ai-tenant-e2e-shared-b-w3-8fd363/rate-test-model-8fd363-kserve-77db559f77-qtxzx"
Nov 01 15:35:39.172972 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:35:39.172886    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f43a7e63-3095-4ad7-935a-bb46a0af45b9-tmp-dir\") pod \"rate-test-model-8fd363-kserve-77db559f77-qtxzx\" (UID: \"f43a7e63-3095-4ad7-935a-bb46a0af45b9\") " pod="ai-tenant-e2e-shared-b-w3-8fd363/rate-test-model-8fd363-kserve-77db559f77-qtxzx"
Nov 01 15:35:39.172972 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:35:39.172926    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr66c\" (UniqueName: \"kubernetes.io/projected/f43a7e63-3095-4ad7-935a-bb46a0af45b9-kube-api-access-lr66c\") pod \"rate-test-model-8fd363-kserve-77db559f77-qtxzx\" (UID: \"f43a7e63-3095-4ad7-935a-bb46a0af45b9\") " pod="ai-tenant-e2e-shared-b-w3-8fd363/rate-test-model-8fd363-kserve-77db559f77-qtxzx"
Nov 01 15:35:39.173142 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:35:39.173083    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f43a7e63-3095-4ad7-935a-bb46a0af45b9-home\") pod \"rate-test-model-8fd363-kserve-77db559f77-qtxzx\" (UID: \"f43a7e63-3095-4ad7-935a-bb46a0af45b9\") " pod="ai-tenant-e2e-shared-b-w3-8fd363/rate-test-model-8fd363-kserve-77db559f77-qtxzx"
Nov 01 15:35:39.173142 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:35:39.173117    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f43a7e63-3095-4ad7-935a-bb46a0af45b9-dshm\") pod \"rate-test-model-8fd363-kserve-77db559f77-qtxzx\" (UID: \"f43a7e63-3095-4ad7-935a-bb46a0af45b9\") " pod="ai-tenant-e2e-shared-b-w3-8fd363/rate-test-model-8fd363-kserve-77db559f77-qtxzx"
Nov 01 15:35:39.173274 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:35:39.173236    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f43a7e63-3095-4ad7-935a-bb46a0af45b9-model-cache\") pod \"rate-test-model-8fd363-kserve-77db559f77-qtxzx\" (UID: \"f43a7e63-3095-4ad7-935a-bb46a0af45b9\") " pod="ai-tenant-e2e-shared-b-w3-8fd363/rate-test-model-8fd363-kserve-77db559f77-qtxzx"
Nov 01 15:35:39.173334 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:35:39.173292    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f43a7e63-3095-4ad7-935a-bb46a0af45b9-tmp-dir\") pod \"rate-test-model-8fd363-kserve-77db559f77-qtxzx\" (UID: \"f43a7e63-3095-4ad7-935a-bb46a0af45b9\") " pod="ai-tenant-e2e-shared-b-w3-8fd363/rate-test-model-8fd363-kserve-77db559f77-qtxzx"
Nov 01 15:35:39.173645 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:35:39.173601    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f43a7e63-3095-4ad7-935a-bb46a0af45b9-home\") pod \"rate-test-model-8fd363-kserve-77db559f77-qtxzx\" (UID: \"f43a7e63-3095-4ad7-935a-bb46a0af45b9\") " pod="ai-tenant-e2e-shared-b-w3-8fd363/rate-test-model-8fd363-kserve-77db559f77-qtxzx"
Nov 01 15:35:39.175671 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:35:39.175645    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f43a7e63-3095-4ad7-935a-bb46a0af45b9-dshm\") pod \"rate-test-model-8fd363-kserve-77db559f77-qtxzx\" (UID: \"f43a7e63-3095-4ad7-935a-bb46a0af45b9\") " pod="ai-tenant-e2e-shared-b-w3-8fd363/rate-test-model-8fd363-kserve-77db559f77-qtxzx"
Nov 01 15:35:39.176781 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:35:39.176754    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f43a7e63-3095-4ad7-935a-bb46a0af45b9-tls-certs\") pod \"rate-test-model-8fd363-kserve-77db559f77-qtxzx\" (UID: \"f43a7e63-3095-4ad7-935a-bb46a0af45b9\") " pod="ai-tenant-e2e-shared-b-w3-8fd363/rate-test-model-8fd363-kserve-77db559f77-qtxzx"
Nov 01 15:35:39.226034 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:35:39.225961    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr66c\" (UniqueName: \"kubernetes.io/projected/f43a7e63-3095-4ad7-935a-bb46a0af45b9-kube-api-access-lr66c\") pod \"rate-test-model-8fd363-kserve-77db559f77-qtxzx\" (UID: \"f43a7e63-3095-4ad7-935a-bb46a0af45b9\") " pod="ai-tenant-e2e-shared-b-w3-8fd363/rate-test-model-8fd363-kserve-77db559f77-qtxzx"
Nov 01 15:35:39.272525 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:35:39.272493    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-8fd363/rate-test-model-8fd363-kserve-77db559f77-qtxzx"
Nov 01 15:35:39.653890 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:35:39.653853    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8fd363/rate-test-model-8fd363-kserve-77db559f77-qtxzx"]
Nov 01 15:35:39.659405 ip-10-0-134-37 kubenswrapper[4993]: W1101 15:35:39.659373    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf43a7e63_3095_4ad7_935a_bb46a0af45b9.slice/crio-1eb1ad7a476188b1ae6aed008761d6239501601f870b141dc35aa48858b1e64e WatchSource:0}: Error finding container 1eb1ad7a476188b1ae6aed008761d6239501601f870b141dc35aa48858b1e64e: Status 404 returned error can't find the container with id 1eb1ad7a476188b1ae6aed008761d6239501601f870b141dc35aa48858b1e64e
Nov 01 15:35:40.563661 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:35:40.563619    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-8fd363/rate-test-model-8fd363-kserve-77db559f77-qtxzx" event={"ID":"f43a7e63-3095-4ad7-935a-bb46a0af45b9","Type":"ContainerStarted","Data":"8e3d3bd34f89853d772792bd5efd3163f5b04cac354b2e060abb82aae6813b84"}
Nov 01 15:35:40.564061 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:35:40.563669    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-8fd363/rate-test-model-8fd363-kserve-77db559f77-qtxzx" event={"ID":"f43a7e63-3095-4ad7-935a-bb46a0af45b9","Type":"ContainerStarted","Data":"1eb1ad7a476188b1ae6aed008761d6239501601f870b141dc35aa48858b1e64e"}
Nov 01 15:35:40.672297 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:35:40.672243    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-8fd363/rate-test-model-8fd363-kserve-77db559f77-qtxzx" podStartSLOduration=2.672229512 podStartE2EDuration="2.672229512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:35:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:35:40.669088897 +0000 UTC m=+4727.998951644" watchObservedRunningTime="2026-11-01 15:35:40.672229512 +0000 UTC m=+4728.002092248"
Nov 01 15:35:41.272981 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:35:41.272935    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-8fd363/rate-test-model-8fd363-kserve-77db559f77-qtxzx"
Nov 01 15:35:49.272958 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:35:49.272919    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-8fd363/rate-test-model-8fd363-kserve-77db559f77-qtxzx"
Nov 01 15:35:49.286127 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:35:49.286096    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-8fd363/rate-test-model-8fd363-kserve-77db559f77-qtxzx"
Nov 01 15:35:49.632787 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:35:49.632710    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-8fd363/rate-test-model-8fd363-kserve-77db559f77-qtxzx"
Nov 01 15:35:57.500137 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:35:57.500101    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:36:27.505198 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:36:27.505147    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:36:53.486492 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:36:53.486464    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xmwtp_bb59051d-c2df-4362-a157-00f14f915e79/console-operator/3.log"
Nov 01 15:36:53.500598 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:36:53.500573    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xmwtp_bb59051d-c2df-4362-a157-00f14f915e79/console-operator/3.log"
Nov 01 15:36:57.500728 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:36:57.500685    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:37:13.777274 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:37:13.777184    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-scdm5"]
Nov 01 15:37:13.777864 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:37:13.777427    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-scdm5" podUID="06108f6d-156c-47cc-b45f-3c421be59030" containerName="istio-proxy" containerID="cri-o://533dbef1755d193b2354d9f4659ddfffde69613ceed7dce0304838b6894d7a57" gracePeriod=30
Nov 01 15:37:15.094679 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:37:15.094632    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-scdm5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 15:37:15.095051 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:37:15.094702    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-scdm5" podUID="06108f6d-156c-47cc-b45f-3c421be59030" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 15:37:19.064328 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:37:19.064272    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-scdm5"
Nov 01 15:37:19.197794 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:37:19.197751    4993 generic.go:356] "Generic (PLEG): container finished" podID="06108f6d-156c-47cc-b45f-3c421be59030" containerID="533dbef1755d193b2354d9f4659ddfffde69613ceed7dce0304838b6894d7a57" exitCode=0
Nov 01 15:37:19.197953 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:37:19.197810    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-scdm5"
Nov 01 15:37:19.197953 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:37:19.197808    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-scdm5" event={"ID":"06108f6d-156c-47cc-b45f-3c421be59030","Type":"ContainerDied","Data":"533dbef1755d193b2354d9f4659ddfffde69613ceed7dce0304838b6894d7a57"}
Nov 01 15:37:19.198043 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:37:19.197965    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-scdm5" event={"ID":"06108f6d-156c-47cc-b45f-3c421be59030","Type":"ContainerDied","Data":"43a24eb8c8795fc2676abd46cdfb3bae5289a891ddaec40fe576990a311ca44b"}
Nov 01 15:37:19.198043 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:37:19.197989    4993 scope.go:122] "RemoveContainer" containerID="533dbef1755d193b2354d9f4659ddfffde69613ceed7dce0304838b6894d7a57"
Nov 01 15:37:19.210905 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:37:19.210888    4993 scope.go:122] "RemoveContainer" containerID="533dbef1755d193b2354d9f4659ddfffde69613ceed7dce0304838b6894d7a57"
Nov 01 15:37:19.211869 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:37:19.211841    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"533dbef1755d193b2354d9f4659ddfffde69613ceed7dce0304838b6894d7a57\": container with ID starting with 533dbef1755d193b2354d9f4659ddfffde69613ceed7dce0304838b6894d7a57 not found: ID does not exist" containerID="533dbef1755d193b2354d9f4659ddfffde69613ceed7dce0304838b6894d7a57"
Nov 01 15:37:19.211993 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:37:19.211877    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"533dbef1755d193b2354d9f4659ddfffde69613ceed7dce0304838b6894d7a57"} err="failed to get container status \"533dbef1755d193b2354d9f4659ddfffde69613ceed7dce0304838b6894d7a57\": rpc error: code = NotFound desc = could not find container \"533dbef1755d193b2354d9f4659ddfffde69613ceed7dce0304838b6894d7a57\": container with ID starting with 533dbef1755d193b2354d9f4659ddfffde69613ceed7dce0304838b6894d7a57 not found: ID does not exist"
Nov 01 15:37:19.212496 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:37:19.212473    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/06108f6d-156c-47cc-b45f-3c421be59030-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06108f6d-156c-47cc-b45f-3c421be59030-istio-podinfo\") pod \"06108f6d-156c-47cc-b45f-3c421be59030\" (UID: \"06108f6d-156c-47cc-b45f-3c421be59030\") "
Nov 01 15:37:19.212568 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:37:19.212551    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06108f6d-156c-47cc-b45f-3c421be59030-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06108f6d-156c-47cc-b45f-3c421be59030-istio-envoy\") pod \"06108f6d-156c-47cc-b45f-3c421be59030\" (UID: \"06108f6d-156c-47cc-b45f-3c421be59030\") "
Nov 01 15:37:19.212611 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:37:19.212601    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06108f6d-156c-47cc-b45f-3c421be59030-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06108f6d-156c-47cc-b45f-3c421be59030-istiod-ca-cert\") pod \"06108f6d-156c-47cc-b45f-3c421be59030\" (UID: \"06108f6d-156c-47cc-b45f-3c421be59030\") "
Nov 01 15:37:19.212668 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:37:19.212637    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06108f6d-156c-47cc-b45f-3c421be59030-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06108f6d-156c-47cc-b45f-3c421be59030-workload-certs\") pod \"06108f6d-156c-47cc-b45f-3c421be59030\" (UID: \"06108f6d-156c-47cc-b45f-3c421be59030\") "
Nov 01 15:37:19.212792 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:37:19.212771    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06108f6d-156c-47cc-b45f-3c421be59030-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06108f6d-156c-47cc-b45f-3c421be59030-workload-socket\") pod \"06108f6d-156c-47cc-b45f-3c421be59030\" (UID: \"06108f6d-156c-47cc-b45f-3c421be59030\") "
Nov 01 15:37:19.212882 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:37:19.212871    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06108f6d-156c-47cc-b45f-3c421be59030-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06108f6d-156c-47cc-b45f-3c421be59030-credential-socket\") pod \"06108f6d-156c-47cc-b45f-3c421be59030\" (UID: \"06108f6d-156c-47cc-b45f-3c421be59030\") "
Nov 01 15:37:19.212948 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:37:19.212911    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06108f6d-156c-47cc-b45f-3c421be59030-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06108f6d-156c-47cc-b45f-3c421be59030-istio-data\") pod \"06108f6d-156c-47cc-b45f-3c421be59030\" (UID: \"06108f6d-156c-47cc-b45f-3c421be59030\") "
Nov 01 15:37:19.212948 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:37:19.212938    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06108f6d-156c-47cc-b45f-3c421be59030-istio-token\" (UniqueName: \"kubernetes.io/projected/06108f6d-156c-47cc-b45f-3c421be59030-istio-token\") pod \"06108f6d-156c-47cc-b45f-3c421be59030\" (UID: \"06108f6d-156c-47cc-b45f-3c421be59030\") "
Nov 01 15:37:19.213067 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:37:19.212994    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06108f6d-156c-47cc-b45f-3c421be59030-workload-certs" pod "06108f6d-156c-47cc-b45f-3c421be59030" (UID: "06108f6d-156c-47cc-b45f-3c421be59030"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:37:19.213067 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:37:19.213027    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06108f6d-156c-47cc-b45f-3c421be59030-kube-api-access-cmbrp\" (UniqueName: \"kubernetes.io/projected/06108f6d-156c-47cc-b45f-3c421be59030-kube-api-access-cmbrp\") pod \"06108f6d-156c-47cc-b45f-3c421be59030\" (UID: \"06108f6d-156c-47cc-b45f-3c421be59030\") "
Nov 01 15:37:19.213067 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:37:19.213039    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06108f6d-156c-47cc-b45f-3c421be59030-workload-socket" pod "06108f6d-156c-47cc-b45f-3c421be59030" (UID: "06108f6d-156c-47cc-b45f-3c421be59030"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:37:19.213402 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:37:19.213158    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06108f6d-156c-47cc-b45f-3c421be59030-credential-socket" pod "06108f6d-156c-47cc-b45f-3c421be59030" (UID: "06108f6d-156c-47cc-b45f-3c421be59030"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:37:19.213402 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:37:19.213273    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06108f6d-156c-47cc-b45f-3c421be59030-istio-data" pod "06108f6d-156c-47cc-b45f-3c421be59030" (UID: "06108f6d-156c-47cc-b45f-3c421be59030"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:37:19.213677 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:37:19.213655    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06108f6d-156c-47cc-b45f-3c421be59030-istiod-ca-cert" pod "06108f6d-156c-47cc-b45f-3c421be59030" (UID: "06108f6d-156c-47cc-b45f-3c421be59030"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:37:19.213779 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:37:19.213733    4993 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06108f6d-156c-47cc-b45f-3c421be59030-credential-socket\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:37:19.213779 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:37:19.213756    4993 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06108f6d-156c-47cc-b45f-3c421be59030-istio-data\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:37:19.213779 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:37:19.213771    4993 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06108f6d-156c-47cc-b45f-3c421be59030-workload-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:37:19.213940 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:37:19.213800    4993 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06108f6d-156c-47cc-b45f-3c421be59030-workload-socket\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:37:19.214847 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:37:19.214824    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/06108f6d-156c-47cc-b45f-3c421be59030-istio-podinfo" pod "06108f6d-156c-47cc-b45f-3c421be59030" (UID: "06108f6d-156c-47cc-b45f-3c421be59030"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 15:37:19.215239 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:37:19.215214    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06108f6d-156c-47cc-b45f-3c421be59030-istio-envoy" pod "06108f6d-156c-47cc-b45f-3c421be59030" (UID: "06108f6d-156c-47cc-b45f-3c421be59030"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:37:19.215496 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:37:19.215476    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06108f6d-156c-47cc-b45f-3c421be59030-istio-token" pod "06108f6d-156c-47cc-b45f-3c421be59030" (UID: "06108f6d-156c-47cc-b45f-3c421be59030"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:37:19.215677 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:37:19.215659    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06108f6d-156c-47cc-b45f-3c421be59030-kube-api-access-cmbrp" pod "06108f6d-156c-47cc-b45f-3c421be59030" (UID: "06108f6d-156c-47cc-b45f-3c421be59030"). InnerVolumeSpecName "kube-api-access-cmbrp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:37:19.314728 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:37:19.314645    4993 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06108f6d-156c-47cc-b45f-3c421be59030-istio-token\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:37:19.314934 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:37:19.314914    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmbrp\" (UniqueName: \"kubernetes.io/projected/06108f6d-156c-47cc-b45f-3c421be59030-kube-api-access-cmbrp\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:37:19.315019 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:37:19.315003    4993 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06108f6d-156c-47cc-b45f-3c421be59030-istio-podinfo\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:37:19.315093 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:37:19.315032    4993 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06108f6d-156c-47cc-b45f-3c421be59030-istio-envoy\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:37:19.315093 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:37:19.315050    4993 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06108f6d-156c-47cc-b45f-3c421be59030-istiod-ca-cert\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:37:19.619677 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:37:19.619587    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-scdm5"]
Nov 01 15:37:19.624291 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:37:19.624261    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-scdm5"]
Nov 01 15:37:20.796062 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:37:20.796028    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06108f6d-156c-47cc-b45f-3c421be59030" path="/var/lib/kubelet/pods/06108f6d-156c-47cc-b45f-3c421be59030/volumes"
Nov 01 15:37:26.797717 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:37:26.797684    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6a0fa146-c48bbd4f7-dh577"]
Nov 01 15:37:26.798228 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:37:26.797855    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a0fa146-c48bbd4f7-dh577" podUID="852aa9ab-4e1d-4798-9d0c-676d3ee14abe" containerName="maas-api" containerID="cri-o://777042721ace705b95c586005dcd3c04bd8157f1a1b9b997557d5ba81c6f5de1" gracePeriod=30
Nov 01 15:37:27.102853 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:37:27.102824    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a0fa146-c48bbd4f7-dh577"
Nov 01 15:37:27.215289 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:37:27.215252    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/852aa9ab-4e1d-4798-9d0c-676d3ee14abe-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/852aa9ab-4e1d-4798-9d0c-676d3ee14abe-maas-api-tls\") pod \"852aa9ab-4e1d-4798-9d0c-676d3ee14abe\" (UID: \"852aa9ab-4e1d-4798-9d0c-676d3ee14abe\") "
Nov 01 15:37:27.216028 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:37:27.216002    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/852aa9ab-4e1d-4798-9d0c-676d3ee14abe-kube-api-access-6czd2\" (UniqueName: \"kubernetes.io/projected/852aa9ab-4e1d-4798-9d0c-676d3ee14abe-kube-api-access-6czd2\") pod \"852aa9ab-4e1d-4798-9d0c-676d3ee14abe\" (UID: \"852aa9ab-4e1d-4798-9d0c-676d3ee14abe\") "
Nov 01 15:37:27.216280 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:37:27.216253    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/852aa9ab-4e1d-4798-9d0c-676d3ee14abe-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/852aa9ab-4e1d-4798-9d0c-676d3ee14abe-openshift-service-ca\") pod \"852aa9ab-4e1d-4798-9d0c-676d3ee14abe\" (UID: \"852aa9ab-4e1d-4798-9d0c-676d3ee14abe\") "
Nov 01 15:37:27.216397 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:37:27.216299    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/852aa9ab-4e1d-4798-9d0c-676d3ee14abe-metrics-certs\" (UniqueName: \"kubernetes.io/secret/852aa9ab-4e1d-4798-9d0c-676d3ee14abe-metrics-certs\") pod \"852aa9ab-4e1d-4798-9d0c-676d3ee14abe\" (UID: \"852aa9ab-4e1d-4798-9d0c-676d3ee14abe\") "
Nov 01 15:37:27.235490 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:37:27.220781    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/852aa9ab-4e1d-4798-9d0c-676d3ee14abe-maas-api-tls" pod "852aa9ab-4e1d-4798-9d0c-676d3ee14abe" (UID: "852aa9ab-4e1d-4798-9d0c-676d3ee14abe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:37:27.235490 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:37:27.221156    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/852aa9ab-4e1d-4798-9d0c-676d3ee14abe-openshift-service-ca" pod "852aa9ab-4e1d-4798-9d0c-676d3ee14abe" (UID: "852aa9ab-4e1d-4798-9d0c-676d3ee14abe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:37:27.235490 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:37:27.224161    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/852aa9ab-4e1d-4798-9d0c-676d3ee14abe-kube-api-access-6czd2" pod "852aa9ab-4e1d-4798-9d0c-676d3ee14abe" (UID: "852aa9ab-4e1d-4798-9d0c-676d3ee14abe"). InnerVolumeSpecName "kube-api-access-6czd2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:37:27.235490 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:37:27.224557    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/852aa9ab-4e1d-4798-9d0c-676d3ee14abe-metrics-certs" pod "852aa9ab-4e1d-4798-9d0c-676d3ee14abe" (UID: "852aa9ab-4e1d-4798-9d0c-676d3ee14abe"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:37:27.253111 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:37:27.253079    4993 generic.go:356] "Generic (PLEG): container finished" podID="852aa9ab-4e1d-4798-9d0c-676d3ee14abe" containerID="777042721ace705b95c586005dcd3c04bd8157f1a1b9b997557d5ba81c6f5de1" exitCode=0
Nov 01 15:37:27.253338 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:37:27.253307    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a0fa146-c48bbd4f7-dh577" event={"ID":"852aa9ab-4e1d-4798-9d0c-676d3ee14abe","Type":"ContainerDied","Data":"777042721ace705b95c586005dcd3c04bd8157f1a1b9b997557d5ba81c6f5de1"}
Nov 01 15:37:27.253425 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:37:27.253376    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a0fa146-c48bbd4f7-dh577" event={"ID":"852aa9ab-4e1d-4798-9d0c-676d3ee14abe","Type":"ContainerDied","Data":"f7c83836a0c9f7b9901684bee2c5ff09244e6185d2f44a1b47897de71559974d"}
Nov 01 15:37:27.253425 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:37:27.253407    4993 scope.go:122] "RemoveContainer" containerID="777042721ace705b95c586005dcd3c04bd8157f1a1b9b997557d5ba81c6f5de1"
Nov 01 15:37:27.253627 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:37:27.253288    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a0fa146-c48bbd4f7-dh577"
Nov 01 15:37:27.266985 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:37:27.266957    4993 scope.go:122] "RemoveContainer" containerID="777042721ace705b95c586005dcd3c04bd8157f1a1b9b997557d5ba81c6f5de1"
Nov 01 15:37:27.267415 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:37:27.267393    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"777042721ace705b95c586005dcd3c04bd8157f1a1b9b997557d5ba81c6f5de1\": container with ID starting with 777042721ace705b95c586005dcd3c04bd8157f1a1b9b997557d5ba81c6f5de1 not found: ID does not exist" containerID="777042721ace705b95c586005dcd3c04bd8157f1a1b9b997557d5ba81c6f5de1"
Nov 01 15:37:27.267530 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:37:27.267422    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"777042721ace705b95c586005dcd3c04bd8157f1a1b9b997557d5ba81c6f5de1"} err="failed to get container status \"777042721ace705b95c586005dcd3c04bd8157f1a1b9b997557d5ba81c6f5de1\": rpc error: code = NotFound desc = could not find container \"777042721ace705b95c586005dcd3c04bd8157f1a1b9b997557d5ba81c6f5de1\": container with ID starting with 777042721ace705b95c586005dcd3c04bd8157f1a1b9b997557d5ba81c6f5de1 not found: ID does not exist"
Nov 01 15:37:27.317883 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:37:27.317852    4993 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/852aa9ab-4e1d-4798-9d0c-676d3ee14abe-openshift-service-ca\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:37:27.317883 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:37:27.317878    4993 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/852aa9ab-4e1d-4798-9d0c-676d3ee14abe-metrics-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:37:27.317883 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:37:27.317889    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/852aa9ab-4e1d-4798-9d0c-676d3ee14abe-maas-api-tls\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:37:27.318138 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:37:27.317899    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6czd2\" (UniqueName: \"kubernetes.io/projected/852aa9ab-4e1d-4798-9d0c-676d3ee14abe-kube-api-access-6czd2\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:37:27.406879 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:37:27.406800    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6a0fa146-c48bbd4f7-dh577"]
Nov 01 15:37:27.410576 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:37:27.410548    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6a0fa146-c48bbd4f7-dh577"]
Nov 01 15:37:27.502432 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:37:27.502389    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:37:28.796691 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:37:28.796656    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="852aa9ab-4e1d-4798-9d0c-676d3ee14abe" path="/var/lib/kubelet/pods/852aa9ab-4e1d-4798-9d0c-676d3ee14abe/volumes"
Nov 01 15:37:57.500475 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:37:57.500434    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:38:12.700162 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:38:12.700117    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-600cc7-6d484f766f-qcf6m"]
Nov 01 15:38:12.700630 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:38:12.700487    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-600cc7-6d484f766f-qcf6m" podUID="8bbc081b-ce18-4d0b-81ec-2bbf576f63f6" containerName="maas-api" containerID="cri-o://71fef5b419223bb1b90ef1c50964f7cbc3fdf5c57203218454ac3ee25678ee91" gracePeriod=30
Nov 01 15:38:13.011448 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:38:13.011415    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-600cc7-6d484f766f-qcf6m"
Nov 01 15:38:13.166085 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:38:13.166052    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bbc081b-ce18-4d0b-81ec-2bbf576f63f6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bbc081b-ce18-4d0b-81ec-2bbf576f63f6-openshift-service-ca\") pod \"8bbc081b-ce18-4d0b-81ec-2bbf576f63f6\" (UID: \"8bbc081b-ce18-4d0b-81ec-2bbf576f63f6\") "
Nov 01 15:38:13.166455 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:38:13.166434    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bbc081b-ce18-4d0b-81ec-2bbf576f63f6-kube-api-access-rx85z\" (UniqueName: \"kubernetes.io/projected/8bbc081b-ce18-4d0b-81ec-2bbf576f63f6-kube-api-access-rx85z\") pod \"8bbc081b-ce18-4d0b-81ec-2bbf576f63f6\" (UID: \"8bbc081b-ce18-4d0b-81ec-2bbf576f63f6\") "
Nov 01 15:38:13.166657 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:38:13.166642    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bbc081b-ce18-4d0b-81ec-2bbf576f63f6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bbc081b-ce18-4d0b-81ec-2bbf576f63f6-maas-api-tls\") pod \"8bbc081b-ce18-4d0b-81ec-2bbf576f63f6\" (UID: \"8bbc081b-ce18-4d0b-81ec-2bbf576f63f6\") "
Nov 01 15:38:13.166877 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:38:13.166863    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bbc081b-ce18-4d0b-81ec-2bbf576f63f6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bbc081b-ce18-4d0b-81ec-2bbf576f63f6-metrics-certs\") pod \"8bbc081b-ce18-4d0b-81ec-2bbf576f63f6\" (UID: \"8bbc081b-ce18-4d0b-81ec-2bbf576f63f6\") "
Nov 01 15:38:13.168401 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:38:13.168373    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bbc081b-ce18-4d0b-81ec-2bbf576f63f6-openshift-service-ca" pod "8bbc081b-ce18-4d0b-81ec-2bbf576f63f6" (UID: "8bbc081b-ce18-4d0b-81ec-2bbf576f63f6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:38:13.181402 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:38:13.181352    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bbc081b-ce18-4d0b-81ec-2bbf576f63f6-metrics-certs" pod "8bbc081b-ce18-4d0b-81ec-2bbf576f63f6" (UID: "8bbc081b-ce18-4d0b-81ec-2bbf576f63f6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:38:13.182718 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:38:13.182695    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bbc081b-ce18-4d0b-81ec-2bbf576f63f6-maas-api-tls" pod "8bbc081b-ce18-4d0b-81ec-2bbf576f63f6" (UID: "8bbc081b-ce18-4d0b-81ec-2bbf576f63f6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:38:13.182930 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:38:13.182915    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bbc081b-ce18-4d0b-81ec-2bbf576f63f6-kube-api-access-rx85z" pod "8bbc081b-ce18-4d0b-81ec-2bbf576f63f6" (UID: "8bbc081b-ce18-4d0b-81ec-2bbf576f63f6"). InnerVolumeSpecName "kube-api-access-rx85z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:38:13.268906 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:38:13.268860    4993 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bbc081b-ce18-4d0b-81ec-2bbf576f63f6-metrics-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:38:13.268906 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:38:13.268897    4993 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bbc081b-ce18-4d0b-81ec-2bbf576f63f6-openshift-service-ca\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:38:13.268906 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:38:13.268911    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rx85z\" (UniqueName: \"kubernetes.io/projected/8bbc081b-ce18-4d0b-81ec-2bbf576f63f6-kube-api-access-rx85z\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:38:13.269162 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:38:13.268923    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bbc081b-ce18-4d0b-81ec-2bbf576f63f6-maas-api-tls\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:38:13.554614 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:38:13.554519    4993 generic.go:356] "Generic (PLEG): container finished" podID="8bbc081b-ce18-4d0b-81ec-2bbf576f63f6" containerID="71fef5b419223bb1b90ef1c50964f7cbc3fdf5c57203218454ac3ee25678ee91" exitCode=0
Nov 01 15:38:13.554614 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:38:13.554557    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-600cc7-6d484f766f-qcf6m"
Nov 01 15:38:13.554614 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:38:13.554596    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-600cc7-6d484f766f-qcf6m" event={"ID":"8bbc081b-ce18-4d0b-81ec-2bbf576f63f6","Type":"ContainerDied","Data":"71fef5b419223bb1b90ef1c50964f7cbc3fdf5c57203218454ac3ee25678ee91"}
Nov 01 15:38:13.554868 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:38:13.554656    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-600cc7-6d484f766f-qcf6m" event={"ID":"8bbc081b-ce18-4d0b-81ec-2bbf576f63f6","Type":"ContainerDied","Data":"753a7eae7bf7d7ec4207e7dc98bd21dd8a2fc4116cb638c05a38fb5c53a9ed4c"}
Nov 01 15:38:13.554868 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:38:13.554685    4993 scope.go:122] "RemoveContainer" containerID="71fef5b419223bb1b90ef1c50964f7cbc3fdf5c57203218454ac3ee25678ee91"
Nov 01 15:38:13.566066 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:38:13.566046    4993 scope.go:122] "RemoveContainer" containerID="71fef5b419223bb1b90ef1c50964f7cbc3fdf5c57203218454ac3ee25678ee91"
Nov 01 15:38:13.566387 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:38:13.566369    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71fef5b419223bb1b90ef1c50964f7cbc3fdf5c57203218454ac3ee25678ee91\": container with ID starting with 71fef5b419223bb1b90ef1c50964f7cbc3fdf5c57203218454ac3ee25678ee91 not found: ID does not exist" containerID="71fef5b419223bb1b90ef1c50964f7cbc3fdf5c57203218454ac3ee25678ee91"
Nov 01 15:38:13.566463 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:38:13.566400    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71fef5b419223bb1b90ef1c50964f7cbc3fdf5c57203218454ac3ee25678ee91"} err="failed to get container status \"71fef5b419223bb1b90ef1c50964f7cbc3fdf5c57203218454ac3ee25678ee91\": rpc error: code = NotFound desc = could not find container \"71fef5b419223bb1b90ef1c50964f7cbc3fdf5c57203218454ac3ee25678ee91\": container with ID starting with 71fef5b419223bb1b90ef1c50964f7cbc3fdf5c57203218454ac3ee25678ee91 not found: ID does not exist"
Nov 01 15:38:13.711457 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:38:13.711410    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-600cc7-6d484f766f-qcf6m"]
Nov 01 15:38:13.714320 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:38:13.714293    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-600cc7-6d484f766f-qcf6m"]
Nov 01 15:38:14.797516 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:38:14.797484    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bbc081b-ce18-4d0b-81ec-2bbf576f63f6" path="/var/lib/kubelet/pods/8bbc081b-ce18-4d0b-81ec-2bbf576f63f6/volumes"
Nov 01 15:38:27.502307 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:38:27.502265    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:38:57.500879 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:38:57.500834    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:39:09.231748 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:39:09.231701    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-b3f06c-76587fff79-m6j9s"]
Nov 01 15:39:14.232353 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:39:14.232284    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b3f06c-76587fff79-m6j9s" podUID="7e3a0173-3e79-4e57-a9bb-0b9eb792da3a" containerName="payload-processing" containerID="cri-o://69d10cbf8d5bac171dfd74182175ac5dfe74e476b4059eadf53c8f9465e73ab0" gracePeriod=25
Nov 01 15:39:21.334063 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:39:21.334020    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b3f06c-76587fff79-m6j9s" podUID="7e3a0173-3e79-4e57-a9bb-0b9eb792da3a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 15:39:27.500227 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:39:27.500189    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:39:31.334302 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:39:31.334260    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b3f06c-76587fff79-m6j9s" podUID="7e3a0173-3e79-4e57-a9bb-0b9eb792da3a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 15:39:34.521179 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:39:34.521141    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b3f06c-76587fff79-m6j9s"
Nov 01 15:39:34.622315 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:39:34.622279    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e3a0173-3e79-4e57-a9bb-0b9eb792da3a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e3a0173-3e79-4e57-a9bb-0b9eb792da3a-plugins-config-volume\") pod \"7e3a0173-3e79-4e57-a9bb-0b9eb792da3a\" (UID: \"7e3a0173-3e79-4e57-a9bb-0b9eb792da3a\") "
Nov 01 15:39:34.622515 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:39:34.622381    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e3a0173-3e79-4e57-a9bb-0b9eb792da3a-kube-api-access-8tzbh\" (UniqueName: \"kubernetes.io/projected/7e3a0173-3e79-4e57-a9bb-0b9eb792da3a-kube-api-access-8tzbh\") pod \"7e3a0173-3e79-4e57-a9bb-0b9eb792da3a\" (UID: \"7e3a0173-3e79-4e57-a9bb-0b9eb792da3a\") "
Nov 01 15:39:34.622676 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:39:34.622641    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e3a0173-3e79-4e57-a9bb-0b9eb792da3a-plugins-config-volume" pod "7e3a0173-3e79-4e57-a9bb-0b9eb792da3a" (UID: "7e3a0173-3e79-4e57-a9bb-0b9eb792da3a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:39:34.622832 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:39:34.622814    4993 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e3a0173-3e79-4e57-a9bb-0b9eb792da3a-plugins-config-volume\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:39:34.624543 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:39:34.624522    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e3a0173-3e79-4e57-a9bb-0b9eb792da3a-kube-api-access-8tzbh" pod "7e3a0173-3e79-4e57-a9bb-0b9eb792da3a" (UID: "7e3a0173-3e79-4e57-a9bb-0b9eb792da3a"). InnerVolumeSpecName "kube-api-access-8tzbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:39:34.724569 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:39:34.724528    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tzbh\" (UniqueName: \"kubernetes.io/projected/7e3a0173-3e79-4e57-a9bb-0b9eb792da3a-kube-api-access-8tzbh\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:39:35.103545 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:39:35.103456    4993 generic.go:356] "Generic (PLEG): container finished" podID="7e3a0173-3e79-4e57-a9bb-0b9eb792da3a" containerID="69d10cbf8d5bac171dfd74182175ac5dfe74e476b4059eadf53c8f9465e73ab0" exitCode=0
Nov 01 15:39:35.103545 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:39:35.103496    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b3f06c-76587fff79-m6j9s"
Nov 01 15:39:35.103545 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:39:35.103512    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b3f06c-76587fff79-m6j9s" event={"ID":"7e3a0173-3e79-4e57-a9bb-0b9eb792da3a","Type":"ContainerDied","Data":"69d10cbf8d5bac171dfd74182175ac5dfe74e476b4059eadf53c8f9465e73ab0"}
Nov 01 15:39:35.103798 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:39:35.103564    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b3f06c-76587fff79-m6j9s" event={"ID":"7e3a0173-3e79-4e57-a9bb-0b9eb792da3a","Type":"ContainerDied","Data":"e735c611ecbad2532c5759db56e19f59253f5e0dabe297d4ae3311296aee734c"}
Nov 01 15:39:35.103798 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:39:35.103593    4993 scope.go:122] "RemoveContainer" containerID="69d10cbf8d5bac171dfd74182175ac5dfe74e476b4059eadf53c8f9465e73ab0"
Nov 01 15:39:35.113344 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:39:35.113322    4993 scope.go:122] "RemoveContainer" containerID="69d10cbf8d5bac171dfd74182175ac5dfe74e476b4059eadf53c8f9465e73ab0"
Nov 01 15:39:35.113614 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:39:35.113590    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69d10cbf8d5bac171dfd74182175ac5dfe74e476b4059eadf53c8f9465e73ab0\": container with ID starting with 69d10cbf8d5bac171dfd74182175ac5dfe74e476b4059eadf53c8f9465e73ab0 not found: ID does not exist" containerID="69d10cbf8d5bac171dfd74182175ac5dfe74e476b4059eadf53c8f9465e73ab0"
Nov 01 15:39:35.113714 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:39:35.113622    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69d10cbf8d5bac171dfd74182175ac5dfe74e476b4059eadf53c8f9465e73ab0"} err="failed to get container status \"69d10cbf8d5bac171dfd74182175ac5dfe74e476b4059eadf53c8f9465e73ab0\": rpc error: code = NotFound desc = could not find container \"69d10cbf8d5bac171dfd74182175ac5dfe74e476b4059eadf53c8f9465e73ab0\": container with ID starting with 69d10cbf8d5bac171dfd74182175ac5dfe74e476b4059eadf53c8f9465e73ab0 not found: ID does not exist"
Nov 01 15:39:35.256148 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:39:35.256113    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-b3f06c-76587fff79-m6j9s"]
Nov 01 15:39:35.302374 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:39:35.302342    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-b3f06c-76587fff79-m6j9s"]
Nov 01 15:39:36.796562 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:39:36.796528    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e3a0173-3e79-4e57-a9bb-0b9eb792da3a" path="/var/lib/kubelet/pods/7e3a0173-3e79-4e57-a9bb-0b9eb792da3a/volumes"
Nov 01 15:39:44.178997 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:39:44.178966    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-4e07cb/sub-test-model-4e07cb-kserve-6647c6c947-dlfnb"]
Nov 01 15:39:55.180978 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:39:55.180934    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1f8e2c/distinct-w0-1f8e2c-kserve-6657447954-rpml6"]
Nov 01 15:39:55.224919 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:39:55.224889    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1f8e2c/simulator-w0-1f8e2c-kserve-755bf645d-zf8t4"]
Nov 01 15:39:57.502011 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:39:57.501902    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:39:59.200640 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:39:59.200596    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-4e07cb/sub-test-model-4e07cb-kserve-6647c6c947-dlfnb" podUID="6f4a1651-eba0-4c6c-a15f-0192f3167fb9" containerName="main" containerID="cri-o://cc1bbb9b91a899df3a586e0276ebb14c20c2462ba7e52338d034bda7d19c8979" gracePeriod=45
Nov 01 15:39:59.480765 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:39:59.480733    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-4e07cb/sub-test-model-4e07cb-kserve-6647c6c947-dlfnb"
Nov 01 15:39:59.628905 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:39:59.628873    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f4a1651-eba0-4c6c-a15f-0192f3167fb9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f4a1651-eba0-4c6c-a15f-0192f3167fb9-dshm\") pod \"6f4a1651-eba0-4c6c-a15f-0192f3167fb9\" (UID: \"6f4a1651-eba0-4c6c-a15f-0192f3167fb9\") "
Nov 01 15:39:59.629103 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:39:59.628981    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f4a1651-eba0-4c6c-a15f-0192f3167fb9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f4a1651-eba0-4c6c-a15f-0192f3167fb9-model-cache\") pod \"6f4a1651-eba0-4c6c-a15f-0192f3167fb9\" (UID: \"6f4a1651-eba0-4c6c-a15f-0192f3167fb9\") "
Nov 01 15:39:59.629103 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:39:59.629042    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f4a1651-eba0-4c6c-a15f-0192f3167fb9-kube-api-access-f69qn\" (UniqueName: \"kubernetes.io/projected/6f4a1651-eba0-4c6c-a15f-0192f3167fb9-kube-api-access-f69qn\") pod \"6f4a1651-eba0-4c6c-a15f-0192f3167fb9\" (UID: \"6f4a1651-eba0-4c6c-a15f-0192f3167fb9\") "
Nov 01 15:39:59.629103 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:39:59.629067    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f4a1651-eba0-4c6c-a15f-0192f3167fb9-tls-certs\" (UniqueName: \"kubernetes.io/secret/6f4a1651-eba0-4c6c-a15f-0192f3167fb9-tls-certs\") pod \"6f4a1651-eba0-4c6c-a15f-0192f3167fb9\" (UID: \"6f4a1651-eba0-4c6c-a15f-0192f3167fb9\") "
Nov 01 15:39:59.629342 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:39:59.629212    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f4a1651-eba0-4c6c-a15f-0192f3167fb9-home\" (UniqueName: \"kubernetes.io/empty-dir/6f4a1651-eba0-4c6c-a15f-0192f3167fb9-home\") pod \"6f4a1651-eba0-4c6c-a15f-0192f3167fb9\" (UID: \"6f4a1651-eba0-4c6c-a15f-0192f3167fb9\") "
Nov 01 15:39:59.629342 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:39:59.629256    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f4a1651-eba0-4c6c-a15f-0192f3167fb9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f4a1651-eba0-4c6c-a15f-0192f3167fb9-tmp-dir\") pod \"6f4a1651-eba0-4c6c-a15f-0192f3167fb9\" (UID: \"6f4a1651-eba0-4c6c-a15f-0192f3167fb9\") "
Nov 01 15:39:59.629342 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:39:59.629305    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f4a1651-eba0-4c6c-a15f-0192f3167fb9-model-cache" pod "6f4a1651-eba0-4c6c-a15f-0192f3167fb9" (UID: "6f4a1651-eba0-4c6c-a15f-0192f3167fb9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:39:59.629540 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:39:59.629512    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f4a1651-eba0-4c6c-a15f-0192f3167fb9-home" pod "6f4a1651-eba0-4c6c-a15f-0192f3167fb9" (UID: "6f4a1651-eba0-4c6c-a15f-0192f3167fb9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:39:59.629612 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:39:59.629579    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f4a1651-eba0-4c6c-a15f-0192f3167fb9-tmp-dir" pod "6f4a1651-eba0-4c6c-a15f-0192f3167fb9" (UID: "6f4a1651-eba0-4c6c-a15f-0192f3167fb9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:39:59.629777 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:39:59.629758    4993 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f4a1651-eba0-4c6c-a15f-0192f3167fb9-model-cache\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:39:59.629858 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:39:59.629781    4993 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f4a1651-eba0-4c6c-a15f-0192f3167fb9-home\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:39:59.629858 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:39:59.629797    4993 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f4a1651-eba0-4c6c-a15f-0192f3167fb9-tmp-dir\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:39:59.631425 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:39:59.631402    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f4a1651-eba0-4c6c-a15f-0192f3167fb9-dshm" pod "6f4a1651-eba0-4c6c-a15f-0192f3167fb9" (UID: "6f4a1651-eba0-4c6c-a15f-0192f3167fb9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:39:59.631712 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:39:59.631691    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f4a1651-eba0-4c6c-a15f-0192f3167fb9-tls-certs" pod "6f4a1651-eba0-4c6c-a15f-0192f3167fb9" (UID: "6f4a1651-eba0-4c6c-a15f-0192f3167fb9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:39:59.631774 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:39:59.631752    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f4a1651-eba0-4c6c-a15f-0192f3167fb9-kube-api-access-f69qn" pod "6f4a1651-eba0-4c6c-a15f-0192f3167fb9" (UID: "6f4a1651-eba0-4c6c-a15f-0192f3167fb9"). InnerVolumeSpecName "kube-api-access-f69qn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:39:59.731207 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:39:59.731101    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f69qn\" (UniqueName: \"kubernetes.io/projected/6f4a1651-eba0-4c6c-a15f-0192f3167fb9-kube-api-access-f69qn\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:39:59.731207 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:39:59.731135    4993 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f4a1651-eba0-4c6c-a15f-0192f3167fb9-tls-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:39:59.731207 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:39:59.731146    4993 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f4a1651-eba0-4c6c-a15f-0192f3167fb9-dshm\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:40:00.284475 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:00.284437    4993 generic.go:356] "Generic (PLEG): container finished" podID="6f4a1651-eba0-4c6c-a15f-0192f3167fb9" containerID="cc1bbb9b91a899df3a586e0276ebb14c20c2462ba7e52338d034bda7d19c8979" exitCode=0
Nov 01 15:40:00.284969 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:00.284502    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-4e07cb/sub-test-model-4e07cb-kserve-6647c6c947-dlfnb"
Nov 01 15:40:00.284969 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:00.284526    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-4e07cb/sub-test-model-4e07cb-kserve-6647c6c947-dlfnb" event={"ID":"6f4a1651-eba0-4c6c-a15f-0192f3167fb9","Type":"ContainerDied","Data":"cc1bbb9b91a899df3a586e0276ebb14c20c2462ba7e52338d034bda7d19c8979"}
Nov 01 15:40:00.284969 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:00.284582    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-4e07cb/sub-test-model-4e07cb-kserve-6647c6c947-dlfnb" event={"ID":"6f4a1651-eba0-4c6c-a15f-0192f3167fb9","Type":"ContainerDied","Data":"08aca74ff0ae583f4f8f1d48cbe54937962210c18f866ff619fd9e7302141b64"}
Nov 01 15:40:00.284969 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:00.284604    4993 scope.go:122] "RemoveContainer" containerID="cc1bbb9b91a899df3a586e0276ebb14c20c2462ba7e52338d034bda7d19c8979"
Nov 01 15:40:00.297421 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:00.297395    4993 scope.go:122] "RemoveContainer" containerID="cc1bbb9b91a899df3a586e0276ebb14c20c2462ba7e52338d034bda7d19c8979"
Nov 01 15:40:00.297740 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:40:00.297716    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc1bbb9b91a899df3a586e0276ebb14c20c2462ba7e52338d034bda7d19c8979\": container with ID starting with cc1bbb9b91a899df3a586e0276ebb14c20c2462ba7e52338d034bda7d19c8979 not found: ID does not exist" containerID="cc1bbb9b91a899df3a586e0276ebb14c20c2462ba7e52338d034bda7d19c8979"
Nov 01 15:40:00.297816 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:00.297755    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc1bbb9b91a899df3a586e0276ebb14c20c2462ba7e52338d034bda7d19c8979"} err="failed to get container status \"cc1bbb9b91a899df3a586e0276ebb14c20c2462ba7e52338d034bda7d19c8979\": rpc error: code = NotFound desc = could not find container \"cc1bbb9b91a899df3a586e0276ebb14c20c2462ba7e52338d034bda7d19c8979\": container with ID starting with cc1bbb9b91a899df3a586e0276ebb14c20c2462ba7e52338d034bda7d19c8979 not found: ID does not exist"
Nov 01 15:40:00.439963 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:00.439931    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-4e07cb/sub-test-model-4e07cb-kserve-6647c6c947-dlfnb"]
Nov 01 15:40:00.443050 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:00.443025    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-4e07cb/sub-test-model-4e07cb-kserve-6647c6c947-dlfnb"]
Nov 01 15:40:00.797580 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:00.797538    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f4a1651-eba0-4c6c-a15f-0192f3167fb9" path="/var/lib/kubelet/pods/6f4a1651-eba0-4c6c-a15f-0192f3167fb9/volumes"
Nov 01 15:40:10.205324 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:10.205225    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-1f8e2c/distinct-w0-1f8e2c-kserve-6657447954-rpml6" podUID="39e0e8e2-87d8-4332-bb5d-86f3849861be" containerName="main" containerID="cri-o://bede0709c95e1ff913eb37843cb9a91670f153f524b63cbb214fea3f29dc4866" gracePeriod=45
Nov 01 15:40:10.246125 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:10.246088    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-1f8e2c/simulator-w0-1f8e2c-kserve-755bf645d-zf8t4" podUID="bbc61e12-d326-4d31-be81-88fb641337c7" containerName="main" containerID="cri-o://b6bce2c001544b5a381a1bd23a0fb0699996dd3fcbf4e57c232d7815539ecfad" gracePeriod=45
Nov 01 15:40:10.286967 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:10.286926    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f8e2c/simulator-w0-1f8e2c-kserve-755bf645d-zf8t4" podUID="bbc61e12-d326-4d31-be81-88fb641337c7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": dial tcp 10.132.0.62:8000: connect: connection refused"
Nov 01 15:40:10.306382 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:10.306341    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f8e2c/distinct-w0-1f8e2c-kserve-6657447954-rpml6" podUID="39e0e8e2-87d8-4332-bb5d-86f3849861be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": dial tcp 10.132.0.77:8000: connect: connection refused"
Nov 01 15:40:10.362694 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:10.362615    4993 generic.go:356] "Generic (PLEG): container finished" podID="39e0e8e2-87d8-4332-bb5d-86f3849861be" containerID="bede0709c95e1ff913eb37843cb9a91670f153f524b63cbb214fea3f29dc4866" exitCode=0
Nov 01 15:40:10.362694 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:10.362678    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1f8e2c/distinct-w0-1f8e2c-kserve-6657447954-rpml6" event={"ID":"39e0e8e2-87d8-4332-bb5d-86f3849861be","Type":"ContainerDied","Data":"bede0709c95e1ff913eb37843cb9a91670f153f524b63cbb214fea3f29dc4866"}
Nov 01 15:40:10.586985 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:10.586944    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1f8e2c/simulator-w0-1f8e2c-kserve-755bf645d-zf8t4"
Nov 01 15:40:10.590667 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:10.590642    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1f8e2c/distinct-w0-1f8e2c-kserve-6657447954-rpml6"
Nov 01 15:40:10.669336 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:10.669306    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39e0e8e2-87d8-4332-bb5d-86f3849861be-home\" (UniqueName: \"kubernetes.io/empty-dir/39e0e8e2-87d8-4332-bb5d-86f3849861be-home\") pod \"39e0e8e2-87d8-4332-bb5d-86f3849861be\" (UID: \"39e0e8e2-87d8-4332-bb5d-86f3849861be\") "
Nov 01 15:40:10.669536 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:10.669345    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbc61e12-d326-4d31-be81-88fb641337c7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbc61e12-d326-4d31-be81-88fb641337c7-tmp-dir\") pod \"bbc61e12-d326-4d31-be81-88fb641337c7\" (UID: \"bbc61e12-d326-4d31-be81-88fb641337c7\") "
Nov 01 15:40:10.669536 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:10.669378    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39e0e8e2-87d8-4332-bb5d-86f3849861be-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39e0e8e2-87d8-4332-bb5d-86f3849861be-tmp-dir\") pod \"39e0e8e2-87d8-4332-bb5d-86f3849861be\" (UID: \"39e0e8e2-87d8-4332-bb5d-86f3849861be\") "
Nov 01 15:40:10.669536 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:10.669416    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbc61e12-d326-4d31-be81-88fb641337c7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbc61e12-d326-4d31-be81-88fb641337c7-dshm\") pod \"bbc61e12-d326-4d31-be81-88fb641337c7\" (UID: \"bbc61e12-d326-4d31-be81-88fb641337c7\") "
Nov 01 15:40:10.669536 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:10.669446    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbc61e12-d326-4d31-be81-88fb641337c7-kube-api-access-fvt7c\" (UniqueName: \"kubernetes.io/projected/bbc61e12-d326-4d31-be81-88fb641337c7-kube-api-access-fvt7c\") pod \"bbc61e12-d326-4d31-be81-88fb641337c7\" (UID: \"bbc61e12-d326-4d31-be81-88fb641337c7\") "
Nov 01 15:40:10.669536 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:10.669478    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39e0e8e2-87d8-4332-bb5d-86f3849861be-dshm\" (UniqueName: \"kubernetes.io/empty-dir/39e0e8e2-87d8-4332-bb5d-86f3849861be-dshm\") pod \"39e0e8e2-87d8-4332-bb5d-86f3849861be\" (UID: \"39e0e8e2-87d8-4332-bb5d-86f3849861be\") "
Nov 01 15:40:10.669536 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:10.669507    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39e0e8e2-87d8-4332-bb5d-86f3849861be-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39e0e8e2-87d8-4332-bb5d-86f3849861be-model-cache\") pod \"39e0e8e2-87d8-4332-bb5d-86f3849861be\" (UID: \"39e0e8e2-87d8-4332-bb5d-86f3849861be\") "
Nov 01 15:40:10.669878 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:10.669602    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbc61e12-d326-4d31-be81-88fb641337c7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbc61e12-d326-4d31-be81-88fb641337c7-model-cache\") pod \"bbc61e12-d326-4d31-be81-88fb641337c7\" (UID: \"bbc61e12-d326-4d31-be81-88fb641337c7\") "
Nov 01 15:40:10.669878 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:10.669653    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbc61e12-d326-4d31-be81-88fb641337c7-tls-certs\" (UniqueName: \"kubernetes.io/secret/bbc61e12-d326-4d31-be81-88fb641337c7-tls-certs\") pod \"bbc61e12-d326-4d31-be81-88fb641337c7\" (UID: \"bbc61e12-d326-4d31-be81-88fb641337c7\") "
Nov 01 15:40:10.669878 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:10.669667    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbc61e12-d326-4d31-be81-88fb641337c7-tmp-dir" pod "bbc61e12-d326-4d31-be81-88fb641337c7" (UID: "bbc61e12-d326-4d31-be81-88fb641337c7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:40:10.669878 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:10.669688    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39e0e8e2-87d8-4332-bb5d-86f3849861be-home" pod "39e0e8e2-87d8-4332-bb5d-86f3849861be" (UID: "39e0e8e2-87d8-4332-bb5d-86f3849861be"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:40:10.669878 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:10.669703    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39e0e8e2-87d8-4332-bb5d-86f3849861be-tmp-dir" pod "39e0e8e2-87d8-4332-bb5d-86f3849861be" (UID: "39e0e8e2-87d8-4332-bb5d-86f3849861be"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:40:10.669878 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:10.669714    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39e0e8e2-87d8-4332-bb5d-86f3849861be-tls-certs\" (UniqueName: \"kubernetes.io/secret/39e0e8e2-87d8-4332-bb5d-86f3849861be-tls-certs\") pod \"39e0e8e2-87d8-4332-bb5d-86f3849861be\" (UID: \"39e0e8e2-87d8-4332-bb5d-86f3849861be\") "
Nov 01 15:40:10.669878 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:10.669773    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39e0e8e2-87d8-4332-bb5d-86f3849861be-kube-api-access-fhxcf\" (UniqueName: \"kubernetes.io/projected/39e0e8e2-87d8-4332-bb5d-86f3849861be-kube-api-access-fhxcf\") pod \"39e0e8e2-87d8-4332-bb5d-86f3849861be\" (UID: \"39e0e8e2-87d8-4332-bb5d-86f3849861be\") "
Nov 01 15:40:10.670285 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:10.669907    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbc61e12-d326-4d31-be81-88fb641337c7-model-cache" pod "bbc61e12-d326-4d31-be81-88fb641337c7" (UID: "bbc61e12-d326-4d31-be81-88fb641337c7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:40:10.670285 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:10.669968    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbc61e12-d326-4d31-be81-88fb641337c7-home\" (UniqueName: \"kubernetes.io/empty-dir/bbc61e12-d326-4d31-be81-88fb641337c7-home\") pod \"bbc61e12-d326-4d31-be81-88fb641337c7\" (UID: \"bbc61e12-d326-4d31-be81-88fb641337c7\") "
Nov 01 15:40:10.670285 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:10.670091    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39e0e8e2-87d8-4332-bb5d-86f3849861be-model-cache" pod "39e0e8e2-87d8-4332-bb5d-86f3849861be" (UID: "39e0e8e2-87d8-4332-bb5d-86f3849861be"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:40:10.670521 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:10.670464    4993 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39e0e8e2-87d8-4332-bb5d-86f3849861be-home\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:40:10.670521 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:10.670484    4993 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbc61e12-d326-4d31-be81-88fb641337c7-tmp-dir\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:40:10.670521 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:10.670498    4993 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39e0e8e2-87d8-4332-bb5d-86f3849861be-tmp-dir\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:40:10.670521 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:10.670514    4993 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39e0e8e2-87d8-4332-bb5d-86f3849861be-model-cache\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:40:10.670739 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:10.670528    4993 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbc61e12-d326-4d31-be81-88fb641337c7-model-cache\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:40:10.670800 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:10.670744    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbc61e12-d326-4d31-be81-88fb641337c7-home" pod "bbc61e12-d326-4d31-be81-88fb641337c7" (UID: "bbc61e12-d326-4d31-be81-88fb641337c7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:40:10.672373 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:10.672346    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39e0e8e2-87d8-4332-bb5d-86f3849861be-dshm" pod "39e0e8e2-87d8-4332-bb5d-86f3849861be" (UID: "39e0e8e2-87d8-4332-bb5d-86f3849861be"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:40:10.672496 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:10.672435    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39e0e8e2-87d8-4332-bb5d-86f3849861be-tls-certs" pod "39e0e8e2-87d8-4332-bb5d-86f3849861be" (UID: "39e0e8e2-87d8-4332-bb5d-86f3849861be"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:40:10.672607 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:10.672585    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbc61e12-d326-4d31-be81-88fb641337c7-dshm" pod "bbc61e12-d326-4d31-be81-88fb641337c7" (UID: "bbc61e12-d326-4d31-be81-88fb641337c7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:40:10.676381 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:10.676355    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39e0e8e2-87d8-4332-bb5d-86f3849861be-kube-api-access-fhxcf" pod "39e0e8e2-87d8-4332-bb5d-86f3849861be" (UID: "39e0e8e2-87d8-4332-bb5d-86f3849861be"). InnerVolumeSpecName "kube-api-access-fhxcf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:40:10.676543 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:10.676379    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbc61e12-d326-4d31-be81-88fb641337c7-kube-api-access-fvt7c" pod "bbc61e12-d326-4d31-be81-88fb641337c7" (UID: "bbc61e12-d326-4d31-be81-88fb641337c7"). InnerVolumeSpecName "kube-api-access-fvt7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:40:10.677715 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:10.677693    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbc61e12-d326-4d31-be81-88fb641337c7-tls-certs" pod "bbc61e12-d326-4d31-be81-88fb641337c7" (UID: "bbc61e12-d326-4d31-be81-88fb641337c7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:40:10.771867 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:10.771821    4993 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbc61e12-d326-4d31-be81-88fb641337c7-tls-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:40:10.771867 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:10.771861    4993 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39e0e8e2-87d8-4332-bb5d-86f3849861be-tls-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:40:10.772125 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:10.771880    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhxcf\" (UniqueName: \"kubernetes.io/projected/39e0e8e2-87d8-4332-bb5d-86f3849861be-kube-api-access-fhxcf\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:40:10.772125 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:10.771895    4993 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbc61e12-d326-4d31-be81-88fb641337c7-home\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:40:10.772125 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:10.771910    4993 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbc61e12-d326-4d31-be81-88fb641337c7-dshm\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:40:10.772125 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:10.771924    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvt7c\" (UniqueName: \"kubernetes.io/projected/bbc61e12-d326-4d31-be81-88fb641337c7-kube-api-access-fvt7c\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:40:10.772125 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:10.771937    4993 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39e0e8e2-87d8-4332-bb5d-86f3849861be-dshm\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:40:11.369630 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:11.369604    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1f8e2c/distinct-w0-1f8e2c-kserve-6657447954-rpml6"
Nov 01 15:40:11.370092 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:11.369634    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1f8e2c/distinct-w0-1f8e2c-kserve-6657447954-rpml6" event={"ID":"39e0e8e2-87d8-4332-bb5d-86f3849861be","Type":"ContainerDied","Data":"c2e65a6e39cf76cc65001d016172eb64442eef4b91e6816b6bba3792e53cbaca"}
Nov 01 15:40:11.370092 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:11.369690    4993 scope.go:122] "RemoveContainer" containerID="bede0709c95e1ff913eb37843cb9a91670f153f524b63cbb214fea3f29dc4866"
Nov 01 15:40:11.371092 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:11.371067    4993 generic.go:356] "Generic (PLEG): container finished" podID="bbc61e12-d326-4d31-be81-88fb641337c7" containerID="b6bce2c001544b5a381a1bd23a0fb0699996dd3fcbf4e57c232d7815539ecfad" exitCode=0
Nov 01 15:40:11.371254 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:11.371132    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1f8e2c/simulator-w0-1f8e2c-kserve-755bf645d-zf8t4"
Nov 01 15:40:11.371254 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:11.371158    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1f8e2c/simulator-w0-1f8e2c-kserve-755bf645d-zf8t4" event={"ID":"bbc61e12-d326-4d31-be81-88fb641337c7","Type":"ContainerDied","Data":"b6bce2c001544b5a381a1bd23a0fb0699996dd3fcbf4e57c232d7815539ecfad"}
Nov 01 15:40:11.371254 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:11.371223    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1f8e2c/simulator-w0-1f8e2c-kserve-755bf645d-zf8t4" event={"ID":"bbc61e12-d326-4d31-be81-88fb641337c7","Type":"ContainerDied","Data":"8d118730bca35b480da1ff596463fa71ca83a0a7255d8ddd32b74074aaad18bb"}
Nov 01 15:40:11.379442 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:11.379397    4993 scope.go:122] "RemoveContainer" containerID="b6bce2c001544b5a381a1bd23a0fb0699996dd3fcbf4e57c232d7815539ecfad"
Nov 01 15:40:11.388036 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:11.388009    4993 scope.go:122] "RemoveContainer" containerID="b6bce2c001544b5a381a1bd23a0fb0699996dd3fcbf4e57c232d7815539ecfad"
Nov 01 15:40:11.388420 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:40:11.388399    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6bce2c001544b5a381a1bd23a0fb0699996dd3fcbf4e57c232d7815539ecfad\": container with ID starting with b6bce2c001544b5a381a1bd23a0fb0699996dd3fcbf4e57c232d7815539ecfad not found: ID does not exist" containerID="b6bce2c001544b5a381a1bd23a0fb0699996dd3fcbf4e57c232d7815539ecfad"
Nov 01 15:40:11.388478 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:11.388430    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6bce2c001544b5a381a1bd23a0fb0699996dd3fcbf4e57c232d7815539ecfad"} err="failed to get container status \"b6bce2c001544b5a381a1bd23a0fb0699996dd3fcbf4e57c232d7815539ecfad\": rpc error: code = NotFound desc = could not find container \"b6bce2c001544b5a381a1bd23a0fb0699996dd3fcbf4e57c232d7815539ecfad\": container with ID starting with b6bce2c001544b5a381a1bd23a0fb0699996dd3fcbf4e57c232d7815539ecfad not found: ID does not exist"
Nov 01 15:40:11.522048 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:11.522009    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1f8e2c/distinct-w0-1f8e2c-kserve-6657447954-rpml6"]
Nov 01 15:40:11.568460 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:11.568422    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-1f8e2c/distinct-w0-1f8e2c-kserve-6657447954-rpml6"]
Nov 01 15:40:11.671036 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:11.670667    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1f8e2c/simulator-w0-1f8e2c-kserve-755bf645d-zf8t4"]
Nov 01 15:40:11.674773 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:11.674732    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-1f8e2c/simulator-w0-1f8e2c-kserve-755bf645d-zf8t4"]
Nov 01 15:40:12.796748 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:12.796709    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39e0e8e2-87d8-4332-bb5d-86f3849861be" path="/var/lib/kubelet/pods/39e0e8e2-87d8-4332-bb5d-86f3849861be/volumes"
Nov 01 15:40:12.797344 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:12.797324    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbc61e12-d326-4d31-be81-88fb641337c7" path="/var/lib/kubelet/pods/bbc61e12-d326-4d31-be81-88fb641337c7/volumes"
Nov 01 15:40:13.510229 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:13.510199    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b3f06c-76c7d95c95-mkfqr"]
Nov 01 15:40:13.510465 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:13.510416    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b3f06c-76c7d95c95-mkfqr" podUID="04ff309b-01de-48c8-a36b-412e5f244b4e" containerName="maas-api" containerID="cri-o://0ac590fc478b3fe1be621c931b5ed4cf8eb4cf8b3dc12f1f2744e21067ab1e0d" gracePeriod=30
Nov 01 15:40:13.806855 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:13.806828    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b3f06c-76c7d95c95-mkfqr"
Nov 01 15:40:13.907059 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:13.907023    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04ff309b-01de-48c8-a36b-412e5f244b4e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04ff309b-01de-48c8-a36b-412e5f244b4e-openshift-service-ca\") pod \"04ff309b-01de-48c8-a36b-412e5f244b4e\" (UID: \"04ff309b-01de-48c8-a36b-412e5f244b4e\") "
Nov 01 15:40:13.907288 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:13.907151    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04ff309b-01de-48c8-a36b-412e5f244b4e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/04ff309b-01de-48c8-a36b-412e5f244b4e-metrics-certs\") pod \"04ff309b-01de-48c8-a36b-412e5f244b4e\" (UID: \"04ff309b-01de-48c8-a36b-412e5f244b4e\") "
Nov 01 15:40:13.907288 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:13.907270    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04ff309b-01de-48c8-a36b-412e5f244b4e-kube-api-access-x6h9q\" (UniqueName: \"kubernetes.io/projected/04ff309b-01de-48c8-a36b-412e5f244b4e-kube-api-access-x6h9q\") pod \"04ff309b-01de-48c8-a36b-412e5f244b4e\" (UID: \"04ff309b-01de-48c8-a36b-412e5f244b4e\") "
Nov 01 15:40:13.907420 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:13.907311    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04ff309b-01de-48c8-a36b-412e5f244b4e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04ff309b-01de-48c8-a36b-412e5f244b4e-maas-api-tls\") pod \"04ff309b-01de-48c8-a36b-412e5f244b4e\" (UID: \"04ff309b-01de-48c8-a36b-412e5f244b4e\") "
Nov 01 15:40:13.907509 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:13.907479    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04ff309b-01de-48c8-a36b-412e5f244b4e-openshift-service-ca" pod "04ff309b-01de-48c8-a36b-412e5f244b4e" (UID: "04ff309b-01de-48c8-a36b-412e5f244b4e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:40:13.907897 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:13.907872    4993 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04ff309b-01de-48c8-a36b-412e5f244b4e-openshift-service-ca\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:40:13.909877 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:13.909843    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04ff309b-01de-48c8-a36b-412e5f244b4e-kube-api-access-x6h9q" pod "04ff309b-01de-48c8-a36b-412e5f244b4e" (UID: "04ff309b-01de-48c8-a36b-412e5f244b4e"). InnerVolumeSpecName "kube-api-access-x6h9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:40:13.919212 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:13.910635    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04ff309b-01de-48c8-a36b-412e5f244b4e-metrics-certs" pod "04ff309b-01de-48c8-a36b-412e5f244b4e" (UID: "04ff309b-01de-48c8-a36b-412e5f244b4e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:40:13.919212 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:13.911566    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04ff309b-01de-48c8-a36b-412e5f244b4e-maas-api-tls" pod "04ff309b-01de-48c8-a36b-412e5f244b4e" (UID: "04ff309b-01de-48c8-a36b-412e5f244b4e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:40:14.009561 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:14.009528    4993 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04ff309b-01de-48c8-a36b-412e5f244b4e-metrics-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:40:14.009561 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:14.009562    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6h9q\" (UniqueName: \"kubernetes.io/projected/04ff309b-01de-48c8-a36b-412e5f244b4e-kube-api-access-x6h9q\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:40:14.009778 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:14.009574    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04ff309b-01de-48c8-a36b-412e5f244b4e-maas-api-tls\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:40:14.395793 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:14.395756    4993 generic.go:356] "Generic (PLEG): container finished" podID="04ff309b-01de-48c8-a36b-412e5f244b4e" containerID="0ac590fc478b3fe1be621c931b5ed4cf8eb4cf8b3dc12f1f2744e21067ab1e0d" exitCode=0
Nov 01 15:40:14.396021 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:14.395799    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b3f06c-76c7d95c95-mkfqr"
Nov 01 15:40:14.396021 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:14.395834    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b3f06c-76c7d95c95-mkfqr" event={"ID":"04ff309b-01de-48c8-a36b-412e5f244b4e","Type":"ContainerDied","Data":"0ac590fc478b3fe1be621c931b5ed4cf8eb4cf8b3dc12f1f2744e21067ab1e0d"}
Nov 01 15:40:14.396021 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:14.395900    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b3f06c-76c7d95c95-mkfqr" event={"ID":"04ff309b-01de-48c8-a36b-412e5f244b4e","Type":"ContainerDied","Data":"2f94b190a5b6e6baacc4451f7b2e9f6a6e8a5605d3ea5db8cb661bc800172ec4"}
Nov 01 15:40:14.396021 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:14.395930    4993 scope.go:122] "RemoveContainer" containerID="0ac590fc478b3fe1be621c931b5ed4cf8eb4cf8b3dc12f1f2744e21067ab1e0d"
Nov 01 15:40:14.406865 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:14.406847    4993 scope.go:122] "RemoveContainer" containerID="0ac590fc478b3fe1be621c931b5ed4cf8eb4cf8b3dc12f1f2744e21067ab1e0d"
Nov 01 15:40:14.407136 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:40:14.407119    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ac590fc478b3fe1be621c931b5ed4cf8eb4cf8b3dc12f1f2744e21067ab1e0d\": container with ID starting with 0ac590fc478b3fe1be621c931b5ed4cf8eb4cf8b3dc12f1f2744e21067ab1e0d not found: ID does not exist" containerID="0ac590fc478b3fe1be621c931b5ed4cf8eb4cf8b3dc12f1f2744e21067ab1e0d"
Nov 01 15:40:14.407181 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:14.407145    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ac590fc478b3fe1be621c931b5ed4cf8eb4cf8b3dc12f1f2744e21067ab1e0d"} err="failed to get container status \"0ac590fc478b3fe1be621c931b5ed4cf8eb4cf8b3dc12f1f2744e21067ab1e0d\": rpc error: code = NotFound desc = could not find container \"0ac590fc478b3fe1be621c931b5ed4cf8eb4cf8b3dc12f1f2744e21067ab1e0d\": container with ID starting with 0ac590fc478b3fe1be621c931b5ed4cf8eb4cf8b3dc12f1f2744e21067ab1e0d not found: ID does not exist"
Nov 01 15:40:14.548856 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:14.548820    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b3f06c-76c7d95c95-mkfqr"]
Nov 01 15:40:14.554074 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:14.554045    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b3f06c-76c7d95c95-mkfqr"]
Nov 01 15:40:14.798892 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:14.798852    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04ff309b-01de-48c8-a36b-412e5f244b4e" path="/var/lib/kubelet/pods/04ff309b-01de-48c8-a36b-412e5f244b4e/volumes"
Nov 01 15:40:16.732312 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:16.732250    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-b3f06c-76c7d95c95-mkfqr" podUID="04ff309b-01de-48c8-a36b-412e5f244b4e" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.119:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 15:40:24.201351 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:24.201313    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-1f8e2c-7bcc77fd5d-8lxsw"]
Nov 01 15:40:24.201779 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:24.201565    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1f8e2c-7bcc77fd5d-8lxsw" podUID="ac70b22c-c696-4be4-84cc-593406d0cda6" containerName="maas-api" containerID="cri-o://3a0ea2e2d87f9b08e4d0796456f943824fb95c18e7b8e9bb038b6ea6c1abfbcb" gracePeriod=30
Nov 01 15:40:24.465083 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:24.464979    4993 generic.go:356] "Generic (PLEG): container finished" podID="ac70b22c-c696-4be4-84cc-593406d0cda6" containerID="3a0ea2e2d87f9b08e4d0796456f943824fb95c18e7b8e9bb038b6ea6c1abfbcb" exitCode=0
Nov 01 15:40:24.465258 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:24.465070    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1f8e2c-7bcc77fd5d-8lxsw" event={"ID":"ac70b22c-c696-4be4-84cc-593406d0cda6","Type":"ContainerDied","Data":"3a0ea2e2d87f9b08e4d0796456f943824fb95c18e7b8e9bb038b6ea6c1abfbcb"}
Nov 01 15:40:24.499596 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:24.499566    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1f8e2c-7bcc77fd5d-8lxsw"
Nov 01 15:40:24.634192 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:24.632103    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac70b22c-c696-4be4-84cc-593406d0cda6-kube-api-access-rwcq6\" (UniqueName: \"kubernetes.io/projected/ac70b22c-c696-4be4-84cc-593406d0cda6-kube-api-access-rwcq6\") pod \"ac70b22c-c696-4be4-84cc-593406d0cda6\" (UID: \"ac70b22c-c696-4be4-84cc-593406d0cda6\") "
Nov 01 15:40:24.634192 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:24.632228    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac70b22c-c696-4be4-84cc-593406d0cda6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac70b22c-c696-4be4-84cc-593406d0cda6-metrics-certs\") pod \"ac70b22c-c696-4be4-84cc-593406d0cda6\" (UID: \"ac70b22c-c696-4be4-84cc-593406d0cda6\") "
Nov 01 15:40:24.634192 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:24.632282    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac70b22c-c696-4be4-84cc-593406d0cda6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac70b22c-c696-4be4-84cc-593406d0cda6-openshift-service-ca\") pod \"ac70b22c-c696-4be4-84cc-593406d0cda6\" (UID: \"ac70b22c-c696-4be4-84cc-593406d0cda6\") "
Nov 01 15:40:24.634192 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:24.632384    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac70b22c-c696-4be4-84cc-593406d0cda6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac70b22c-c696-4be4-84cc-593406d0cda6-maas-api-tls\") pod \"ac70b22c-c696-4be4-84cc-593406d0cda6\" (UID: \"ac70b22c-c696-4be4-84cc-593406d0cda6\") "
Nov 01 15:40:24.634192 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:24.633960    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac70b22c-c696-4be4-84cc-593406d0cda6-openshift-service-ca" pod "ac70b22c-c696-4be4-84cc-593406d0cda6" (UID: "ac70b22c-c696-4be4-84cc-593406d0cda6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:40:24.656189 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:24.654009    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac70b22c-c696-4be4-84cc-593406d0cda6-kube-api-access-rwcq6" pod "ac70b22c-c696-4be4-84cc-593406d0cda6" (UID: "ac70b22c-c696-4be4-84cc-593406d0cda6"). InnerVolumeSpecName "kube-api-access-rwcq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:40:24.662274 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:24.662237    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac70b22c-c696-4be4-84cc-593406d0cda6-maas-api-tls" pod "ac70b22c-c696-4be4-84cc-593406d0cda6" (UID: "ac70b22c-c696-4be4-84cc-593406d0cda6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:40:24.664270 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:24.664240    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac70b22c-c696-4be4-84cc-593406d0cda6-metrics-certs" pod "ac70b22c-c696-4be4-84cc-593406d0cda6" (UID: "ac70b22c-c696-4be4-84cc-593406d0cda6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:40:24.733426 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:24.733371    4993 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac70b22c-c696-4be4-84cc-593406d0cda6-openshift-service-ca\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:40:24.733426 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:24.733408    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac70b22c-c696-4be4-84cc-593406d0cda6-maas-api-tls\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:40:24.733426 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:24.733423    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwcq6\" (UniqueName: \"kubernetes.io/projected/ac70b22c-c696-4be4-84cc-593406d0cda6-kube-api-access-rwcq6\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:40:24.733651 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:24.733438    4993 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac70b22c-c696-4be4-84cc-593406d0cda6-metrics-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:40:25.481317 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:25.481100    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1f8e2c-7bcc77fd5d-8lxsw" event={"ID":"ac70b22c-c696-4be4-84cc-593406d0cda6","Type":"ContainerDied","Data":"c4e2d56a5a31ef22c78187e3b91b6c0e0742cee298ce0c8a75d441e6c174853a"}
Nov 01 15:40:25.481317 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:25.481253    4993 scope.go:122] "RemoveContainer" containerID="3a0ea2e2d87f9b08e4d0796456f943824fb95c18e7b8e9bb038b6ea6c1abfbcb"
Nov 01 15:40:25.481846 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:25.481463    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1f8e2c-7bcc77fd5d-8lxsw"
Nov 01 15:40:25.637336 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:25.637302    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-1f8e2c-7bcc77fd5d-8lxsw"]
Nov 01 15:40:25.683913 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:25.683880    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-1f8e2c-7bcc77fd5d-8lxsw"]
Nov 01 15:40:26.796052 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:26.796015    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac70b22c-c696-4be4-84cc-593406d0cda6" path="/var/lib/kubelet/pods/ac70b22c-c696-4be4-84cc-593406d0cda6/volumes"
Nov 01 15:40:27.502643 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:40:27.502601    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:40:29.851106 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:29.851070    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-536a87/distinct-2-w4-536a87-kserve-6ff7c9fb94-scgv7"]
Nov 01 15:40:29.898017 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:29.897973    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-536a87/distinct-w4-536a87-kserve-6bdb5c8897-lmcls"]
Nov 01 15:40:29.901011 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:29.900977    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-536a87/premium-w4-536a87-kserve-74c6fdbc8-sf89f"]
Nov 01 15:40:42.938694 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:42.938658    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8fd363/rate-test-model-8fd363-kserve-77db559f77-qtxzx"]
Nov 01 15:40:44.878494 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:44.878439    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-536a87/distinct-2-w4-536a87-kserve-6ff7c9fb94-scgv7" podUID="56b01618-1df3-4ba6-8766-46fa85c27773" containerName="main" containerID="cri-o://96631854d96fbc1fb56b5d7015a6d7cc20a6b6b6e1e4d36508f3af75dcddf9b1" gracePeriod=45
Nov 01 15:40:44.919102 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:44.919051    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-536a87/distinct-w4-536a87-kserve-6bdb5c8897-lmcls" podUID="3cdf53f6-df5a-4cdc-b751-e26c71484700" containerName="main" containerID="cri-o://34c3e8d99f425379366e0d6ea908a239eae0c724bf8d28b0e49305a464d85806" gracePeriod=45
Nov 01 15:40:44.921100 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:44.921054    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-536a87/premium-w4-536a87-kserve-74c6fdbc8-sf89f" podUID="8a720a76-488a-4a65-98ad-31ca84fa7842" containerName="main" containerID="cri-o://f3959116fe019c5c61eaac8ca70d68bed208bfa32a91dbf4d6f0c3dad66f46ef" gracePeriod=45
Nov 01 15:40:45.009152 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.009102    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536a87/distinct-w4-536a87-kserve-6bdb5c8897-lmcls" podUID="3cdf53f6-df5a-4cdc-b751-e26c71484700" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": dial tcp 10.132.0.87:8000: connect: connection refused"
Nov 01 15:40:45.264339 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.264312    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-536a87/distinct-2-w4-536a87-kserve-6ff7c9fb94-scgv7"
Nov 01 15:40:45.270241 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.270215    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-536a87/premium-w4-536a87-kserve-74c6fdbc8-sf89f"
Nov 01 15:40:45.273909 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.273886    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-536a87/distinct-w4-536a87-kserve-6bdb5c8897-lmcls"
Nov 01 15:40:45.393645 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.393566    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a720a76-488a-4a65-98ad-31ca84fa7842-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a720a76-488a-4a65-98ad-31ca84fa7842-dshm\") pod \"8a720a76-488a-4a65-98ad-31ca84fa7842\" (UID: \"8a720a76-488a-4a65-98ad-31ca84fa7842\") "
Nov 01 15:40:45.393645 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.393619    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56b01618-1df3-4ba6-8766-46fa85c27773-tls-certs\" (UniqueName: \"kubernetes.io/secret/56b01618-1df3-4ba6-8766-46fa85c27773-tls-certs\") pod \"56b01618-1df3-4ba6-8766-46fa85c27773\" (UID: \"56b01618-1df3-4ba6-8766-46fa85c27773\") "
Nov 01 15:40:45.393645 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.393640    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cdf53f6-df5a-4cdc-b751-e26c71484700-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cdf53f6-df5a-4cdc-b751-e26c71484700-model-cache\") pod \"3cdf53f6-df5a-4cdc-b751-e26c71484700\" (UID: \"3cdf53f6-df5a-4cdc-b751-e26c71484700\") "
Nov 01 15:40:45.393912 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.393660    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56b01618-1df3-4ba6-8766-46fa85c27773-kube-api-access-rvj9t\" (UniqueName: \"kubernetes.io/projected/56b01618-1df3-4ba6-8766-46fa85c27773-kube-api-access-rvj9t\") pod \"56b01618-1df3-4ba6-8766-46fa85c27773\" (UID: \"56b01618-1df3-4ba6-8766-46fa85c27773\") "
Nov 01 15:40:45.393912 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.393682    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cdf53f6-df5a-4cdc-b751-e26c71484700-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cdf53f6-df5a-4cdc-b751-e26c71484700-dshm\") pod \"3cdf53f6-df5a-4cdc-b751-e26c71484700\" (UID: \"3cdf53f6-df5a-4cdc-b751-e26c71484700\") "
Nov 01 15:40:45.393912 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.393716    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3cdf53f6-df5a-4cdc-b751-e26c71484700-tls-certs\" (UniqueName: \"kubernetes.io/secret/3cdf53f6-df5a-4cdc-b751-e26c71484700-tls-certs\") pod \"3cdf53f6-df5a-4cdc-b751-e26c71484700\" (UID: \"3cdf53f6-df5a-4cdc-b751-e26c71484700\") "
Nov 01 15:40:45.393912 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.393785    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56b01618-1df3-4ba6-8766-46fa85c27773-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56b01618-1df3-4ba6-8766-46fa85c27773-model-cache\") pod \"56b01618-1df3-4ba6-8766-46fa85c27773\" (UID: \"56b01618-1df3-4ba6-8766-46fa85c27773\") "
Nov 01 15:40:45.393912 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.393823    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cdf53f6-df5a-4cdc-b751-e26c71484700-home\" (UniqueName: \"kubernetes.io/empty-dir/3cdf53f6-df5a-4cdc-b751-e26c71484700-home\") pod \"3cdf53f6-df5a-4cdc-b751-e26c71484700\" (UID: \"3cdf53f6-df5a-4cdc-b751-e26c71484700\") "
Nov 01 15:40:45.393912 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.393864    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a720a76-488a-4a65-98ad-31ca84fa7842-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a720a76-488a-4a65-98ad-31ca84fa7842-model-cache\") pod \"8a720a76-488a-4a65-98ad-31ca84fa7842\" (UID: \"8a720a76-488a-4a65-98ad-31ca84fa7842\") "
Nov 01 15:40:45.393912 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.393904    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56b01618-1df3-4ba6-8766-46fa85c27773-home\" (UniqueName: \"kubernetes.io/empty-dir/56b01618-1df3-4ba6-8766-46fa85c27773-home\") pod \"56b01618-1df3-4ba6-8766-46fa85c27773\" (UID: \"56b01618-1df3-4ba6-8766-46fa85c27773\") "
Nov 01 15:40:45.394313 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.393934    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cdf53f6-df5a-4cdc-b751-e26c71484700-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cdf53f6-df5a-4cdc-b751-e26c71484700-tmp-dir\") pod \"3cdf53f6-df5a-4cdc-b751-e26c71484700\" (UID: \"3cdf53f6-df5a-4cdc-b751-e26c71484700\") "
Nov 01 15:40:45.394313 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.393971    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cdf53f6-df5a-4cdc-b751-e26c71484700-model-cache" pod "3cdf53f6-df5a-4cdc-b751-e26c71484700" (UID: "3cdf53f6-df5a-4cdc-b751-e26c71484700"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:40:45.394313 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.393986    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a720a76-488a-4a65-98ad-31ca84fa7842-tls-certs\" (UniqueName: \"kubernetes.io/secret/8a720a76-488a-4a65-98ad-31ca84fa7842-tls-certs\") pod \"8a720a76-488a-4a65-98ad-31ca84fa7842\" (UID: \"8a720a76-488a-4a65-98ad-31ca84fa7842\") "
Nov 01 15:40:45.394313 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.394077    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56b01618-1df3-4ba6-8766-46fa85c27773-dshm\" (UniqueName: \"kubernetes.io/empty-dir/56b01618-1df3-4ba6-8766-46fa85c27773-dshm\") pod \"56b01618-1df3-4ba6-8766-46fa85c27773\" (UID: \"56b01618-1df3-4ba6-8766-46fa85c27773\") "
Nov 01 15:40:45.394313 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.394118    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cdf53f6-df5a-4cdc-b751-e26c71484700-kube-api-access-27dvp\" (UniqueName: \"kubernetes.io/projected/3cdf53f6-df5a-4cdc-b751-e26c71484700-kube-api-access-27dvp\") pod \"3cdf53f6-df5a-4cdc-b751-e26c71484700\" (UID: \"3cdf53f6-df5a-4cdc-b751-e26c71484700\") "
Nov 01 15:40:45.394313 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.394156    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56b01618-1df3-4ba6-8766-46fa85c27773-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56b01618-1df3-4ba6-8766-46fa85c27773-tmp-dir\") pod \"56b01618-1df3-4ba6-8766-46fa85c27773\" (UID: \"56b01618-1df3-4ba6-8766-46fa85c27773\") "
Nov 01 15:40:45.394313 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.394222    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a720a76-488a-4a65-98ad-31ca84fa7842-home\" (UniqueName: \"kubernetes.io/empty-dir/8a720a76-488a-4a65-98ad-31ca84fa7842-home\") pod \"8a720a76-488a-4a65-98ad-31ca84fa7842\" (UID: \"8a720a76-488a-4a65-98ad-31ca84fa7842\") "
Nov 01 15:40:45.394313 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.394261    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a720a76-488a-4a65-98ad-31ca84fa7842-kube-api-access-d9qjj\" (UniqueName: \"kubernetes.io/projected/8a720a76-488a-4a65-98ad-31ca84fa7842-kube-api-access-d9qjj\") pod \"8a720a76-488a-4a65-98ad-31ca84fa7842\" (UID: \"8a720a76-488a-4a65-98ad-31ca84fa7842\") "
Nov 01 15:40:45.394738 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.394346    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a720a76-488a-4a65-98ad-31ca84fa7842-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a720a76-488a-4a65-98ad-31ca84fa7842-tmp-dir\") pod \"8a720a76-488a-4a65-98ad-31ca84fa7842\" (UID: \"8a720a76-488a-4a65-98ad-31ca84fa7842\") "
Nov 01 15:40:45.396332 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.394856    4993 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cdf53f6-df5a-4cdc-b751-e26c71484700-model-cache\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:40:45.396332 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.395109    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a720a76-488a-4a65-98ad-31ca84fa7842-tmp-dir" pod "8a720a76-488a-4a65-98ad-31ca84fa7842" (UID: "8a720a76-488a-4a65-98ad-31ca84fa7842"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:40:45.396332 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.395344    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a720a76-488a-4a65-98ad-31ca84fa7842-home" pod "8a720a76-488a-4a65-98ad-31ca84fa7842" (UID: "8a720a76-488a-4a65-98ad-31ca84fa7842"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:40:45.396332 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.395488    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a720a76-488a-4a65-98ad-31ca84fa7842-model-cache" pod "8a720a76-488a-4a65-98ad-31ca84fa7842" (UID: "8a720a76-488a-4a65-98ad-31ca84fa7842"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:40:45.396332 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.395701    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56b01618-1df3-4ba6-8766-46fa85c27773-model-cache" pod "56b01618-1df3-4ba6-8766-46fa85c27773" (UID: "56b01618-1df3-4ba6-8766-46fa85c27773"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:40:45.396332 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.396067    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56b01618-1df3-4ba6-8766-46fa85c27773-home" pod "56b01618-1df3-4ba6-8766-46fa85c27773" (UID: "56b01618-1df3-4ba6-8766-46fa85c27773"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:40:45.396332 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.396250    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56b01618-1df3-4ba6-8766-46fa85c27773-tmp-dir" pod "56b01618-1df3-4ba6-8766-46fa85c27773" (UID: "56b01618-1df3-4ba6-8766-46fa85c27773"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:40:45.396332 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.396273    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cdf53f6-df5a-4cdc-b751-e26c71484700-tmp-dir" pod "3cdf53f6-df5a-4cdc-b751-e26c71484700" (UID: "3cdf53f6-df5a-4cdc-b751-e26c71484700"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:40:45.396332 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.396307    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cdf53f6-df5a-4cdc-b751-e26c71484700-home" pod "3cdf53f6-df5a-4cdc-b751-e26c71484700" (UID: "3cdf53f6-df5a-4cdc-b751-e26c71484700"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:40:45.396814 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.396472    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56b01618-1df3-4ba6-8766-46fa85c27773-tls-certs" pod "56b01618-1df3-4ba6-8766-46fa85c27773" (UID: "56b01618-1df3-4ba6-8766-46fa85c27773"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:40:45.396814 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.396490    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a720a76-488a-4a65-98ad-31ca84fa7842-tls-certs" pod "8a720a76-488a-4a65-98ad-31ca84fa7842" (UID: "8a720a76-488a-4a65-98ad-31ca84fa7842"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:40:45.396814 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.396758    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a720a76-488a-4a65-98ad-31ca84fa7842-dshm" pod "8a720a76-488a-4a65-98ad-31ca84fa7842" (UID: "8a720a76-488a-4a65-98ad-31ca84fa7842"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:40:45.400110 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.400084    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56b01618-1df3-4ba6-8766-46fa85c27773-dshm" pod "56b01618-1df3-4ba6-8766-46fa85c27773" (UID: "56b01618-1df3-4ba6-8766-46fa85c27773"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:40:45.400520 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.400495    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cdf53f6-df5a-4cdc-b751-e26c71484700-tls-certs" pod "3cdf53f6-df5a-4cdc-b751-e26c71484700" (UID: "3cdf53f6-df5a-4cdc-b751-e26c71484700"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:40:45.400520 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.400509    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cdf53f6-df5a-4cdc-b751-e26c71484700-dshm" pod "3cdf53f6-df5a-4cdc-b751-e26c71484700" (UID: "3cdf53f6-df5a-4cdc-b751-e26c71484700"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:40:45.400684 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.400502    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56b01618-1df3-4ba6-8766-46fa85c27773-kube-api-access-rvj9t" pod "56b01618-1df3-4ba6-8766-46fa85c27773" (UID: "56b01618-1df3-4ba6-8766-46fa85c27773"). InnerVolumeSpecName "kube-api-access-rvj9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:40:45.400684 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.400562    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a720a76-488a-4a65-98ad-31ca84fa7842-kube-api-access-d9qjj" pod "8a720a76-488a-4a65-98ad-31ca84fa7842" (UID: "8a720a76-488a-4a65-98ad-31ca84fa7842"). InnerVolumeSpecName "kube-api-access-d9qjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:40:45.401747 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.401720    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cdf53f6-df5a-4cdc-b751-e26c71484700-kube-api-access-27dvp" pod "3cdf53f6-df5a-4cdc-b751-e26c71484700" (UID: "3cdf53f6-df5a-4cdc-b751-e26c71484700"). InnerVolumeSpecName "kube-api-access-27dvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:40:45.495787 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.495743    4993 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cdf53f6-df5a-4cdc-b751-e26c71484700-tls-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:40:45.495787 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.495785    4993 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56b01618-1df3-4ba6-8766-46fa85c27773-model-cache\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:40:45.495989 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.495800    4993 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3cdf53f6-df5a-4cdc-b751-e26c71484700-home\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:40:45.495989 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.495814    4993 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a720a76-488a-4a65-98ad-31ca84fa7842-model-cache\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:40:45.495989 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.495827    4993 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56b01618-1df3-4ba6-8766-46fa85c27773-home\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:40:45.495989 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.495842    4993 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cdf53f6-df5a-4cdc-b751-e26c71484700-tmp-dir\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:40:45.495989 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.495856    4993 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a720a76-488a-4a65-98ad-31ca84fa7842-tls-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:40:45.495989 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.495868    4993 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56b01618-1df3-4ba6-8766-46fa85c27773-dshm\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:40:45.495989 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.495881    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27dvp\" (UniqueName: \"kubernetes.io/projected/3cdf53f6-df5a-4cdc-b751-e26c71484700-kube-api-access-27dvp\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:40:45.495989 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.495895    4993 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56b01618-1df3-4ba6-8766-46fa85c27773-tmp-dir\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:40:45.495989 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.495908    4993 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a720a76-488a-4a65-98ad-31ca84fa7842-home\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:40:45.495989 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.495923    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9qjj\" (UniqueName: \"kubernetes.io/projected/8a720a76-488a-4a65-98ad-31ca84fa7842-kube-api-access-d9qjj\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:40:45.495989 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.495935    4993 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a720a76-488a-4a65-98ad-31ca84fa7842-tmp-dir\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:40:45.495989 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.495947    4993 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a720a76-488a-4a65-98ad-31ca84fa7842-dshm\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:40:45.495989 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.495959    4993 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56b01618-1df3-4ba6-8766-46fa85c27773-tls-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:40:45.495989 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.495972    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvj9t\" (UniqueName: \"kubernetes.io/projected/56b01618-1df3-4ba6-8766-46fa85c27773-kube-api-access-rvj9t\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:40:45.495989 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.495986    4993 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cdf53f6-df5a-4cdc-b751-e26c71484700-dshm\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:40:45.614604 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.614567    4993 generic.go:356] "Generic (PLEG): container finished" podID="3cdf53f6-df5a-4cdc-b751-e26c71484700" containerID="34c3e8d99f425379366e0d6ea908a239eae0c724bf8d28b0e49305a464d85806" exitCode=0
Nov 01 15:40:45.614794 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.614625    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-536a87/distinct-w4-536a87-kserve-6bdb5c8897-lmcls"
Nov 01 15:40:45.614794 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.614650    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-536a87/distinct-w4-536a87-kserve-6bdb5c8897-lmcls" event={"ID":"3cdf53f6-df5a-4cdc-b751-e26c71484700","Type":"ContainerDied","Data":"34c3e8d99f425379366e0d6ea908a239eae0c724bf8d28b0e49305a464d85806"}
Nov 01 15:40:45.614794 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.614710    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-536a87/distinct-w4-536a87-kserve-6bdb5c8897-lmcls" event={"ID":"3cdf53f6-df5a-4cdc-b751-e26c71484700","Type":"ContainerDied","Data":"36396d5e1673024628bff726e102c63c50fb859a1a918fab907ce843a22e2034"}
Nov 01 15:40:45.614794 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.614735    4993 scope.go:122] "RemoveContainer" containerID="34c3e8d99f425379366e0d6ea908a239eae0c724bf8d28b0e49305a464d85806"
Nov 01 15:40:45.616564 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.616473    4993 generic.go:356] "Generic (PLEG): container finished" podID="56b01618-1df3-4ba6-8766-46fa85c27773" containerID="96631854d96fbc1fb56b5d7015a6d7cc20a6b6b6e1e4d36508f3af75dcddf9b1" exitCode=0
Nov 01 15:40:45.616564 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.616499    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-536a87/distinct-2-w4-536a87-kserve-6ff7c9fb94-scgv7" event={"ID":"56b01618-1df3-4ba6-8766-46fa85c27773","Type":"ContainerDied","Data":"96631854d96fbc1fb56b5d7015a6d7cc20a6b6b6e1e4d36508f3af75dcddf9b1"}
Nov 01 15:40:45.616564 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.616547    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-536a87/distinct-2-w4-536a87-kserve-6ff7c9fb94-scgv7"
Nov 01 15:40:45.616934 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.616548    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-536a87/distinct-2-w4-536a87-kserve-6ff7c9fb94-scgv7" event={"ID":"56b01618-1df3-4ba6-8766-46fa85c27773","Type":"ContainerDied","Data":"baac5501299abe77b8031a135aed18f58528dceb6c547cf04efd1011fc730869"}
Nov 01 15:40:45.618408 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.618382    4993 generic.go:356] "Generic (PLEG): container finished" podID="8a720a76-488a-4a65-98ad-31ca84fa7842" containerID="f3959116fe019c5c61eaac8ca70d68bed208bfa32a91dbf4d6f0c3dad66f46ef" exitCode=0
Nov 01 15:40:45.618520 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.618421    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-536a87/premium-w4-536a87-kserve-74c6fdbc8-sf89f" event={"ID":"8a720a76-488a-4a65-98ad-31ca84fa7842","Type":"ContainerDied","Data":"f3959116fe019c5c61eaac8ca70d68bed208bfa32a91dbf4d6f0c3dad66f46ef"}
Nov 01 15:40:45.618520 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.618447    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-536a87/premium-w4-536a87-kserve-74c6fdbc8-sf89f" event={"ID":"8a720a76-488a-4a65-98ad-31ca84fa7842","Type":"ContainerDied","Data":"a5dd081048f8953ac160f157e50966ada50a13d66abf577500f38a6329849bbc"}
Nov 01 15:40:45.618520 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.618506    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-536a87/premium-w4-536a87-kserve-74c6fdbc8-sf89f"
Nov 01 15:40:45.636400 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.632538    4993 scope.go:122] "RemoveContainer" containerID="34c3e8d99f425379366e0d6ea908a239eae0c724bf8d28b0e49305a464d85806"
Nov 01 15:40:45.636818 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:40:45.636748    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34c3e8d99f425379366e0d6ea908a239eae0c724bf8d28b0e49305a464d85806\": container with ID starting with 34c3e8d99f425379366e0d6ea908a239eae0c724bf8d28b0e49305a464d85806 not found: ID does not exist" containerID="34c3e8d99f425379366e0d6ea908a239eae0c724bf8d28b0e49305a464d85806"
Nov 01 15:40:45.636818 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.636792    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34c3e8d99f425379366e0d6ea908a239eae0c724bf8d28b0e49305a464d85806"} err="failed to get container status \"34c3e8d99f425379366e0d6ea908a239eae0c724bf8d28b0e49305a464d85806\": rpc error: code = NotFound desc = could not find container \"34c3e8d99f425379366e0d6ea908a239eae0c724bf8d28b0e49305a464d85806\": container with ID starting with 34c3e8d99f425379366e0d6ea908a239eae0c724bf8d28b0e49305a464d85806 not found: ID does not exist"
Nov 01 15:40:45.636931 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.636818    4993 scope.go:122] "RemoveContainer" containerID="96631854d96fbc1fb56b5d7015a6d7cc20a6b6b6e1e4d36508f3af75dcddf9b1"
Nov 01 15:40:45.657115 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.657077    4993 scope.go:122] "RemoveContainer" containerID="96631854d96fbc1fb56b5d7015a6d7cc20a6b6b6e1e4d36508f3af75dcddf9b1"
Nov 01 15:40:45.657549 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:40:45.657500    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96631854d96fbc1fb56b5d7015a6d7cc20a6b6b6e1e4d36508f3af75dcddf9b1\": container with ID starting with 96631854d96fbc1fb56b5d7015a6d7cc20a6b6b6e1e4d36508f3af75dcddf9b1 not found: ID does not exist" containerID="96631854d96fbc1fb56b5d7015a6d7cc20a6b6b6e1e4d36508f3af75dcddf9b1"
Nov 01 15:40:45.657629 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.657563    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96631854d96fbc1fb56b5d7015a6d7cc20a6b6b6e1e4d36508f3af75dcddf9b1"} err="failed to get container status \"96631854d96fbc1fb56b5d7015a6d7cc20a6b6b6e1e4d36508f3af75dcddf9b1\": rpc error: code = NotFound desc = could not find container \"96631854d96fbc1fb56b5d7015a6d7cc20a6b6b6e1e4d36508f3af75dcddf9b1\": container with ID starting with 96631854d96fbc1fb56b5d7015a6d7cc20a6b6b6e1e4d36508f3af75dcddf9b1 not found: ID does not exist"
Nov 01 15:40:45.657629 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.657605    4993 scope.go:122] "RemoveContainer" containerID="f3959116fe019c5c61eaac8ca70d68bed208bfa32a91dbf4d6f0c3dad66f46ef"
Nov 01 15:40:45.666431 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.666411    4993 scope.go:122] "RemoveContainer" containerID="f3959116fe019c5c61eaac8ca70d68bed208bfa32a91dbf4d6f0c3dad66f46ef"
Nov 01 15:40:45.666714 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:40:45.666693    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3959116fe019c5c61eaac8ca70d68bed208bfa32a91dbf4d6f0c3dad66f46ef\": container with ID starting with f3959116fe019c5c61eaac8ca70d68bed208bfa32a91dbf4d6f0c3dad66f46ef not found: ID does not exist" containerID="f3959116fe019c5c61eaac8ca70d68bed208bfa32a91dbf4d6f0c3dad66f46ef"
Nov 01 15:40:45.666773 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.666722    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3959116fe019c5c61eaac8ca70d68bed208bfa32a91dbf4d6f0c3dad66f46ef"} err="failed to get container status \"f3959116fe019c5c61eaac8ca70d68bed208bfa32a91dbf4d6f0c3dad66f46ef\": rpc error: code = NotFound desc = could not find container \"f3959116fe019c5c61eaac8ca70d68bed208bfa32a91dbf4d6f0c3dad66f46ef\": container with ID starting with f3959116fe019c5c61eaac8ca70d68bed208bfa32a91dbf4d6f0c3dad66f46ef not found: ID does not exist"
Nov 01 15:40:45.768723 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.768685    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-536a87/premium-w4-536a87-kserve-74c6fdbc8-sf89f"]
Nov 01 15:40:45.772539 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.772509    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-536a87/premium-w4-536a87-kserve-74c6fdbc8-sf89f"]
Nov 01 15:40:45.873355 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.873301    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-536a87/distinct-w4-536a87-kserve-6bdb5c8897-lmcls"]
Nov 01 15:40:45.876948 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.876921    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-536a87/distinct-w4-536a87-kserve-6bdb5c8897-lmcls"]
Nov 01 15:40:45.977657 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.977614    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-536a87/distinct-2-w4-536a87-kserve-6ff7c9fb94-scgv7"]
Nov 01 15:40:45.981721 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:45.981689    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-536a87/distinct-2-w4-536a87-kserve-6ff7c9fb94-scgv7"]
Nov 01 15:40:46.131544 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:46.131503    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-536a87/distinct-2-w4-536a87-kserve-6ff7c9fb94-scgv7" podUID="56b01618-1df3-4ba6-8766-46fa85c27773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded"
Nov 01 15:40:46.797604 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:46.797573    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cdf53f6-df5a-4cdc-b751-e26c71484700" path="/var/lib/kubelet/pods/3cdf53f6-df5a-4cdc-b751-e26c71484700/volumes"
Nov 01 15:40:46.798081 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:46.798065    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56b01618-1df3-4ba6-8766-46fa85c27773" path="/var/lib/kubelet/pods/56b01618-1df3-4ba6-8766-46fa85c27773/volumes"
Nov 01 15:40:46.798521 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:46.798506    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a720a76-488a-4a65-98ad-31ca84fa7842" path="/var/lib/kubelet/pods/8a720a76-488a-4a65-98ad-31ca84fa7842/volumes"
Nov 01 15:40:57.500640 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:40:57.500593    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:40:57.959357 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:57.959236    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-8fd363/rate-test-model-8fd363-kserve-77db559f77-qtxzx" podUID="f43a7e63-3095-4ad7-935a-bb46a0af45b9" containerName="main" containerID="cri-o://8e3d3bd34f89853d772792bd5efd3163f5b04cac354b2e060abb82aae6813b84" gracePeriod=45
Nov 01 15:40:58.239163 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:58.239135    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-8fd363/rate-test-model-8fd363-kserve-77db559f77-qtxzx"
Nov 01 15:40:58.335044 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:58.335007    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f43a7e63-3095-4ad7-935a-bb46a0af45b9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f43a7e63-3095-4ad7-935a-bb46a0af45b9-model-cache\") pod \"f43a7e63-3095-4ad7-935a-bb46a0af45b9\" (UID: \"f43a7e63-3095-4ad7-935a-bb46a0af45b9\") "
Nov 01 15:40:58.335271 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:58.335088    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f43a7e63-3095-4ad7-935a-bb46a0af45b9-home\" (UniqueName: \"kubernetes.io/empty-dir/f43a7e63-3095-4ad7-935a-bb46a0af45b9-home\") pod \"f43a7e63-3095-4ad7-935a-bb46a0af45b9\" (UID: \"f43a7e63-3095-4ad7-935a-bb46a0af45b9\") "
Nov 01 15:40:58.335271 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:58.335119    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f43a7e63-3095-4ad7-935a-bb46a0af45b9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f43a7e63-3095-4ad7-935a-bb46a0af45b9-tmp-dir\") pod \"f43a7e63-3095-4ad7-935a-bb46a0af45b9\" (UID: \"f43a7e63-3095-4ad7-935a-bb46a0af45b9\") "
Nov 01 15:40:58.335271 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:58.335136    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f43a7e63-3095-4ad7-935a-bb46a0af45b9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f43a7e63-3095-4ad7-935a-bb46a0af45b9-dshm\") pod \"f43a7e63-3095-4ad7-935a-bb46a0af45b9\" (UID: \"f43a7e63-3095-4ad7-935a-bb46a0af45b9\") "
Nov 01 15:40:58.335271 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:58.335194    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f43a7e63-3095-4ad7-935a-bb46a0af45b9-kube-api-access-lr66c\" (UniqueName: \"kubernetes.io/projected/f43a7e63-3095-4ad7-935a-bb46a0af45b9-kube-api-access-lr66c\") pod \"f43a7e63-3095-4ad7-935a-bb46a0af45b9\" (UID: \"f43a7e63-3095-4ad7-935a-bb46a0af45b9\") "
Nov 01 15:40:58.335271 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:58.335250    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f43a7e63-3095-4ad7-935a-bb46a0af45b9-tls-certs\" (UniqueName: \"kubernetes.io/secret/f43a7e63-3095-4ad7-935a-bb46a0af45b9-tls-certs\") pod \"f43a7e63-3095-4ad7-935a-bb46a0af45b9\" (UID: \"f43a7e63-3095-4ad7-935a-bb46a0af45b9\") "
Nov 01 15:40:58.335559 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:58.335363    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f43a7e63-3095-4ad7-935a-bb46a0af45b9-model-cache" pod "f43a7e63-3095-4ad7-935a-bb46a0af45b9" (UID: "f43a7e63-3095-4ad7-935a-bb46a0af45b9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:40:58.335612 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:58.335591    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f43a7e63-3095-4ad7-935a-bb46a0af45b9-home" pod "f43a7e63-3095-4ad7-935a-bb46a0af45b9" (UID: "f43a7e63-3095-4ad7-935a-bb46a0af45b9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:40:58.335705 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:58.335685    4993 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f43a7e63-3095-4ad7-935a-bb46a0af45b9-home\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:40:58.335764 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:58.335710    4993 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f43a7e63-3095-4ad7-935a-bb46a0af45b9-model-cache\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:40:58.336062 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:58.336040    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f43a7e63-3095-4ad7-935a-bb46a0af45b9-tmp-dir" pod "f43a7e63-3095-4ad7-935a-bb46a0af45b9" (UID: "f43a7e63-3095-4ad7-935a-bb46a0af45b9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:40:58.338121 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:58.338091    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f43a7e63-3095-4ad7-935a-bb46a0af45b9-kube-api-access-lr66c" pod "f43a7e63-3095-4ad7-935a-bb46a0af45b9" (UID: "f43a7e63-3095-4ad7-935a-bb46a0af45b9"). InnerVolumeSpecName "kube-api-access-lr66c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:40:58.338279 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:58.338207    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f43a7e63-3095-4ad7-935a-bb46a0af45b9-tls-certs" pod "f43a7e63-3095-4ad7-935a-bb46a0af45b9" (UID: "f43a7e63-3095-4ad7-935a-bb46a0af45b9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:40:58.338774 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:58.338741    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f43a7e63-3095-4ad7-935a-bb46a0af45b9-dshm" pod "f43a7e63-3095-4ad7-935a-bb46a0af45b9" (UID: "f43a7e63-3095-4ad7-935a-bb46a0af45b9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:40:58.436811 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:58.436778    4993 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f43a7e63-3095-4ad7-935a-bb46a0af45b9-tmp-dir\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:40:58.436811 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:58.436804    4993 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f43a7e63-3095-4ad7-935a-bb46a0af45b9-dshm\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:40:58.436811 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:58.436814    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lr66c\" (UniqueName: \"kubernetes.io/projected/f43a7e63-3095-4ad7-935a-bb46a0af45b9-kube-api-access-lr66c\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:40:58.437045 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:58.436823    4993 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f43a7e63-3095-4ad7-935a-bb46a0af45b9-tls-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:40:58.706331 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:58.706295    4993 generic.go:356] "Generic (PLEG): container finished" podID="f43a7e63-3095-4ad7-935a-bb46a0af45b9" containerID="8e3d3bd34f89853d772792bd5efd3163f5b04cac354b2e060abb82aae6813b84" exitCode=0
Nov 01 15:40:58.706762 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:58.706349    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-8fd363/rate-test-model-8fd363-kserve-77db559f77-qtxzx"
Nov 01 15:40:58.706762 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:58.706363    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-8fd363/rate-test-model-8fd363-kserve-77db559f77-qtxzx" event={"ID":"f43a7e63-3095-4ad7-935a-bb46a0af45b9","Type":"ContainerDied","Data":"8e3d3bd34f89853d772792bd5efd3163f5b04cac354b2e060abb82aae6813b84"}
Nov 01 15:40:58.706762 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:58.706409    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-8fd363/rate-test-model-8fd363-kserve-77db559f77-qtxzx" event={"ID":"f43a7e63-3095-4ad7-935a-bb46a0af45b9","Type":"ContainerDied","Data":"1eb1ad7a476188b1ae6aed008761d6239501601f870b141dc35aa48858b1e64e"}
Nov 01 15:40:58.706762 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:58.706427    4993 scope.go:122] "RemoveContainer" containerID="8e3d3bd34f89853d772792bd5efd3163f5b04cac354b2e060abb82aae6813b84"
Nov 01 15:40:58.716145 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:58.716125    4993 scope.go:122] "RemoveContainer" containerID="8e3d3bd34f89853d772792bd5efd3163f5b04cac354b2e060abb82aae6813b84"
Nov 01 15:40:58.716439 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:40:58.716420    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e3d3bd34f89853d772792bd5efd3163f5b04cac354b2e060abb82aae6813b84\": container with ID starting with 8e3d3bd34f89853d772792bd5efd3163f5b04cac354b2e060abb82aae6813b84 not found: ID does not exist" containerID="8e3d3bd34f89853d772792bd5efd3163f5b04cac354b2e060abb82aae6813b84"
Nov 01 15:40:58.716512 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:58.716447    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e3d3bd34f89853d772792bd5efd3163f5b04cac354b2e060abb82aae6813b84"} err="failed to get container status \"8e3d3bd34f89853d772792bd5efd3163f5b04cac354b2e060abb82aae6813b84\": rpc error: code = NotFound desc = could not find container \"8e3d3bd34f89853d772792bd5efd3163f5b04cac354b2e060abb82aae6813b84\": container with ID starting with 8e3d3bd34f89853d772792bd5efd3163f5b04cac354b2e060abb82aae6813b84 not found: ID does not exist"
Nov 01 15:40:58.859732 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:58.859690    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8fd363/rate-test-model-8fd363-kserve-77db559f77-qtxzx"]
Nov 01 15:40:58.862673 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:58.862641    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8fd363/rate-test-model-8fd363-kserve-77db559f77-qtxzx"]
Nov 01 15:40:59.849899 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:59.849862    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca0f9e-645d8b74c-4lpp2"]
Nov 01 15:40:59.850411 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:40:59.850103    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca0f9e-645d8b74c-4lpp2" podUID="0e34d9fc-4d6c-496b-95db-6ddd2b2f2c10" containerName="maas-api" containerID="cri-o://23768c861603b16576ab042e9b5f0ec433d10efa64c5a78faf44f76dc216d9d2" gracePeriod=30
Nov 01 15:41:00.139445 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:00.139417    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca0f9e-645d8b74c-4lpp2"
Nov 01 15:41:00.167945 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:00.167909    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e34d9fc-4d6c-496b-95db-6ddd2b2f2c10-kube-api-access-hx85s\" (UniqueName: \"kubernetes.io/projected/0e34d9fc-4d6c-496b-95db-6ddd2b2f2c10-kube-api-access-hx85s\") pod \"0e34d9fc-4d6c-496b-95db-6ddd2b2f2c10\" (UID: \"0e34d9fc-4d6c-496b-95db-6ddd2b2f2c10\") "
Nov 01 15:41:00.168156 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:00.167999    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e34d9fc-4d6c-496b-95db-6ddd2b2f2c10-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e34d9fc-4d6c-496b-95db-6ddd2b2f2c10-metrics-certs\") pod \"0e34d9fc-4d6c-496b-95db-6ddd2b2f2c10\" (UID: \"0e34d9fc-4d6c-496b-95db-6ddd2b2f2c10\") "
Nov 01 15:41:00.168156 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:00.168069    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e34d9fc-4d6c-496b-95db-6ddd2b2f2c10-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e34d9fc-4d6c-496b-95db-6ddd2b2f2c10-openshift-service-ca\") pod \"0e34d9fc-4d6c-496b-95db-6ddd2b2f2c10\" (UID: \"0e34d9fc-4d6c-496b-95db-6ddd2b2f2c10\") "
Nov 01 15:41:00.168517 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:00.168495    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e34d9fc-4d6c-496b-95db-6ddd2b2f2c10-openshift-service-ca" pod "0e34d9fc-4d6c-496b-95db-6ddd2b2f2c10" (UID: "0e34d9fc-4d6c-496b-95db-6ddd2b2f2c10"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:41:00.170614 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:00.170585    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e34d9fc-4d6c-496b-95db-6ddd2b2f2c10-metrics-certs" pod "0e34d9fc-4d6c-496b-95db-6ddd2b2f2c10" (UID: "0e34d9fc-4d6c-496b-95db-6ddd2b2f2c10"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:41:00.170736 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:00.170618    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e34d9fc-4d6c-496b-95db-6ddd2b2f2c10-kube-api-access-hx85s" pod "0e34d9fc-4d6c-496b-95db-6ddd2b2f2c10" (UID: "0e34d9fc-4d6c-496b-95db-6ddd2b2f2c10"). InnerVolumeSpecName "kube-api-access-hx85s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:41:00.268975 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:00.268942    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e34d9fc-4d6c-496b-95db-6ddd2b2f2c10-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e34d9fc-4d6c-496b-95db-6ddd2b2f2c10-maas-api-tls\") pod \"0e34d9fc-4d6c-496b-95db-6ddd2b2f2c10\" (UID: \"0e34d9fc-4d6c-496b-95db-6ddd2b2f2c10\") "
Nov 01 15:41:00.269279 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:00.269261    4993 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e34d9fc-4d6c-496b-95db-6ddd2b2f2c10-metrics-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:41:00.269279 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:00.269280    4993 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e34d9fc-4d6c-496b-95db-6ddd2b2f2c10-openshift-service-ca\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:41:00.269442 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:00.269309    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hx85s\" (UniqueName: \"kubernetes.io/projected/0e34d9fc-4d6c-496b-95db-6ddd2b2f2c10-kube-api-access-hx85s\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:41:00.271018 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:00.270989    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e34d9fc-4d6c-496b-95db-6ddd2b2f2c10-maas-api-tls" pod "0e34d9fc-4d6c-496b-95db-6ddd2b2f2c10" (UID: "0e34d9fc-4d6c-496b-95db-6ddd2b2f2c10"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:41:00.370606 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:00.370488    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e34d9fc-4d6c-496b-95db-6ddd2b2f2c10-maas-api-tls\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:41:00.728091 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:00.727983    4993 generic.go:356] "Generic (PLEG): container finished" podID="0e34d9fc-4d6c-496b-95db-6ddd2b2f2c10" containerID="23768c861603b16576ab042e9b5f0ec433d10efa64c5a78faf44f76dc216d9d2" exitCode=0
Nov 01 15:41:00.728091 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:00.728052    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca0f9e-645d8b74c-4lpp2" event={"ID":"0e34d9fc-4d6c-496b-95db-6ddd2b2f2c10","Type":"ContainerDied","Data":"23768c861603b16576ab042e9b5f0ec433d10efa64c5a78faf44f76dc216d9d2"}
Nov 01 15:41:00.728362 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:00.728088    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca0f9e-645d8b74c-4lpp2"
Nov 01 15:41:00.728362 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:00.728136    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca0f9e-645d8b74c-4lpp2" event={"ID":"0e34d9fc-4d6c-496b-95db-6ddd2b2f2c10","Type":"ContainerDied","Data":"dfc75b0d95a4e50afd138d77a58b879e2f01e87b098bd710377a177002939d28"}
Nov 01 15:41:00.728362 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:00.728182    4993 scope.go:122] "RemoveContainer" containerID="23768c861603b16576ab042e9b5f0ec433d10efa64c5a78faf44f76dc216d9d2"
Nov 01 15:41:00.744444 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:00.744125    4993 scope.go:122] "RemoveContainer" containerID="23768c861603b16576ab042e9b5f0ec433d10efa64c5a78faf44f76dc216d9d2"
Nov 01 15:41:00.744812 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:41:00.744776    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23768c861603b16576ab042e9b5f0ec433d10efa64c5a78faf44f76dc216d9d2\": container with ID starting with 23768c861603b16576ab042e9b5f0ec433d10efa64c5a78faf44f76dc216d9d2 not found: ID does not exist" containerID="23768c861603b16576ab042e9b5f0ec433d10efa64c5a78faf44f76dc216d9d2"
Nov 01 15:41:00.744982 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:00.744820    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23768c861603b16576ab042e9b5f0ec433d10efa64c5a78faf44f76dc216d9d2"} err="failed to get container status \"23768c861603b16576ab042e9b5f0ec433d10efa64c5a78faf44f76dc216d9d2\": rpc error: code = NotFound desc = could not find container \"23768c861603b16576ab042e9b5f0ec433d10efa64c5a78faf44f76dc216d9d2\": container with ID starting with 23768c861603b16576ab042e9b5f0ec433d10efa64c5a78faf44f76dc216d9d2 not found: ID does not exist"
Nov 01 15:41:00.795538 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:00.795506    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f43a7e63-3095-4ad7-935a-bb46a0af45b9" path="/var/lib/kubelet/pods/f43a7e63-3095-4ad7-935a-bb46a0af45b9/volumes"
Nov 01 15:41:00.883278 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:00.883241    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca0f9e-645d8b74c-4lpp2"]
Nov 01 15:41:00.888979 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:00.888955    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca0f9e-645d8b74c-4lpp2"]
Nov 01 15:41:02.796958 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:02.796920    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e34d9fc-4d6c-496b-95db-6ddd2b2f2c10" path="/var/lib/kubelet/pods/0e34d9fc-4d6c-496b-95db-6ddd2b2f2c10/volumes"
Nov 01 15:41:10.404425 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:10.404380    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb"]
Nov 01 15:41:10.404941 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:10.404642    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb" podUID="4196a39a-6cde-4b40-b2b2-c4792aceb17e" containerName="istio-proxy" containerID="cri-o://39bc79121cbef6faffb528cc2034b9a978a4707319f0bcfda02e52ee6b334d97" gracePeriod=30
Nov 01 15:41:13.162658 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:13.162615    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-007270/simulator-w2-007270-kserve-5bf8844f79-c7v9v"]
Nov 01 15:41:13.993539 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:13.993503    4993 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 15:41:13.993710 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:13.993576    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb" podUID="4196a39a-6cde-4b40-b2b2-c4792aceb17e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 15:41:15.660623 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:15.660593    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb"
Nov 01 15:41:15.739264 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:15.739228    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4196a39a-6cde-4b40-b2b2-c4792aceb17e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4196a39a-6cde-4b40-b2b2-c4792aceb17e-istio-envoy\") pod \"4196a39a-6cde-4b40-b2b2-c4792aceb17e\" (UID: \"4196a39a-6cde-4b40-b2b2-c4792aceb17e\") "
Nov 01 15:41:15.739464 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:15.739277    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4196a39a-6cde-4b40-b2b2-c4792aceb17e-istio-token\" (UniqueName: \"kubernetes.io/projected/4196a39a-6cde-4b40-b2b2-c4792aceb17e-istio-token\") pod \"4196a39a-6cde-4b40-b2b2-c4792aceb17e\" (UID: \"4196a39a-6cde-4b40-b2b2-c4792aceb17e\") "
Nov 01 15:41:15.739464 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:15.739365    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4196a39a-6cde-4b40-b2b2-c4792aceb17e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4196a39a-6cde-4b40-b2b2-c4792aceb17e-istio-data\") pod \"4196a39a-6cde-4b40-b2b2-c4792aceb17e\" (UID: \"4196a39a-6cde-4b40-b2b2-c4792aceb17e\") "
Nov 01 15:41:15.739464 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:15.739417    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4196a39a-6cde-4b40-b2b2-c4792aceb17e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4196a39a-6cde-4b40-b2b2-c4792aceb17e-istiod-ca-cert\") pod \"4196a39a-6cde-4b40-b2b2-c4792aceb17e\" (UID: \"4196a39a-6cde-4b40-b2b2-c4792aceb17e\") "
Nov 01 15:41:15.739464 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:15.739458    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4196a39a-6cde-4b40-b2b2-c4792aceb17e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4196a39a-6cde-4b40-b2b2-c4792aceb17e-credential-socket\") pod \"4196a39a-6cde-4b40-b2b2-c4792aceb17e\" (UID: \"4196a39a-6cde-4b40-b2b2-c4792aceb17e\") "
Nov 01 15:41:15.739683 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:15.739529    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4196a39a-6cde-4b40-b2b2-c4792aceb17e-kube-api-access-dfncq\" (UniqueName: \"kubernetes.io/projected/4196a39a-6cde-4b40-b2b2-c4792aceb17e-kube-api-access-dfncq\") pod \"4196a39a-6cde-4b40-b2b2-c4792aceb17e\" (UID: \"4196a39a-6cde-4b40-b2b2-c4792aceb17e\") "
Nov 01 15:41:15.739683 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:15.739568    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4196a39a-6cde-4b40-b2b2-c4792aceb17e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4196a39a-6cde-4b40-b2b2-c4792aceb17e-workload-socket\") pod \"4196a39a-6cde-4b40-b2b2-c4792aceb17e\" (UID: \"4196a39a-6cde-4b40-b2b2-c4792aceb17e\") "
Nov 01 15:41:15.739683 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:15.739601    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4196a39a-6cde-4b40-b2b2-c4792aceb17e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4196a39a-6cde-4b40-b2b2-c4792aceb17e-workload-certs\") pod \"4196a39a-6cde-4b40-b2b2-c4792aceb17e\" (UID: \"4196a39a-6cde-4b40-b2b2-c4792aceb17e\") "
Nov 01 15:41:15.739683 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:15.739615    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4196a39a-6cde-4b40-b2b2-c4792aceb17e-istio-data" pod "4196a39a-6cde-4b40-b2b2-c4792aceb17e" (UID: "4196a39a-6cde-4b40-b2b2-c4792aceb17e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:41:15.739683 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:15.739636    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4196a39a-6cde-4b40-b2b2-c4792aceb17e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4196a39a-6cde-4b40-b2b2-c4792aceb17e-istio-podinfo\") pod \"4196a39a-6cde-4b40-b2b2-c4792aceb17e\" (UID: \"4196a39a-6cde-4b40-b2b2-c4792aceb17e\") "
Nov 01 15:41:15.739918 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:15.739823    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4196a39a-6cde-4b40-b2b2-c4792aceb17e-workload-socket" pod "4196a39a-6cde-4b40-b2b2-c4792aceb17e" (UID: "4196a39a-6cde-4b40-b2b2-c4792aceb17e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:41:15.739918 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:15.739856    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4196a39a-6cde-4b40-b2b2-c4792aceb17e-istiod-ca-cert" pod "4196a39a-6cde-4b40-b2b2-c4792aceb17e" (UID: "4196a39a-6cde-4b40-b2b2-c4792aceb17e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:41:15.740025 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:15.740008    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4196a39a-6cde-4b40-b2b2-c4792aceb17e-credential-socket" pod "4196a39a-6cde-4b40-b2b2-c4792aceb17e" (UID: "4196a39a-6cde-4b40-b2b2-c4792aceb17e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:41:15.740087 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:15.740022    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4196a39a-6cde-4b40-b2b2-c4792aceb17e-workload-certs" pod "4196a39a-6cde-4b40-b2b2-c4792aceb17e" (UID: "4196a39a-6cde-4b40-b2b2-c4792aceb17e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:41:15.740259 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:15.740237    4993 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4196a39a-6cde-4b40-b2b2-c4792aceb17e-istio-data\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:41:15.740334 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:15.740264    4993 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4196a39a-6cde-4b40-b2b2-c4792aceb17e-istiod-ca-cert\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:41:15.740334 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:15.740281    4993 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4196a39a-6cde-4b40-b2b2-c4792aceb17e-credential-socket\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:41:15.740334 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:15.740296    4993 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4196a39a-6cde-4b40-b2b2-c4792aceb17e-workload-socket\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:41:15.740334 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:15.740312    4993 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4196a39a-6cde-4b40-b2b2-c4792aceb17e-workload-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:41:15.741721 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:15.741690    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4196a39a-6cde-4b40-b2b2-c4792aceb17e-istio-envoy" pod "4196a39a-6cde-4b40-b2b2-c4792aceb17e" (UID: "4196a39a-6cde-4b40-b2b2-c4792aceb17e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:41:15.741721 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:15.741708    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4196a39a-6cde-4b40-b2b2-c4792aceb17e-istio-token" pod "4196a39a-6cde-4b40-b2b2-c4792aceb17e" (UID: "4196a39a-6cde-4b40-b2b2-c4792aceb17e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:41:15.741888 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:15.741847    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4196a39a-6cde-4b40-b2b2-c4792aceb17e-kube-api-access-dfncq" pod "4196a39a-6cde-4b40-b2b2-c4792aceb17e" (UID: "4196a39a-6cde-4b40-b2b2-c4792aceb17e"). InnerVolumeSpecName "kube-api-access-dfncq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:41:15.741945 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:15.741883    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4196a39a-6cde-4b40-b2b2-c4792aceb17e-istio-podinfo" pod "4196a39a-6cde-4b40-b2b2-c4792aceb17e" (UID: "4196a39a-6cde-4b40-b2b2-c4792aceb17e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 15:41:15.824362 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:15.824325    4993 generic.go:356] "Generic (PLEG): container finished" podID="4196a39a-6cde-4b40-b2b2-c4792aceb17e" containerID="39bc79121cbef6faffb528cc2034b9a978a4707319f0bcfda02e52ee6b334d97" exitCode=0
Nov 01 15:41:15.824559 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:15.824379    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb"
Nov 01 15:41:15.824559 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:15.824412    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb" event={"ID":"4196a39a-6cde-4b40-b2b2-c4792aceb17e","Type":"ContainerDied","Data":"39bc79121cbef6faffb528cc2034b9a978a4707319f0bcfda02e52ee6b334d97"}
Nov 01 15:41:15.824559 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:15.824471    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb" event={"ID":"4196a39a-6cde-4b40-b2b2-c4792aceb17e","Type":"ContainerDied","Data":"e2e6b07f5d475f23200e246afa786edb71207437dcc83c9dc94d9694a917a5e0"}
Nov 01 15:41:15.824559 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:15.824494    4993 scope.go:122] "RemoveContainer" containerID="39bc79121cbef6faffb528cc2034b9a978a4707319f0bcfda02e52ee6b334d97"
Nov 01 15:41:15.834865 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:15.834840    4993 scope.go:122] "RemoveContainer" containerID="39bc79121cbef6faffb528cc2034b9a978a4707319f0bcfda02e52ee6b334d97"
Nov 01 15:41:15.835247 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:41:15.835218    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39bc79121cbef6faffb528cc2034b9a978a4707319f0bcfda02e52ee6b334d97\": container with ID starting with 39bc79121cbef6faffb528cc2034b9a978a4707319f0bcfda02e52ee6b334d97 not found: ID does not exist" containerID="39bc79121cbef6faffb528cc2034b9a978a4707319f0bcfda02e52ee6b334d97"
Nov 01 15:41:15.835380 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:15.835254    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39bc79121cbef6faffb528cc2034b9a978a4707319f0bcfda02e52ee6b334d97"} err="failed to get container status \"39bc79121cbef6faffb528cc2034b9a978a4707319f0bcfda02e52ee6b334d97\": rpc error: code = NotFound desc = could not find container \"39bc79121cbef6faffb528cc2034b9a978a4707319f0bcfda02e52ee6b334d97\": container with ID starting with 39bc79121cbef6faffb528cc2034b9a978a4707319f0bcfda02e52ee6b334d97 not found: ID does not exist"
Nov 01 15:41:15.841799 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:15.841733    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfncq\" (UniqueName: \"kubernetes.io/projected/4196a39a-6cde-4b40-b2b2-c4792aceb17e-kube-api-access-dfncq\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:41:15.841799 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:15.841759    4993 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4196a39a-6cde-4b40-b2b2-c4792aceb17e-istio-podinfo\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:41:15.841799 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:15.841773    4993 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4196a39a-6cde-4b40-b2b2-c4792aceb17e-istio-envoy\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:41:15.841799 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:15.841783    4993 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4196a39a-6cde-4b40-b2b2-c4792aceb17e-istio-token\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:41:15.978290 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:15.978251    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb"]
Nov 01 15:41:15.980692 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:15.980661    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-m4fpb"]
Nov 01 15:41:16.796455 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:16.796413    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4196a39a-6cde-4b40-b2b2-c4792aceb17e" path="/var/lib/kubelet/pods/4196a39a-6cde-4b40-b2b2-c4792aceb17e/volumes"
Nov 01 15:41:20.489462 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:20.489425    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-96165a-6cccf67998-zjh5q"]
Nov 01 15:41:20.489862 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:20.489647    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-96165a-6cccf67998-zjh5q" podUID="8cebd6df-9dc4-47f7-a67d-9b1f21551d8b" containerName="maas-api" containerID="cri-o://2608f1421a8e50446b78a0a028588a8aa4c580832c23bd90dc26d2085cd3315b" gracePeriod=30
Nov 01 15:41:20.772055 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:20.772021    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-96165a-6cccf67998-zjh5q"
Nov 01 15:41:20.855920 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:20.855803    4993 generic.go:356] "Generic (PLEG): container finished" podID="8cebd6df-9dc4-47f7-a67d-9b1f21551d8b" containerID="2608f1421a8e50446b78a0a028588a8aa4c580832c23bd90dc26d2085cd3315b" exitCode=0
Nov 01 15:41:20.855920 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:20.855851    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-96165a-6cccf67998-zjh5q" event={"ID":"8cebd6df-9dc4-47f7-a67d-9b1f21551d8b","Type":"ContainerDied","Data":"2608f1421a8e50446b78a0a028588a8aa4c580832c23bd90dc26d2085cd3315b"}
Nov 01 15:41:20.855920 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:20.855908    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-96165a-6cccf67998-zjh5q"
Nov 01 15:41:20.856260 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:20.855931    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-96165a-6cccf67998-zjh5q" event={"ID":"8cebd6df-9dc4-47f7-a67d-9b1f21551d8b","Type":"ContainerDied","Data":"b90ffa8b25a475a5e11c6111eca6f8e505470eed8ee625defe9049e8e0f79228"}
Nov 01 15:41:20.856260 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:20.855962    4993 scope.go:122] "RemoveContainer" containerID="2608f1421a8e50446b78a0a028588a8aa4c580832c23bd90dc26d2085cd3315b"
Nov 01 15:41:20.866614 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:20.866586    4993 scope.go:122] "RemoveContainer" containerID="2608f1421a8e50446b78a0a028588a8aa4c580832c23bd90dc26d2085cd3315b"
Nov 01 15:41:20.866900 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:41:20.866884    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2608f1421a8e50446b78a0a028588a8aa4c580832c23bd90dc26d2085cd3315b\": container with ID starting with 2608f1421a8e50446b78a0a028588a8aa4c580832c23bd90dc26d2085cd3315b not found: ID does not exist" containerID="2608f1421a8e50446b78a0a028588a8aa4c580832c23bd90dc26d2085cd3315b"
Nov 01 15:41:20.866953 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:20.866910    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2608f1421a8e50446b78a0a028588a8aa4c580832c23bd90dc26d2085cd3315b"} err="failed to get container status \"2608f1421a8e50446b78a0a028588a8aa4c580832c23bd90dc26d2085cd3315b\": rpc error: code = NotFound desc = could not find container \"2608f1421a8e50446b78a0a028588a8aa4c580832c23bd90dc26d2085cd3315b\": container with ID starting with 2608f1421a8e50446b78a0a028588a8aa4c580832c23bd90dc26d2085cd3315b not found: ID does not exist"
Nov 01 15:41:20.899931 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:20.899897    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cebd6df-9dc4-47f7-a67d-9b1f21551d8b-kube-api-access-zhwmc\" (UniqueName: \"kubernetes.io/projected/8cebd6df-9dc4-47f7-a67d-9b1f21551d8b-kube-api-access-zhwmc\") pod \"8cebd6df-9dc4-47f7-a67d-9b1f21551d8b\" (UID: \"8cebd6df-9dc4-47f7-a67d-9b1f21551d8b\") "
Nov 01 15:41:20.900130 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:20.899968    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cebd6df-9dc4-47f7-a67d-9b1f21551d8b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cebd6df-9dc4-47f7-a67d-9b1f21551d8b-openshift-service-ca\") pod \"8cebd6df-9dc4-47f7-a67d-9b1f21551d8b\" (UID: \"8cebd6df-9dc4-47f7-a67d-9b1f21551d8b\") "
Nov 01 15:41:20.900130 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:20.900019    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8cebd6df-9dc4-47f7-a67d-9b1f21551d8b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cebd6df-9dc4-47f7-a67d-9b1f21551d8b-maas-api-tls\") pod \"8cebd6df-9dc4-47f7-a67d-9b1f21551d8b\" (UID: \"8cebd6df-9dc4-47f7-a67d-9b1f21551d8b\") "
Nov 01 15:41:20.900130 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:20.900096    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8cebd6df-9dc4-47f7-a67d-9b1f21551d8b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cebd6df-9dc4-47f7-a67d-9b1f21551d8b-metrics-certs\") pod \"8cebd6df-9dc4-47f7-a67d-9b1f21551d8b\" (UID: \"8cebd6df-9dc4-47f7-a67d-9b1f21551d8b\") "
Nov 01 15:41:20.900404 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:20.900378    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cebd6df-9dc4-47f7-a67d-9b1f21551d8b-openshift-service-ca" pod "8cebd6df-9dc4-47f7-a67d-9b1f21551d8b" (UID: "8cebd6df-9dc4-47f7-a67d-9b1f21551d8b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:41:20.900608 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:20.900584    4993 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cebd6df-9dc4-47f7-a67d-9b1f21551d8b-openshift-service-ca\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:41:20.902344 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:20.902307    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cebd6df-9dc4-47f7-a67d-9b1f21551d8b-metrics-certs" pod "8cebd6df-9dc4-47f7-a67d-9b1f21551d8b" (UID: "8cebd6df-9dc4-47f7-a67d-9b1f21551d8b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:41:20.902344 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:20.902336    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cebd6df-9dc4-47f7-a67d-9b1f21551d8b-maas-api-tls" pod "8cebd6df-9dc4-47f7-a67d-9b1f21551d8b" (UID: "8cebd6df-9dc4-47f7-a67d-9b1f21551d8b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:41:20.902575 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:20.902559    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cebd6df-9dc4-47f7-a67d-9b1f21551d8b-kube-api-access-zhwmc" pod "8cebd6df-9dc4-47f7-a67d-9b1f21551d8b" (UID: "8cebd6df-9dc4-47f7-a67d-9b1f21551d8b"). InnerVolumeSpecName "kube-api-access-zhwmc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:41:21.001590 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:21.001495    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cebd6df-9dc4-47f7-a67d-9b1f21551d8b-maas-api-tls\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:41:21.001590 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:21.001537    4993 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cebd6df-9dc4-47f7-a67d-9b1f21551d8b-metrics-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:41:21.001590 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:21.001556    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhwmc\" (UniqueName: \"kubernetes.io/projected/8cebd6df-9dc4-47f7-a67d-9b1f21551d8b-kube-api-access-zhwmc\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:41:21.268507 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:21.268426    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-96165a-6cccf67998-zjh5q"]
Nov 01 15:41:21.272476 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:21.272447    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-96165a-6cccf67998-zjh5q"]
Nov 01 15:41:22.795377 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:22.795343    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cebd6df-9dc4-47f7-a67d-9b1f21551d8b" path="/var/lib/kubelet/pods/8cebd6df-9dc4-47f7-a67d-9b1f21551d8b/volumes"
Nov 01 15:41:27.501212 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:41:27.501147    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:41:28.183959 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:28.183907    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-007270/simulator-w2-007270-kserve-5bf8844f79-c7v9v" podUID="2ae81a81-e763-40ac-8f5e-4bd38f9a3e68" containerName="main" containerID="cri-o://09998d05d6ddb8e95f6dad6a29b774ff321d60f92a894e18aa4bd1da2ec938ae" gracePeriod=45
Nov 01 15:41:28.191064 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:28.190726    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-007270/simulator-w2-007270-kserve-5bf8844f79-c7v9v" podUID="2ae81a81-e763-40ac-8f5e-4bd38f9a3e68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": dial tcp 10.132.0.60:8000: connect: connection refused"
Nov 01 15:41:28.454744 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:28.454716    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-007270/simulator-w2-007270-kserve-5bf8844f79-c7v9v"
Nov 01 15:41:28.585024 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:28.584957    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ae81a81-e763-40ac-8f5e-4bd38f9a3e68-home\" (UniqueName: \"kubernetes.io/empty-dir/2ae81a81-e763-40ac-8f5e-4bd38f9a3e68-home\") pod \"2ae81a81-e763-40ac-8f5e-4bd38f9a3e68\" (UID: \"2ae81a81-e763-40ac-8f5e-4bd38f9a3e68\") "
Nov 01 15:41:28.585799 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:28.585099    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ae81a81-e763-40ac-8f5e-4bd38f9a3e68-kube-api-access-cx68x\" (UniqueName: \"kubernetes.io/projected/2ae81a81-e763-40ac-8f5e-4bd38f9a3e68-kube-api-access-cx68x\") pod \"2ae81a81-e763-40ac-8f5e-4bd38f9a3e68\" (UID: \"2ae81a81-e763-40ac-8f5e-4bd38f9a3e68\") "
Nov 01 15:41:28.585799 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:28.585128    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ae81a81-e763-40ac-8f5e-4bd38f9a3e68-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ae81a81-e763-40ac-8f5e-4bd38f9a3e68-model-cache\") pod \"2ae81a81-e763-40ac-8f5e-4bd38f9a3e68\" (UID: \"2ae81a81-e763-40ac-8f5e-4bd38f9a3e68\") "
Nov 01 15:41:28.585799 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:28.585212    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ae81a81-e763-40ac-8f5e-4bd38f9a3e68-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ae81a81-e763-40ac-8f5e-4bd38f9a3e68-dshm\") pod \"2ae81a81-e763-40ac-8f5e-4bd38f9a3e68\" (UID: \"2ae81a81-e763-40ac-8f5e-4bd38f9a3e68\") "
Nov 01 15:41:28.585799 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:28.585253    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ae81a81-e763-40ac-8f5e-4bd38f9a3e68-tls-certs\" (UniqueName: \"kubernetes.io/secret/2ae81a81-e763-40ac-8f5e-4bd38f9a3e68-tls-certs\") pod \"2ae81a81-e763-40ac-8f5e-4bd38f9a3e68\" (UID: \"2ae81a81-e763-40ac-8f5e-4bd38f9a3e68\") "
Nov 01 15:41:28.585799 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:28.585301    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ae81a81-e763-40ac-8f5e-4bd38f9a3e68-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ae81a81-e763-40ac-8f5e-4bd38f9a3e68-tmp-dir\") pod \"2ae81a81-e763-40ac-8f5e-4bd38f9a3e68\" (UID: \"2ae81a81-e763-40ac-8f5e-4bd38f9a3e68\") "
Nov 01 15:41:28.586369 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:28.585875    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ae81a81-e763-40ac-8f5e-4bd38f9a3e68-tmp-dir" pod "2ae81a81-e763-40ac-8f5e-4bd38f9a3e68" (UID: "2ae81a81-e763-40ac-8f5e-4bd38f9a3e68"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:41:28.586369 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:28.586055    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ae81a81-e763-40ac-8f5e-4bd38f9a3e68-home" pod "2ae81a81-e763-40ac-8f5e-4bd38f9a3e68" (UID: "2ae81a81-e763-40ac-8f5e-4bd38f9a3e68"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:41:28.586549 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:28.586423    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ae81a81-e763-40ac-8f5e-4bd38f9a3e68-model-cache" pod "2ae81a81-e763-40ac-8f5e-4bd38f9a3e68" (UID: "2ae81a81-e763-40ac-8f5e-4bd38f9a3e68"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:41:28.588989 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:28.588965    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ae81a81-e763-40ac-8f5e-4bd38f9a3e68-kube-api-access-cx68x" pod "2ae81a81-e763-40ac-8f5e-4bd38f9a3e68" (UID: "2ae81a81-e763-40ac-8f5e-4bd38f9a3e68"). InnerVolumeSpecName "kube-api-access-cx68x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:41:28.590038 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:28.589997    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ae81a81-e763-40ac-8f5e-4bd38f9a3e68-dshm" pod "2ae81a81-e763-40ac-8f5e-4bd38f9a3e68" (UID: "2ae81a81-e763-40ac-8f5e-4bd38f9a3e68"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:41:28.590613 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:28.590576    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ae81a81-e763-40ac-8f5e-4bd38f9a3e68-tls-certs" pod "2ae81a81-e763-40ac-8f5e-4bd38f9a3e68" (UID: "2ae81a81-e763-40ac-8f5e-4bd38f9a3e68"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:41:28.686918 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:28.686833    4993 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ae81a81-e763-40ac-8f5e-4bd38f9a3e68-dshm\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:41:28.686918 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:28.686868    4993 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ae81a81-e763-40ac-8f5e-4bd38f9a3e68-tls-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:41:28.686918 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:28.686881    4993 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ae81a81-e763-40ac-8f5e-4bd38f9a3e68-tmp-dir\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:41:28.686918 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:28.686889    4993 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ae81a81-e763-40ac-8f5e-4bd38f9a3e68-home\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:41:28.686918 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:28.686899    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cx68x\" (UniqueName: \"kubernetes.io/projected/2ae81a81-e763-40ac-8f5e-4bd38f9a3e68-kube-api-access-cx68x\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:41:28.686918 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:28.686909    4993 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ae81a81-e763-40ac-8f5e-4bd38f9a3e68-model-cache\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:41:28.911877 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:28.911838    4993 generic.go:356] "Generic (PLEG): container finished" podID="2ae81a81-e763-40ac-8f5e-4bd38f9a3e68" containerID="09998d05d6ddb8e95f6dad6a29b774ff321d60f92a894e18aa4bd1da2ec938ae" exitCode=0
Nov 01 15:41:28.912097 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:28.911884    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-007270/simulator-w2-007270-kserve-5bf8844f79-c7v9v"
Nov 01 15:41:28.912097 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:28.911914    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-007270/simulator-w2-007270-kserve-5bf8844f79-c7v9v" event={"ID":"2ae81a81-e763-40ac-8f5e-4bd38f9a3e68","Type":"ContainerDied","Data":"09998d05d6ddb8e95f6dad6a29b774ff321d60f92a894e18aa4bd1da2ec938ae"}
Nov 01 15:41:28.912097 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:28.911956    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-007270/simulator-w2-007270-kserve-5bf8844f79-c7v9v" event={"ID":"2ae81a81-e763-40ac-8f5e-4bd38f9a3e68","Type":"ContainerDied","Data":"6dca24665904dca4595aaecb558fec9634ccac1ceda21aff7a7787099046d6c0"}
Nov 01 15:41:28.912097 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:28.911978    4993 scope.go:122] "RemoveContainer" containerID="09998d05d6ddb8e95f6dad6a29b774ff321d60f92a894e18aa4bd1da2ec938ae"
Nov 01 15:41:28.925388 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:28.925368    4993 scope.go:122] "RemoveContainer" containerID="09998d05d6ddb8e95f6dad6a29b774ff321d60f92a894e18aa4bd1da2ec938ae"
Nov 01 15:41:28.925680 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:41:28.925655    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09998d05d6ddb8e95f6dad6a29b774ff321d60f92a894e18aa4bd1da2ec938ae\": container with ID starting with 09998d05d6ddb8e95f6dad6a29b774ff321d60f92a894e18aa4bd1da2ec938ae not found: ID does not exist" containerID="09998d05d6ddb8e95f6dad6a29b774ff321d60f92a894e18aa4bd1da2ec938ae"
Nov 01 15:41:28.925727 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:28.925694    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09998d05d6ddb8e95f6dad6a29b774ff321d60f92a894e18aa4bd1da2ec938ae"} err="failed to get container status \"09998d05d6ddb8e95f6dad6a29b774ff321d60f92a894e18aa4bd1da2ec938ae\": rpc error: code = NotFound desc = could not find container \"09998d05d6ddb8e95f6dad6a29b774ff321d60f92a894e18aa4bd1da2ec938ae\": container with ID starting with 09998d05d6ddb8e95f6dad6a29b774ff321d60f92a894e18aa4bd1da2ec938ae not found: ID does not exist"
Nov 01 15:41:29.064811 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:29.064775    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-007270/simulator-w2-007270-kserve-5bf8844f79-c7v9v"]
Nov 01 15:41:29.069698 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:29.069659    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-007270/simulator-w2-007270-kserve-5bf8844f79-c7v9v"]
Nov 01 15:41:30.079613 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:30.079574    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jqv2m"]
Nov 01 15:41:30.080148 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:30.079768    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jqv2m" podUID="8d18c69f-6d32-40d7-b857-d0d686235013" containerName="istio-proxy" containerID="cri-o://a800ea4265bea44c90a5e6b419b173fad026aac68c05ad9a9237d682eb74022d" gracePeriod=30
Nov 01 15:41:30.194295 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:30.194253    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-jqv2m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 15:41:30.194489 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:30.194311    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jqv2m" podUID="8d18c69f-6d32-40d7-b857-d0d686235013" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 15:41:30.795148 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:30.795112    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ae81a81-e763-40ac-8f5e-4bd38f9a3e68" path="/var/lib/kubelet/pods/2ae81a81-e763-40ac-8f5e-4bd38f9a3e68/volumes"
Nov 01 15:41:35.344726 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:35.344696    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jqv2m"
Nov 01 15:41:35.355296 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:35.355238    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d18c69f-6d32-40d7-b857-d0d686235013-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d18c69f-6d32-40d7-b857-d0d686235013-istiod-ca-cert\") pod \"8d18c69f-6d32-40d7-b857-d0d686235013\" (UID: \"8d18c69f-6d32-40d7-b857-d0d686235013\") "
Nov 01 15:41:35.355296 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:35.355280    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d18c69f-6d32-40d7-b857-d0d686235013-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d18c69f-6d32-40d7-b857-d0d686235013-istio-envoy\") pod \"8d18c69f-6d32-40d7-b857-d0d686235013\" (UID: \"8d18c69f-6d32-40d7-b857-d0d686235013\") "
Nov 01 15:41:35.355480 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:35.355327    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8d18c69f-6d32-40d7-b857-d0d686235013-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d18c69f-6d32-40d7-b857-d0d686235013-istio-podinfo\") pod \"8d18c69f-6d32-40d7-b857-d0d686235013\" (UID: \"8d18c69f-6d32-40d7-b857-d0d686235013\") "
Nov 01 15:41:35.355480 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:35.355444    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d18c69f-6d32-40d7-b857-d0d686235013-istio-token\" (UniqueName: \"kubernetes.io/projected/8d18c69f-6d32-40d7-b857-d0d686235013-istio-token\") pod \"8d18c69f-6d32-40d7-b857-d0d686235013\" (UID: \"8d18c69f-6d32-40d7-b857-d0d686235013\") "
Nov 01 15:41:35.355594 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:35.355492    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d18c69f-6d32-40d7-b857-d0d686235013-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d18c69f-6d32-40d7-b857-d0d686235013-credential-socket\") pod \"8d18c69f-6d32-40d7-b857-d0d686235013\" (UID: \"8d18c69f-6d32-40d7-b857-d0d686235013\") "
Nov 01 15:41:35.355594 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:35.355533    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d18c69f-6d32-40d7-b857-d0d686235013-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d18c69f-6d32-40d7-b857-d0d686235013-workload-socket\") pod \"8d18c69f-6d32-40d7-b857-d0d686235013\" (UID: \"8d18c69f-6d32-40d7-b857-d0d686235013\") "
Nov 01 15:41:35.355705 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:35.355624    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d18c69f-6d32-40d7-b857-d0d686235013-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d18c69f-6d32-40d7-b857-d0d686235013-workload-certs\") pod \"8d18c69f-6d32-40d7-b857-d0d686235013\" (UID: \"8d18c69f-6d32-40d7-b857-d0d686235013\") "
Nov 01 15:41:35.355705 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:35.355659    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d18c69f-6d32-40d7-b857-d0d686235013-istiod-ca-cert" pod "8d18c69f-6d32-40d7-b857-d0d686235013" (UID: "8d18c69f-6d32-40d7-b857-d0d686235013"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:41:35.355705 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:35.355667    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d18c69f-6d32-40d7-b857-d0d686235013-kube-api-access-jbz5b\" (UniqueName: \"kubernetes.io/projected/8d18c69f-6d32-40d7-b857-d0d686235013-kube-api-access-jbz5b\") pod \"8d18c69f-6d32-40d7-b857-d0d686235013\" (UID: \"8d18c69f-6d32-40d7-b857-d0d686235013\") "
Nov 01 15:41:35.355860 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:35.355741    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d18c69f-6d32-40d7-b857-d0d686235013-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d18c69f-6d32-40d7-b857-d0d686235013-istio-data\") pod \"8d18c69f-6d32-40d7-b857-d0d686235013\" (UID: \"8d18c69f-6d32-40d7-b857-d0d686235013\") "
Nov 01 15:41:35.356136 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:35.356110    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d18c69f-6d32-40d7-b857-d0d686235013-workload-socket" pod "8d18c69f-6d32-40d7-b857-d0d686235013" (UID: "8d18c69f-6d32-40d7-b857-d0d686235013"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:41:35.356326 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:35.356306    4993 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d18c69f-6d32-40d7-b857-d0d686235013-workload-socket\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:41:35.356410 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:35.356331    4993 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d18c69f-6d32-40d7-b857-d0d686235013-istiod-ca-cert\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:41:35.356410 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:35.356314    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d18c69f-6d32-40d7-b857-d0d686235013-credential-socket" pod "8d18c69f-6d32-40d7-b857-d0d686235013" (UID: "8d18c69f-6d32-40d7-b857-d0d686235013"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:41:35.356410 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:35.356120    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d18c69f-6d32-40d7-b857-d0d686235013-workload-certs" pod "8d18c69f-6d32-40d7-b857-d0d686235013" (UID: "8d18c69f-6d32-40d7-b857-d0d686235013"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:41:35.356410 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:35.356328    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d18c69f-6d32-40d7-b857-d0d686235013-istio-data" pod "8d18c69f-6d32-40d7-b857-d0d686235013" (UID: "8d18c69f-6d32-40d7-b857-d0d686235013"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:41:35.358044 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:35.358020    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d18c69f-6d32-40d7-b857-d0d686235013-istio-envoy" pod "8d18c69f-6d32-40d7-b857-d0d686235013" (UID: "8d18c69f-6d32-40d7-b857-d0d686235013"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:41:35.358128 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:35.358030    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d18c69f-6d32-40d7-b857-d0d686235013-istio-token" pod "8d18c69f-6d32-40d7-b857-d0d686235013" (UID: "8d18c69f-6d32-40d7-b857-d0d686235013"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:41:35.358128 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:35.358062    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d18c69f-6d32-40d7-b857-d0d686235013-istio-podinfo" pod "8d18c69f-6d32-40d7-b857-d0d686235013" (UID: "8d18c69f-6d32-40d7-b857-d0d686235013"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 15:41:35.358245 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:35.358134    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d18c69f-6d32-40d7-b857-d0d686235013-kube-api-access-jbz5b" pod "8d18c69f-6d32-40d7-b857-d0d686235013" (UID: "8d18c69f-6d32-40d7-b857-d0d686235013"). InnerVolumeSpecName "kube-api-access-jbz5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:41:35.457337 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:35.457296    4993 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d18c69f-6d32-40d7-b857-d0d686235013-workload-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:41:35.457337 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:35.457334    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbz5b\" (UniqueName: \"kubernetes.io/projected/8d18c69f-6d32-40d7-b857-d0d686235013-kube-api-access-jbz5b\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:41:35.457580 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:35.457352    4993 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d18c69f-6d32-40d7-b857-d0d686235013-istio-data\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:41:35.457580 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:35.457364    4993 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d18c69f-6d32-40d7-b857-d0d686235013-istio-envoy\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:41:35.457580 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:35.457378    4993 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d18c69f-6d32-40d7-b857-d0d686235013-istio-podinfo\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:41:35.457580 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:35.457390    4993 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d18c69f-6d32-40d7-b857-d0d686235013-istio-token\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:41:35.457580 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:35.457402    4993 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d18c69f-6d32-40d7-b857-d0d686235013-credential-socket\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:41:35.954699 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:35.954662    4993 generic.go:356] "Generic (PLEG): container finished" podID="8d18c69f-6d32-40d7-b857-d0d686235013" containerID="a800ea4265bea44c90a5e6b419b173fad026aac68c05ad9a9237d682eb74022d" exitCode=0
Nov 01 15:41:35.954891 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:35.954713    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jqv2m"
Nov 01 15:41:35.954891 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:35.954723    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jqv2m" event={"ID":"8d18c69f-6d32-40d7-b857-d0d686235013","Type":"ContainerDied","Data":"a800ea4265bea44c90a5e6b419b173fad026aac68c05ad9a9237d682eb74022d"}
Nov 01 15:41:35.954891 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:35.954776    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jqv2m" event={"ID":"8d18c69f-6d32-40d7-b857-d0d686235013","Type":"ContainerDied","Data":"0d9b9d79fbbd55e537e3d4b9cf86ad128093269058bfb571e59a3ec3751e2c3c"}
Nov 01 15:41:35.954891 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:35.954799    4993 scope.go:122] "RemoveContainer" containerID="a800ea4265bea44c90a5e6b419b173fad026aac68c05ad9a9237d682eb74022d"
Nov 01 15:41:35.965075 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:35.965058    4993 scope.go:122] "RemoveContainer" containerID="a800ea4265bea44c90a5e6b419b173fad026aac68c05ad9a9237d682eb74022d"
Nov 01 15:41:35.965366 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:41:35.965349    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a800ea4265bea44c90a5e6b419b173fad026aac68c05ad9a9237d682eb74022d\": container with ID starting with a800ea4265bea44c90a5e6b419b173fad026aac68c05ad9a9237d682eb74022d not found: ID does not exist" containerID="a800ea4265bea44c90a5e6b419b173fad026aac68c05ad9a9237d682eb74022d"
Nov 01 15:41:35.965432 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:35.965374    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a800ea4265bea44c90a5e6b419b173fad026aac68c05ad9a9237d682eb74022d"} err="failed to get container status \"a800ea4265bea44c90a5e6b419b173fad026aac68c05ad9a9237d682eb74022d\": rpc error: code = NotFound desc = could not find container \"a800ea4265bea44c90a5e6b419b173fad026aac68c05ad9a9237d682eb74022d\": container with ID starting with a800ea4265bea44c90a5e6b419b173fad026aac68c05ad9a9237d682eb74022d not found: ID does not exist"
Nov 01 15:41:36.107070 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:36.107030    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jqv2m"]
Nov 01 15:41:36.110110 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:36.110080    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-jqv2m"]
Nov 01 15:41:36.795799 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:36.795763    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d18c69f-6d32-40d7-b857-d0d686235013" path="/var/lib/kubelet/pods/8d18c69f-6d32-40d7-b857-d0d686235013/volumes"
Nov 01 15:41:53.522797 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:53.522769    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xmwtp_bb59051d-c2df-4362-a157-00f14f915e79/console-operator/3.log"
Nov 01 15:41:53.536676 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:41:53.536655    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xmwtp_bb59051d-c2df-4362-a157-00f14f915e79/console-operator/3.log"
Nov 01 15:41:57.500560 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:41:57.500518    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:42:02.553494 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:02.553457    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-af2dd2-865d4789d4-2cvq5"]
Nov 01 15:42:02.553915 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:02.553699    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-af2dd2-865d4789d4-2cvq5" podUID="4808fe31-322b-4ae7-bf3b-f46024a38aa2" containerName="maas-api" containerID="cri-o://c527f0cc6281fdb509985094a6c3b0cb2b801190c2b3eff2fc13c795336bff2d" gracePeriod=30
Nov 01 15:42:02.820444 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:02.820372    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-af2dd2-865d4789d4-2cvq5"
Nov 01 15:42:02.941515 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:02.941477    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4808fe31-322b-4ae7-bf3b-f46024a38aa2-kube-api-access-5wqjv\" (UniqueName: \"kubernetes.io/projected/4808fe31-322b-4ae7-bf3b-f46024a38aa2-kube-api-access-5wqjv\") pod \"4808fe31-322b-4ae7-bf3b-f46024a38aa2\" (UID: \"4808fe31-322b-4ae7-bf3b-f46024a38aa2\") "
Nov 01 15:42:02.941682 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:02.941563    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4808fe31-322b-4ae7-bf3b-f46024a38aa2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4808fe31-322b-4ae7-bf3b-f46024a38aa2-metrics-certs\") pod \"4808fe31-322b-4ae7-bf3b-f46024a38aa2\" (UID: \"4808fe31-322b-4ae7-bf3b-f46024a38aa2\") "
Nov 01 15:42:02.941731 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:02.941710    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4808fe31-322b-4ae7-bf3b-f46024a38aa2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4808fe31-322b-4ae7-bf3b-f46024a38aa2-maas-api-tls\") pod \"4808fe31-322b-4ae7-bf3b-f46024a38aa2\" (UID: \"4808fe31-322b-4ae7-bf3b-f46024a38aa2\") "
Nov 01 15:42:02.941779 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:02.941744    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4808fe31-322b-4ae7-bf3b-f46024a38aa2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4808fe31-322b-4ae7-bf3b-f46024a38aa2-openshift-service-ca\") pod \"4808fe31-322b-4ae7-bf3b-f46024a38aa2\" (UID: \"4808fe31-322b-4ae7-bf3b-f46024a38aa2\") "
Nov 01 15:42:02.942131 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:02.942100    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4808fe31-322b-4ae7-bf3b-f46024a38aa2-openshift-service-ca" pod "4808fe31-322b-4ae7-bf3b-f46024a38aa2" (UID: "4808fe31-322b-4ae7-bf3b-f46024a38aa2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:42:02.943766 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:02.943745    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4808fe31-322b-4ae7-bf3b-f46024a38aa2-maas-api-tls" pod "4808fe31-322b-4ae7-bf3b-f46024a38aa2" (UID: "4808fe31-322b-4ae7-bf3b-f46024a38aa2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:42:02.944152 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:02.944130    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4808fe31-322b-4ae7-bf3b-f46024a38aa2-metrics-certs" pod "4808fe31-322b-4ae7-bf3b-f46024a38aa2" (UID: "4808fe31-322b-4ae7-bf3b-f46024a38aa2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:42:02.944280 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:02.944152    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4808fe31-322b-4ae7-bf3b-f46024a38aa2-kube-api-access-5wqjv" pod "4808fe31-322b-4ae7-bf3b-f46024a38aa2" (UID: "4808fe31-322b-4ae7-bf3b-f46024a38aa2"). InnerVolumeSpecName "kube-api-access-5wqjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:42:03.043560 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:03.043514    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4808fe31-322b-4ae7-bf3b-f46024a38aa2-maas-api-tls\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:42:03.043560 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:03.043557    4993 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4808fe31-322b-4ae7-bf3b-f46024a38aa2-openshift-service-ca\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:42:03.043783 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:03.043572    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wqjv\" (UniqueName: \"kubernetes.io/projected/4808fe31-322b-4ae7-bf3b-f46024a38aa2-kube-api-access-5wqjv\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:42:03.043783 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:03.043589    4993 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4808fe31-322b-4ae7-bf3b-f46024a38aa2-metrics-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:42:03.114199 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:03.114093    4993 generic.go:356] "Generic (PLEG): container finished" podID="4808fe31-322b-4ae7-bf3b-f46024a38aa2" containerID="c527f0cc6281fdb509985094a6c3b0cb2b801190c2b3eff2fc13c795336bff2d" exitCode=0
Nov 01 15:42:03.114199 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:03.114142    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-af2dd2-865d4789d4-2cvq5"
Nov 01 15:42:03.114389 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:03.114213    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-af2dd2-865d4789d4-2cvq5" event={"ID":"4808fe31-322b-4ae7-bf3b-f46024a38aa2","Type":"ContainerDied","Data":"c527f0cc6281fdb509985094a6c3b0cb2b801190c2b3eff2fc13c795336bff2d"}
Nov 01 15:42:03.114389 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:03.114271    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-af2dd2-865d4789d4-2cvq5" event={"ID":"4808fe31-322b-4ae7-bf3b-f46024a38aa2","Type":"ContainerDied","Data":"a7cdabd4c74df9f0a7c7eeac48182f3e1e8172a4fab21c3e3f40c34fc154af78"}
Nov 01 15:42:03.114389 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:03.114301    4993 scope.go:122] "RemoveContainer" containerID="c527f0cc6281fdb509985094a6c3b0cb2b801190c2b3eff2fc13c795336bff2d"
Nov 01 15:42:03.124251 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:03.124229    4993 scope.go:122] "RemoveContainer" containerID="c527f0cc6281fdb509985094a6c3b0cb2b801190c2b3eff2fc13c795336bff2d"
Nov 01 15:42:03.124573 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:42:03.124552    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c527f0cc6281fdb509985094a6c3b0cb2b801190c2b3eff2fc13c795336bff2d\": container with ID starting with c527f0cc6281fdb509985094a6c3b0cb2b801190c2b3eff2fc13c795336bff2d not found: ID does not exist" containerID="c527f0cc6281fdb509985094a6c3b0cb2b801190c2b3eff2fc13c795336bff2d"
Nov 01 15:42:03.124624 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:03.124582    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c527f0cc6281fdb509985094a6c3b0cb2b801190c2b3eff2fc13c795336bff2d"} err="failed to get container status \"c527f0cc6281fdb509985094a6c3b0cb2b801190c2b3eff2fc13c795336bff2d\": rpc error: code = NotFound desc = could not find container \"c527f0cc6281fdb509985094a6c3b0cb2b801190c2b3eff2fc13c795336bff2d\": container with ID starting with c527f0cc6281fdb509985094a6c3b0cb2b801190c2b3eff2fc13c795336bff2d not found: ID does not exist"
Nov 01 15:42:03.265507 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:03.265466    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-af2dd2-865d4789d4-2cvq5"]
Nov 01 15:42:03.269322 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:03.269292    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-af2dd2-865d4789d4-2cvq5"]
Nov 01 15:42:03.600462 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:03.600428    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3f45c3-6fb65f9b8c-6gx8x"]
Nov 01 15:42:03.600880 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:03.600623    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3f45c3-6fb65f9b8c-6gx8x" podUID="e2108b10-03af-450f-b750-e56bab9faff2" containerName="maas-api" containerID="cri-o://5b4d7d6057d656f65cc690a3fd7c99cf7bd1dfd54c50562b0ed4701940125ba3" gracePeriod=30
Nov 01 15:42:03.875317 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:03.875287    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3f45c3-6fb65f9b8c-6gx8x"
Nov 01 15:42:04.054480 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:04.054446    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2108b10-03af-450f-b750-e56bab9faff2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2108b10-03af-450f-b750-e56bab9faff2-openshift-service-ca\") pod \"e2108b10-03af-450f-b750-e56bab9faff2\" (UID: \"e2108b10-03af-450f-b750-e56bab9faff2\") "
Nov 01 15:42:04.054664 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:04.054514    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2108b10-03af-450f-b750-e56bab9faff2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2108b10-03af-450f-b750-e56bab9faff2-maas-api-tls\") pod \"e2108b10-03af-450f-b750-e56bab9faff2\" (UID: \"e2108b10-03af-450f-b750-e56bab9faff2\") "
Nov 01 15:42:04.054664 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:04.054533    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2108b10-03af-450f-b750-e56bab9faff2-kube-api-access-h4828\" (UniqueName: \"kubernetes.io/projected/e2108b10-03af-450f-b750-e56bab9faff2-kube-api-access-h4828\") pod \"e2108b10-03af-450f-b750-e56bab9faff2\" (UID: \"e2108b10-03af-450f-b750-e56bab9faff2\") "
Nov 01 15:42:04.054664 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:04.054592    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2108b10-03af-450f-b750-e56bab9faff2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2108b10-03af-450f-b750-e56bab9faff2-metrics-certs\") pod \"e2108b10-03af-450f-b750-e56bab9faff2\" (UID: \"e2108b10-03af-450f-b750-e56bab9faff2\") "
Nov 01 15:42:04.055009 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:04.054894    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2108b10-03af-450f-b750-e56bab9faff2-openshift-service-ca" pod "e2108b10-03af-450f-b750-e56bab9faff2" (UID: "e2108b10-03af-450f-b750-e56bab9faff2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:42:04.055144 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:04.055110    4993 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2108b10-03af-450f-b750-e56bab9faff2-openshift-service-ca\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:42:04.056822 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:04.056797    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2108b10-03af-450f-b750-e56bab9faff2-maas-api-tls" pod "e2108b10-03af-450f-b750-e56bab9faff2" (UID: "e2108b10-03af-450f-b750-e56bab9faff2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:42:04.057227 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:04.057200    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2108b10-03af-450f-b750-e56bab9faff2-kube-api-access-h4828" pod "e2108b10-03af-450f-b750-e56bab9faff2" (UID: "e2108b10-03af-450f-b750-e56bab9faff2"). InnerVolumeSpecName "kube-api-access-h4828". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:42:04.057311 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:04.057227    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2108b10-03af-450f-b750-e56bab9faff2-metrics-certs" pod "e2108b10-03af-450f-b750-e56bab9faff2" (UID: "e2108b10-03af-450f-b750-e56bab9faff2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:42:04.120838 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:04.120740    4993 generic.go:356] "Generic (PLEG): container finished" podID="e2108b10-03af-450f-b750-e56bab9faff2" containerID="5b4d7d6057d656f65cc690a3fd7c99cf7bd1dfd54c50562b0ed4701940125ba3" exitCode=0
Nov 01 15:42:04.120838 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:04.120792    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3f45c3-6fb65f9b8c-6gx8x"
Nov 01 15:42:04.120838 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:04.120817    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3f45c3-6fb65f9b8c-6gx8x" event={"ID":"e2108b10-03af-450f-b750-e56bab9faff2","Type":"ContainerDied","Data":"5b4d7d6057d656f65cc690a3fd7c99cf7bd1dfd54c50562b0ed4701940125ba3"}
Nov 01 15:42:04.121127 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:04.120874    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3f45c3-6fb65f9b8c-6gx8x" event={"ID":"e2108b10-03af-450f-b750-e56bab9faff2","Type":"ContainerDied","Data":"65c04f7841ae6bf40d52816e7c8088201b9d8bfd4123c56872fa2813a286b530"}
Nov 01 15:42:04.121127 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:04.120900    4993 scope.go:122] "RemoveContainer" containerID="5b4d7d6057d656f65cc690a3fd7c99cf7bd1dfd54c50562b0ed4701940125ba3"
Nov 01 15:42:04.131071 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:04.131051    4993 scope.go:122] "RemoveContainer" containerID="5b4d7d6057d656f65cc690a3fd7c99cf7bd1dfd54c50562b0ed4701940125ba3"
Nov 01 15:42:04.131361 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:42:04.131338    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b4d7d6057d656f65cc690a3fd7c99cf7bd1dfd54c50562b0ed4701940125ba3\": container with ID starting with 5b4d7d6057d656f65cc690a3fd7c99cf7bd1dfd54c50562b0ed4701940125ba3 not found: ID does not exist" containerID="5b4d7d6057d656f65cc690a3fd7c99cf7bd1dfd54c50562b0ed4701940125ba3"
Nov 01 15:42:04.131472 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:04.131368    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b4d7d6057d656f65cc690a3fd7c99cf7bd1dfd54c50562b0ed4701940125ba3"} err="failed to get container status \"5b4d7d6057d656f65cc690a3fd7c99cf7bd1dfd54c50562b0ed4701940125ba3\": rpc error: code = NotFound desc = could not find container \"5b4d7d6057d656f65cc690a3fd7c99cf7bd1dfd54c50562b0ed4701940125ba3\": container with ID starting with 5b4d7d6057d656f65cc690a3fd7c99cf7bd1dfd54c50562b0ed4701940125ba3 not found: ID does not exist"
Nov 01 15:42:04.156304 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:04.156271    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2108b10-03af-450f-b750-e56bab9faff2-maas-api-tls\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:42:04.156304 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:04.156305    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4828\" (UniqueName: \"kubernetes.io/projected/e2108b10-03af-450f-b750-e56bab9faff2-kube-api-access-h4828\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:42:04.156482 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:04.156321    4993 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2108b10-03af-450f-b750-e56bab9faff2-metrics-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:42:04.272834 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:04.272799    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3f45c3-6fb65f9b8c-6gx8x"]
Nov 01 15:42:04.278462 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:04.278436    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3f45c3-6fb65f9b8c-6gx8x"]
Nov 01 15:42:04.794704 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:04.794671    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4808fe31-322b-4ae7-bf3b-f46024a38aa2" path="/var/lib/kubelet/pods/4808fe31-322b-4ae7-bf3b-f46024a38aa2/volumes"
Nov 01 15:42:04.795105 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:04.795091    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2108b10-03af-450f-b750-e56bab9faff2" path="/var/lib/kubelet/pods/e2108b10-03af-450f-b750-e56bab9faff2/volumes"
Nov 01 15:42:08.995256 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:08.995217    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-007270-7bf55ff5fc-88krj"]
Nov 01 15:42:08.995694 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:08.995589    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-007270-7bf55ff5fc-88krj" podUID="ea582656-480c-4e92-868d-c1f36225e433" containerName="maas-api" containerID="cri-o://5816b8dc1eaad31d3aa4c76157985729dd1cfc85d6aae5e3545bda160df0f5d8" gracePeriod=30
Nov 01 15:42:09.152449 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:09.152408    4993 generic.go:356] "Generic (PLEG): container finished" podID="ea582656-480c-4e92-868d-c1f36225e433" containerID="5816b8dc1eaad31d3aa4c76157985729dd1cfc85d6aae5e3545bda160df0f5d8" exitCode=0
Nov 01 15:42:09.152639 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:09.152501    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-007270-7bf55ff5fc-88krj" event={"ID":"ea582656-480c-4e92-868d-c1f36225e433","Type":"ContainerDied","Data":"5816b8dc1eaad31d3aa4c76157985729dd1cfc85d6aae5e3545bda160df0f5d8"}
Nov 01 15:42:09.289968 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:09.289936    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-007270-7bf55ff5fc-88krj"
Nov 01 15:42:09.312529 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:09.312497    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea582656-480c-4e92-868d-c1f36225e433-kube-api-access-m2qvv\" (UniqueName: \"kubernetes.io/projected/ea582656-480c-4e92-868d-c1f36225e433-kube-api-access-m2qvv\") pod \"ea582656-480c-4e92-868d-c1f36225e433\" (UID: \"ea582656-480c-4e92-868d-c1f36225e433\") "
Nov 01 15:42:09.312714 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:09.312581    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea582656-480c-4e92-868d-c1f36225e433-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea582656-480c-4e92-868d-c1f36225e433-metrics-certs\") pod \"ea582656-480c-4e92-868d-c1f36225e433\" (UID: \"ea582656-480c-4e92-868d-c1f36225e433\") "
Nov 01 15:42:09.312714 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:09.312619    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea582656-480c-4e92-868d-c1f36225e433-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea582656-480c-4e92-868d-c1f36225e433-openshift-service-ca\") pod \"ea582656-480c-4e92-868d-c1f36225e433\" (UID: \"ea582656-480c-4e92-868d-c1f36225e433\") "
Nov 01 15:42:09.312714 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:09.312667    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea582656-480c-4e92-868d-c1f36225e433-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea582656-480c-4e92-868d-c1f36225e433-maas-api-tls\") pod \"ea582656-480c-4e92-868d-c1f36225e433\" (UID: \"ea582656-480c-4e92-868d-c1f36225e433\") "
Nov 01 15:42:09.313011 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:09.312981    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea582656-480c-4e92-868d-c1f36225e433-openshift-service-ca" pod "ea582656-480c-4e92-868d-c1f36225e433" (UID: "ea582656-480c-4e92-868d-c1f36225e433"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:42:09.314809 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:09.314783    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea582656-480c-4e92-868d-c1f36225e433-metrics-certs" pod "ea582656-480c-4e92-868d-c1f36225e433" (UID: "ea582656-480c-4e92-868d-c1f36225e433"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:42:09.315006 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:09.314981    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea582656-480c-4e92-868d-c1f36225e433-maas-api-tls" pod "ea582656-480c-4e92-868d-c1f36225e433" (UID: "ea582656-480c-4e92-868d-c1f36225e433"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:42:09.315006 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:09.314987    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea582656-480c-4e92-868d-c1f36225e433-kube-api-access-m2qvv" pod "ea582656-480c-4e92-868d-c1f36225e433" (UID: "ea582656-480c-4e92-868d-c1f36225e433"). InnerVolumeSpecName "kube-api-access-m2qvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:42:09.414495 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:09.414463    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2qvv\" (UniqueName: \"kubernetes.io/projected/ea582656-480c-4e92-868d-c1f36225e433-kube-api-access-m2qvv\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:42:09.414495 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:09.414493    4993 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea582656-480c-4e92-868d-c1f36225e433-metrics-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:42:09.414698 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:09.414503    4993 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea582656-480c-4e92-868d-c1f36225e433-openshift-service-ca\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:42:09.414698 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:09.414513    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea582656-480c-4e92-868d-c1f36225e433-maas-api-tls\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:42:10.160113 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:10.160077    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-007270-7bf55ff5fc-88krj"
Nov 01 15:42:10.160586 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:10.160082    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-007270-7bf55ff5fc-88krj" event={"ID":"ea582656-480c-4e92-868d-c1f36225e433","Type":"ContainerDied","Data":"9f74ef8749865c7d51b96f699dffb082b41b7d6b97ac36e5a11dc52faefb94e0"}
Nov 01 15:42:10.160586 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:10.160232    4993 scope.go:122] "RemoveContainer" containerID="5816b8dc1eaad31d3aa4c76157985729dd1cfc85d6aae5e3545bda160df0f5d8"
Nov 01 15:42:10.312183 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:10.312134    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-007270-7bf55ff5fc-88krj"]
Nov 01 15:42:10.358432 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:10.358395    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-007270-7bf55ff5fc-88krj"]
Nov 01 15:42:10.795056 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:10.795019    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea582656-480c-4e92-868d-c1f36225e433" path="/var/lib/kubelet/pods/ea582656-480c-4e92-868d-c1f36225e433/volumes"
Nov 01 15:42:27.500516 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:42:27.500473    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:42:37.418861 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:37.418818    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1ac11c-76b994fd57-mhmp5"]
Nov 01 15:42:37.419385 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:37.419088    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1ac11c-76b994fd57-mhmp5" podUID="6ad8a332-6847-48a9-9249-415156dff36d" containerName="maas-api" containerID="cri-o://b8d32f611baf4308b14e4141f1815b91b6ac3719c8f40c99c461db83ad8a6210" gracePeriod=30
Nov 01 15:42:37.691657 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:37.691627    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1ac11c-76b994fd57-mhmp5"
Nov 01 15:42:37.719016 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:37.718988    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ad8a332-6847-48a9-9249-415156dff36d-kube-api-access-q72nt\" (UniqueName: \"kubernetes.io/projected/6ad8a332-6847-48a9-9249-415156dff36d-kube-api-access-q72nt\") pod \"6ad8a332-6847-48a9-9249-415156dff36d\" (UID: \"6ad8a332-6847-48a9-9249-415156dff36d\") "
Nov 01 15:42:37.719196 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:37.719032    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ad8a332-6847-48a9-9249-415156dff36d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ad8a332-6847-48a9-9249-415156dff36d-openshift-service-ca\") pod \"6ad8a332-6847-48a9-9249-415156dff36d\" (UID: \"6ad8a332-6847-48a9-9249-415156dff36d\") "
Nov 01 15:42:37.719196 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:37.719074    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ad8a332-6847-48a9-9249-415156dff36d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ad8a332-6847-48a9-9249-415156dff36d-maas-api-tls\") pod \"6ad8a332-6847-48a9-9249-415156dff36d\" (UID: \"6ad8a332-6847-48a9-9249-415156dff36d\") "
Nov 01 15:42:37.719196 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:37.719138    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ad8a332-6847-48a9-9249-415156dff36d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ad8a332-6847-48a9-9249-415156dff36d-metrics-certs\") pod \"6ad8a332-6847-48a9-9249-415156dff36d\" (UID: \"6ad8a332-6847-48a9-9249-415156dff36d\") "
Nov 01 15:42:37.719534 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:37.719501    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ad8a332-6847-48a9-9249-415156dff36d-openshift-service-ca" pod "6ad8a332-6847-48a9-9249-415156dff36d" (UID: "6ad8a332-6847-48a9-9249-415156dff36d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:42:37.721333 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:37.721297    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ad8a332-6847-48a9-9249-415156dff36d-maas-api-tls" pod "6ad8a332-6847-48a9-9249-415156dff36d" (UID: "6ad8a332-6847-48a9-9249-415156dff36d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:42:37.721333 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:37.721323    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ad8a332-6847-48a9-9249-415156dff36d-metrics-certs" pod "6ad8a332-6847-48a9-9249-415156dff36d" (UID: "6ad8a332-6847-48a9-9249-415156dff36d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:42:37.721480 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:37.721391    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ad8a332-6847-48a9-9249-415156dff36d-kube-api-access-q72nt" pod "6ad8a332-6847-48a9-9249-415156dff36d" (UID: "6ad8a332-6847-48a9-9249-415156dff36d"). InnerVolumeSpecName "kube-api-access-q72nt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:42:37.820457 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:37.820418    4993 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ad8a332-6847-48a9-9249-415156dff36d-openshift-service-ca\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:42:37.820457 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:37.820456    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ad8a332-6847-48a9-9249-415156dff36d-maas-api-tls\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:42:37.820712 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:37.820471    4993 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ad8a332-6847-48a9-9249-415156dff36d-metrics-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:42:37.820712 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:37.820488    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q72nt\" (UniqueName: \"kubernetes.io/projected/6ad8a332-6847-48a9-9249-415156dff36d-kube-api-access-q72nt\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:42:38.313785 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:38.313754    4993 generic.go:356] "Generic (PLEG): container finished" podID="6ad8a332-6847-48a9-9249-415156dff36d" containerID="b8d32f611baf4308b14e4141f1815b91b6ac3719c8f40c99c461db83ad8a6210" exitCode=0
Nov 01 15:42:38.313973 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:38.313806    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1ac11c-76b994fd57-mhmp5"
Nov 01 15:42:38.313973 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:38.313849    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1ac11c-76b994fd57-mhmp5" event={"ID":"6ad8a332-6847-48a9-9249-415156dff36d","Type":"ContainerDied","Data":"b8d32f611baf4308b14e4141f1815b91b6ac3719c8f40c99c461db83ad8a6210"}
Nov 01 15:42:38.313973 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:38.313905    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1ac11c-76b994fd57-mhmp5" event={"ID":"6ad8a332-6847-48a9-9249-415156dff36d","Type":"ContainerDied","Data":"09ecb773d90a34938e10313a09eb4ed2369b5a9169cf9bbe2366da16103eee81"}
Nov 01 15:42:38.313973 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:38.313929    4993 scope.go:122] "RemoveContainer" containerID="b8d32f611baf4308b14e4141f1815b91b6ac3719c8f40c99c461db83ad8a6210"
Nov 01 15:42:38.329106 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:38.329085    4993 scope.go:122] "RemoveContainer" containerID="b8d32f611baf4308b14e4141f1815b91b6ac3719c8f40c99c461db83ad8a6210"
Nov 01 15:42:38.329530 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:42:38.329487    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8d32f611baf4308b14e4141f1815b91b6ac3719c8f40c99c461db83ad8a6210\": container with ID starting with b8d32f611baf4308b14e4141f1815b91b6ac3719c8f40c99c461db83ad8a6210 not found: ID does not exist" containerID="b8d32f611baf4308b14e4141f1815b91b6ac3719c8f40c99c461db83ad8a6210"
Nov 01 15:42:38.329702 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:38.329527    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8d32f611baf4308b14e4141f1815b91b6ac3719c8f40c99c461db83ad8a6210"} err="failed to get container status \"b8d32f611baf4308b14e4141f1815b91b6ac3719c8f40c99c461db83ad8a6210\": rpc error: code = NotFound desc = could not find container \"b8d32f611baf4308b14e4141f1815b91b6ac3719c8f40c99c461db83ad8a6210\": container with ID starting with b8d32f611baf4308b14e4141f1815b91b6ac3719c8f40c99c461db83ad8a6210 not found: ID does not exist"
Nov 01 15:42:38.487926 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:38.487893    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1ac11c-76b994fd57-mhmp5"]
Nov 01 15:42:38.533950 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:38.533910    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1ac11c-76b994fd57-mhmp5"]
Nov 01 15:42:38.794776 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:42:38.794741    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ad8a332-6847-48a9-9249-415156dff36d" path="/var/lib/kubelet/pods/6ad8a332-6847-48a9-9249-415156dff36d/volumes"
Nov 01 15:42:57.502579 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:42:57.502538    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:43:02.377471 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:02.377436    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8fd363-c4bc9fcf8-pm6vm"]
Nov 01 15:43:02.377954 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:02.377639    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8fd363-c4bc9fcf8-pm6vm" podUID="1896a74d-cf1e-48cc-8ab8-330e11570845" containerName="maas-api" containerID="cri-o://344ab2c54a9fd135fde8842986ad8ac27ca1f2ace7e593ab39d4aaa90c16c054" gracePeriod=30
Nov 01 15:43:02.677494 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:02.677449    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8fd363-c4bc9fcf8-pm6vm"
Nov 01 15:43:02.783982 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:02.783913    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1896a74d-cf1e-48cc-8ab8-330e11570845-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1896a74d-cf1e-48cc-8ab8-330e11570845-maas-api-tls\") pod \"1896a74d-cf1e-48cc-8ab8-330e11570845\" (UID: \"1896a74d-cf1e-48cc-8ab8-330e11570845\") "
Nov 01 15:43:02.784234 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:02.783993    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1896a74d-cf1e-48cc-8ab8-330e11570845-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1896a74d-cf1e-48cc-8ab8-330e11570845-openshift-service-ca\") pod \"1896a74d-cf1e-48cc-8ab8-330e11570845\" (UID: \"1896a74d-cf1e-48cc-8ab8-330e11570845\") "
Nov 01 15:43:02.784234 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:02.784035    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1896a74d-cf1e-48cc-8ab8-330e11570845-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1896a74d-cf1e-48cc-8ab8-330e11570845-metrics-certs\") pod \"1896a74d-cf1e-48cc-8ab8-330e11570845\" (UID: \"1896a74d-cf1e-48cc-8ab8-330e11570845\") "
Nov 01 15:43:02.784234 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:02.784144    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1896a74d-cf1e-48cc-8ab8-330e11570845-kube-api-access-rp9w6\" (UniqueName: \"kubernetes.io/projected/1896a74d-cf1e-48cc-8ab8-330e11570845-kube-api-access-rp9w6\") pod \"1896a74d-cf1e-48cc-8ab8-330e11570845\" (UID: \"1896a74d-cf1e-48cc-8ab8-330e11570845\") "
Nov 01 15:43:02.784676 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:02.784642    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1896a74d-cf1e-48cc-8ab8-330e11570845-openshift-service-ca" pod "1896a74d-cf1e-48cc-8ab8-330e11570845" (UID: "1896a74d-cf1e-48cc-8ab8-330e11570845"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:43:02.786896 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:02.786870    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1896a74d-cf1e-48cc-8ab8-330e11570845-kube-api-access-rp9w6" pod "1896a74d-cf1e-48cc-8ab8-330e11570845" (UID: "1896a74d-cf1e-48cc-8ab8-330e11570845"). InnerVolumeSpecName "kube-api-access-rp9w6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:43:02.787257 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:02.787228    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1896a74d-cf1e-48cc-8ab8-330e11570845-maas-api-tls" pod "1896a74d-cf1e-48cc-8ab8-330e11570845" (UID: "1896a74d-cf1e-48cc-8ab8-330e11570845"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:43:02.787759 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:02.787719    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1896a74d-cf1e-48cc-8ab8-330e11570845-metrics-certs" pod "1896a74d-cf1e-48cc-8ab8-330e11570845" (UID: "1896a74d-cf1e-48cc-8ab8-330e11570845"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:43:02.885213 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:02.885148    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1896a74d-cf1e-48cc-8ab8-330e11570845-maas-api-tls\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:43:02.885213 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:02.885208    4993 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1896a74d-cf1e-48cc-8ab8-330e11570845-openshift-service-ca\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:43:02.885213 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:02.885220    4993 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1896a74d-cf1e-48cc-8ab8-330e11570845-metrics-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:43:02.885457 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:02.885229    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rp9w6\" (UniqueName: \"kubernetes.io/projected/1896a74d-cf1e-48cc-8ab8-330e11570845-kube-api-access-rp9w6\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:43:03.450791 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:03.450759    4993 generic.go:356] "Generic (PLEG): container finished" podID="1896a74d-cf1e-48cc-8ab8-330e11570845" containerID="344ab2c54a9fd135fde8842986ad8ac27ca1f2ace7e593ab39d4aaa90c16c054" exitCode=0
Nov 01 15:43:03.451294 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:03.450815    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8fd363-c4bc9fcf8-pm6vm"
Nov 01 15:43:03.451294 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:03.450853    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8fd363-c4bc9fcf8-pm6vm" event={"ID":"1896a74d-cf1e-48cc-8ab8-330e11570845","Type":"ContainerDied","Data":"344ab2c54a9fd135fde8842986ad8ac27ca1f2ace7e593ab39d4aaa90c16c054"}
Nov 01 15:43:03.451294 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:03.450923    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8fd363-c4bc9fcf8-pm6vm" event={"ID":"1896a74d-cf1e-48cc-8ab8-330e11570845","Type":"ContainerDied","Data":"a0a2652c181b58e59686213bb7cc0b73d542e6e5aa614ceef214257e90574819"}
Nov 01 15:43:03.451294 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:03.450955    4993 scope.go:122] "RemoveContainer" containerID="344ab2c54a9fd135fde8842986ad8ac27ca1f2ace7e593ab39d4aaa90c16c054"
Nov 01 15:43:03.460079 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:03.460058    4993 scope.go:122] "RemoveContainer" containerID="344ab2c54a9fd135fde8842986ad8ac27ca1f2ace7e593ab39d4aaa90c16c054"
Nov 01 15:43:03.460374 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:43:03.460344    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"344ab2c54a9fd135fde8842986ad8ac27ca1f2ace7e593ab39d4aaa90c16c054\": container with ID starting with 344ab2c54a9fd135fde8842986ad8ac27ca1f2ace7e593ab39d4aaa90c16c054 not found: ID does not exist" containerID="344ab2c54a9fd135fde8842986ad8ac27ca1f2ace7e593ab39d4aaa90c16c054"
Nov 01 15:43:03.460458 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:03.460383    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"344ab2c54a9fd135fde8842986ad8ac27ca1f2ace7e593ab39d4aaa90c16c054"} err="failed to get container status \"344ab2c54a9fd135fde8842986ad8ac27ca1f2ace7e593ab39d4aaa90c16c054\": rpc error: code = NotFound desc = could not find container \"344ab2c54a9fd135fde8842986ad8ac27ca1f2ace7e593ab39d4aaa90c16c054\": container with ID starting with 344ab2c54a9fd135fde8842986ad8ac27ca1f2ace7e593ab39d4aaa90c16c054 not found: ID does not exist"
Nov 01 15:43:03.604305 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:03.604234    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8fd363-c4bc9fcf8-pm6vm"]
Nov 01 15:43:03.649283 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:03.649244    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8fd363-c4bc9fcf8-pm6vm"]
Nov 01 15:43:04.795237 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:04.795203    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1896a74d-cf1e-48cc-8ab8-330e11570845" path="/var/lib/kubelet/pods/1896a74d-cf1e-48cc-8ab8-330e11570845/volumes"
Nov 01 15:43:07.768515 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:07.768473    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-52fffa-5cfddf5d94-6mlwc"]
Nov 01 15:43:07.768979 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:07.768721    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-52fffa-5cfddf5d94-6mlwc" podUID="2c28e8f1-bdac-437b-9c48-82171b8754f7" containerName="maas-api" containerID="cri-o://77b3d270038d4af4b48b2308c0866a495230ee71106247effbcc6cc8d4a458d1" gracePeriod=30
Nov 01 15:43:08.045143 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:08.045116    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-52fffa-5cfddf5d94-6mlwc"
Nov 01 15:43:08.138664 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:08.138631    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c28e8f1-bdac-437b-9c48-82171b8754f7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c28e8f1-bdac-437b-9c48-82171b8754f7-openshift-service-ca\") pod \"2c28e8f1-bdac-437b-9c48-82171b8754f7\" (UID: \"2c28e8f1-bdac-437b-9c48-82171b8754f7\") "
Nov 01 15:43:08.138878 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:08.138697    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c28e8f1-bdac-437b-9c48-82171b8754f7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c28e8f1-bdac-437b-9c48-82171b8754f7-metrics-certs\") pod \"2c28e8f1-bdac-437b-9c48-82171b8754f7\" (UID: \"2c28e8f1-bdac-437b-9c48-82171b8754f7\") "
Nov 01 15:43:08.138878 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:08.138803    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c28e8f1-bdac-437b-9c48-82171b8754f7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c28e8f1-bdac-437b-9c48-82171b8754f7-maas-api-tls\") pod \"2c28e8f1-bdac-437b-9c48-82171b8754f7\" (UID: \"2c28e8f1-bdac-437b-9c48-82171b8754f7\") "
Nov 01 15:43:08.139052 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:08.139022    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c28e8f1-bdac-437b-9c48-82171b8754f7-openshift-service-ca" pod "2c28e8f1-bdac-437b-9c48-82171b8754f7" (UID: "2c28e8f1-bdac-437b-9c48-82171b8754f7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:43:08.139129 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:08.139116    4993 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c28e8f1-bdac-437b-9c48-82171b8754f7-openshift-service-ca\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:43:08.140956 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:08.140927    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c28e8f1-bdac-437b-9c48-82171b8754f7-maas-api-tls" pod "2c28e8f1-bdac-437b-9c48-82171b8754f7" (UID: "2c28e8f1-bdac-437b-9c48-82171b8754f7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:43:08.141085 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:08.141021    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c28e8f1-bdac-437b-9c48-82171b8754f7-metrics-certs" pod "2c28e8f1-bdac-437b-9c48-82171b8754f7" (UID: "2c28e8f1-bdac-437b-9c48-82171b8754f7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:43:08.239789 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:08.239749    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c28e8f1-bdac-437b-9c48-82171b8754f7-kube-api-access-jr4dl\" (UniqueName: \"kubernetes.io/projected/2c28e8f1-bdac-437b-9c48-82171b8754f7-kube-api-access-jr4dl\") pod \"2c28e8f1-bdac-437b-9c48-82171b8754f7\" (UID: \"2c28e8f1-bdac-437b-9c48-82171b8754f7\") "
Nov 01 15:43:08.240037 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:08.240019    4993 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c28e8f1-bdac-437b-9c48-82171b8754f7-metrics-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:43:08.240112 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:08.240042    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c28e8f1-bdac-437b-9c48-82171b8754f7-maas-api-tls\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:43:08.241800 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:08.241777    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c28e8f1-bdac-437b-9c48-82171b8754f7-kube-api-access-jr4dl" pod "2c28e8f1-bdac-437b-9c48-82171b8754f7" (UID: "2c28e8f1-bdac-437b-9c48-82171b8754f7"). InnerVolumeSpecName "kube-api-access-jr4dl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:43:08.340549 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:08.340462    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jr4dl\" (UniqueName: \"kubernetes.io/projected/2c28e8f1-bdac-437b-9c48-82171b8754f7-kube-api-access-jr4dl\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:43:08.485969 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:08.485928    4993 generic.go:356] "Generic (PLEG): container finished" podID="2c28e8f1-bdac-437b-9c48-82171b8754f7" containerID="77b3d270038d4af4b48b2308c0866a495230ee71106247effbcc6cc8d4a458d1" exitCode=0
Nov 01 15:43:08.486129 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:08.485973    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-52fffa-5cfddf5d94-6mlwc"
Nov 01 15:43:08.486129 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:08.486021    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-52fffa-5cfddf5d94-6mlwc" event={"ID":"2c28e8f1-bdac-437b-9c48-82171b8754f7","Type":"ContainerDied","Data":"77b3d270038d4af4b48b2308c0866a495230ee71106247effbcc6cc8d4a458d1"}
Nov 01 15:43:08.486129 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:08.486088    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-52fffa-5cfddf5d94-6mlwc" event={"ID":"2c28e8f1-bdac-437b-9c48-82171b8754f7","Type":"ContainerDied","Data":"9fc89839f54aec63c2e39a043146c614bb324e0e57fc3ed821b36f030e197149"}
Nov 01 15:43:08.486129 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:08.486119    4993 scope.go:122] "RemoveContainer" containerID="77b3d270038d4af4b48b2308c0866a495230ee71106247effbcc6cc8d4a458d1"
Nov 01 15:43:08.496876 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:08.496860    4993 scope.go:122] "RemoveContainer" containerID="77b3d270038d4af4b48b2308c0866a495230ee71106247effbcc6cc8d4a458d1"
Nov 01 15:43:08.497214 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:43:08.497195    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77b3d270038d4af4b48b2308c0866a495230ee71106247effbcc6cc8d4a458d1\": container with ID starting with 77b3d270038d4af4b48b2308c0866a495230ee71106247effbcc6cc8d4a458d1 not found: ID does not exist" containerID="77b3d270038d4af4b48b2308c0866a495230ee71106247effbcc6cc8d4a458d1"
Nov 01 15:43:08.497293 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:08.497225    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77b3d270038d4af4b48b2308c0866a495230ee71106247effbcc6cc8d4a458d1"} err="failed to get container status \"77b3d270038d4af4b48b2308c0866a495230ee71106247effbcc6cc8d4a458d1\": rpc error: code = NotFound desc = could not find container \"77b3d270038d4af4b48b2308c0866a495230ee71106247effbcc6cc8d4a458d1\": container with ID starting with 77b3d270038d4af4b48b2308c0866a495230ee71106247effbcc6cc8d4a458d1 not found: ID does not exist"
Nov 01 15:43:08.638619 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:08.638525    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-52fffa-5cfddf5d94-6mlwc"]
Nov 01 15:43:08.684407 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:08.684334    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-52fffa-5cfddf5d94-6mlwc"]
Nov 01 15:43:08.793870 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:08.793835    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c28e8f1-bdac-437b-9c48-82171b8754f7" path="/var/lib/kubelet/pods/2c28e8f1-bdac-437b-9c48-82171b8754f7/volumes"
Nov 01 15:43:16.463301 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:16.463268    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-5b0dcc/distinct-w3-5b0dcc-kserve-57cb5cb5cf-msjth"]
Nov 01 15:43:24.410349 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:24.410307    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-00a787/distinct-2-w1-00a787-kserve-56f5f9dc7d-bhxxr"]
Nov 01 15:43:24.456800 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:24.456759    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-00a787/distinct-w1-00a787-kserve-dbd65bf4f-lbcwn"]
Nov 01 15:43:24.458960 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:24.458935    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-00a787/simulator-w1-00a787-kserve-5bdcf4b567-4xsn7"]
Nov 01 15:43:27.500395 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:43:27.500357    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:43:31.485213 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:31.485131    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-w3-5b0dcc-kserve-57cb5cb5cf-msjth" podUID="e3ac5585-dc23-445d-89fc-b68f90dac95a" containerName="main" containerID="cri-o://08ec96cbfd6ef12925ec063c35ae354ff867b41b162fc4c4fb4166a67a6be41f" gracePeriod=45
Nov 01 15:43:31.615986 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:31.615949    4993 generic.go:356] "Generic (PLEG): container finished" podID="e3ac5585-dc23-445d-89fc-b68f90dac95a" containerID="08ec96cbfd6ef12925ec063c35ae354ff867b41b162fc4c4fb4166a67a6be41f" exitCode=0
Nov 01 15:43:31.616157 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:31.616006    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-w3-5b0dcc-kserve-57cb5cb5cf-msjth" event={"ID":"e3ac5585-dc23-445d-89fc-b68f90dac95a","Type":"ContainerDied","Data":"08ec96cbfd6ef12925ec063c35ae354ff867b41b162fc4c4fb4166a67a6be41f"}
Nov 01 15:43:31.646215 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:31.646177    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-w3-5b0dcc-kserve-57cb5cb5cf-msjth" podUID="e3ac5585-dc23-445d-89fc-b68f90dac95a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": dial tcp 10.132.0.67:8000: connect: connection refused"
Nov 01 15:43:31.746968 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:31.746894    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-w3-5b0dcc-kserve-57cb5cb5cf-msjth"
Nov 01 15:43:31.857622 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:31.857586    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3ac5585-dc23-445d-89fc-b68f90dac95a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3ac5585-dc23-445d-89fc-b68f90dac95a-tmp-dir\") pod \"e3ac5585-dc23-445d-89fc-b68f90dac95a\" (UID: \"e3ac5585-dc23-445d-89fc-b68f90dac95a\") "
Nov 01 15:43:31.857818 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:31.857640    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3ac5585-dc23-445d-89fc-b68f90dac95a-tls-certs\" (UniqueName: \"kubernetes.io/secret/e3ac5585-dc23-445d-89fc-b68f90dac95a-tls-certs\") pod \"e3ac5585-dc23-445d-89fc-b68f90dac95a\" (UID: \"e3ac5585-dc23-445d-89fc-b68f90dac95a\") "
Nov 01 15:43:31.857818 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:31.857679    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3ac5585-dc23-445d-89fc-b68f90dac95a-home\" (UniqueName: \"kubernetes.io/empty-dir/e3ac5585-dc23-445d-89fc-b68f90dac95a-home\") pod \"e3ac5585-dc23-445d-89fc-b68f90dac95a\" (UID: \"e3ac5585-dc23-445d-89fc-b68f90dac95a\") "
Nov 01 15:43:31.857818 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:31.857711    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3ac5585-dc23-445d-89fc-b68f90dac95a-kube-api-access-4pb98\" (UniqueName: \"kubernetes.io/projected/e3ac5585-dc23-445d-89fc-b68f90dac95a-kube-api-access-4pb98\") pod \"e3ac5585-dc23-445d-89fc-b68f90dac95a\" (UID: \"e3ac5585-dc23-445d-89fc-b68f90dac95a\") "
Nov 01 15:43:31.857818 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:31.857750    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3ac5585-dc23-445d-89fc-b68f90dac95a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3ac5585-dc23-445d-89fc-b68f90dac95a-model-cache\") pod \"e3ac5585-dc23-445d-89fc-b68f90dac95a\" (UID: \"e3ac5585-dc23-445d-89fc-b68f90dac95a\") "
Nov 01 15:43:31.858007 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:31.857839    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3ac5585-dc23-445d-89fc-b68f90dac95a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3ac5585-dc23-445d-89fc-b68f90dac95a-dshm\") pod \"e3ac5585-dc23-445d-89fc-b68f90dac95a\" (UID: \"e3ac5585-dc23-445d-89fc-b68f90dac95a\") "
Nov 01 15:43:31.858374 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:31.858346    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3ac5585-dc23-445d-89fc-b68f90dac95a-home" pod "e3ac5585-dc23-445d-89fc-b68f90dac95a" (UID: "e3ac5585-dc23-445d-89fc-b68f90dac95a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:43:31.858525 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:31.858355    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3ac5585-dc23-445d-89fc-b68f90dac95a-model-cache" pod "e3ac5585-dc23-445d-89fc-b68f90dac95a" (UID: "e3ac5585-dc23-445d-89fc-b68f90dac95a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:43:31.858642 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:31.858499    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3ac5585-dc23-445d-89fc-b68f90dac95a-tmp-dir" pod "e3ac5585-dc23-445d-89fc-b68f90dac95a" (UID: "e3ac5585-dc23-445d-89fc-b68f90dac95a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:43:31.862873 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:31.862830    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3ac5585-dc23-445d-89fc-b68f90dac95a-dshm" pod "e3ac5585-dc23-445d-89fc-b68f90dac95a" (UID: "e3ac5585-dc23-445d-89fc-b68f90dac95a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:43:31.862873 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:31.862859    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3ac5585-dc23-445d-89fc-b68f90dac95a-tls-certs" pod "e3ac5585-dc23-445d-89fc-b68f90dac95a" (UID: "e3ac5585-dc23-445d-89fc-b68f90dac95a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:43:31.864678 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:31.864637    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3ac5585-dc23-445d-89fc-b68f90dac95a-kube-api-access-4pb98" pod "e3ac5585-dc23-445d-89fc-b68f90dac95a" (UID: "e3ac5585-dc23-445d-89fc-b68f90dac95a"). InnerVolumeSpecName "kube-api-access-4pb98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:43:31.958942 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:31.958903    4993 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3ac5585-dc23-445d-89fc-b68f90dac95a-dshm\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:43:31.958942 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:31.958943    4993 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3ac5585-dc23-445d-89fc-b68f90dac95a-tmp-dir\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:43:31.959155 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:31.958958    4993 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3ac5585-dc23-445d-89fc-b68f90dac95a-tls-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:43:31.959155 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:31.958973    4993 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3ac5585-dc23-445d-89fc-b68f90dac95a-home\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:43:31.959155 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:31.958988    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pb98\" (UniqueName: \"kubernetes.io/projected/e3ac5585-dc23-445d-89fc-b68f90dac95a-kube-api-access-4pb98\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:43:31.959155 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:31.959002    4993 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3ac5585-dc23-445d-89fc-b68f90dac95a-model-cache\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:43:32.623146 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:32.623113    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-w3-5b0dcc-kserve-57cb5cb5cf-msjth"
Nov 01 15:43:32.623608 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:32.623155    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5b0dcc/distinct-w3-5b0dcc-kserve-57cb5cb5cf-msjth" event={"ID":"e3ac5585-dc23-445d-89fc-b68f90dac95a","Type":"ContainerDied","Data":"b06defadfe07e00f98dba3f1dba806e84dc1f7baf682bced39bef9f598cf0001"}
Nov 01 15:43:32.623608 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:32.623221    4993 scope.go:122] "RemoveContainer" containerID="08ec96cbfd6ef12925ec063c35ae354ff867b41b162fc4c4fb4166a67a6be41f"
Nov 01 15:43:32.774100 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:32.774062    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-5b0dcc/distinct-w3-5b0dcc-kserve-57cb5cb5cf-msjth"]
Nov 01 15:43:32.778104 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:32.778079    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-5b0dcc/distinct-w3-5b0dcc-kserve-57cb5cb5cf-msjth"]
Nov 01 15:43:32.795577 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:32.795551    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3ac5585-dc23-445d-89fc-b68f90dac95a" path="/var/lib/kubelet/pods/e3ac5585-dc23-445d-89fc-b68f90dac95a/volumes"
Nov 01 15:43:35.040701 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.040632    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k"]
Nov 01 15:43:35.041261 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.041134    4993 cpu_manager.go:414] "containerMap: removing container" podUID="06108f6d-156c-47cc-b45f-3c421be59030" containerName="istio-proxy"
Nov 01 15:43:35.041261 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.041155    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06108f6d-156c-47cc-b45f-3c421be59030" containerName="istio-proxy"
Nov 01 15:43:35.041261 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.041186    4993 cpu_manager.go:414] "containerMap: removing container" podUID="0e34d9fc-4d6c-496b-95db-6ddd2b2f2c10" containerName="maas-api"
Nov 01 15:43:35.041261 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.041192    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e34d9fc-4d6c-496b-95db-6ddd2b2f2c10" containerName="maas-api"
Nov 01 15:43:35.041261 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.041203    4993 cpu_manager.go:414] "containerMap: removing container" podUID="bbc61e12-d326-4d31-be81-88fb641337c7" containerName="main"
Nov 01 15:43:35.041261 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.041211    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbc61e12-d326-4d31-be81-88fb641337c7" containerName="main"
Nov 01 15:43:35.041261 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.041221    4993 cpu_manager.go:414] "containerMap: removing container" podUID="39e0e8e2-87d8-4332-bb5d-86f3849861be" containerName="main"
Nov 01 15:43:35.041261 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.041227    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39e0e8e2-87d8-4332-bb5d-86f3849861be" containerName="main"
Nov 01 15:43:35.041261 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.041233    4993 cpu_manager.go:414] "containerMap: removing container" podUID="e2108b10-03af-450f-b750-e56bab9faff2" containerName="maas-api"
Nov 01 15:43:35.041261 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.041237    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2108b10-03af-450f-b750-e56bab9faff2" containerName="maas-api"
Nov 01 15:43:35.041261 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.041245    4993 cpu_manager.go:414] "containerMap: removing container" podUID="3cdf53f6-df5a-4cdc-b751-e26c71484700" containerName="main"
Nov 01 15:43:35.041261 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.041250    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cdf53f6-df5a-4cdc-b751-e26c71484700" containerName="main"
Nov 01 15:43:35.041261 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.041257    4993 cpu_manager.go:414] "containerMap: removing container" podUID="ea582656-480c-4e92-868d-c1f36225e433" containerName="maas-api"
Nov 01 15:43:35.041261 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.041262    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea582656-480c-4e92-868d-c1f36225e433" containerName="maas-api"
Nov 01 15:43:35.041261 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.041272    4993 cpu_manager.go:414] "containerMap: removing container" podUID="1896a74d-cf1e-48cc-8ab8-330e11570845" containerName="maas-api"
Nov 01 15:43:35.041830 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.041278    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1896a74d-cf1e-48cc-8ab8-330e11570845" containerName="maas-api"
Nov 01 15:43:35.041830 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.041287    4993 cpu_manager.go:414] "containerMap: removing container" podUID="4196a39a-6cde-4b40-b2b2-c4792aceb17e" containerName="istio-proxy"
Nov 01 15:43:35.041830 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.041291    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4196a39a-6cde-4b40-b2b2-c4792aceb17e" containerName="istio-proxy"
Nov 01 15:43:35.041830 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.041298    4993 cpu_manager.go:414] "containerMap: removing container" podUID="4808fe31-322b-4ae7-bf3b-f46024a38aa2" containerName="maas-api"
Nov 01 15:43:35.041830 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.041303    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4808fe31-322b-4ae7-bf3b-f46024a38aa2" containerName="maas-api"
Nov 01 15:43:35.041830 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.041311    4993 cpu_manager.go:414] "containerMap: removing container" podUID="8bbc081b-ce18-4d0b-81ec-2bbf576f63f6" containerName="maas-api"
Nov 01 15:43:35.041830 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.041315    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bbc081b-ce18-4d0b-81ec-2bbf576f63f6" containerName="maas-api"
Nov 01 15:43:35.041830 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.041331    4993 cpu_manager.go:414] "containerMap: removing container" podUID="56b01618-1df3-4ba6-8766-46fa85c27773" containerName="main"
Nov 01 15:43:35.041830 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.041336    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56b01618-1df3-4ba6-8766-46fa85c27773" containerName="main"
Nov 01 15:43:35.041830 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.041345    4993 cpu_manager.go:414] "containerMap: removing container" podUID="2ae81a81-e763-40ac-8f5e-4bd38f9a3e68" containerName="main"
Nov 01 15:43:35.041830 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.041349    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ae81a81-e763-40ac-8f5e-4bd38f9a3e68" containerName="main"
Nov 01 15:43:35.041830 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.041355    4993 cpu_manager.go:414] "containerMap: removing container" podUID="6ad8a332-6847-48a9-9249-415156dff36d" containerName="maas-api"
Nov 01 15:43:35.041830 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.041359    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ad8a332-6847-48a9-9249-415156dff36d" containerName="maas-api"
Nov 01 15:43:35.041830 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.041370    4993 cpu_manager.go:414] "containerMap: removing container" podUID="6f4a1651-eba0-4c6c-a15f-0192f3167fb9" containerName="main"
Nov 01 15:43:35.041830 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.041374    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f4a1651-eba0-4c6c-a15f-0192f3167fb9" containerName="main"
Nov 01 15:43:35.041830 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.041380    4993 cpu_manager.go:414] "containerMap: removing container" podUID="f43a7e63-3095-4ad7-935a-bb46a0af45b9" containerName="main"
Nov 01 15:43:35.041830 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.041385    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f43a7e63-3095-4ad7-935a-bb46a0af45b9" containerName="main"
Nov 01 15:43:35.041830 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.041392    4993 cpu_manager.go:414] "containerMap: removing container" podUID="ac70b22c-c696-4be4-84cc-593406d0cda6" containerName="maas-api"
Nov 01 15:43:35.041830 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.041396    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac70b22c-c696-4be4-84cc-593406d0cda6" containerName="maas-api"
Nov 01 15:43:35.041830 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.041404    4993 cpu_manager.go:414] "containerMap: removing container" podUID="8cebd6df-9dc4-47f7-a67d-9b1f21551d8b" containerName="maas-api"
Nov 01 15:43:35.041830 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.041408    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cebd6df-9dc4-47f7-a67d-9b1f21551d8b" containerName="maas-api"
Nov 01 15:43:35.041830 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.041417    4993 cpu_manager.go:414] "containerMap: removing container" podUID="8a720a76-488a-4a65-98ad-31ca84fa7842" containerName="main"
Nov 01 15:43:35.041830 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.041422    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a720a76-488a-4a65-98ad-31ca84fa7842" containerName="main"
Nov 01 15:43:35.041830 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.041430    4993 cpu_manager.go:414] "containerMap: removing container" podUID="e3ac5585-dc23-445d-89fc-b68f90dac95a" containerName="main"
Nov 01 15:43:35.041830 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.041434    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3ac5585-dc23-445d-89fc-b68f90dac95a" containerName="main"
Nov 01 15:43:35.041830 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.041443    4993 cpu_manager.go:414] "containerMap: removing container" podUID="7e3a0173-3e79-4e57-a9bb-0b9eb792da3a" containerName="payload-processing"
Nov 01 15:43:35.041830 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.041449    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e3a0173-3e79-4e57-a9bb-0b9eb792da3a" containerName="payload-processing"
Nov 01 15:43:35.041830 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.041465    4993 cpu_manager.go:414] "containerMap: removing container" podUID="04ff309b-01de-48c8-a36b-412e5f244b4e" containerName="maas-api"
Nov 01 15:43:35.041830 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.041470    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04ff309b-01de-48c8-a36b-412e5f244b4e" containerName="maas-api"
Nov 01 15:43:35.041830 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.041477    4993 cpu_manager.go:414] "containerMap: removing container" podUID="8d18c69f-6d32-40d7-b857-d0d686235013" containerName="istio-proxy"
Nov 01 15:43:35.041830 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.041480    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d18c69f-6d32-40d7-b857-d0d686235013" containerName="istio-proxy"
Nov 01 15:43:35.041830 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.041486    4993 cpu_manager.go:414] "containerMap: removing container" podUID="852aa9ab-4e1d-4798-9d0c-676d3ee14abe" containerName="maas-api"
Nov 01 15:43:35.041830 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.041490    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="852aa9ab-4e1d-4798-9d0c-676d3ee14abe" containerName="maas-api"
Nov 01 15:43:35.041830 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.041496    4993 cpu_manager.go:414] "containerMap: removing container" podUID="2c28e8f1-bdac-437b-9c48-82171b8754f7" containerName="maas-api"
Nov 01 15:43:35.041830 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.041501    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c28e8f1-bdac-437b-9c48-82171b8754f7" containerName="maas-api"
Nov 01 15:43:35.041830 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.041575    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="4808fe31-322b-4ae7-bf3b-f46024a38aa2" containerName="maas-api"
Nov 01 15:43:35.041830 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.041584    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bbc081b-ce18-4d0b-81ec-2bbf576f63f6" containerName="maas-api"
Nov 01 15:43:35.041830 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.041592    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="56b01618-1df3-4ba6-8766-46fa85c27773" containerName="main"
Nov 01 15:43:35.041830 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.041598    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ae81a81-e763-40ac-8f5e-4bd38f9a3e68" containerName="main"
Nov 01 15:43:35.041830 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.041603    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ad8a332-6847-48a9-9249-415156dff36d" containerName="maas-api"
Nov 01 15:43:35.041830 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.041609    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f4a1651-eba0-4c6c-a15f-0192f3167fb9" containerName="main"
Nov 01 15:43:35.041830 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.041614    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="f43a7e63-3095-4ad7-935a-bb46a0af45b9" containerName="main"
Nov 01 15:43:35.041830 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.041619    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac70b22c-c696-4be4-84cc-593406d0cda6" containerName="maas-api"
Nov 01 15:43:35.041830 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.041625    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cebd6df-9dc4-47f7-a67d-9b1f21551d8b" containerName="maas-api"
Nov 01 15:43:35.041830 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.041631    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a720a76-488a-4a65-98ad-31ca84fa7842" containerName="main"
Nov 01 15:43:35.041830 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.041637    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3ac5585-dc23-445d-89fc-b68f90dac95a" containerName="main"
Nov 01 15:43:35.041830 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.041643    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e3a0173-3e79-4e57-a9bb-0b9eb792da3a" containerName="payload-processing"
Nov 01 15:43:35.041830 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.041649    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="04ff309b-01de-48c8-a36b-412e5f244b4e" containerName="maas-api"
Nov 01 15:43:35.041830 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.041654    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d18c69f-6d32-40d7-b857-d0d686235013" containerName="istio-proxy"
Nov 01 15:43:35.041830 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.041660    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="852aa9ab-4e1d-4798-9d0c-676d3ee14abe" containerName="maas-api"
Nov 01 15:43:35.041830 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.041665    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c28e8f1-bdac-437b-9c48-82171b8754f7" containerName="maas-api"
Nov 01 15:43:35.041830 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.041670    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="06108f6d-156c-47cc-b45f-3c421be59030" containerName="istio-proxy"
Nov 01 15:43:35.041830 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.041676    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e34d9fc-4d6c-496b-95db-6ddd2b2f2c10" containerName="maas-api"
Nov 01 15:43:35.041830 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.041682    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbc61e12-d326-4d31-be81-88fb641337c7" containerName="main"
Nov 01 15:43:35.041830 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.041696    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="39e0e8e2-87d8-4332-bb5d-86f3849861be" containerName="main"
Nov 01 15:43:35.041830 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.041702    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2108b10-03af-450f-b750-e56bab9faff2" containerName="maas-api"
Nov 01 15:43:35.041830 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.041707    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cdf53f6-df5a-4cdc-b751-e26c71484700" containerName="main"
Nov 01 15:43:35.041830 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.041713    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea582656-480c-4e92-868d-c1f36225e433" containerName="maas-api"
Nov 01 15:43:35.041830 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.041718    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="1896a74d-cf1e-48cc-8ab8-330e11570845" containerName="maas-api"
Nov 01 15:43:35.041830 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.041723    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="4196a39a-6cde-4b40-b2b2-c4792aceb17e" containerName="istio-proxy"
Nov 01 15:43:35.046225 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.046206    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k"
Nov 01 15:43:35.092070 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.092034    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dc8aaf6-7922-409a-8aec-99bf7446a947-istiod-ca-cert\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k\" (UID: \"7dc8aaf6-7922-409a-8aec-99bf7446a947\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k"
Nov 01 15:43:35.092235 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.092146    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dc8aaf6-7922-409a-8aec-99bf7446a947-istio-data\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k\" (UID: \"7dc8aaf6-7922-409a-8aec-99bf7446a947\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k"
Nov 01 15:43:35.092235 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.092212    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dc8aaf6-7922-409a-8aec-99bf7446a947-istio-podinfo\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k\" (UID: \"7dc8aaf6-7922-409a-8aec-99bf7446a947\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k"
Nov 01 15:43:35.092364 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.092247    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4cl2\" (UniqueName: \"kubernetes.io/projected/7dc8aaf6-7922-409a-8aec-99bf7446a947-kube-api-access-v4cl2\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k\" (UID: \"7dc8aaf6-7922-409a-8aec-99bf7446a947\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k"
Nov 01 15:43:35.092425 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.092385    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dc8aaf6-7922-409a-8aec-99bf7446a947-istio-token\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k\" (UID: \"7dc8aaf6-7922-409a-8aec-99bf7446a947\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k"
Nov 01 15:43:35.092515 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.092483    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dc8aaf6-7922-409a-8aec-99bf7446a947-workload-socket\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k\" (UID: \"7dc8aaf6-7922-409a-8aec-99bf7446a947\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k"
Nov 01 15:43:35.092572 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.092539    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dc8aaf6-7922-409a-8aec-99bf7446a947-istio-envoy\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k\" (UID: \"7dc8aaf6-7922-409a-8aec-99bf7446a947\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k"
Nov 01 15:43:35.092572 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.092564    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dc8aaf6-7922-409a-8aec-99bf7446a947-credential-socket\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k\" (UID: \"7dc8aaf6-7922-409a-8aec-99bf7446a947\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k"
Nov 01 15:43:35.092674 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.092580    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dc8aaf6-7922-409a-8aec-99bf7446a947-workload-certs\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k\" (UID: \"7dc8aaf6-7922-409a-8aec-99bf7446a947\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k"
Nov 01 15:43:35.144658 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.144611    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k"]
Nov 01 15:43:35.188086 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.188059    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-5b0dcc-openshift-default-dockercfg-4d7r2\""
Nov 01 15:43:35.193533 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.193502    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dc8aaf6-7922-409a-8aec-99bf7446a947-istio-token\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k\" (UID: \"7dc8aaf6-7922-409a-8aec-99bf7446a947\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k"
Nov 01 15:43:35.193674 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.193559    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dc8aaf6-7922-409a-8aec-99bf7446a947-workload-socket\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k\" (UID: \"7dc8aaf6-7922-409a-8aec-99bf7446a947\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k"
Nov 01 15:43:35.193674 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.193589    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dc8aaf6-7922-409a-8aec-99bf7446a947-istio-envoy\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k\" (UID: \"7dc8aaf6-7922-409a-8aec-99bf7446a947\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k"
Nov 01 15:43:35.193674 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.193606    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dc8aaf6-7922-409a-8aec-99bf7446a947-credential-socket\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k\" (UID: \"7dc8aaf6-7922-409a-8aec-99bf7446a947\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k"
Nov 01 15:43:35.193674 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.193625    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dc8aaf6-7922-409a-8aec-99bf7446a947-workload-certs\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k\" (UID: \"7dc8aaf6-7922-409a-8aec-99bf7446a947\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k"
Nov 01 15:43:35.193674 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.193662    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dc8aaf6-7922-409a-8aec-99bf7446a947-istiod-ca-cert\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k\" (UID: \"7dc8aaf6-7922-409a-8aec-99bf7446a947\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k"
Nov 01 15:43:35.194011 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.193720    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dc8aaf6-7922-409a-8aec-99bf7446a947-istio-data\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k\" (UID: \"7dc8aaf6-7922-409a-8aec-99bf7446a947\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k"
Nov 01 15:43:35.194011 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.193757    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dc8aaf6-7922-409a-8aec-99bf7446a947-istio-podinfo\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k\" (UID: \"7dc8aaf6-7922-409a-8aec-99bf7446a947\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k"
Nov 01 15:43:35.194011 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.193790    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4cl2\" (UniqueName: \"kubernetes.io/projected/7dc8aaf6-7922-409a-8aec-99bf7446a947-kube-api-access-v4cl2\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k\" (UID: \"7dc8aaf6-7922-409a-8aec-99bf7446a947\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k"
Nov 01 15:43:35.194197 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.194145    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dc8aaf6-7922-409a-8aec-99bf7446a947-workload-socket\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k\" (UID: \"7dc8aaf6-7922-409a-8aec-99bf7446a947\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k"
Nov 01 15:43:35.194260 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.194206    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dc8aaf6-7922-409a-8aec-99bf7446a947-credential-socket\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k\" (UID: \"7dc8aaf6-7922-409a-8aec-99bf7446a947\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k"
Nov 01 15:43:35.194306 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.194288    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dc8aaf6-7922-409a-8aec-99bf7446a947-workload-certs\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k\" (UID: \"7dc8aaf6-7922-409a-8aec-99bf7446a947\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k"
Nov 01 15:43:35.194453 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.194434    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dc8aaf6-7922-409a-8aec-99bf7446a947-istio-data\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k\" (UID: \"7dc8aaf6-7922-409a-8aec-99bf7446a947\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k"
Nov 01 15:43:35.194748 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.194724    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dc8aaf6-7922-409a-8aec-99bf7446a947-istiod-ca-cert\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k\" (UID: \"7dc8aaf6-7922-409a-8aec-99bf7446a947\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k"
Nov 01 15:43:35.195987 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.195968    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dc8aaf6-7922-409a-8aec-99bf7446a947-istio-envoy\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k\" (UID: \"7dc8aaf6-7922-409a-8aec-99bf7446a947\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k"
Nov 01 15:43:35.196326 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.196304    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dc8aaf6-7922-409a-8aec-99bf7446a947-istio-podinfo\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k\" (UID: \"7dc8aaf6-7922-409a-8aec-99bf7446a947\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k"
Nov 01 15:43:35.246422 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.246393    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dc8aaf6-7922-409a-8aec-99bf7446a947-istio-token\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k\" (UID: \"7dc8aaf6-7922-409a-8aec-99bf7446a947\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k"
Nov 01 15:43:35.246565 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.246499    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4cl2\" (UniqueName: \"kubernetes.io/projected/7dc8aaf6-7922-409a-8aec-99bf7446a947-kube-api-access-v4cl2\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k\" (UID: \"7dc8aaf6-7922-409a-8aec-99bf7446a947\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k"
Nov 01 15:43:35.359737 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.359645    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k"
Nov 01 15:43:35.779392 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.779357    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k"]
Nov 01 15:43:35.785338 ip-10-0-134-37 kubenswrapper[4993]: W1101 15:43:35.785310    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dc8aaf6_7922_409a_8aec_99bf7446a947.slice/crio-07fad7a56b2e132efc4aa5266d6d436390d25398e779cf7edd9c33f8b8cbbaca WatchSource:0}: Error finding container 07fad7a56b2e132efc4aa5266d6d436390d25398e779cf7edd9c33f8b8cbbaca: Status 404 returned error can't find the container with id 07fad7a56b2e132efc4aa5266d6d436390d25398e779cf7edd9c33f8b8cbbaca
Nov 01 15:43:35.787673 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.787638    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 15:43:35.787762 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.787717    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 15:43:35.787762 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:35.787748    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 15:43:36.646780 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:36.646745    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k" event={"ID":"7dc8aaf6-7922-409a-8aec-99bf7446a947","Type":"ContainerStarted","Data":"4306f36776b72366d9c99ce2180ec093d2de807dfd24959705845f46a1a62ccb"}
Nov 01 15:43:36.647201 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:36.646795    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k" event={"ID":"7dc8aaf6-7922-409a-8aec-99bf7446a947","Type":"ContainerStarted","Data":"07fad7a56b2e132efc4aa5266d6d436390d25398e779cf7edd9c33f8b8cbbaca"}
Nov 01 15:43:37.363193 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:37.363068    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k"
Nov 01 15:43:37.363395 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:37.363229    4993 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body=
Nov 01 15:43:37.363395 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:37.363289    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k" podUID="7dc8aaf6-7922-409a-8aec-99bf7446a947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused"
Nov 01 15:43:38.365202 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:38.364560    4993 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body=
Nov 01 15:43:38.365202 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:38.364640    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k" podUID="7dc8aaf6-7922-409a-8aec-99bf7446a947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused"
Nov 01 15:43:39.362559 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:39.362517    4993 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body=
Nov 01 15:43:39.362849 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:39.362821    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k" podUID="7dc8aaf6-7922-409a-8aec-99bf7446a947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused"
Nov 01 15:43:39.430657 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:39.430617    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-00a787/distinct-2-w1-00a787-kserve-56f5f9dc7d-bhxxr" podUID="c0180369-b893-47c4-a683-ad9b6da82101" containerName="main" containerID="cri-o://97665b126c6408fa7d4fcd310e7991e451940b112dac9caa4c2a1763f6043a73" gracePeriod=45
Nov 01 15:43:39.479162 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:39.478986    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-00a787/simulator-w1-00a787-kserve-5bdcf4b567-4xsn7" podUID="6b8fc4b9-5b38-4d33-8f56-c8913e7e403f" containerName="main" containerID="cri-o://ae7fe7833bf1c036a266b82ea50175d7d93c5fccb7c95a7b935f77a5e853624d" gracePeriod=45
Nov 01 15:43:39.480492 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:39.480455    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-00a787/distinct-w1-00a787-kserve-dbd65bf4f-lbcwn" podUID="e1c97a57-0e89-4370-bc30-c75368d340f9" containerName="main" containerID="cri-o://581910a8db537325e9bf269cc934ccfaf9d759c6989acb3257fe69f698851061" gracePeriod=45
Nov 01 15:43:39.689347 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:39.689225    4993 generic.go:356] "Generic (PLEG): container finished" podID="6b8fc4b9-5b38-4d33-8f56-c8913e7e403f" containerID="ae7fe7833bf1c036a266b82ea50175d7d93c5fccb7c95a7b935f77a5e853624d" exitCode=0
Nov 01 15:43:39.689347 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:39.689286    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-00a787/simulator-w1-00a787-kserve-5bdcf4b567-4xsn7" event={"ID":"6b8fc4b9-5b38-4d33-8f56-c8913e7e403f","Type":"ContainerDied","Data":"ae7fe7833bf1c036a266b82ea50175d7d93c5fccb7c95a7b935f77a5e853624d"}
Nov 01 15:43:39.691151 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:39.691131    4993 generic.go:356] "Generic (PLEG): container finished" podID="e1c97a57-0e89-4370-bc30-c75368d340f9" containerID="581910a8db537325e9bf269cc934ccfaf9d759c6989acb3257fe69f698851061" exitCode=0
Nov 01 15:43:39.691309 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:39.691276    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-00a787/distinct-w1-00a787-kserve-dbd65bf4f-lbcwn" event={"ID":"e1c97a57-0e89-4370-bc30-c75368d340f9","Type":"ContainerDied","Data":"581910a8db537325e9bf269cc934ccfaf9d759c6989acb3257fe69f698851061"}
Nov 01 15:43:39.693120 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:39.693020    4993 generic.go:356] "Generic (PLEG): container finished" podID="c0180369-b893-47c4-a683-ad9b6da82101" containerID="97665b126c6408fa7d4fcd310e7991e451940b112dac9caa4c2a1763f6043a73" exitCode=0
Nov 01 15:43:39.693120 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:39.693086    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-00a787/distinct-2-w1-00a787-kserve-56f5f9dc7d-bhxxr" event={"ID":"c0180369-b893-47c4-a683-ad9b6da82101","Type":"ContainerDied","Data":"97665b126c6408fa7d4fcd310e7991e451940b112dac9caa4c2a1763f6043a73"}
Nov 01 15:43:39.817663 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:39.817389    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-00a787/distinct-2-w1-00a787-kserve-56f5f9dc7d-bhxxr"
Nov 01 15:43:39.840215 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:39.840190    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-00a787/distinct-w1-00a787-kserve-dbd65bf4f-lbcwn"
Nov 01 15:43:39.877367 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:39.850921    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-00a787/simulator-w1-00a787-kserve-5bdcf4b567-4xsn7"
Nov 01 15:43:39.938427 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:39.938312    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k" podStartSLOduration=4.938292378 podStartE2EDuration="4.938292378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:43:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:43:36.77624405 +0000 UTC m=+5204.106106806" watchObservedRunningTime="2026-11-01 15:43:39.938292378 +0000 UTC m=+5207.268155117"
Nov 01 15:43:39.948031 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:39.948000    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1c97a57-0e89-4370-bc30-c75368d340f9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1c97a57-0e89-4370-bc30-c75368d340f9-dshm\") pod \"e1c97a57-0e89-4370-bc30-c75368d340f9\" (UID: \"e1c97a57-0e89-4370-bc30-c75368d340f9\") "
Nov 01 15:43:39.948224 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:39.948059    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0180369-b893-47c4-a683-ad9b6da82101-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0180369-b893-47c4-a683-ad9b6da82101-dshm\") pod \"c0180369-b893-47c4-a683-ad9b6da82101\" (UID: \"c0180369-b893-47c4-a683-ad9b6da82101\") "
Nov 01 15:43:39.948224 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:39.948084    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0180369-b893-47c4-a683-ad9b6da82101-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0180369-b893-47c4-a683-ad9b6da82101-model-cache\") pod \"c0180369-b893-47c4-a683-ad9b6da82101\" (UID: \"c0180369-b893-47c4-a683-ad9b6da82101\") "
Nov 01 15:43:39.948224 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:39.948133    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b8fc4b9-5b38-4d33-8f56-c8913e7e403f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b8fc4b9-5b38-4d33-8f56-c8913e7e403f-model-cache\") pod \"6b8fc4b9-5b38-4d33-8f56-c8913e7e403f\" (UID: \"6b8fc4b9-5b38-4d33-8f56-c8913e7e403f\") "
Nov 01 15:43:39.948224 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:39.948192    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0180369-b893-47c4-a683-ad9b6da82101-home\" (UniqueName: \"kubernetes.io/empty-dir/c0180369-b893-47c4-a683-ad9b6da82101-home\") pod \"c0180369-b893-47c4-a683-ad9b6da82101\" (UID: \"c0180369-b893-47c4-a683-ad9b6da82101\") "
Nov 01 15:43:39.948224 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:39.948218    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0180369-b893-47c4-a683-ad9b6da82101-kube-api-access-7f5kx\" (UniqueName: \"kubernetes.io/projected/c0180369-b893-47c4-a683-ad9b6da82101-kube-api-access-7f5kx\") pod \"c0180369-b893-47c4-a683-ad9b6da82101\" (UID: \"c0180369-b893-47c4-a683-ad9b6da82101\") "
Nov 01 15:43:39.948508 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:39.948257    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1c97a57-0e89-4370-bc30-c75368d340f9-kube-api-access-6cv5n\" (UniqueName: \"kubernetes.io/projected/e1c97a57-0e89-4370-bc30-c75368d340f9-kube-api-access-6cv5n\") pod \"e1c97a57-0e89-4370-bc30-c75368d340f9\" (UID: \"e1c97a57-0e89-4370-bc30-c75368d340f9\") "
Nov 01 15:43:39.948508 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:39.948302    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1c97a57-0e89-4370-bc30-c75368d340f9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1c97a57-0e89-4370-bc30-c75368d340f9-tmp-dir\") pod \"e1c97a57-0e89-4370-bc30-c75368d340f9\" (UID: \"e1c97a57-0e89-4370-bc30-c75368d340f9\") "
Nov 01 15:43:39.948508 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:39.948339    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0180369-b893-47c4-a683-ad9b6da82101-tls-certs\" (UniqueName: \"kubernetes.io/secret/c0180369-b893-47c4-a683-ad9b6da82101-tls-certs\") pod \"c0180369-b893-47c4-a683-ad9b6da82101\" (UID: \"c0180369-b893-47c4-a683-ad9b6da82101\") "
Nov 01 15:43:39.948508 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:39.948383    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b8fc4b9-5b38-4d33-8f56-c8913e7e403f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b8fc4b9-5b38-4d33-8f56-c8913e7e403f-tmp-dir\") pod \"6b8fc4b9-5b38-4d33-8f56-c8913e7e403f\" (UID: \"6b8fc4b9-5b38-4d33-8f56-c8913e7e403f\") "
Nov 01 15:43:39.948508 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:39.948415    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b8fc4b9-5b38-4d33-8f56-c8913e7e403f-tls-certs\" (UniqueName: \"kubernetes.io/secret/6b8fc4b9-5b38-4d33-8f56-c8913e7e403f-tls-certs\") pod \"6b8fc4b9-5b38-4d33-8f56-c8913e7e403f\" (UID: \"6b8fc4b9-5b38-4d33-8f56-c8913e7e403f\") "
Nov 01 15:43:39.948508 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:39.948461    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1c97a57-0e89-4370-bc30-c75368d340f9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1c97a57-0e89-4370-bc30-c75368d340f9-model-cache\") pod \"e1c97a57-0e89-4370-bc30-c75368d340f9\" (UID: \"e1c97a57-0e89-4370-bc30-c75368d340f9\") "
Nov 01 15:43:39.948827 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:39.948542    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b8fc4b9-5b38-4d33-8f56-c8913e7e403f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b8fc4b9-5b38-4d33-8f56-c8913e7e403f-dshm\") pod \"6b8fc4b9-5b38-4d33-8f56-c8913e7e403f\" (UID: \"6b8fc4b9-5b38-4d33-8f56-c8913e7e403f\") "
Nov 01 15:43:39.948827 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:39.948571    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b8fc4b9-5b38-4d33-8f56-c8913e7e403f-kube-api-access-gnd8g\" (UniqueName: \"kubernetes.io/projected/6b8fc4b9-5b38-4d33-8f56-c8913e7e403f-kube-api-access-gnd8g\") pod \"6b8fc4b9-5b38-4d33-8f56-c8913e7e403f\" (UID: \"6b8fc4b9-5b38-4d33-8f56-c8913e7e403f\") "
Nov 01 15:43:39.948827 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:39.948603    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1c97a57-0e89-4370-bc30-c75368d340f9-tls-certs\" (UniqueName: \"kubernetes.io/secret/e1c97a57-0e89-4370-bc30-c75368d340f9-tls-certs\") pod \"e1c97a57-0e89-4370-bc30-c75368d340f9\" (UID: \"e1c97a57-0e89-4370-bc30-c75368d340f9\") "
Nov 01 15:43:39.948827 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:39.948629    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0180369-b893-47c4-a683-ad9b6da82101-home" pod "c0180369-b893-47c4-a683-ad9b6da82101" (UID: "c0180369-b893-47c4-a683-ad9b6da82101"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:43:39.948827 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:39.948639    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1c97a57-0e89-4370-bc30-c75368d340f9-home\" (UniqueName: \"kubernetes.io/empty-dir/e1c97a57-0e89-4370-bc30-c75368d340f9-home\") pod \"e1c97a57-0e89-4370-bc30-c75368d340f9\" (UID: \"e1c97a57-0e89-4370-bc30-c75368d340f9\") "
Nov 01 15:43:39.948827 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:39.948730    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0180369-b893-47c4-a683-ad9b6da82101-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0180369-b893-47c4-a683-ad9b6da82101-tmp-dir\") pod \"c0180369-b893-47c4-a683-ad9b6da82101\" (UID: \"c0180369-b893-47c4-a683-ad9b6da82101\") "
Nov 01 15:43:39.948827 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:39.948783    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b8fc4b9-5b38-4d33-8f56-c8913e7e403f-home\" (UniqueName: \"kubernetes.io/empty-dir/6b8fc4b9-5b38-4d33-8f56-c8913e7e403f-home\") pod \"6b8fc4b9-5b38-4d33-8f56-c8913e7e403f\" (UID: \"6b8fc4b9-5b38-4d33-8f56-c8913e7e403f\") "
Nov 01 15:43:39.949233 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:39.948851    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1c97a57-0e89-4370-bc30-c75368d340f9-home" pod "e1c97a57-0e89-4370-bc30-c75368d340f9" (UID: "e1c97a57-0e89-4370-bc30-c75368d340f9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:43:39.949233 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:39.949225    4993 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0180369-b893-47c4-a683-ad9b6da82101-home\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:43:39.949337 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:39.949240    4993 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1c97a57-0e89-4370-bc30-c75368d340f9-home\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:43:39.950098 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:39.949547    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b8fc4b9-5b38-4d33-8f56-c8913e7e403f-home" pod "6b8fc4b9-5b38-4d33-8f56-c8913e7e403f" (UID: "6b8fc4b9-5b38-4d33-8f56-c8913e7e403f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:43:39.950098 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:39.949619    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0180369-b893-47c4-a683-ad9b6da82101-model-cache" pod "c0180369-b893-47c4-a683-ad9b6da82101" (UID: "c0180369-b893-47c4-a683-ad9b6da82101"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:43:39.950098 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:39.949766    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b8fc4b9-5b38-4d33-8f56-c8913e7e403f-model-cache" pod "6b8fc4b9-5b38-4d33-8f56-c8913e7e403f" (UID: "6b8fc4b9-5b38-4d33-8f56-c8913e7e403f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:43:39.950098 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:39.949811    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0180369-b893-47c4-a683-ad9b6da82101-tmp-dir" pod "c0180369-b893-47c4-a683-ad9b6da82101" (UID: "c0180369-b893-47c4-a683-ad9b6da82101"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:43:39.950098 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:39.949839    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1c97a57-0e89-4370-bc30-c75368d340f9-tmp-dir" pod "e1c97a57-0e89-4370-bc30-c75368d340f9" (UID: "e1c97a57-0e89-4370-bc30-c75368d340f9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:43:39.951054 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:39.950469    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b8fc4b9-5b38-4d33-8f56-c8913e7e403f-tmp-dir" pod "6b8fc4b9-5b38-4d33-8f56-c8913e7e403f" (UID: "6b8fc4b9-5b38-4d33-8f56-c8913e7e403f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:43:39.951054 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:39.950672    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1c97a57-0e89-4370-bc30-c75368d340f9-model-cache" pod "e1c97a57-0e89-4370-bc30-c75368d340f9" (UID: "e1c97a57-0e89-4370-bc30-c75368d340f9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:43:39.963739 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:39.963692    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0180369-b893-47c4-a683-ad9b6da82101-kube-api-access-7f5kx" pod "c0180369-b893-47c4-a683-ad9b6da82101" (UID: "c0180369-b893-47c4-a683-ad9b6da82101"). InnerVolumeSpecName "kube-api-access-7f5kx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:43:39.978524 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:39.978471    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1c97a57-0e89-4370-bc30-c75368d340f9-tls-certs" pod "e1c97a57-0e89-4370-bc30-c75368d340f9" (UID: "e1c97a57-0e89-4370-bc30-c75368d340f9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:43:39.979052 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:39.978997    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1c97a57-0e89-4370-bc30-c75368d340f9-kube-api-access-6cv5n" pod "e1c97a57-0e89-4370-bc30-c75368d340f9" (UID: "e1c97a57-0e89-4370-bc30-c75368d340f9"). InnerVolumeSpecName "kube-api-access-6cv5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:43:39.979052 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:39.979013    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b8fc4b9-5b38-4d33-8f56-c8913e7e403f-kube-api-access-gnd8g" pod "6b8fc4b9-5b38-4d33-8f56-c8913e7e403f" (UID: "6b8fc4b9-5b38-4d33-8f56-c8913e7e403f"). InnerVolumeSpecName "kube-api-access-gnd8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:43:39.979286 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:39.979195    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0180369-b893-47c4-a683-ad9b6da82101-dshm" pod "c0180369-b893-47c4-a683-ad9b6da82101" (UID: "c0180369-b893-47c4-a683-ad9b6da82101"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:43:39.979496 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:39.979472    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1c97a57-0e89-4370-bc30-c75368d340f9-dshm" pod "e1c97a57-0e89-4370-bc30-c75368d340f9" (UID: "e1c97a57-0e89-4370-bc30-c75368d340f9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:43:39.981461 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:39.981434    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b8fc4b9-5b38-4d33-8f56-c8913e7e403f-tls-certs" pod "6b8fc4b9-5b38-4d33-8f56-c8913e7e403f" (UID: "6b8fc4b9-5b38-4d33-8f56-c8913e7e403f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:43:39.981546 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:39.981486    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0180369-b893-47c4-a683-ad9b6da82101-tls-certs" pod "c0180369-b893-47c4-a683-ad9b6da82101" (UID: "c0180369-b893-47c4-a683-ad9b6da82101"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:43:39.982931 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:39.982899    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b8fc4b9-5b38-4d33-8f56-c8913e7e403f-dshm" pod "6b8fc4b9-5b38-4d33-8f56-c8913e7e403f" (UID: "6b8fc4b9-5b38-4d33-8f56-c8913e7e403f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:43:40.050731 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:40.050699    4993 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1c97a57-0e89-4370-bc30-c75368d340f9-tls-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:43:40.050731 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:40.050730    4993 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0180369-b893-47c4-a683-ad9b6da82101-tmp-dir\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:43:40.050939 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:40.050740    4993 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b8fc4b9-5b38-4d33-8f56-c8913e7e403f-home\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:43:40.050939 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:40.050748    4993 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1c97a57-0e89-4370-bc30-c75368d340f9-dshm\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:43:40.050939 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:40.050756    4993 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0180369-b893-47c4-a683-ad9b6da82101-dshm\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:43:40.050939 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:40.050764    4993 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0180369-b893-47c4-a683-ad9b6da82101-model-cache\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:43:40.050939 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:40.050772    4993 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b8fc4b9-5b38-4d33-8f56-c8913e7e403f-model-cache\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:43:40.050939 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:40.050782    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7f5kx\" (UniqueName: \"kubernetes.io/projected/c0180369-b893-47c4-a683-ad9b6da82101-kube-api-access-7f5kx\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:43:40.050939 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:40.050791    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cv5n\" (UniqueName: \"kubernetes.io/projected/e1c97a57-0e89-4370-bc30-c75368d340f9-kube-api-access-6cv5n\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:43:40.050939 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:40.050804    4993 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1c97a57-0e89-4370-bc30-c75368d340f9-tmp-dir\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:43:40.050939 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:40.050814    4993 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0180369-b893-47c4-a683-ad9b6da82101-tls-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:43:40.050939 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:40.050822    4993 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b8fc4b9-5b38-4d33-8f56-c8913e7e403f-tmp-dir\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:43:40.050939 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:40.050832    4993 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b8fc4b9-5b38-4d33-8f56-c8913e7e403f-tls-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:43:40.050939 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:40.050839    4993 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1c97a57-0e89-4370-bc30-c75368d340f9-model-cache\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:43:40.050939 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:40.050847    4993 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b8fc4b9-5b38-4d33-8f56-c8913e7e403f-dshm\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:43:40.050939 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:40.050856    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnd8g\" (UniqueName: \"kubernetes.io/projected/6b8fc4b9-5b38-4d33-8f56-c8913e7e403f-kube-api-access-gnd8g\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:43:40.373193 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:40.371043    4993 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body=
Nov 01 15:43:40.373193 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:40.371104    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k" podUID="7dc8aaf6-7922-409a-8aec-99bf7446a947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused"
Nov 01 15:43:40.701824 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:40.701718    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-00a787/distinct-2-w1-00a787-kserve-56f5f9dc7d-bhxxr" event={"ID":"c0180369-b893-47c4-a683-ad9b6da82101","Type":"ContainerDied","Data":"ba76b3d8f4fe38d5ecd52eff5b9d5014fd38585ed71464754c9649c4703379b7"}
Nov 01 15:43:40.701824 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:40.701797    4993 scope.go:122] "RemoveContainer" containerID="97665b126c6408fa7d4fcd310e7991e451940b112dac9caa4c2a1763f6043a73"
Nov 01 15:43:40.702302 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:40.701936    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-00a787/distinct-2-w1-00a787-kserve-56f5f9dc7d-bhxxr"
Nov 01 15:43:40.707212 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:40.706633    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-00a787/simulator-w1-00a787-kserve-5bdcf4b567-4xsn7" event={"ID":"6b8fc4b9-5b38-4d33-8f56-c8913e7e403f","Type":"ContainerDied","Data":"8c168e65c7d637449c6b7c7899a61ad7d17b049ab08f03b442cbac9d1ccf6806"}
Nov 01 15:43:40.707212 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:40.706710    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-00a787/simulator-w1-00a787-kserve-5bdcf4b567-4xsn7"
Nov 01 15:43:40.710821 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:40.710797    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-00a787/distinct-w1-00a787-kserve-dbd65bf4f-lbcwn" event={"ID":"e1c97a57-0e89-4370-bc30-c75368d340f9","Type":"ContainerDied","Data":"8b5917d7895cdde55cc565fc0429bc9c0e67da83610f8388bb429f14048abdd9"}
Nov 01 15:43:40.710926 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:40.710862    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-00a787/distinct-w1-00a787-kserve-dbd65bf4f-lbcwn"
Nov 01 15:43:40.723248 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:40.714753    4993 scope.go:122] "RemoveContainer" containerID="ae7fe7833bf1c036a266b82ea50175d7d93c5fccb7c95a7b935f77a5e853624d"
Nov 01 15:43:40.732055 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:40.732025    4993 scope.go:122] "RemoveContainer" containerID="581910a8db537325e9bf269cc934ccfaf9d759c6989acb3257fe69f698851061"
Nov 01 15:43:40.877284 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:40.877248    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-00a787/simulator-w1-00a787-kserve-5bdcf4b567-4xsn7"]
Nov 01 15:43:40.884193 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:40.883572    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-00a787/simulator-w1-00a787-kserve-5bdcf4b567-4xsn7"]
Nov 01 15:43:40.983374 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:40.983237    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-00a787/distinct-w1-00a787-kserve-dbd65bf4f-lbcwn"]
Nov 01 15:43:40.985558 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:40.985484    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-00a787/distinct-w1-00a787-kserve-dbd65bf4f-lbcwn"]
Nov 01 15:43:41.078194 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:41.077222    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-00a787/distinct-2-w1-00a787-kserve-56f5f9dc7d-bhxxr"]
Nov 01 15:43:41.083632 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:41.083557    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-00a787/distinct-2-w1-00a787-kserve-56f5f9dc7d-bhxxr"]
Nov 01 15:43:41.361705 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:41.361610    4993 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body=
Nov 01 15:43:41.361999 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:41.361962    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k" podUID="7dc8aaf6-7922-409a-8aec-99bf7446a947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused"
Nov 01 15:43:42.360356 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:42.360314    4993 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body=
Nov 01 15:43:42.360767 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:42.360373    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k" podUID="7dc8aaf6-7922-409a-8aec-99bf7446a947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused"
Nov 01 15:43:42.688640 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:42.688546    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-00a787-65c44bcd97-thn6r"]
Nov 01 15:43:42.688826 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:42.688798    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-00a787-65c44bcd97-thn6r" podUID="0da21f81-aebc-40fd-9991-60e5631cf2d5" containerName="maas-api" containerID="cri-o://ec2dd19821ba88da2d66c7e7f473b6f1dc56b9dbd2b98f1514093b28aef08eb8" gracePeriod=30
Nov 01 15:43:42.796611 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:42.796572    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b8fc4b9-5b38-4d33-8f56-c8913e7e403f" path="/var/lib/kubelet/pods/6b8fc4b9-5b38-4d33-8f56-c8913e7e403f/volumes"
Nov 01 15:43:42.797250 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:42.797224    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0180369-b893-47c4-a683-ad9b6da82101" path="/var/lib/kubelet/pods/c0180369-b893-47c4-a683-ad9b6da82101/volumes"
Nov 01 15:43:42.797854 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:42.797788    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1c97a57-0e89-4370-bc30-c75368d340f9" path="/var/lib/kubelet/pods/e1c97a57-0e89-4370-bc30-c75368d340f9/volumes"
Nov 01 15:43:42.978140 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:42.978104    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-00a787-65c44bcd97-thn6r"
Nov 01 15:43:43.108832 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:43.108782    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0da21f81-aebc-40fd-9991-60e5631cf2d5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0da21f81-aebc-40fd-9991-60e5631cf2d5-metrics-certs\") pod \"0da21f81-aebc-40fd-9991-60e5631cf2d5\" (UID: \"0da21f81-aebc-40fd-9991-60e5631cf2d5\") "
Nov 01 15:43:43.109008 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:43.108884    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0da21f81-aebc-40fd-9991-60e5631cf2d5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0da21f81-aebc-40fd-9991-60e5631cf2d5-maas-api-tls\") pod \"0da21f81-aebc-40fd-9991-60e5631cf2d5\" (UID: \"0da21f81-aebc-40fd-9991-60e5631cf2d5\") "
Nov 01 15:43:43.109008 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:43.108986    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0da21f81-aebc-40fd-9991-60e5631cf2d5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0da21f81-aebc-40fd-9991-60e5631cf2d5-openshift-service-ca\") pod \"0da21f81-aebc-40fd-9991-60e5631cf2d5\" (UID: \"0da21f81-aebc-40fd-9991-60e5631cf2d5\") "
Nov 01 15:43:43.109162 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:43.109022    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0da21f81-aebc-40fd-9991-60e5631cf2d5-kube-api-access-h2pq5\" (UniqueName: \"kubernetes.io/projected/0da21f81-aebc-40fd-9991-60e5631cf2d5-kube-api-access-h2pq5\") pod \"0da21f81-aebc-40fd-9991-60e5631cf2d5\" (UID: \"0da21f81-aebc-40fd-9991-60e5631cf2d5\") "
Nov 01 15:43:43.109396 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:43.109366    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0da21f81-aebc-40fd-9991-60e5631cf2d5-openshift-service-ca" pod "0da21f81-aebc-40fd-9991-60e5631cf2d5" (UID: "0da21f81-aebc-40fd-9991-60e5631cf2d5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:43:43.111198 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:43.111143    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0da21f81-aebc-40fd-9991-60e5631cf2d5-kube-api-access-h2pq5" pod "0da21f81-aebc-40fd-9991-60e5631cf2d5" (UID: "0da21f81-aebc-40fd-9991-60e5631cf2d5"). InnerVolumeSpecName "kube-api-access-h2pq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:43:43.111304 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:43.111213    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0da21f81-aebc-40fd-9991-60e5631cf2d5-maas-api-tls" pod "0da21f81-aebc-40fd-9991-60e5631cf2d5" (UID: "0da21f81-aebc-40fd-9991-60e5631cf2d5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:43:43.111458 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:43.111433    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0da21f81-aebc-40fd-9991-60e5631cf2d5-metrics-certs" pod "0da21f81-aebc-40fd-9991-60e5631cf2d5" (UID: "0da21f81-aebc-40fd-9991-60e5631cf2d5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:43:43.210574 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:43.210473    4993 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0da21f81-aebc-40fd-9991-60e5631cf2d5-openshift-service-ca\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:43:43.210574 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:43.210514    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2pq5\" (UniqueName: \"kubernetes.io/projected/0da21f81-aebc-40fd-9991-60e5631cf2d5-kube-api-access-h2pq5\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:43:43.210574 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:43.210533    4993 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0da21f81-aebc-40fd-9991-60e5631cf2d5-metrics-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:43:43.210574 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:43.210547    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0da21f81-aebc-40fd-9991-60e5631cf2d5-maas-api-tls\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:43:43.732188 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:43.732135    4993 generic.go:356] "Generic (PLEG): container finished" podID="0da21f81-aebc-40fd-9991-60e5631cf2d5" containerID="ec2dd19821ba88da2d66c7e7f473b6f1dc56b9dbd2b98f1514093b28aef08eb8" exitCode=0
Nov 01 15:43:43.732656 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:43.732195    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-00a787-65c44bcd97-thn6r" event={"ID":"0da21f81-aebc-40fd-9991-60e5631cf2d5","Type":"ContainerDied","Data":"ec2dd19821ba88da2d66c7e7f473b6f1dc56b9dbd2b98f1514093b28aef08eb8"}
Nov 01 15:43:43.732656 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:43.732212    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-00a787-65c44bcd97-thn6r"
Nov 01 15:43:43.732656 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:43.732236    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-00a787-65c44bcd97-thn6r" event={"ID":"0da21f81-aebc-40fd-9991-60e5631cf2d5","Type":"ContainerDied","Data":"f101720edea263ec66474c0a4f6d88dd5626235f8916f0de545549d85a2834b7"}
Nov 01 15:43:43.732656 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:43.732257    4993 scope.go:122] "RemoveContainer" containerID="ec2dd19821ba88da2d66c7e7f473b6f1dc56b9dbd2b98f1514093b28aef08eb8"
Nov 01 15:43:43.743523 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:43.743491    4993 scope.go:122] "RemoveContainer" containerID="ec2dd19821ba88da2d66c7e7f473b6f1dc56b9dbd2b98f1514093b28aef08eb8"
Nov 01 15:43:43.743820 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:43:43.743792    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec2dd19821ba88da2d66c7e7f473b6f1dc56b9dbd2b98f1514093b28aef08eb8\": container with ID starting with ec2dd19821ba88da2d66c7e7f473b6f1dc56b9dbd2b98f1514093b28aef08eb8 not found: ID does not exist" containerID="ec2dd19821ba88da2d66c7e7f473b6f1dc56b9dbd2b98f1514093b28aef08eb8"
Nov 01 15:43:43.743905 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:43.743825    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec2dd19821ba88da2d66c7e7f473b6f1dc56b9dbd2b98f1514093b28aef08eb8"} err="failed to get container status \"ec2dd19821ba88da2d66c7e7f473b6f1dc56b9dbd2b98f1514093b28aef08eb8\": rpc error: code = NotFound desc = could not find container \"ec2dd19821ba88da2d66c7e7f473b6f1dc56b9dbd2b98f1514093b28aef08eb8\": container with ID starting with ec2dd19821ba88da2d66c7e7f473b6f1dc56b9dbd2b98f1514093b28aef08eb8 not found: ID does not exist"
Nov 01 15:43:43.886642 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:43.886588    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-00a787-65c44bcd97-thn6r"]
Nov 01 15:43:43.930985 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:43.930951    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-00a787-65c44bcd97-thn6r"]
Nov 01 15:43:44.248592 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:44.248561    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k"
Nov 01 15:43:44.249021 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:44.248991    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k"
Nov 01 15:43:44.250087 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:44.250062    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k"
Nov 01 15:43:44.793596 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:44.793566    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0da21f81-aebc-40fd-9991-60e5631cf2d5" path="/var/lib/kubelet/pods/0da21f81-aebc-40fd-9991-60e5631cf2d5/volumes"
Nov 01 15:43:56.283388 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:56.283355    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k"]
Nov 01 15:43:56.283875 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:56.283596    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k" podUID="7dc8aaf6-7922-409a-8aec-99bf7446a947" containerName="istio-proxy" containerID="cri-o://4306f36776b72366d9c99ce2180ec093d2de807dfd24959705845f46a1a62ccb" gracePeriod=30
Nov 01 15:43:57.501315 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:43:57.501274    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:43:59.251617 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:59.251575    4993 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 15:43:59.251994 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:43:59.251641    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k" podUID="7dc8aaf6-7922-409a-8aec-99bf7446a947" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 15:44:01.549400 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:44:01.549372    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k"
Nov 01 15:44:01.571527 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:44:01.571440    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dc8aaf6-7922-409a-8aec-99bf7446a947-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dc8aaf6-7922-409a-8aec-99bf7446a947-credential-socket\") pod \"7dc8aaf6-7922-409a-8aec-99bf7446a947\" (UID: \"7dc8aaf6-7922-409a-8aec-99bf7446a947\") "
Nov 01 15:44:01.571677 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:44:01.571545    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7dc8aaf6-7922-409a-8aec-99bf7446a947-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dc8aaf6-7922-409a-8aec-99bf7446a947-istiod-ca-cert\") pod \"7dc8aaf6-7922-409a-8aec-99bf7446a947\" (UID: \"7dc8aaf6-7922-409a-8aec-99bf7446a947\") "
Nov 01 15:44:01.571677 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:44:01.571579    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dc8aaf6-7922-409a-8aec-99bf7446a947-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dc8aaf6-7922-409a-8aec-99bf7446a947-istio-data\") pod \"7dc8aaf6-7922-409a-8aec-99bf7446a947\" (UID: \"7dc8aaf6-7922-409a-8aec-99bf7446a947\") "
Nov 01 15:44:01.571677 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:44:01.571617    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dc8aaf6-7922-409a-8aec-99bf7446a947-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dc8aaf6-7922-409a-8aec-99bf7446a947-istio-envoy\") pod \"7dc8aaf6-7922-409a-8aec-99bf7446a947\" (UID: \"7dc8aaf6-7922-409a-8aec-99bf7446a947\") "
Nov 01 15:44:01.571677 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:44:01.571650    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7dc8aaf6-7922-409a-8aec-99bf7446a947-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dc8aaf6-7922-409a-8aec-99bf7446a947-istio-podinfo\") pod \"7dc8aaf6-7922-409a-8aec-99bf7446a947\" (UID: \"7dc8aaf6-7922-409a-8aec-99bf7446a947\") "
Nov 01 15:44:01.571820 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:44:01.571678    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dc8aaf6-7922-409a-8aec-99bf7446a947-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dc8aaf6-7922-409a-8aec-99bf7446a947-workload-certs\") pod \"7dc8aaf6-7922-409a-8aec-99bf7446a947\" (UID: \"7dc8aaf6-7922-409a-8aec-99bf7446a947\") "
Nov 01 15:44:01.571820 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:44:01.571715    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dc8aaf6-7922-409a-8aec-99bf7446a947-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dc8aaf6-7922-409a-8aec-99bf7446a947-workload-socket\") pod \"7dc8aaf6-7922-409a-8aec-99bf7446a947\" (UID: \"7dc8aaf6-7922-409a-8aec-99bf7446a947\") "
Nov 01 15:44:01.571820 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:44:01.571755    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dc8aaf6-7922-409a-8aec-99bf7446a947-credential-socket" pod "7dc8aaf6-7922-409a-8aec-99bf7446a947" (UID: "7dc8aaf6-7922-409a-8aec-99bf7446a947"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:44:01.571820 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:44:01.571804    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dc8aaf6-7922-409a-8aec-99bf7446a947-istio-token\" (UniqueName: \"kubernetes.io/projected/7dc8aaf6-7922-409a-8aec-99bf7446a947-istio-token\") pod \"7dc8aaf6-7922-409a-8aec-99bf7446a947\" (UID: \"7dc8aaf6-7922-409a-8aec-99bf7446a947\") "
Nov 01 15:44:01.572021 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:44:01.571860    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dc8aaf6-7922-409a-8aec-99bf7446a947-kube-api-access-v4cl2\" (UniqueName: \"kubernetes.io/projected/7dc8aaf6-7922-409a-8aec-99bf7446a947-kube-api-access-v4cl2\") pod \"7dc8aaf6-7922-409a-8aec-99bf7446a947\" (UID: \"7dc8aaf6-7922-409a-8aec-99bf7446a947\") "
Nov 01 15:44:01.572021 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:44:01.571917    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dc8aaf6-7922-409a-8aec-99bf7446a947-istiod-ca-cert" pod "7dc8aaf6-7922-409a-8aec-99bf7446a947" (UID: "7dc8aaf6-7922-409a-8aec-99bf7446a947"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:44:01.572204 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:44:01.572141    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dc8aaf6-7922-409a-8aec-99bf7446a947-workload-socket" pod "7dc8aaf6-7922-409a-8aec-99bf7446a947" (UID: "7dc8aaf6-7922-409a-8aec-99bf7446a947"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:44:01.572204 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:44:01.572151    4993 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dc8aaf6-7922-409a-8aec-99bf7446a947-credential-socket\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:44:01.572339 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:44:01.572238    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dc8aaf6-7922-409a-8aec-99bf7446a947-istio-data" pod "7dc8aaf6-7922-409a-8aec-99bf7446a947" (UID: "7dc8aaf6-7922-409a-8aec-99bf7446a947"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:44:01.572339 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:44:01.572267    4993 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dc8aaf6-7922-409a-8aec-99bf7446a947-istiod-ca-cert\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:44:01.572339 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:44:01.572273    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dc8aaf6-7922-409a-8aec-99bf7446a947-workload-certs" pod "7dc8aaf6-7922-409a-8aec-99bf7446a947" (UID: "7dc8aaf6-7922-409a-8aec-99bf7446a947"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:44:01.574275 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:44:01.574252    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dc8aaf6-7922-409a-8aec-99bf7446a947-kube-api-access-v4cl2" pod "7dc8aaf6-7922-409a-8aec-99bf7446a947" (UID: "7dc8aaf6-7922-409a-8aec-99bf7446a947"). InnerVolumeSpecName "kube-api-access-v4cl2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:44:01.574531 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:44:01.574504    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7dc8aaf6-7922-409a-8aec-99bf7446a947-istio-podinfo" pod "7dc8aaf6-7922-409a-8aec-99bf7446a947" (UID: "7dc8aaf6-7922-409a-8aec-99bf7446a947"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 15:44:01.574640 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:44:01.574529    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dc8aaf6-7922-409a-8aec-99bf7446a947-istio-token" pod "7dc8aaf6-7922-409a-8aec-99bf7446a947" (UID: "7dc8aaf6-7922-409a-8aec-99bf7446a947"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:44:01.574640 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:44:01.574571    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dc8aaf6-7922-409a-8aec-99bf7446a947-istio-envoy" pod "7dc8aaf6-7922-409a-8aec-99bf7446a947" (UID: "7dc8aaf6-7922-409a-8aec-99bf7446a947"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:44:01.673756 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:44:01.673719    4993 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dc8aaf6-7922-409a-8aec-99bf7446a947-istio-token\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:44:01.673756 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:44:01.673753    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4cl2\" (UniqueName: \"kubernetes.io/projected/7dc8aaf6-7922-409a-8aec-99bf7446a947-kube-api-access-v4cl2\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:44:01.673978 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:44:01.673766    4993 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dc8aaf6-7922-409a-8aec-99bf7446a947-istio-data\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:44:01.673978 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:44:01.673779    4993 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dc8aaf6-7922-409a-8aec-99bf7446a947-istio-envoy\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:44:01.673978 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:44:01.673790    4993 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dc8aaf6-7922-409a-8aec-99bf7446a947-istio-podinfo\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:44:01.673978 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:44:01.673803    4993 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dc8aaf6-7922-409a-8aec-99bf7446a947-workload-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:44:01.673978 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:44:01.673815    4993 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dc8aaf6-7922-409a-8aec-99bf7446a947-workload-socket\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:44:01.824198 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:44:01.824029    4993 generic.go:356] "Generic (PLEG): container finished" podID="7dc8aaf6-7922-409a-8aec-99bf7446a947" containerID="4306f36776b72366d9c99ce2180ec093d2de807dfd24959705845f46a1a62ccb" exitCode=0
Nov 01 15:44:01.824198 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:44:01.824074    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k" event={"ID":"7dc8aaf6-7922-409a-8aec-99bf7446a947","Type":"ContainerDied","Data":"4306f36776b72366d9c99ce2180ec093d2de807dfd24959705845f46a1a62ccb"}
Nov 01 15:44:01.824198 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:44:01.824087    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k"
Nov 01 15:44:01.824198 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:44:01.824143    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k" event={"ID":"7dc8aaf6-7922-409a-8aec-99bf7446a947","Type":"ContainerDied","Data":"07fad7a56b2e132efc4aa5266d6d436390d25398e779cf7edd9c33f8b8cbbaca"}
Nov 01 15:44:01.824198 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:44:01.824190    4993 scope.go:122] "RemoveContainer" containerID="4306f36776b72366d9c99ce2180ec093d2de807dfd24959705845f46a1a62ccb"
Nov 01 15:44:01.834259 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:44:01.834238    4993 scope.go:122] "RemoveContainer" containerID="4306f36776b72366d9c99ce2180ec093d2de807dfd24959705845f46a1a62ccb"
Nov 01 15:44:01.834521 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:44:01.834500    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4306f36776b72366d9c99ce2180ec093d2de807dfd24959705845f46a1a62ccb\": container with ID starting with 4306f36776b72366d9c99ce2180ec093d2de807dfd24959705845f46a1a62ccb not found: ID does not exist" containerID="4306f36776b72366d9c99ce2180ec093d2de807dfd24959705845f46a1a62ccb"
Nov 01 15:44:01.834574 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:44:01.834530    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4306f36776b72366d9c99ce2180ec093d2de807dfd24959705845f46a1a62ccb"} err="failed to get container status \"4306f36776b72366d9c99ce2180ec093d2de807dfd24959705845f46a1a62ccb\": rpc error: code = NotFound desc = could not find container \"4306f36776b72366d9c99ce2180ec093d2de807dfd24959705845f46a1a62ccb\": container with ID starting with 4306f36776b72366d9c99ce2180ec093d2de807dfd24959705845f46a1a62ccb not found: ID does not exist"
Nov 01 15:44:01.977349 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:44:01.977310    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k"]
Nov 01 15:44:01.983066 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:44:01.983025    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fc29k"]
Nov 01 15:44:02.796209 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:44:02.796148    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7dc8aaf6-7922-409a-8aec-99bf7446a947" path="/var/lib/kubelet/pods/7dc8aaf6-7922-409a-8aec-99bf7446a947/volumes"
Nov 01 15:44:05.327384 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:44:05.327351    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-8dvhs_9c1bd12c-d218-4715-9239-1c18ce556fe9/manager/0.log"
Nov 01 15:44:05.662230 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:44:05.662131    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-d8qkk_e5667cdf-da6e-4930-93c0-f6dfc6910d45/manager/0.log"
Nov 01 15:44:06.269493 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:44:06.269457    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-cbwwd_c535f9df-00b8-4b69-91b7-e38837ab6802/manager/0.log"
Nov 01 15:44:08.400695 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:44:08.400659    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-hg8lm_4a9c6ca9-d1f4-41b1-9496-2e32c99edfce/manager/0.log"
Nov 01 15:44:08.697927 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:44:08.697843    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-2chcp_31f198c9-adac-4cd9-8cd9-2551a89aad9c/manager/0.log"
Nov 01 15:44:09.030693 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:44:09.030657    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-5csbz_93dc6178-9868-4299-bad6-0ee49fc5d2ad/manager/0.log"
Nov 01 15:44:09.988563 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:44:09.988534    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-lr4hb_6c32c11d-d77f-458d-9cf0-fe262d31aeeb/server/0.log"
Nov 01 15:44:10.481449 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:44:10.481370    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-cmbwq_6489d076-e1bd-402d-9d85-4e3bce4123b0/proxy/0.log"
Nov 01 15:44:27.501281 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:44:27.501238    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:44:57.500093 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:44:57.500056    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:45:00.154006 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:45:00.153969    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892465-zb7bf"]
Nov 01 15:45:00.156373 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:45:00.154404    4993 cpu_manager.go:414] "containerMap: removing container" podUID="6b8fc4b9-5b38-4d33-8f56-c8913e7e403f" containerName="main"
Nov 01 15:45:00.156373 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:45:00.154417    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b8fc4b9-5b38-4d33-8f56-c8913e7e403f" containerName="main"
Nov 01 15:45:00.156373 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:45:00.154435    4993 cpu_manager.go:414] "containerMap: removing container" podUID="e1c97a57-0e89-4370-bc30-c75368d340f9" containerName="main"
Nov 01 15:45:00.156373 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:45:00.154441    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1c97a57-0e89-4370-bc30-c75368d340f9" containerName="main"
Nov 01 15:45:00.156373 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:45:00.154454    4993 cpu_manager.go:414] "containerMap: removing container" podUID="7dc8aaf6-7922-409a-8aec-99bf7446a947" containerName="istio-proxy"
Nov 01 15:45:00.156373 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:45:00.154459    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dc8aaf6-7922-409a-8aec-99bf7446a947" containerName="istio-proxy"
Nov 01 15:45:00.156373 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:45:00.154472    4993 cpu_manager.go:414] "containerMap: removing container" podUID="c0180369-b893-47c4-a683-ad9b6da82101" containerName="main"
Nov 01 15:45:00.156373 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:45:00.154476    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0180369-b893-47c4-a683-ad9b6da82101" containerName="main"
Nov 01 15:45:00.156373 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:45:00.154484    4993 cpu_manager.go:414] "containerMap: removing container" podUID="0da21f81-aebc-40fd-9991-60e5631cf2d5" containerName="maas-api"
Nov 01 15:45:00.156373 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:45:00.154488    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0da21f81-aebc-40fd-9991-60e5631cf2d5" containerName="maas-api"
Nov 01 15:45:00.156373 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:45:00.154541    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b8fc4b9-5b38-4d33-8f56-c8913e7e403f" containerName="main"
Nov 01 15:45:00.156373 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:45:00.154551    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1c97a57-0e89-4370-bc30-c75368d340f9" containerName="main"
Nov 01 15:45:00.156373 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:45:00.154558    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dc8aaf6-7922-409a-8aec-99bf7446a947" containerName="istio-proxy"
Nov 01 15:45:00.156373 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:45:00.154565    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0180369-b893-47c4-a683-ad9b6da82101" containerName="main"
Nov 01 15:45:00.156373 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:45:00.154571    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="0da21f81-aebc-40fd-9991-60e5631cf2d5" containerName="maas-api"
Nov 01 15:45:00.157405 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:45:00.157386    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892465-zb7bf"
Nov 01 15:45:00.208773 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:45:00.208740    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d6ds8\""
Nov 01 15:45:00.208773 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:45:00.208769    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 15:45:00.227158 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:45:00.227119    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wss22\" (UniqueName: \"kubernetes.io/projected/7fde97a9-43a6-40db-930f-3afa0d456e94-kube-api-access-wss22\") pod \"maas-api-key-cleanup-29892465-zb7bf\" (UID: \"7fde97a9-43a6-40db-930f-3afa0d456e94\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892465-zb7bf"
Nov 01 15:45:00.227346 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:45:00.227254    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fde97a9-43a6-40db-930f-3afa0d456e94-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892465-zb7bf\" (UID: \"7fde97a9-43a6-40db-930f-3afa0d456e94\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892465-zb7bf"
Nov 01 15:45:00.254931 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:45:00.254900    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 15:45:00.257142 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:45:00.257114    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892465-zb7bf"]
Nov 01 15:45:00.327968 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:45:00.327932    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fde97a9-43a6-40db-930f-3afa0d456e94-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892465-zb7bf\" (UID: \"7fde97a9-43a6-40db-930f-3afa0d456e94\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892465-zb7bf"
Nov 01 15:45:00.328201 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:45:00.328098    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wss22\" (UniqueName: \"kubernetes.io/projected/7fde97a9-43a6-40db-930f-3afa0d456e94-kube-api-access-wss22\") pod \"maas-api-key-cleanup-29892465-zb7bf\" (UID: \"7fde97a9-43a6-40db-930f-3afa0d456e94\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892465-zb7bf"
Nov 01 15:45:00.328655 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:45:00.328629    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fde97a9-43a6-40db-930f-3afa0d456e94-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892465-zb7bf\" (UID: \"7fde97a9-43a6-40db-930f-3afa0d456e94\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892465-zb7bf"
Nov 01 15:45:00.381072 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:45:00.381041    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wss22\" (UniqueName: \"kubernetes.io/projected/7fde97a9-43a6-40db-930f-3afa0d456e94-kube-api-access-wss22\") pod \"maas-api-key-cleanup-29892465-zb7bf\" (UID: \"7fde97a9-43a6-40db-930f-3afa0d456e94\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892465-zb7bf"
Nov 01 15:45:00.470213 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:45:00.470101    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892465-zb7bf"
Nov 01 15:45:00.835179 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:45:00.835133    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892465-zb7bf"]
Nov 01 15:45:00.841257 ip-10-0-134-37 kubenswrapper[4993]: W1101 15:45:00.841228    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fde97a9_43a6_40db_930f_3afa0d456e94.slice/crio-e41fa81093bf2573e45ba3c252a216b7cb8b79a1c8c3f01776fb0ff1fa41582c WatchSource:0}: Error finding container e41fa81093bf2573e45ba3c252a216b7cb8b79a1c8c3f01776fb0ff1fa41582c: Status 404 returned error can't find the container with id e41fa81093bf2573e45ba3c252a216b7cb8b79a1c8c3f01776fb0ff1fa41582c
Nov 01 15:45:01.126768 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:45:01.126684    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892465-zb7bf" event={"ID":"7fde97a9-43a6-40db-930f-3afa0d456e94","Type":"ContainerStarted","Data":"e41fa81093bf2573e45ba3c252a216b7cb8b79a1c8c3f01776fb0ff1fa41582c"}
Nov 01 15:45:02.133366 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:45:02.133328    4993 generic.go:356] "Generic (PLEG): container finished" podID="7fde97a9-43a6-40db-930f-3afa0d456e94" containerID="e4a877d1aebdafb09ba162830f1e1389b9f4d8ec203e76395804757c9cb5fb3e" exitCode=60
Nov 01 15:45:02.133786 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:45:02.133420    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892465-zb7bf" event={"ID":"7fde97a9-43a6-40db-930f-3afa0d456e94","Type":"ContainerDied","Data":"e4a877d1aebdafb09ba162830f1e1389b9f4d8ec203e76395804757c9cb5fb3e"}
Nov 01 15:45:02.133786 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:45:02.133610    4993 scope.go:122] "RemoveContainer" containerID="e4a877d1aebdafb09ba162830f1e1389b9f4d8ec203e76395804757c9cb5fb3e"
Nov 01 15:45:03.140185 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:45:03.140144    4993 generic.go:356] "Generic (PLEG): container finished" podID="7fde97a9-43a6-40db-930f-3afa0d456e94" containerID="128d811981b6f7767029dd1142496051229cdb4ab79f8560d2b19ba40d207fda" exitCode=60
Nov 01 15:45:03.140687 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:45:03.140211    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892465-zb7bf" event={"ID":"7fde97a9-43a6-40db-930f-3afa0d456e94","Type":"ContainerDied","Data":"128d811981b6f7767029dd1142496051229cdb4ab79f8560d2b19ba40d207fda"}
Nov 01 15:45:03.140687 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:45:03.140261    4993 scope.go:122] "RemoveContainer" containerID="e4a877d1aebdafb09ba162830f1e1389b9f4d8ec203e76395804757c9cb5fb3e"
Nov 01 15:45:03.140687 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:45:03.140550    4993 scope.go:122] "RemoveContainer" containerID="128d811981b6f7767029dd1142496051229cdb4ab79f8560d2b19ba40d207fda"
Nov 01 15:45:03.140851 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:45:03.140827    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892465-zb7bf_odh-ai-gateway-infra(7fde97a9-43a6-40db-930f-3afa0d456e94)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892465-zb7bf" podUID="7fde97a9-43a6-40db-930f-3afa0d456e94"
Nov 01 15:45:04.146782 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:45:04.146753    4993 scope.go:122] "RemoveContainer" containerID="128d811981b6f7767029dd1142496051229cdb4ab79f8560d2b19ba40d207fda"
Nov 01 15:45:04.147210 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:45:04.146969    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892465-zb7bf_odh-ai-gateway-infra(7fde97a9-43a6-40db-930f-3afa0d456e94)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892465-zb7bf" podUID="7fde97a9-43a6-40db-930f-3afa0d456e94"
Nov 01 15:45:14.788295 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:45:14.788258    4993 scope.go:122] "RemoveContainer" containerID="128d811981b6f7767029dd1142496051229cdb4ab79f8560d2b19ba40d207fda"
Nov 01 15:45:15.201439 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:45:15.201408    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892465-zb7bf" event={"ID":"7fde97a9-43a6-40db-930f-3afa0d456e94","Type":"ContainerStarted","Data":"ca9f0b0d9e039b5c1e78d14afe2eaf30048fc34ee48f1c7f094666b07b971e2a"}
Nov 01 15:45:15.306655 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:45:15.306582    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892465-zb7bf" podStartSLOduration=15.306566342 podStartE2EDuration="15.306566342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:45:15.305952401 +0000 UTC m=+5302.635815139" watchObservedRunningTime="2026-11-01 15:45:15.306566342 +0000 UTC m=+5302.636429080"
Nov 01 15:45:16.208417 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:45:16.208382    4993 generic.go:356] "Generic (PLEG): container finished" podID="7fde97a9-43a6-40db-930f-3afa0d456e94" containerID="ca9f0b0d9e039b5c1e78d14afe2eaf30048fc34ee48f1c7f094666b07b971e2a" exitCode=60
Nov 01 15:45:16.208924 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:45:16.208450    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892465-zb7bf" event={"ID":"7fde97a9-43a6-40db-930f-3afa0d456e94","Type":"ContainerDied","Data":"ca9f0b0d9e039b5c1e78d14afe2eaf30048fc34ee48f1c7f094666b07b971e2a"}
Nov 01 15:45:16.208924 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:45:16.208496    4993 scope.go:122] "RemoveContainer" containerID="128d811981b6f7767029dd1142496051229cdb4ab79f8560d2b19ba40d207fda"
Nov 01 15:45:16.208924 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:45:16.208857    4993 scope.go:122] "RemoveContainer" containerID="ca9f0b0d9e039b5c1e78d14afe2eaf30048fc34ee48f1c7f094666b07b971e2a"
Nov 01 15:45:16.209162 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:45:16.209137    4993 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29892465-zb7bf_odh-ai-gateway-infra(7fde97a9-43a6-40db-930f-3afa0d456e94)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892465-zb7bf" podUID="7fde97a9-43a6-40db-930f-3afa0d456e94"
Nov 01 15:45:16.315443 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:45:16.315408    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892465-zb7bf"]
Nov 01 15:45:17.354519 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:45:17.354489    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892465-zb7bf"
Nov 01 15:45:17.492672 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:45:17.492634    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fde97a9-43a6-40db-930f-3afa0d456e94-kube-api-access-wss22\" (UniqueName: \"kubernetes.io/projected/7fde97a9-43a6-40db-930f-3afa0d456e94-kube-api-access-wss22\") pod \"7fde97a9-43a6-40db-930f-3afa0d456e94\" (UID: \"7fde97a9-43a6-40db-930f-3afa0d456e94\") "
Nov 01 15:45:17.492866 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:45:17.492681    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fde97a9-43a6-40db-930f-3afa0d456e94-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fde97a9-43a6-40db-930f-3afa0d456e94-maas-api-service-ca\") pod \"7fde97a9-43a6-40db-930f-3afa0d456e94\" (UID: \"7fde97a9-43a6-40db-930f-3afa0d456e94\") "
Nov 01 15:45:17.493211 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:45:17.493163    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fde97a9-43a6-40db-930f-3afa0d456e94-maas-api-service-ca" pod "7fde97a9-43a6-40db-930f-3afa0d456e94" (UID: "7fde97a9-43a6-40db-930f-3afa0d456e94"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:45:17.495108 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:45:17.495081    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fde97a9-43a6-40db-930f-3afa0d456e94-kube-api-access-wss22" pod "7fde97a9-43a6-40db-930f-3afa0d456e94" (UID: "7fde97a9-43a6-40db-930f-3afa0d456e94"). InnerVolumeSpecName "kube-api-access-wss22". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:45:17.593691 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:45:17.593654    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wss22\" (UniqueName: \"kubernetes.io/projected/7fde97a9-43a6-40db-930f-3afa0d456e94-kube-api-access-wss22\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:45:17.593691 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:45:17.593685    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fde97a9-43a6-40db-930f-3afa0d456e94-maas-api-service-ca\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:45:18.223153 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:45:18.223112    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892465-zb7bf" event={"ID":"7fde97a9-43a6-40db-930f-3afa0d456e94","Type":"ContainerDied","Data":"e41fa81093bf2573e45ba3c252a216b7cb8b79a1c8c3f01776fb0ff1fa41582c"}
Nov 01 15:45:18.223153 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:45:18.223138    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892465-zb7bf"
Nov 01 15:45:18.223457 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:45:18.223199    4993 scope.go:122] "RemoveContainer" containerID="ca9f0b0d9e039b5c1e78d14afe2eaf30048fc34ee48f1c7f094666b07b971e2a"
Nov 01 15:45:18.374064 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:45:18.374030    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892465-zb7bf"]
Nov 01 15:45:18.419886 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:45:18.419849    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892465-zb7bf"]
Nov 01 15:45:18.794844 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:45:18.794801    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fde97a9-43a6-40db-930f-3afa0d456e94" path="/var/lib/kubelet/pods/7fde97a9-43a6-40db-930f-3afa0d456e94/volumes"
Nov 01 15:45:27.499513 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:45:27.499476    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:45:57.502762 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:45:57.502703    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:46:00.117012 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:00.116971    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q9j4g"]
Nov 01 15:46:00.117478 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:00.117398    4993 cpu_manager.go:414] "containerMap: removing container" podUID="7fde97a9-43a6-40db-930f-3afa0d456e94" containerName="cleanup"
Nov 01 15:46:00.117478 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:00.117409    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fde97a9-43a6-40db-930f-3afa0d456e94" containerName="cleanup"
Nov 01 15:46:00.117478 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:00.117418    4993 cpu_manager.go:414] "containerMap: removing container" podUID="7fde97a9-43a6-40db-930f-3afa0d456e94" containerName="cleanup"
Nov 01 15:46:00.117478 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:00.117423    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fde97a9-43a6-40db-930f-3afa0d456e94" containerName="cleanup"
Nov 01 15:46:00.117636 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:00.117516    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fde97a9-43a6-40db-930f-3afa0d456e94" containerName="cleanup"
Nov 01 15:46:00.117636 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:00.117526    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fde97a9-43a6-40db-930f-3afa0d456e94" containerName="cleanup"
Nov 01 15:46:00.117636 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:00.117532    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fde97a9-43a6-40db-930f-3afa0d456e94" containerName="cleanup"
Nov 01 15:46:00.121892 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:00.121869    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q9j4g"
Nov 01 15:46:00.214313 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:00.214277    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q9j4g"]
Nov 01 15:46:00.276187 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:00.276130    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29c351ad-629b-418c-b64d-9ff0e52ccdcb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-q9j4g\" (UID: \"29c351ad-629b-418c-b64d-9ff0e52ccdcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q9j4g"
Nov 01 15:46:00.276369 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:00.276267    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29c351ad-629b-418c-b64d-9ff0e52ccdcb-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-q9j4g\" (UID: \"29c351ad-629b-418c-b64d-9ff0e52ccdcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q9j4g"
Nov 01 15:46:00.276369 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:00.276301    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29c351ad-629b-418c-b64d-9ff0e52ccdcb-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-q9j4g\" (UID: \"29c351ad-629b-418c-b64d-9ff0e52ccdcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q9j4g"
Nov 01 15:46:00.276369 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:00.276318    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29c351ad-629b-418c-b64d-9ff0e52ccdcb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-q9j4g\" (UID: \"29c351ad-629b-418c-b64d-9ff0e52ccdcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q9j4g"
Nov 01 15:46:00.276369 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:00.276336    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29c351ad-629b-418c-b64d-9ff0e52ccdcb-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-q9j4g\" (UID: \"29c351ad-629b-418c-b64d-9ff0e52ccdcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q9j4g"
Nov 01 15:46:00.276369 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:00.276358    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29c351ad-629b-418c-b64d-9ff0e52ccdcb-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-q9j4g\" (UID: \"29c351ad-629b-418c-b64d-9ff0e52ccdcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q9j4g"
Nov 01 15:46:00.276599 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:00.276417    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbnfr\" (UniqueName: \"kubernetes.io/projected/29c351ad-629b-418c-b64d-9ff0e52ccdcb-kube-api-access-qbnfr\") pod \"maas-default-gateway-openshift-default-7f5667f86c-q9j4g\" (UID: \"29c351ad-629b-418c-b64d-9ff0e52ccdcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q9j4g"
Nov 01 15:46:00.276599 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:00.276466    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29c351ad-629b-418c-b64d-9ff0e52ccdcb-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-q9j4g\" (UID: \"29c351ad-629b-418c-b64d-9ff0e52ccdcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q9j4g"
Nov 01 15:46:00.276599 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:00.276522    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29c351ad-629b-418c-b64d-9ff0e52ccdcb-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-q9j4g\" (UID: \"29c351ad-629b-418c-b64d-9ff0e52ccdcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q9j4g"
Nov 01 15:46:00.377525 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:00.377428    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29c351ad-629b-418c-b64d-9ff0e52ccdcb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-q9j4g\" (UID: \"29c351ad-629b-418c-b64d-9ff0e52ccdcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q9j4g"
Nov 01 15:46:00.377525 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:00.377498    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29c351ad-629b-418c-b64d-9ff0e52ccdcb-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-q9j4g\" (UID: \"29c351ad-629b-418c-b64d-9ff0e52ccdcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q9j4g"
Nov 01 15:46:00.377770 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:00.377529    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29c351ad-629b-418c-b64d-9ff0e52ccdcb-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-q9j4g\" (UID: \"29c351ad-629b-418c-b64d-9ff0e52ccdcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q9j4g"
Nov 01 15:46:00.377770 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:00.377553    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29c351ad-629b-418c-b64d-9ff0e52ccdcb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-q9j4g\" (UID: \"29c351ad-629b-418c-b64d-9ff0e52ccdcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q9j4g"
Nov 01 15:46:00.377770 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:00.377574    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29c351ad-629b-418c-b64d-9ff0e52ccdcb-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-q9j4g\" (UID: \"29c351ad-629b-418c-b64d-9ff0e52ccdcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q9j4g"
Nov 01 15:46:00.377770 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:00.377607    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29c351ad-629b-418c-b64d-9ff0e52ccdcb-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-q9j4g\" (UID: \"29c351ad-629b-418c-b64d-9ff0e52ccdcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q9j4g"
Nov 01 15:46:00.377770 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:00.377631    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbnfr\" (UniqueName: \"kubernetes.io/projected/29c351ad-629b-418c-b64d-9ff0e52ccdcb-kube-api-access-qbnfr\") pod \"maas-default-gateway-openshift-default-7f5667f86c-q9j4g\" (UID: \"29c351ad-629b-418c-b64d-9ff0e52ccdcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q9j4g"
Nov 01 15:46:00.377770 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:00.377669    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29c351ad-629b-418c-b64d-9ff0e52ccdcb-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-q9j4g\" (UID: \"29c351ad-629b-418c-b64d-9ff0e52ccdcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q9j4g"
Nov 01 15:46:00.377770 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:00.377696    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29c351ad-629b-418c-b64d-9ff0e52ccdcb-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-q9j4g\" (UID: \"29c351ad-629b-418c-b64d-9ff0e52ccdcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q9j4g"
Nov 01 15:46:00.378079 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:00.377992    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29c351ad-629b-418c-b64d-9ff0e52ccdcb-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-q9j4g\" (UID: \"29c351ad-629b-418c-b64d-9ff0e52ccdcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q9j4g"
Nov 01 15:46:00.378079 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:00.378064    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29c351ad-629b-418c-b64d-9ff0e52ccdcb-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-q9j4g\" (UID: \"29c351ad-629b-418c-b64d-9ff0e52ccdcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q9j4g"
Nov 01 15:46:00.378229 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:00.378094    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29c351ad-629b-418c-b64d-9ff0e52ccdcb-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-q9j4g\" (UID: \"29c351ad-629b-418c-b64d-9ff0e52ccdcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q9j4g"
Nov 01 15:46:00.378298 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:00.378221    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29c351ad-629b-418c-b64d-9ff0e52ccdcb-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-q9j4g\" (UID: \"29c351ad-629b-418c-b64d-9ff0e52ccdcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q9j4g"
Nov 01 15:46:00.378352 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:00.378330    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29c351ad-629b-418c-b64d-9ff0e52ccdcb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-q9j4g\" (UID: \"29c351ad-629b-418c-b64d-9ff0e52ccdcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q9j4g"
Nov 01 15:46:00.380105 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:00.380081    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29c351ad-629b-418c-b64d-9ff0e52ccdcb-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-q9j4g\" (UID: \"29c351ad-629b-418c-b64d-9ff0e52ccdcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q9j4g"
Nov 01 15:46:00.380201 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:00.380159    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29c351ad-629b-418c-b64d-9ff0e52ccdcb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-q9j4g\" (UID: \"29c351ad-629b-418c-b64d-9ff0e52ccdcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q9j4g"
Nov 01 15:46:00.431738 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:00.431695    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29c351ad-629b-418c-b64d-9ff0e52ccdcb-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-q9j4g\" (UID: \"29c351ad-629b-418c-b64d-9ff0e52ccdcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q9j4g"
Nov 01 15:46:00.431891 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:00.431843    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbnfr\" (UniqueName: \"kubernetes.io/projected/29c351ad-629b-418c-b64d-9ff0e52ccdcb-kube-api-access-qbnfr\") pod \"maas-default-gateway-openshift-default-7f5667f86c-q9j4g\" (UID: \"29c351ad-629b-418c-b64d-9ff0e52ccdcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q9j4g"
Nov 01 15:46:00.434187 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:00.434151    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q9j4g"
Nov 01 15:46:00.854013 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:00.853976    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q9j4g"]
Nov 01 15:46:00.898723 ip-10-0-134-37 kubenswrapper[4993]: W1101 15:46:00.898694    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29c351ad_629b_418c_b64d_9ff0e52ccdcb.slice/crio-bb220fe7419632268fc1719ee1975f0bbbed5794d4e9fefb9593d0510571a767 WatchSource:0}: Error finding container bb220fe7419632268fc1719ee1975f0bbbed5794d4e9fefb9593d0510571a767: Status 404 returned error can't find the container with id bb220fe7419632268fc1719ee1975f0bbbed5794d4e9fefb9593d0510571a767
Nov 01 15:46:00.900715 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:00.900685    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 15:46:00.900785 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:00.900756    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 15:46:00.900824 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:00.900786    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 15:46:01.441726 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:01.441685    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q9j4g" event={"ID":"29c351ad-629b-418c-b64d-9ff0e52ccdcb","Type":"ContainerStarted","Data":"73021efa8dcec9d51231cb76de1bc7e0ab7732e36e3dd31950a0a4799f1eb0b9"}
Nov 01 15:46:01.442133 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:01.441752    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q9j4g" event={"ID":"29c351ad-629b-418c-b64d-9ff0e52ccdcb","Type":"ContainerStarted","Data":"bb220fe7419632268fc1719ee1975f0bbbed5794d4e9fefb9593d0510571a767"}
Nov 01 15:46:01.576781 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:01.576699    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q9j4g" podStartSLOduration=1.5766784729999999 podStartE2EDuration="1.576678473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:46:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:46:01.550507521 +0000 UTC m=+5348.880370257" watchObservedRunningTime="2026-11-01 15:46:01.576678473 +0000 UTC m=+5348.906541210"
Nov 01 15:46:02.434871 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:02.434831    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q9j4g"
Nov 01 15:46:02.436444 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:02.436413    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-q9j4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Nov 01 15:46:02.436592 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:02.436471    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q9j4g" podUID="29c351ad-629b-418c-b64d-9ff0e52ccdcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Nov 01 15:46:03.435087 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:03.435054    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-q9j4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Nov 01 15:46:03.435502 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:03.435113    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q9j4g" podUID="29c351ad-629b-418c-b64d-9ff0e52ccdcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Nov 01 15:46:04.435505 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:04.435467    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-q9j4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Nov 01 15:46:04.435919 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:04.435519    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q9j4g" podUID="29c351ad-629b-418c-b64d-9ff0e52ccdcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Nov 01 15:46:05.435268 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:05.435222    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-q9j4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Nov 01 15:46:05.435482 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:05.435286    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q9j4g" podUID="29c351ad-629b-418c-b64d-9ff0e52ccdcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Nov 01 15:46:06.435347 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:06.435311    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-q9j4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Nov 01 15:46:06.435746 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:06.435367    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q9j4g" podUID="29c351ad-629b-418c-b64d-9ff0e52ccdcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Nov 01 15:46:07.434818 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:07.434777    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-q9j4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Nov 01 15:46:07.435007 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:07.434845    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q9j4g" podUID="29c351ad-629b-418c-b64d-9ff0e52ccdcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Nov 01 15:46:09.130648 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:09.130594    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q9j4g"
Nov 01 15:46:09.131147 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:09.131122    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q9j4g"
Nov 01 15:46:09.132002 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:09.131978    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q9j4g"
Nov 01 15:46:27.527079 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:46:27.527042    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:46:45.129442 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:45.129399    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-flzf5"]
Nov 01 15:46:45.129874 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:45.129787    4993 cpu_manager.go:414] "containerMap: removing container" podUID="7fde97a9-43a6-40db-930f-3afa0d456e94" containerName="cleanup"
Nov 01 15:46:45.129874 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:45.129798    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fde97a9-43a6-40db-930f-3afa0d456e94" containerName="cleanup"
Nov 01 15:46:45.137417 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:45.137385    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-flzf5"
Nov 01 15:46:45.284737 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:45.284693    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3d2334c-8e83-4a16-a126-cbe45120efa7-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-flzf5\" (UID: \"e3d2334c-8e83-4a16-a126-cbe45120efa7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-flzf5"
Nov 01 15:46:45.284927 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:45.284751    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3d2334c-8e83-4a16-a126-cbe45120efa7-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-flzf5\" (UID: \"e3d2334c-8e83-4a16-a126-cbe45120efa7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-flzf5"
Nov 01 15:46:45.284927 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:45.284813    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3d2334c-8e83-4a16-a126-cbe45120efa7-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-flzf5\" (UID: \"e3d2334c-8e83-4a16-a126-cbe45120efa7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-flzf5"
Nov 01 15:46:45.284927 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:45.284865    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9v9p8\" (UniqueName: \"kubernetes.io/projected/e3d2334c-8e83-4a16-a126-cbe45120efa7-kube-api-access-9v9p8\") pod \"maas-default-gateway-openshift-default-7f5667f86c-flzf5\" (UID: \"e3d2334c-8e83-4a16-a126-cbe45120efa7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-flzf5"
Nov 01 15:46:45.284927 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:45.284907    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3d2334c-8e83-4a16-a126-cbe45120efa7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-flzf5\" (UID: \"e3d2334c-8e83-4a16-a126-cbe45120efa7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-flzf5"
Nov 01 15:46:45.285188 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:45.284935    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3d2334c-8e83-4a16-a126-cbe45120efa7-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-flzf5\" (UID: \"e3d2334c-8e83-4a16-a126-cbe45120efa7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-flzf5"
Nov 01 15:46:45.285188 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:45.285039    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3d2334c-8e83-4a16-a126-cbe45120efa7-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-flzf5\" (UID: \"e3d2334c-8e83-4a16-a126-cbe45120efa7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-flzf5"
Nov 01 15:46:45.285188 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:45.285154    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3d2334c-8e83-4a16-a126-cbe45120efa7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-flzf5\" (UID: \"e3d2334c-8e83-4a16-a126-cbe45120efa7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-flzf5"
Nov 01 15:46:45.285367 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:45.285203    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3d2334c-8e83-4a16-a126-cbe45120efa7-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-flzf5\" (UID: \"e3d2334c-8e83-4a16-a126-cbe45120efa7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-flzf5"
Nov 01 15:46:45.321452 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:45.321417    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-flzf5"]
Nov 01 15:46:45.385833 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:45.385795    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3d2334c-8e83-4a16-a126-cbe45120efa7-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-flzf5\" (UID: \"e3d2334c-8e83-4a16-a126-cbe45120efa7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-flzf5"
Nov 01 15:46:45.385833 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:45.385836    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3d2334c-8e83-4a16-a126-cbe45120efa7-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-flzf5\" (UID: \"e3d2334c-8e83-4a16-a126-cbe45120efa7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-flzf5"
Nov 01 15:46:45.386103 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:45.385874    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9v9p8\" (UniqueName: \"kubernetes.io/projected/e3d2334c-8e83-4a16-a126-cbe45120efa7-kube-api-access-9v9p8\") pod \"maas-default-gateway-openshift-default-7f5667f86c-flzf5\" (UID: \"e3d2334c-8e83-4a16-a126-cbe45120efa7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-flzf5"
Nov 01 15:46:45.386103 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:45.385898    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3d2334c-8e83-4a16-a126-cbe45120efa7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-flzf5\" (UID: \"e3d2334c-8e83-4a16-a126-cbe45120efa7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-flzf5"
Nov 01 15:46:45.386103 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:45.385924    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3d2334c-8e83-4a16-a126-cbe45120efa7-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-flzf5\" (UID: \"e3d2334c-8e83-4a16-a126-cbe45120efa7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-flzf5"
Nov 01 15:46:45.386103 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:45.385978    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3d2334c-8e83-4a16-a126-cbe45120efa7-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-flzf5\" (UID: \"e3d2334c-8e83-4a16-a126-cbe45120efa7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-flzf5"
Nov 01 15:46:45.386103 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:45.386066    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3d2334c-8e83-4a16-a126-cbe45120efa7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-flzf5\" (UID: \"e3d2334c-8e83-4a16-a126-cbe45120efa7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-flzf5"
Nov 01 15:46:45.386103 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:45.386090    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3d2334c-8e83-4a16-a126-cbe45120efa7-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-flzf5\" (UID: \"e3d2334c-8e83-4a16-a126-cbe45120efa7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-flzf5"
Nov 01 15:46:45.386522 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:45.386145    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3d2334c-8e83-4a16-a126-cbe45120efa7-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-flzf5\" (UID: \"e3d2334c-8e83-4a16-a126-cbe45120efa7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-flzf5"
Nov 01 15:46:45.386587 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:45.386570    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3d2334c-8e83-4a16-a126-cbe45120efa7-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-flzf5\" (UID: \"e3d2334c-8e83-4a16-a126-cbe45120efa7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-flzf5"
Nov 01 15:46:45.386655 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:45.386582    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3d2334c-8e83-4a16-a126-cbe45120efa7-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-flzf5\" (UID: \"e3d2334c-8e83-4a16-a126-cbe45120efa7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-flzf5"
Nov 01 15:46:45.386889 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:45.386864    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3d2334c-8e83-4a16-a126-cbe45120efa7-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-flzf5\" (UID: \"e3d2334c-8e83-4a16-a126-cbe45120efa7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-flzf5"
Nov 01 15:46:45.387766 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:45.387740    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3d2334c-8e83-4a16-a126-cbe45120efa7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-flzf5\" (UID: \"e3d2334c-8e83-4a16-a126-cbe45120efa7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-flzf5"
Nov 01 15:46:45.387929 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:45.387793    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3d2334c-8e83-4a16-a126-cbe45120efa7-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-flzf5\" (UID: \"e3d2334c-8e83-4a16-a126-cbe45120efa7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-flzf5"
Nov 01 15:46:45.389453 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:45.389406    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3d2334c-8e83-4a16-a126-cbe45120efa7-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-flzf5\" (UID: \"e3d2334c-8e83-4a16-a126-cbe45120efa7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-flzf5"
Nov 01 15:46:45.390273 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:45.390248    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3d2334c-8e83-4a16-a126-cbe45120efa7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-flzf5\" (UID: \"e3d2334c-8e83-4a16-a126-cbe45120efa7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-flzf5"
Nov 01 15:46:45.438837 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:45.438762    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3d2334c-8e83-4a16-a126-cbe45120efa7-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-flzf5\" (UID: \"e3d2334c-8e83-4a16-a126-cbe45120efa7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-flzf5"
Nov 01 15:46:45.439696 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:45.439669    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9v9p8\" (UniqueName: \"kubernetes.io/projected/e3d2334c-8e83-4a16-a126-cbe45120efa7-kube-api-access-9v9p8\") pod \"maas-default-gateway-openshift-default-7f5667f86c-flzf5\" (UID: \"e3d2334c-8e83-4a16-a126-cbe45120efa7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-flzf5"
Nov 01 15:46:45.467928 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:45.467885    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-flzf5"
Nov 01 15:46:45.944321 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:45.944285    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-flzf5"]
Nov 01 15:46:45.951238 ip-10-0-134-37 kubenswrapper[4993]: W1101 15:46:45.951205    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3d2334c_8e83_4a16_a126_cbe45120efa7.slice/crio-1f699e063f0e18182dbf1fd19364007686d245af81fbac07924282ff1416bb06 WatchSource:0}: Error finding container 1f699e063f0e18182dbf1fd19364007686d245af81fbac07924282ff1416bb06: Status 404 returned error can't find the container with id 1f699e063f0e18182dbf1fd19364007686d245af81fbac07924282ff1416bb06
Nov 01 15:46:45.953852 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:45.953812    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 15:46:45.953943 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:45.953902    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 15:46:45.953987 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:45.953953    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 15:46:46.688827 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:46.688794    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-flzf5" event={"ID":"e3d2334c-8e83-4a16-a126-cbe45120efa7","Type":"ContainerStarted","Data":"e166aab2c0ffc6e3d26cfc3350434b816e10a4be39d345acb970f755191fe718"}
Nov 01 15:46:46.689287 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:46.688854    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-flzf5" event={"ID":"e3d2334c-8e83-4a16-a126-cbe45120efa7","Type":"ContainerStarted","Data":"1f699e063f0e18182dbf1fd19364007686d245af81fbac07924282ff1416bb06"}
Nov 01 15:46:46.845805 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:46.845750    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-flzf5" podStartSLOduration=1.845734517 podStartE2EDuration="1.845734517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:46:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:46:46.84334385 +0000 UTC m=+5394.173206588" watchObservedRunningTime="2026-11-01 15:46:46.845734517 +0000 UTC m=+5394.175597254"
Nov 01 15:46:47.473193 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:47.471528    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-flzf5"
Nov 01 15:46:47.476208 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:47.473954    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-flzf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Nov 01 15:46:47.476208 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:47.474013    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-flzf5" podUID="e3d2334c-8e83-4a16-a126-cbe45120efa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Nov 01 15:46:48.470191 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:48.469395    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-flzf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Nov 01 15:46:48.470191 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:48.469463    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-flzf5" podUID="e3d2334c-8e83-4a16-a126-cbe45120efa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Nov 01 15:46:49.469660 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:49.469615    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-flzf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Nov 01 15:46:49.469931 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:49.469901    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-flzf5" podUID="e3d2334c-8e83-4a16-a126-cbe45120efa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Nov 01 15:46:50.471192 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:50.470645    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-flzf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Nov 01 15:46:50.471192 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:50.470728    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-flzf5" podUID="e3d2334c-8e83-4a16-a126-cbe45120efa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Nov 01 15:46:51.484193 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:51.480301    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-flzf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Nov 01 15:46:51.484193 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:51.480384    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-flzf5" podUID="e3d2334c-8e83-4a16-a126-cbe45120efa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Nov 01 15:46:52.469091 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:52.469050    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-flzf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Nov 01 15:46:52.469326 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:52.469132    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-flzf5" podUID="e3d2334c-8e83-4a16-a126-cbe45120efa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Nov 01 15:46:53.468532 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:53.468491    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-flzf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Nov 01 15:46:53.469020 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:53.468550    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-flzf5" podUID="e3d2334c-8e83-4a16-a126-cbe45120efa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Nov 01 15:46:53.557052 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:53.557023    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xmwtp_bb59051d-c2df-4362-a157-00f14f915e79/console-operator/3.log"
Nov 01 15:46:53.575186 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:53.575146    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xmwtp_bb59051d-c2df-4362-a157-00f14f915e79/console-operator/3.log"
Nov 01 15:46:55.509402 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:55.509347    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-flzf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 15:46:55.509934 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:55.509420    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-flzf5" podUID="e3d2334c-8e83-4a16-a126-cbe45120efa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 15:46:56.368560 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:56.368511    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-flzf5"
Nov 01 15:46:56.368969 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:56.368947    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-flzf5"
Nov 01 15:46:56.369769 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:46:56.369743    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-flzf5"
Nov 01 15:46:57.499341 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:46:57.499310    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:47:27.498570 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:47:27.498533    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:47:57.499084 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:47:57.499043    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:48:27.500547 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:48:27.500511    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:48:45.894301 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:48:45.894251    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3ce2275-mm4ft"]
Nov 01 15:48:45.899311 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:48:45.899287    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3ce2275-mm4ft"
Nov 01 15:48:45.928509 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:48:45.928481    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd078c7f-1a30-4190-920c-19c2259a5f97-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3ce2275-mm4ft\" (UID: \"bd078c7f-1a30-4190-920c-19c2259a5f97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3ce2275-mm4ft"
Nov 01 15:48:45.928620 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:48:45.928519    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84pqb\" (UniqueName: \"kubernetes.io/projected/bd078c7f-1a30-4190-920c-19c2259a5f97-kube-api-access-84pqb\") pod \"maas-api-revoke-sub-a3ce2275-mm4ft\" (UID: \"bd078c7f-1a30-4190-920c-19c2259a5f97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3ce2275-mm4ft"
Nov 01 15:48:45.928620 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:48:45.928548    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd078c7f-1a30-4190-920c-19c2259a5f97-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3ce2275-mm4ft\" (UID: \"bd078c7f-1a30-4190-920c-19c2259a5f97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3ce2275-mm4ft"
Nov 01 15:48:45.949556 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:48:45.949530    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 15:48:45.949645 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:48:45.949557    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 15:48:45.982586 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:48:45.982557    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 15:48:45.996042 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:48:45.996006    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d6ds8\""
Nov 01 15:48:46.001103 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:48:46.001054    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3ce2275-mm4ft"]
Nov 01 15:48:46.029135 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:48:46.029103    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd078c7f-1a30-4190-920c-19c2259a5f97-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3ce2275-mm4ft\" (UID: \"bd078c7f-1a30-4190-920c-19c2259a5f97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3ce2275-mm4ft"
Nov 01 15:48:46.029304 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:48:46.029145    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84pqb\" (UniqueName: \"kubernetes.io/projected/bd078c7f-1a30-4190-920c-19c2259a5f97-kube-api-access-84pqb\") pod \"maas-api-revoke-sub-a3ce2275-mm4ft\" (UID: \"bd078c7f-1a30-4190-920c-19c2259a5f97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3ce2275-mm4ft"
Nov 01 15:48:46.029304 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:48:46.029205    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd078c7f-1a30-4190-920c-19c2259a5f97-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3ce2275-mm4ft\" (UID: \"bd078c7f-1a30-4190-920c-19c2259a5f97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3ce2275-mm4ft"
Nov 01 15:48:46.029840 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:48:46.029813    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd078c7f-1a30-4190-920c-19c2259a5f97-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3ce2275-mm4ft\" (UID: \"bd078c7f-1a30-4190-920c-19c2259a5f97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3ce2275-mm4ft"
Nov 01 15:48:46.031441 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:48:46.031414    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd078c7f-1a30-4190-920c-19c2259a5f97-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3ce2275-mm4ft\" (UID: \"bd078c7f-1a30-4190-920c-19c2259a5f97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3ce2275-mm4ft"
Nov 01 15:48:46.081998 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:48:46.081962    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84pqb\" (UniqueName: \"kubernetes.io/projected/bd078c7f-1a30-4190-920c-19c2259a5f97-kube-api-access-84pqb\") pod \"maas-api-revoke-sub-a3ce2275-mm4ft\" (UID: \"bd078c7f-1a30-4190-920c-19c2259a5f97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3ce2275-mm4ft"
Nov 01 15:48:46.211826 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:48:46.211742    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3ce2275-mm4ft"
Nov 01 15:48:46.578725 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:48:46.578689    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3ce2275-mm4ft"]
Nov 01 15:48:46.585226 ip-10-0-134-37 kubenswrapper[4993]: W1101 15:48:46.585194    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd078c7f_1a30_4190_920c_19c2259a5f97.slice/crio-595ebf3d05f1625e61186de7b19bad78e841885e4ddbebb611182491f1171f01 WatchSource:0}: Error finding container 595ebf3d05f1625e61186de7b19bad78e841885e4ddbebb611182491f1171f01: Status 404 returned error can't find the container with id 595ebf3d05f1625e61186de7b19bad78e841885e4ddbebb611182491f1171f01
Nov 01 15:48:47.486804 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:48:47.486769    4993 generic.go:356] "Generic (PLEG): container finished" podID="bd078c7f-1a30-4190-920c-19c2259a5f97" containerID="c8333c3a75e3943eb287e584fff7330e58c96bcb80ed9157b7f178a308212995" exitCode=0
Nov 01 15:48:47.487228 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:48:47.486854    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3ce2275-mm4ft" event={"ID":"bd078c7f-1a30-4190-920c-19c2259a5f97","Type":"ContainerDied","Data":"c8333c3a75e3943eb287e584fff7330e58c96bcb80ed9157b7f178a308212995"}
Nov 01 15:48:47.487228 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:48:47.486916    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3ce2275-mm4ft" event={"ID":"bd078c7f-1a30-4190-920c-19c2259a5f97","Type":"ContainerStarted","Data":"595ebf3d05f1625e61186de7b19bad78e841885e4ddbebb611182491f1171f01"}
Nov 01 15:48:48.624392 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:48:48.624362    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3ce2275-mm4ft"
Nov 01 15:48:48.653125 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:48:48.653104    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd078c7f-1a30-4190-920c-19c2259a5f97-kube-api-access-84pqb\" (UniqueName: \"kubernetes.io/projected/bd078c7f-1a30-4190-920c-19c2259a5f97-kube-api-access-84pqb\") pod \"bd078c7f-1a30-4190-920c-19c2259a5f97\" (UID: \"bd078c7f-1a30-4190-920c-19c2259a5f97\") "
Nov 01 15:48:48.653250 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:48:48.653145    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd078c7f-1a30-4190-920c-19c2259a5f97-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd078c7f-1a30-4190-920c-19c2259a5f97-maas-api-openshift-ca\") pod \"bd078c7f-1a30-4190-920c-19c2259a5f97\" (UID: \"bd078c7f-1a30-4190-920c-19c2259a5f97\") "
Nov 01 15:48:48.653250 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:48:48.653244    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd078c7f-1a30-4190-920c-19c2259a5f97-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd078c7f-1a30-4190-920c-19c2259a5f97-maas-api-xks-ca\") pod \"bd078c7f-1a30-4190-920c-19c2259a5f97\" (UID: \"bd078c7f-1a30-4190-920c-19c2259a5f97\") "
Nov 01 15:48:48.653603 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:48:48.653577    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd078c7f-1a30-4190-920c-19c2259a5f97-maas-api-openshift-ca" pod "bd078c7f-1a30-4190-920c-19c2259a5f97" (UID: "bd078c7f-1a30-4190-920c-19c2259a5f97"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:48:48.655193 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:48:48.655150    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd078c7f-1a30-4190-920c-19c2259a5f97-maas-api-xks-ca" pod "bd078c7f-1a30-4190-920c-19c2259a5f97" (UID: "bd078c7f-1a30-4190-920c-19c2259a5f97"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:48:48.655298 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:48:48.655284    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd078c7f-1a30-4190-920c-19c2259a5f97-kube-api-access-84pqb" pod "bd078c7f-1a30-4190-920c-19c2259a5f97" (UID: "bd078c7f-1a30-4190-920c-19c2259a5f97"). InnerVolumeSpecName "kube-api-access-84pqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:48:48.754158 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:48:48.754088    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd078c7f-1a30-4190-920c-19c2259a5f97-maas-api-xks-ca\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:48:48.754158 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:48:48.754116    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84pqb\" (UniqueName: \"kubernetes.io/projected/bd078c7f-1a30-4190-920c-19c2259a5f97-kube-api-access-84pqb\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:48:48.754158 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:48:48.754126    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd078c7f-1a30-4190-920c-19c2259a5f97-maas-api-openshift-ca\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:48:49.498432 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:48:49.498400    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3ce2275-mm4ft"
Nov 01 15:48:49.498609 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:48:49.498410    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3ce2275-mm4ft" event={"ID":"bd078c7f-1a30-4190-920c-19c2259a5f97","Type":"ContainerDied","Data":"595ebf3d05f1625e61186de7b19bad78e841885e4ddbebb611182491f1171f01"}
Nov 01 15:48:49.498609 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:48:49.498521    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="595ebf3d05f1625e61186de7b19bad78e841885e4ddbebb611182491f1171f01"
Nov 01 15:48:57.500079 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:48:57.500043    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:49:04.159553 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:49:04.159307    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d03ddfd-pj4r6"]
Nov 01 15:49:04.161000 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:49:04.160955    4993 cpu_manager.go:414] "containerMap: removing container" podUID="bd078c7f-1a30-4190-920c-19c2259a5f97" containerName="revoke-keys"
Nov 01 15:49:04.161190 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:49:04.161149    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd078c7f-1a30-4190-920c-19c2259a5f97" containerName="revoke-keys"
Nov 01 15:49:04.161593 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:49:04.161575    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd078c7f-1a30-4190-920c-19c2259a5f97" containerName="revoke-keys"
Nov 01 15:49:04.165495 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:49:04.165476    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d03ddfd-pj4r6"
Nov 01 15:49:04.214991 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:49:04.214956    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 15:49:04.215265 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:49:04.215237    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 15:49:04.261281 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:49:04.261234    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d6ds8\""
Nov 01 15:49:04.262059 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:49:04.261939    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 15:49:04.267265 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:49:04.267241    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d03ddfd-pj4r6"]
Nov 01 15:49:04.295514 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:49:04.295471    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/294b9dc2-3774-4de4-94d4-2f86e90134bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d03ddfd-pj4r6\" (UID: \"294b9dc2-3774-4de4-94d4-2f86e90134bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d03ddfd-pj4r6"
Nov 01 15:49:04.295706 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:49:04.295597    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42llz\" (UniqueName: \"kubernetes.io/projected/294b9dc2-3774-4de4-94d4-2f86e90134bb-kube-api-access-42llz\") pod \"maas-api-revoke-sub-0d03ddfd-pj4r6\" (UID: \"294b9dc2-3774-4de4-94d4-2f86e90134bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d03ddfd-pj4r6"
Nov 01 15:49:04.295706 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:49:04.295689    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/294b9dc2-3774-4de4-94d4-2f86e90134bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d03ddfd-pj4r6\" (UID: \"294b9dc2-3774-4de4-94d4-2f86e90134bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d03ddfd-pj4r6"
Nov 01 15:49:04.396904 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:49:04.396852    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/294b9dc2-3774-4de4-94d4-2f86e90134bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d03ddfd-pj4r6\" (UID: \"294b9dc2-3774-4de4-94d4-2f86e90134bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d03ddfd-pj4r6"
Nov 01 15:49:04.397115 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:49:04.396935    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42llz\" (UniqueName: \"kubernetes.io/projected/294b9dc2-3774-4de4-94d4-2f86e90134bb-kube-api-access-42llz\") pod \"maas-api-revoke-sub-0d03ddfd-pj4r6\" (UID: \"294b9dc2-3774-4de4-94d4-2f86e90134bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d03ddfd-pj4r6"
Nov 01 15:49:04.397115 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:49:04.396985    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/294b9dc2-3774-4de4-94d4-2f86e90134bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d03ddfd-pj4r6\" (UID: \"294b9dc2-3774-4de4-94d4-2f86e90134bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d03ddfd-pj4r6"
Nov 01 15:49:04.397912 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:49:04.397879    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/294b9dc2-3774-4de4-94d4-2f86e90134bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d03ddfd-pj4r6\" (UID: \"294b9dc2-3774-4de4-94d4-2f86e90134bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d03ddfd-pj4r6"
Nov 01 15:49:04.400372 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:49:04.400323    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/294b9dc2-3774-4de4-94d4-2f86e90134bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d03ddfd-pj4r6\" (UID: \"294b9dc2-3774-4de4-94d4-2f86e90134bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d03ddfd-pj4r6"
Nov 01 15:49:04.450548 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:49:04.450461    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42llz\" (UniqueName: \"kubernetes.io/projected/294b9dc2-3774-4de4-94d4-2f86e90134bb-kube-api-access-42llz\") pod \"maas-api-revoke-sub-0d03ddfd-pj4r6\" (UID: \"294b9dc2-3774-4de4-94d4-2f86e90134bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d03ddfd-pj4r6"
Nov 01 15:49:04.486266 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:49:04.486223    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d03ddfd-pj4r6"
Nov 01 15:49:04.942692 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:49:04.942657    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d03ddfd-pj4r6"]
Nov 01 15:49:05.728192 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:49:05.727789    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d03ddfd-pj4r6" event={"ID":"294b9dc2-3774-4de4-94d4-2f86e90134bb","Type":"ContainerStarted","Data":"214922dcb2f6a4e59c4ac19cfaf4bcc25eabbef2e3b12daab40224bf8496bd71"}
Nov 01 15:49:05.728192 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:49:05.727876    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d03ddfd-pj4r6" event={"ID":"294b9dc2-3774-4de4-94d4-2f86e90134bb","Type":"ContainerStarted","Data":"e98892ba1c31ab8b7141845a9ee59ec52c2b488bbb297227af1a01c41b15fbd2"}
Nov 01 15:49:05.850519 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:49:05.850470    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d03ddfd-pj4r6" podStartSLOduration=1.85045494 podStartE2EDuration="1.85045494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:49:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:49:05.84914167 +0000 UTC m=+5533.179004410" watchObservedRunningTime="2026-11-01 15:49:05.85045494 +0000 UTC m=+5533.180317677"
Nov 01 15:49:06.735644 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:49:06.735609    4993 generic.go:356] "Generic (PLEG): container finished" podID="294b9dc2-3774-4de4-94d4-2f86e90134bb" containerID="214922dcb2f6a4e59c4ac19cfaf4bcc25eabbef2e3b12daab40224bf8496bd71" exitCode=0
Nov 01 15:49:06.736081 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:49:06.735688    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d03ddfd-pj4r6" event={"ID":"294b9dc2-3774-4de4-94d4-2f86e90134bb","Type":"ContainerDied","Data":"214922dcb2f6a4e59c4ac19cfaf4bcc25eabbef2e3b12daab40224bf8496bd71"}
Nov 01 15:49:07.871218 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:49:07.871186    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d03ddfd-pj4r6"
Nov 01 15:49:07.938755 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:49:07.938722    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/294b9dc2-3774-4de4-94d4-2f86e90134bb-kube-api-access-42llz\" (UniqueName: \"kubernetes.io/projected/294b9dc2-3774-4de4-94d4-2f86e90134bb-kube-api-access-42llz\") pod \"294b9dc2-3774-4de4-94d4-2f86e90134bb\" (UID: \"294b9dc2-3774-4de4-94d4-2f86e90134bb\") "
Nov 01 15:49:07.938967 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:49:07.938816    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/294b9dc2-3774-4de4-94d4-2f86e90134bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/294b9dc2-3774-4de4-94d4-2f86e90134bb-maas-api-openshift-ca\") pod \"294b9dc2-3774-4de4-94d4-2f86e90134bb\" (UID: \"294b9dc2-3774-4de4-94d4-2f86e90134bb\") "
Nov 01 15:49:07.938967 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:49:07.938896    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/294b9dc2-3774-4de4-94d4-2f86e90134bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/294b9dc2-3774-4de4-94d4-2f86e90134bb-maas-api-xks-ca\") pod \"294b9dc2-3774-4de4-94d4-2f86e90134bb\" (UID: \"294b9dc2-3774-4de4-94d4-2f86e90134bb\") "
Nov 01 15:49:07.939232 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:49:07.939211    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/294b9dc2-3774-4de4-94d4-2f86e90134bb-maas-api-openshift-ca" pod "294b9dc2-3774-4de4-94d4-2f86e90134bb" (UID: "294b9dc2-3774-4de4-94d4-2f86e90134bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:49:07.941057 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:49:07.941021    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/294b9dc2-3774-4de4-94d4-2f86e90134bb-kube-api-access-42llz" pod "294b9dc2-3774-4de4-94d4-2f86e90134bb" (UID: "294b9dc2-3774-4de4-94d4-2f86e90134bb"). InnerVolumeSpecName "kube-api-access-42llz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:49:07.941193 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:49:07.941084    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/294b9dc2-3774-4de4-94d4-2f86e90134bb-maas-api-xks-ca" pod "294b9dc2-3774-4de4-94d4-2f86e90134bb" (UID: "294b9dc2-3774-4de4-94d4-2f86e90134bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 15:49:08.040228 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:49:08.040129    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/294b9dc2-3774-4de4-94d4-2f86e90134bb-maas-api-openshift-ca\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:49:08.040228 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:49:08.040161    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/294b9dc2-3774-4de4-94d4-2f86e90134bb-maas-api-xks-ca\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:49:08.040228 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:49:08.040192    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42llz\" (UniqueName: \"kubernetes.io/projected/294b9dc2-3774-4de4-94d4-2f86e90134bb-kube-api-access-42llz\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:49:08.747112 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:49:08.747082    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d03ddfd-pj4r6"
Nov 01 15:49:08.747298 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:49:08.747106    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d03ddfd-pj4r6" event={"ID":"294b9dc2-3774-4de4-94d4-2f86e90134bb","Type":"ContainerDied","Data":"e98892ba1c31ab8b7141845a9ee59ec52c2b488bbb297227af1a01c41b15fbd2"}
Nov 01 15:49:08.747298 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:49:08.747154    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e98892ba1c31ab8b7141845a9ee59ec52c2b488bbb297227af1a01c41b15fbd2"
Nov 01 15:49:27.507018 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:49:27.506971    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:49:57.500619 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:49:57.500583    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:50:27.500251 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:50:27.500214    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:50:57.499382 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:50:57.499346    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:51:27.499045 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:51:27.499000    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:51:53.599290 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:51:53.599254    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xmwtp_bb59051d-c2df-4362-a157-00f14f915e79/console-operator/3.log"
Nov 01 15:51:53.615490 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:51:53.615455    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xmwtp_bb59051d-c2df-4362-a157-00f14f915e79/console-operator/3.log"
Nov 01 15:51:57.499138 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:51:57.499103    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:52:27.500798 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:52:27.500715    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:52:42.426384 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:52:42.426313    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-flzf5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 15:52:42.427057 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:52:42.427020    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-flzf5" podUID="e3d2334c-8e83-4a16-a126-cbe45120efa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 15:52:57.423650 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:52:57.423609    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-flzf5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 15:52:57.424337 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:52:57.424283    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-flzf5" podUID="e3d2334c-8e83-4a16-a126-cbe45120efa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 15:52:57.623980 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:52:57.623922    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:53:13.857192 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:53:13.856266    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-4pmtn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 15:53:13.857192 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:53:13.856359    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4pmtn" podUID="306b0405-fb73-435a-ae73-42a90cd2d2f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 15:53:27.501041 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:53:27.500999    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:53:49.095959 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:53:49.095922    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3ce2275-mm4ft"]
Nov 01 15:53:49.143408 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:53:49.143371    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3ce2275-mm4ft"]
Nov 01 15:53:50.793870 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:53:50.793833    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd078c7f-1a30-4190-920c-19c2259a5f97" path="/var/lib/kubelet/pods/bd078c7f-1a30-4190-920c-19c2259a5f97/volumes"
Nov 01 15:53:57.499650 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:53:57.499615    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:54:05.801326 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:05.801289    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-h7dbs"]
Nov 01 15:54:05.801946 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:05.801920    4993 cpu_manager.go:414] "containerMap: removing container" podUID="294b9dc2-3774-4de4-94d4-2f86e90134bb" containerName="revoke-keys"
Nov 01 15:54:05.802030 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:05.801949    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="294b9dc2-3774-4de4-94d4-2f86e90134bb" containerName="revoke-keys"
Nov 01 15:54:05.802089 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:05.802073    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="294b9dc2-3774-4de4-94d4-2f86e90134bb" containerName="revoke-keys"
Nov 01 15:54:05.805633 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:05.805608    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-h7dbs"
Nov 01 15:54:05.881788 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:05.881741    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6e524b3-6d28-42a6-975d-d19758cf1378-istio-envoy\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-h7dbs\" (UID: \"a6e524b3-6d28-42a6-975d-d19758cf1378\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-h7dbs"
Nov 01 15:54:05.881788 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:05.881785    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6e524b3-6d28-42a6-975d-d19758cf1378-workload-certs\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-h7dbs\" (UID: \"a6e524b3-6d28-42a6-975d-d19758cf1378\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-h7dbs"
Nov 01 15:54:05.882005 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:05.881887    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6e524b3-6d28-42a6-975d-d19758cf1378-workload-socket\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-h7dbs\" (UID: \"a6e524b3-6d28-42a6-975d-d19758cf1378\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-h7dbs"
Nov 01 15:54:05.882005 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:05.881936    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6e524b3-6d28-42a6-975d-d19758cf1378-istio-token\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-h7dbs\" (UID: \"a6e524b3-6d28-42a6-975d-d19758cf1378\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-h7dbs"
Nov 01 15:54:05.882079 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:05.882056    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6e524b3-6d28-42a6-975d-d19758cf1378-credential-socket\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-h7dbs\" (UID: \"a6e524b3-6d28-42a6-975d-d19758cf1378\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-h7dbs"
Nov 01 15:54:05.882118 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:05.882078    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6e524b3-6d28-42a6-975d-d19758cf1378-istiod-ca-cert\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-h7dbs\" (UID: \"a6e524b3-6d28-42a6-975d-d19758cf1378\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-h7dbs"
Nov 01 15:54:05.882327 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:05.882293    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6e524b3-6d28-42a6-975d-d19758cf1378-istio-podinfo\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-h7dbs\" (UID: \"a6e524b3-6d28-42a6-975d-d19758cf1378\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-h7dbs"
Nov 01 15:54:05.882472 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:05.882350    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2q6x\" (UniqueName: \"kubernetes.io/projected/a6e524b3-6d28-42a6-975d-d19758cf1378-kube-api-access-l2q6x\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-h7dbs\" (UID: \"a6e524b3-6d28-42a6-975d-d19758cf1378\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-h7dbs"
Nov 01 15:54:05.882472 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:05.882373    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6e524b3-6d28-42a6-975d-d19758cf1378-istio-data\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-h7dbs\" (UID: \"a6e524b3-6d28-42a6-975d-d19758cf1378\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-h7dbs"
Nov 01 15:54:05.901761 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:05.901727    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-dual-3436ba-openshift-default-dockercfg-hdq2b\""
Nov 01 15:54:05.950587 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:05.950553    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-h7dbs"]
Nov 01 15:54:05.983352 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:05.983312    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6e524b3-6d28-42a6-975d-d19758cf1378-credential-socket\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-h7dbs\" (UID: \"a6e524b3-6d28-42a6-975d-d19758cf1378\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-h7dbs"
Nov 01 15:54:05.983549 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:05.983364    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6e524b3-6d28-42a6-975d-d19758cf1378-istiod-ca-cert\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-h7dbs\" (UID: \"a6e524b3-6d28-42a6-975d-d19758cf1378\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-h7dbs"
Nov 01 15:54:05.983549 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:05.983413    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6e524b3-6d28-42a6-975d-d19758cf1378-istio-podinfo\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-h7dbs\" (UID: \"a6e524b3-6d28-42a6-975d-d19758cf1378\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-h7dbs"
Nov 01 15:54:05.983549 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:05.983439    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2q6x\" (UniqueName: \"kubernetes.io/projected/a6e524b3-6d28-42a6-975d-d19758cf1378-kube-api-access-l2q6x\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-h7dbs\" (UID: \"a6e524b3-6d28-42a6-975d-d19758cf1378\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-h7dbs"
Nov 01 15:54:05.983549 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:05.983465    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6e524b3-6d28-42a6-975d-d19758cf1378-istio-data\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-h7dbs\" (UID: \"a6e524b3-6d28-42a6-975d-d19758cf1378\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-h7dbs"
Nov 01 15:54:05.983549 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:05.983511    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6e524b3-6d28-42a6-975d-d19758cf1378-istio-envoy\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-h7dbs\" (UID: \"a6e524b3-6d28-42a6-975d-d19758cf1378\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-h7dbs"
Nov 01 15:54:05.983549 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:05.983535    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6e524b3-6d28-42a6-975d-d19758cf1378-workload-certs\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-h7dbs\" (UID: \"a6e524b3-6d28-42a6-975d-d19758cf1378\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-h7dbs"
Nov 01 15:54:05.983893 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:05.983585    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6e524b3-6d28-42a6-975d-d19758cf1378-workload-socket\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-h7dbs\" (UID: \"a6e524b3-6d28-42a6-975d-d19758cf1378\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-h7dbs"
Nov 01 15:54:05.983893 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:05.983614    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6e524b3-6d28-42a6-975d-d19758cf1378-istio-token\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-h7dbs\" (UID: \"a6e524b3-6d28-42a6-975d-d19758cf1378\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-h7dbs"
Nov 01 15:54:05.984005 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:05.983892    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6e524b3-6d28-42a6-975d-d19758cf1378-istio-data\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-h7dbs\" (UID: \"a6e524b3-6d28-42a6-975d-d19758cf1378\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-h7dbs"
Nov 01 15:54:05.984005 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:05.983905    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6e524b3-6d28-42a6-975d-d19758cf1378-workload-certs\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-h7dbs\" (UID: \"a6e524b3-6d28-42a6-975d-d19758cf1378\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-h7dbs"
Nov 01 15:54:05.984265 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:05.984216    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6e524b3-6d28-42a6-975d-d19758cf1378-workload-socket\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-h7dbs\" (UID: \"a6e524b3-6d28-42a6-975d-d19758cf1378\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-h7dbs"
Nov 01 15:54:05.984363 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:05.984290    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6e524b3-6d28-42a6-975d-d19758cf1378-credential-socket\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-h7dbs\" (UID: \"a6e524b3-6d28-42a6-975d-d19758cf1378\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-h7dbs"
Nov 01 15:54:05.984699 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:05.984667    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6e524b3-6d28-42a6-975d-d19758cf1378-istiod-ca-cert\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-h7dbs\" (UID: \"a6e524b3-6d28-42a6-975d-d19758cf1378\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-h7dbs"
Nov 01 15:54:05.986034 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:05.986012    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6e524b3-6d28-42a6-975d-d19758cf1378-istio-envoy\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-h7dbs\" (UID: \"a6e524b3-6d28-42a6-975d-d19758cf1378\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-h7dbs"
Nov 01 15:54:05.986288 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:05.986270    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6e524b3-6d28-42a6-975d-d19758cf1378-istio-podinfo\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-h7dbs\" (UID: \"a6e524b3-6d28-42a6-975d-d19758cf1378\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-h7dbs"
Nov 01 15:54:06.080317 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:06.080245    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6e524b3-6d28-42a6-975d-d19758cf1378-istio-token\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-h7dbs\" (UID: \"a6e524b3-6d28-42a6-975d-d19758cf1378\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-h7dbs"
Nov 01 15:54:06.080317 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:06.080245    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2q6x\" (UniqueName: \"kubernetes.io/projected/a6e524b3-6d28-42a6-975d-d19758cf1378-kube-api-access-l2q6x\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-h7dbs\" (UID: \"a6e524b3-6d28-42a6-975d-d19758cf1378\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-h7dbs"
Nov 01 15:54:06.118257 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:06.118223    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-h7dbs"
Nov 01 15:54:06.542395 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:06.542354    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-h7dbs"]
Nov 01 15:54:06.548792 ip-10-0-134-37 kubenswrapper[4993]: W1101 15:54:06.548758    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6e524b3_6d28_42a6_975d_d19758cf1378.slice/crio-893376d71c4b04f74806235458d6227c2210338e1e9dbdb099d16d7d1ab59c08 WatchSource:0}: Error finding container 893376d71c4b04f74806235458d6227c2210338e1e9dbdb099d16d7d1ab59c08: Status 404 returned error can't find the container with id 893376d71c4b04f74806235458d6227c2210338e1e9dbdb099d16d7d1ab59c08
Nov 01 15:54:06.551207 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:06.551156    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 15:54:06.551300 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:06.551247    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 15:54:06.551300 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:06.551278    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 15:54:07.444284 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:07.444241    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-h7dbs" event={"ID":"a6e524b3-6d28-42a6-975d-d19758cf1378","Type":"ContainerStarted","Data":"1fe78710aa2a3ed537b4c4edc49a10ad52b181820d14d8bee3d6d4664a307fab"}
Nov 01 15:54:07.444688 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:07.444311    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-h7dbs" event={"ID":"a6e524b3-6d28-42a6-975d-d19758cf1378","Type":"ContainerStarted","Data":"893376d71c4b04f74806235458d6227c2210338e1e9dbdb099d16d7d1ab59c08"}
Nov 01 15:54:07.683323 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:07.683272    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-h7dbs" podStartSLOduration=2.683257611 podStartE2EDuration="2.683257611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:54:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:54:07.636300144 +0000 UTC m=+5834.966162880" watchObservedRunningTime="2026-11-01 15:54:07.683257611 +0000 UTC m=+5835.013120348"
Nov 01 15:54:08.044660 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:08.044622    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d03ddfd-pj4r6"]
Nov 01 15:54:08.076823 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:08.076780    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d03ddfd-pj4r6"]
Nov 01 15:54:08.118875 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:08.118837    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-h7dbs"
Nov 01 15:54:08.120501 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:08.120466    4993 patch_prober.go:29] interesting pod/e2e-dual-3436ba-openshift-default-8446b54f85-h7dbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body=
Nov 01 15:54:08.120675 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:08.120534    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-h7dbs" podUID="a6e524b3-6d28-42a6-975d-d19758cf1378" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused"
Nov 01 15:54:08.793998 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:08.793964    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="294b9dc2-3774-4de4-94d4-2f86e90134bb" path="/var/lib/kubelet/pods/294b9dc2-3774-4de4-94d4-2f86e90134bb/volumes"
Nov 01 15:54:09.118916 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:09.118838    4993 patch_prober.go:29] interesting pod/e2e-dual-3436ba-openshift-default-8446b54f85-h7dbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body=
Nov 01 15:54:09.119066 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:09.118905    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-h7dbs" podUID="a6e524b3-6d28-42a6-975d-d19758cf1378" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused"
Nov 01 15:54:10.118861 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:10.118820    4993 patch_prober.go:29] interesting pod/e2e-dual-3436ba-openshift-default-8446b54f85-h7dbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body=
Nov 01 15:54:10.119296 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:10.118882    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-h7dbs" podUID="a6e524b3-6d28-42a6-975d-d19758cf1378" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused"
Nov 01 15:54:11.119487 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:11.119452    4993 patch_prober.go:29] interesting pod/e2e-dual-3436ba-openshift-default-8446b54f85-h7dbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body=
Nov 01 15:54:11.119903 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:11.119507    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-h7dbs" podUID="a6e524b3-6d28-42a6-975d-d19758cf1378" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused"
Nov 01 15:54:12.119263 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:12.119229    4993 patch_prober.go:29] interesting pod/e2e-dual-3436ba-openshift-default-8446b54f85-h7dbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body=
Nov 01 15:54:12.119446 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:12.119279    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-h7dbs" podUID="a6e524b3-6d28-42a6-975d-d19758cf1378" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused"
Nov 01 15:54:13.123914 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:13.123878    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-h7dbs"
Nov 01 15:54:13.124396 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:13.124299    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-h7dbs"
Nov 01 15:54:13.125256 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:13.125232    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-h7dbs"
Nov 01 15:54:27.500509 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:54:27.500467    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:54:36.046352 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:36.046315    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-pnd69"]
Nov 01 15:54:36.049600 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:36.049575    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-pnd69"
Nov 01 15:54:36.175819 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:36.175782    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38-istio-token\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-pnd69\" (UID: \"aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-pnd69"
Nov 01 15:54:36.175819 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:36.175822    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38-istio-envoy\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-pnd69\" (UID: \"aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-pnd69"
Nov 01 15:54:36.176061 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:36.175852    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38-istio-data\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-pnd69\" (UID: \"aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-pnd69"
Nov 01 15:54:36.176061 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:36.175938    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38-istiod-ca-cert\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-pnd69\" (UID: \"aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-pnd69"
Nov 01 15:54:36.176061 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:36.176036    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38-credential-socket\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-pnd69\" (UID: \"aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-pnd69"
Nov 01 15:54:36.176200 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:36.176079    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38-workload-certs\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-pnd69\" (UID: \"aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-pnd69"
Nov 01 15:54:36.176242 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:36.176192    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2jr6\" (UniqueName: \"kubernetes.io/projected/aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38-kube-api-access-m2jr6\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-pnd69\" (UID: \"aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-pnd69"
Nov 01 15:54:36.176242 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:36.176223    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38-workload-socket\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-pnd69\" (UID: \"aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-pnd69"
Nov 01 15:54:36.176314 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:36.176243    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38-istio-podinfo\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-pnd69\" (UID: \"aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-pnd69"
Nov 01 15:54:36.240451 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:36.240418    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-pnd69"]
Nov 01 15:54:36.277126 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:36.277093    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38-istio-token\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-pnd69\" (UID: \"aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-pnd69"
Nov 01 15:54:36.277126 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:36.277129    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38-istio-envoy\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-pnd69\" (UID: \"aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-pnd69"
Nov 01 15:54:36.277399 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:36.277157    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38-istio-data\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-pnd69\" (UID: \"aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-pnd69"
Nov 01 15:54:36.277399 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:36.277292    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38-istiod-ca-cert\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-pnd69\" (UID: \"aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-pnd69"
Nov 01 15:54:36.277399 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:36.277339    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38-credential-socket\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-pnd69\" (UID: \"aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-pnd69"
Nov 01 15:54:36.277570 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:36.277409    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38-workload-certs\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-pnd69\" (UID: \"aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-pnd69"
Nov 01 15:54:36.277570 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:36.277475    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2jr6\" (UniqueName: \"kubernetes.io/projected/aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38-kube-api-access-m2jr6\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-pnd69\" (UID: \"aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-pnd69"
Nov 01 15:54:36.277570 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:36.277507    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38-workload-socket\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-pnd69\" (UID: \"aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-pnd69"
Nov 01 15:54:36.277570 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:36.277532    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38-istio-podinfo\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-pnd69\" (UID: \"aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-pnd69"
Nov 01 15:54:36.277773 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:36.277611    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38-istio-data\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-pnd69\" (UID: \"aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-pnd69"
Nov 01 15:54:36.278046 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:36.278008    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38-credential-socket\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-pnd69\" (UID: \"aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-pnd69"
Nov 01 15:54:36.278196 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:36.278090    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38-workload-certs\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-pnd69\" (UID: \"aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-pnd69"
Nov 01 15:54:36.278196 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:36.278137    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38-workload-socket\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-pnd69\" (UID: \"aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-pnd69"
Nov 01 15:54:36.278429 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:36.278408    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38-istiod-ca-cert\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-pnd69\" (UID: \"aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-pnd69"
Nov 01 15:54:36.279495 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:36.279477    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38-istio-envoy\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-pnd69\" (UID: \"aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-pnd69"
Nov 01 15:54:36.280226 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:36.280209    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38-istio-podinfo\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-pnd69\" (UID: \"aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-pnd69"
Nov 01 15:54:36.329999 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:36.329945    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38-istio-token\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-pnd69\" (UID: \"aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-pnd69"
Nov 01 15:54:36.330111 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:36.330090    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2jr6\" (UniqueName: \"kubernetes.io/projected/aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38-kube-api-access-m2jr6\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-pnd69\" (UID: \"aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-pnd69"
Nov 01 15:54:36.362247 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:36.362218    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-pnd69"
Nov 01 15:54:36.784430 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:36.784396    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-pnd69"]
Nov 01 15:54:36.793072 ip-10-0-134-37 kubenswrapper[4993]: W1101 15:54:36.793043    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaae6e7f9_ab68_4cb7_8c3b_4c372e0f5b38.slice/crio-b0c4a45ff61fe4c3f9c67071bc5af5d508b8c1fd29ad78f64a8a2ff6c4e07e39 WatchSource:0}: Error finding container b0c4a45ff61fe4c3f9c67071bc5af5d508b8c1fd29ad78f64a8a2ff6c4e07e39: Status 404 returned error can't find the container with id b0c4a45ff61fe4c3f9c67071bc5af5d508b8c1fd29ad78f64a8a2ff6c4e07e39
Nov 01 15:54:36.795223 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:36.795190    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 15:54:36.795304 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:36.795265    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 15:54:36.795304 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:36.795300    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 15:54:37.624702 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:37.624667    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-pnd69" event={"ID":"aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38","Type":"ContainerStarted","Data":"76accbdb06292d1a97ff24d91b26149c711c992a2b893e34336a0ec4d315b666"}
Nov 01 15:54:37.625092 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:37.624720    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-pnd69" event={"ID":"aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38","Type":"ContainerStarted","Data":"b0c4a45ff61fe4c3f9c67071bc5af5d508b8c1fd29ad78f64a8a2ff6c4e07e39"}
Nov 01 15:54:37.828048 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:37.828001    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-pnd69" podStartSLOduration=1.8279873420000001 podStartE2EDuration="1.827987342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:54:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:54:37.827848408 +0000 UTC m=+5865.157711145" watchObservedRunningTime="2026-11-01 15:54:37.827987342 +0000 UTC m=+5865.157850079"
Nov 01 15:54:38.371204 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:38.370161    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-pnd69"
Nov 01 15:54:38.371204 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:38.370311    4993 patch_prober.go:29] interesting pod/e2e-dual-3436ba-openshift-default-8446b54f85-pnd69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Nov 01 15:54:38.371204 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:38.370363    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-pnd69" podUID="aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Nov 01 15:54:39.372596 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:39.372561    4993 patch_prober.go:29] interesting pod/e2e-dual-3436ba-openshift-default-8446b54f85-pnd69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Nov 01 15:54:39.373215 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:39.373183    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-pnd69" podUID="aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Nov 01 15:54:40.376193 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:40.375331    4993 patch_prober.go:29] interesting pod/e2e-dual-3436ba-openshift-default-8446b54f85-pnd69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Nov 01 15:54:40.376193 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:40.375420    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-pnd69" podUID="aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Nov 01 15:54:41.372193 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:41.364607    4993 patch_prober.go:29] interesting pod/e2e-dual-3436ba-openshift-default-8446b54f85-pnd69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Nov 01 15:54:41.372193 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:41.364674    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-pnd69" podUID="aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Nov 01 15:54:42.371661 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:42.371622    4993 patch_prober.go:29] interesting pod/e2e-dual-3436ba-openshift-default-8446b54f85-pnd69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Nov 01 15:54:42.372273 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:42.372234    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-pnd69" podUID="aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Nov 01 15:54:43.366765 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:43.366731    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-pnd69"
Nov 01 15:54:43.367118 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:43.367091    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-pnd69"
Nov 01 15:54:43.367674 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:43.367655    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-pnd69"
Nov 01 15:54:51.140410 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:51.140369    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-5pw89"]
Nov 01 15:54:51.144584 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:51.144558    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-5pw89"
Nov 01 15:54:51.216292 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:51.216249    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c09c8668-89c0-48ed-86da-fe404dc989e7-credential-socket\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-5pw89\" (UID: \"c09c8668-89c0-48ed-86da-fe404dc989e7\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-5pw89"
Nov 01 15:54:51.216292 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:51.216295    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c09c8668-89c0-48ed-86da-fe404dc989e7-istio-envoy\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-5pw89\" (UID: \"c09c8668-89c0-48ed-86da-fe404dc989e7\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-5pw89"
Nov 01 15:54:51.216519 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:51.216375    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c09c8668-89c0-48ed-86da-fe404dc989e7-istio-token\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-5pw89\" (UID: \"c09c8668-89c0-48ed-86da-fe404dc989e7\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-5pw89"
Nov 01 15:54:51.216519 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:51.216444    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c09c8668-89c0-48ed-86da-fe404dc989e7-istio-podinfo\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-5pw89\" (UID: \"c09c8668-89c0-48ed-86da-fe404dc989e7\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-5pw89"
Nov 01 15:54:51.216519 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:51.216468    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c09c8668-89c0-48ed-86da-fe404dc989e7-workload-socket\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-5pw89\" (UID: \"c09c8668-89c0-48ed-86da-fe404dc989e7\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-5pw89"
Nov 01 15:54:51.216519 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:51.216506    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c09c8668-89c0-48ed-86da-fe404dc989e7-workload-certs\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-5pw89\" (UID: \"c09c8668-89c0-48ed-86da-fe404dc989e7\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-5pw89"
Nov 01 15:54:51.216677 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:51.216544    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c09c8668-89c0-48ed-86da-fe404dc989e7-istiod-ca-cert\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-5pw89\" (UID: \"c09c8668-89c0-48ed-86da-fe404dc989e7\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-5pw89"
Nov 01 15:54:51.216677 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:51.216600    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c09c8668-89c0-48ed-86da-fe404dc989e7-istio-data\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-5pw89\" (UID: \"c09c8668-89c0-48ed-86da-fe404dc989e7\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-5pw89"
Nov 01 15:54:51.216677 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:51.216656    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ggcz\" (UniqueName: \"kubernetes.io/projected/c09c8668-89c0-48ed-86da-fe404dc989e7-kube-api-access-9ggcz\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-5pw89\" (UID: \"c09c8668-89c0-48ed-86da-fe404dc989e7\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-5pw89"
Nov 01 15:54:51.241437 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:51.241404    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-5pw89"]
Nov 01 15:54:51.317554 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:51.317519    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ggcz\" (UniqueName: \"kubernetes.io/projected/c09c8668-89c0-48ed-86da-fe404dc989e7-kube-api-access-9ggcz\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-5pw89\" (UID: \"c09c8668-89c0-48ed-86da-fe404dc989e7\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-5pw89"
Nov 01 15:54:51.317554 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:51.317561    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c09c8668-89c0-48ed-86da-fe404dc989e7-credential-socket\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-5pw89\" (UID: \"c09c8668-89c0-48ed-86da-fe404dc989e7\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-5pw89"
Nov 01 15:54:51.317820 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:51.317585    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c09c8668-89c0-48ed-86da-fe404dc989e7-istio-envoy\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-5pw89\" (UID: \"c09c8668-89c0-48ed-86da-fe404dc989e7\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-5pw89"
Nov 01 15:54:51.317820 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:51.317618    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c09c8668-89c0-48ed-86da-fe404dc989e7-istio-token\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-5pw89\" (UID: \"c09c8668-89c0-48ed-86da-fe404dc989e7\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-5pw89"
Nov 01 15:54:51.317820 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:51.317655    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c09c8668-89c0-48ed-86da-fe404dc989e7-istio-podinfo\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-5pw89\" (UID: \"c09c8668-89c0-48ed-86da-fe404dc989e7\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-5pw89"
Nov 01 15:54:51.317820 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:51.317687    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c09c8668-89c0-48ed-86da-fe404dc989e7-workload-socket\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-5pw89\" (UID: \"c09c8668-89c0-48ed-86da-fe404dc989e7\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-5pw89"
Nov 01 15:54:51.317820 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:51.317716    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c09c8668-89c0-48ed-86da-fe404dc989e7-workload-certs\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-5pw89\" (UID: \"c09c8668-89c0-48ed-86da-fe404dc989e7\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-5pw89"
Nov 01 15:54:51.317820 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:51.317764    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c09c8668-89c0-48ed-86da-fe404dc989e7-istiod-ca-cert\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-5pw89\" (UID: \"c09c8668-89c0-48ed-86da-fe404dc989e7\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-5pw89"
Nov 01 15:54:51.317820 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:51.317817    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c09c8668-89c0-48ed-86da-fe404dc989e7-istio-data\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-5pw89\" (UID: \"c09c8668-89c0-48ed-86da-fe404dc989e7\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-5pw89"
Nov 01 15:54:51.318287 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:51.318055    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c09c8668-89c0-48ed-86da-fe404dc989e7-credential-socket\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-5pw89\" (UID: \"c09c8668-89c0-48ed-86da-fe404dc989e7\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-5pw89"
Nov 01 15:54:51.318287 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:51.318109    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c09c8668-89c0-48ed-86da-fe404dc989e7-workload-certs\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-5pw89\" (UID: \"c09c8668-89c0-48ed-86da-fe404dc989e7\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-5pw89"
Nov 01 15:54:51.318287 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:51.318136    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c09c8668-89c0-48ed-86da-fe404dc989e7-workload-socket\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-5pw89\" (UID: \"c09c8668-89c0-48ed-86da-fe404dc989e7\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-5pw89"
Nov 01 15:54:51.318287 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:51.318257    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c09c8668-89c0-48ed-86da-fe404dc989e7-istio-data\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-5pw89\" (UID: \"c09c8668-89c0-48ed-86da-fe404dc989e7\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-5pw89"
Nov 01 15:54:51.318669 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:51.318647    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c09c8668-89c0-48ed-86da-fe404dc989e7-istiod-ca-cert\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-5pw89\" (UID: \"c09c8668-89c0-48ed-86da-fe404dc989e7\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-5pw89"
Nov 01 15:54:51.320035 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:51.320013    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c09c8668-89c0-48ed-86da-fe404dc989e7-istio-envoy\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-5pw89\" (UID: \"c09c8668-89c0-48ed-86da-fe404dc989e7\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-5pw89"
Nov 01 15:54:51.320134 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:51.320069    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c09c8668-89c0-48ed-86da-fe404dc989e7-istio-podinfo\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-5pw89\" (UID: \"c09c8668-89c0-48ed-86da-fe404dc989e7\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-5pw89"
Nov 01 15:54:51.370927 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:51.370884    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c09c8668-89c0-48ed-86da-fe404dc989e7-istio-token\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-5pw89\" (UID: \"c09c8668-89c0-48ed-86da-fe404dc989e7\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-5pw89"
Nov 01 15:54:51.371100 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:51.371069    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ggcz\" (UniqueName: \"kubernetes.io/projected/c09c8668-89c0-48ed-86da-fe404dc989e7-kube-api-access-9ggcz\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-5pw89\" (UID: \"c09c8668-89c0-48ed-86da-fe404dc989e7\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-5pw89"
Nov 01 15:54:51.458332 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:51.458244    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-5pw89"
Nov 01 15:54:51.879502 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:51.879466    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-5pw89"]
Nov 01 15:54:51.886635 ip-10-0-134-37 kubenswrapper[4993]: W1101 15:54:51.886604    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc09c8668_89c0_48ed_86da_fe404dc989e7.slice/crio-f0c8172474a20e2246b3cfacac48622a9263bd25cdcb0d6b271482cde8ff8baf WatchSource:0}: Error finding container f0c8172474a20e2246b3cfacac48622a9263bd25cdcb0d6b271482cde8ff8baf: Status 404 returned error can't find the container with id f0c8172474a20e2246b3cfacac48622a9263bd25cdcb0d6b271482cde8ff8baf
Nov 01 15:54:51.888679 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:51.888640    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 15:54:51.888753 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:51.888711    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 15:54:51.888753 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:51.888739    4993 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 15:54:52.705807 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:52.705767    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-5pw89" event={"ID":"c09c8668-89c0-48ed-86da-fe404dc989e7","Type":"ContainerStarted","Data":"6fddad4873ecef8a5eb635d6f52310a58c58332ece82ce156f85653fef172fe3"}
Nov 01 15:54:52.706244 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:52.705829    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-5pw89" event={"ID":"c09c8668-89c0-48ed-86da-fe404dc989e7","Type":"ContainerStarted","Data":"f0c8172474a20e2246b3cfacac48622a9263bd25cdcb0d6b271482cde8ff8baf"}
Nov 01 15:54:52.908964 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:52.908912    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-5pw89" podStartSLOduration=1.908898564 podStartE2EDuration="1.908898564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:54:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:54:52.861624877 +0000 UTC m=+5880.191487615" watchObservedRunningTime="2026-11-01 15:54:52.908898564 +0000 UTC m=+5880.238761360"
Nov 01 15:54:53.468193 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:53.462851    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-5pw89"
Nov 01 15:54:53.468193 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:53.463001    4993 patch_prober.go:29] interesting pod/e2e-dual-3436ba-openshift-default-8446b54f85-5pw89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Nov 01 15:54:53.468193 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:53.463047    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-5pw89" podUID="c09c8668-89c0-48ed-86da-fe404dc989e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Nov 01 15:54:54.421386 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:54.421347    4993 scope.go:122] "RemoveContainer" containerID="c8333c3a75e3943eb287e584fff7330e58c96bcb80ed9157b7f178a308212995"
Nov 01 15:54:54.461603 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:54.461563    4993 patch_prober.go:29] interesting pod/e2e-dual-3436ba-openshift-default-8446b54f85-5pw89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Nov 01 15:54:54.461889 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:54.461857    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-5pw89" podUID="c09c8668-89c0-48ed-86da-fe404dc989e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Nov 01 15:54:55.463620 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:55.463578    4993 patch_prober.go:29] interesting pod/e2e-dual-3436ba-openshift-default-8446b54f85-5pw89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Nov 01 15:54:55.464287 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:55.464253    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-5pw89" podUID="c09c8668-89c0-48ed-86da-fe404dc989e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Nov 01 15:54:56.470191 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:56.464624    4993 patch_prober.go:29] interesting pod/e2e-dual-3436ba-openshift-default-8446b54f85-5pw89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Nov 01 15:54:56.470191 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:56.464684    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-5pw89" podUID="c09c8668-89c0-48ed-86da-fe404dc989e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Nov 01 15:54:57.469195 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:57.467273    4993 patch_prober.go:29] interesting pod/e2e-dual-3436ba-openshift-default-8446b54f85-5pw89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Nov 01 15:54:57.469195 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:57.467339    4993 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-5pw89" podUID="c09c8668-89c0-48ed-86da-fe404dc989e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Nov 01 15:54:57.502800 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:54:57.502766    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:54:58.463273 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:58.463240    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-5pw89"
Nov 01 15:54:58.463543 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:58.463522    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-5pw89"
Nov 01 15:54:58.464337 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:54:58.464320    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-5pw89"
Nov 01 15:55:27.499429 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:55:27.499393    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:55:54.443201 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:55:54.443152    4993 scope.go:122] "RemoveContainer" containerID="214922dcb2f6a4e59c4ac19cfaf4bcc25eabbef2e3b12daab40224bf8496bd71"
Nov 01 15:55:57.499533 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:55:57.499494    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:56:13.218560 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:13.218526    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-5pw89"]
Nov 01 15:56:13.219114 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:13.218790    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-5pw89" podUID="c09c8668-89c0-48ed-86da-fe404dc989e7" containerName="istio-proxy" containerID="cri-o://6fddad4873ecef8a5eb635d6f52310a58c58332ece82ce156f85653fef172fe3" gracePeriod=30
Nov 01 15:56:13.234092 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:13.234064    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-h7dbs"]
Nov 01 15:56:13.234324 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:13.234289    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-h7dbs" podUID="a6e524b3-6d28-42a6-975d-d19758cf1378" containerName="istio-proxy" containerID="cri-o://1fe78710aa2a3ed537b4c4edc49a10ad52b181820d14d8bee3d6d4664a307fab" gracePeriod=30
Nov 01 15:56:13.257851 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:13.257824    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-pnd69"]
Nov 01 15:56:13.258025 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:13.257994    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-pnd69" podUID="aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38" containerName="istio-proxy" containerID="cri-o://76accbdb06292d1a97ff24d91b26149c711c992a2b893e34336a0ec4d315b666" gracePeriod=30
Nov 01 15:56:13.369566 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:13.369531    4993 patch_prober.go:29] interesting pod/e2e-dual-3436ba-openshift-default-8446b54f85-pnd69 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 15:56:13.369725 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:13.369586    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-pnd69" podUID="aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 15:56:13.466594 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:13.466555    4993 patch_prober.go:29] interesting pod/e2e-dual-3436ba-openshift-default-8446b54f85-5pw89 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 15:56:13.466763 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:13.466607    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-5pw89" podUID="c09c8668-89c0-48ed-86da-fe404dc989e7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 15:56:18.518714 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:18.518684    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-5pw89"
Nov 01 15:56:18.571503 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:18.571471    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-h7dbs"
Nov 01 15:56:18.574693 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:18.574675    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-pnd69"
Nov 01 15:56:18.600596 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:18.600576    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c09c8668-89c0-48ed-86da-fe404dc989e7-istio-token\" (UniqueName: \"kubernetes.io/projected/c09c8668-89c0-48ed-86da-fe404dc989e7-istio-token\") pod \"c09c8668-89c0-48ed-86da-fe404dc989e7\" (UID: \"c09c8668-89c0-48ed-86da-fe404dc989e7\") "
Nov 01 15:56:18.600696 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:18.600619    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6e524b3-6d28-42a6-975d-d19758cf1378-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6e524b3-6d28-42a6-975d-d19758cf1378-istio-envoy\") pod \"a6e524b3-6d28-42a6-975d-d19758cf1378\" (UID: \"a6e524b3-6d28-42a6-975d-d19758cf1378\") "
Nov 01 15:56:18.600696 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:18.600657    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6e524b3-6d28-42a6-975d-d19758cf1378-kube-api-access-l2q6x\" (UniqueName: \"kubernetes.io/projected/a6e524b3-6d28-42a6-975d-d19758cf1378-kube-api-access-l2q6x\") pod \"a6e524b3-6d28-42a6-975d-d19758cf1378\" (UID: \"a6e524b3-6d28-42a6-975d-d19758cf1378\") "
Nov 01 15:56:18.600696 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:18.600686    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c09c8668-89c0-48ed-86da-fe404dc989e7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c09c8668-89c0-48ed-86da-fe404dc989e7-workload-certs\") pod \"c09c8668-89c0-48ed-86da-fe404dc989e7\" (UID: \"c09c8668-89c0-48ed-86da-fe404dc989e7\") "
Nov 01 15:56:18.600838 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:18.600724    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6e524b3-6d28-42a6-975d-d19758cf1378-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6e524b3-6d28-42a6-975d-d19758cf1378-workload-certs\") pod \"a6e524b3-6d28-42a6-975d-d19758cf1378\" (UID: \"a6e524b3-6d28-42a6-975d-d19758cf1378\") "
Nov 01 15:56:18.600838 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:18.600745    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a6e524b3-6d28-42a6-975d-d19758cf1378-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6e524b3-6d28-42a6-975d-d19758cf1378-istio-podinfo\") pod \"a6e524b3-6d28-42a6-975d-d19758cf1378\" (UID: \"a6e524b3-6d28-42a6-975d-d19758cf1378\") "
Nov 01 15:56:18.600838 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:18.600775    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c09c8668-89c0-48ed-86da-fe404dc989e7-kube-api-access-9ggcz\" (UniqueName: \"kubernetes.io/projected/c09c8668-89c0-48ed-86da-fe404dc989e7-kube-api-access-9ggcz\") pod \"c09c8668-89c0-48ed-86da-fe404dc989e7\" (UID: \"c09c8668-89c0-48ed-86da-fe404dc989e7\") "
Nov 01 15:56:18.600838 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:18.600813    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6e524b3-6d28-42a6-975d-d19758cf1378-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6e524b3-6d28-42a6-975d-d19758cf1378-istiod-ca-cert\") pod \"a6e524b3-6d28-42a6-975d-d19758cf1378\" (UID: \"a6e524b3-6d28-42a6-975d-d19758cf1378\") "
Nov 01 15:56:18.601049 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:18.600862    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6e524b3-6d28-42a6-975d-d19758cf1378-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6e524b3-6d28-42a6-975d-d19758cf1378-istio-data\") pod \"a6e524b3-6d28-42a6-975d-d19758cf1378\" (UID: \"a6e524b3-6d28-42a6-975d-d19758cf1378\") "
Nov 01 15:56:18.601049 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:18.600890    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6e524b3-6d28-42a6-975d-d19758cf1378-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6e524b3-6d28-42a6-975d-d19758cf1378-credential-socket\") pod \"a6e524b3-6d28-42a6-975d-d19758cf1378\" (UID: \"a6e524b3-6d28-42a6-975d-d19758cf1378\") "
Nov 01 15:56:18.601049 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:18.600922    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c09c8668-89c0-48ed-86da-fe404dc989e7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c09c8668-89c0-48ed-86da-fe404dc989e7-credential-socket\") pod \"c09c8668-89c0-48ed-86da-fe404dc989e7\" (UID: \"c09c8668-89c0-48ed-86da-fe404dc989e7\") "
Nov 01 15:56:18.601049 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:18.600973    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c09c8668-89c0-48ed-86da-fe404dc989e7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c09c8668-89c0-48ed-86da-fe404dc989e7-istio-podinfo\") pod \"c09c8668-89c0-48ed-86da-fe404dc989e7\" (UID: \"c09c8668-89c0-48ed-86da-fe404dc989e7\") "
Nov 01 15:56:18.601049 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:18.601001    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c09c8668-89c0-48ed-86da-fe404dc989e7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c09c8668-89c0-48ed-86da-fe404dc989e7-istiod-ca-cert\") pod \"c09c8668-89c0-48ed-86da-fe404dc989e7\" (UID: \"c09c8668-89c0-48ed-86da-fe404dc989e7\") "
Nov 01 15:56:18.601049 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:18.601028    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6e524b3-6d28-42a6-975d-d19758cf1378-workload-certs" pod "a6e524b3-6d28-42a6-975d-d19758cf1378" (UID: "a6e524b3-6d28-42a6-975d-d19758cf1378"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:56:18.601049 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:18.601046    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c09c8668-89c0-48ed-86da-fe404dc989e7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c09c8668-89c0-48ed-86da-fe404dc989e7-istio-data\") pod \"c09c8668-89c0-48ed-86da-fe404dc989e7\" (UID: \"c09c8668-89c0-48ed-86da-fe404dc989e7\") "
Nov 01 15:56:18.601461 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:18.601087    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c09c8668-89c0-48ed-86da-fe404dc989e7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c09c8668-89c0-48ed-86da-fe404dc989e7-workload-socket\") pod \"c09c8668-89c0-48ed-86da-fe404dc989e7\" (UID: \"c09c8668-89c0-48ed-86da-fe404dc989e7\") "
Nov 01 15:56:18.601461 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:18.601119    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6e524b3-6d28-42a6-975d-d19758cf1378-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6e524b3-6d28-42a6-975d-d19758cf1378-workload-socket\") pod \"a6e524b3-6d28-42a6-975d-d19758cf1378\" (UID: \"a6e524b3-6d28-42a6-975d-d19758cf1378\") "
Nov 01 15:56:18.601461 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:18.601146    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c09c8668-89c0-48ed-86da-fe404dc989e7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c09c8668-89c0-48ed-86da-fe404dc989e7-istio-envoy\") pod \"c09c8668-89c0-48ed-86da-fe404dc989e7\" (UID: \"c09c8668-89c0-48ed-86da-fe404dc989e7\") "
Nov 01 15:56:18.601461 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:18.601243    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6e524b3-6d28-42a6-975d-d19758cf1378-istio-token\" (UniqueName: \"kubernetes.io/projected/a6e524b3-6d28-42a6-975d-d19758cf1378-istio-token\") pod \"a6e524b3-6d28-42a6-975d-d19758cf1378\" (UID: \"a6e524b3-6d28-42a6-975d-d19758cf1378\") "
Nov 01 15:56:18.601461 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:18.601284    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c09c8668-89c0-48ed-86da-fe404dc989e7-workload-certs" pod "c09c8668-89c0-48ed-86da-fe404dc989e7" (UID: "c09c8668-89c0-48ed-86da-fe404dc989e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:56:18.601461 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:18.601294    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6e524b3-6d28-42a6-975d-d19758cf1378-credential-socket" pod "a6e524b3-6d28-42a6-975d-d19758cf1378" (UID: "a6e524b3-6d28-42a6-975d-d19758cf1378"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:56:18.601765 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:18.601495    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6e524b3-6d28-42a6-975d-d19758cf1378-istio-data" pod "a6e524b3-6d28-42a6-975d-d19758cf1378" (UID: "a6e524b3-6d28-42a6-975d-d19758cf1378"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:56:18.601765 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:18.601639    4993 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6e524b3-6d28-42a6-975d-d19758cf1378-workload-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:56:18.601765 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:18.601643    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6e524b3-6d28-42a6-975d-d19758cf1378-istiod-ca-cert" pod "a6e524b3-6d28-42a6-975d-d19758cf1378" (UID: "a6e524b3-6d28-42a6-975d-d19758cf1378"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:56:18.601765 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:18.601657    4993 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6e524b3-6d28-42a6-975d-d19758cf1378-istio-data\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:56:18.601765 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:18.601673    4993 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6e524b3-6d28-42a6-975d-d19758cf1378-credential-socket\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:56:18.601765 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:18.601689    4993 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c09c8668-89c0-48ed-86da-fe404dc989e7-workload-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:56:18.602132 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:18.601890    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6e524b3-6d28-42a6-975d-d19758cf1378-workload-socket" pod "a6e524b3-6d28-42a6-975d-d19758cf1378" (UID: "a6e524b3-6d28-42a6-975d-d19758cf1378"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:56:18.602132 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:18.601902    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c09c8668-89c0-48ed-86da-fe404dc989e7-istio-data" pod "c09c8668-89c0-48ed-86da-fe404dc989e7" (UID: "c09c8668-89c0-48ed-86da-fe404dc989e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:56:18.602132 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:18.602094    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c09c8668-89c0-48ed-86da-fe404dc989e7-workload-socket" pod "c09c8668-89c0-48ed-86da-fe404dc989e7" (UID: "c09c8668-89c0-48ed-86da-fe404dc989e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:56:18.602132 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:18.602111    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c09c8668-89c0-48ed-86da-fe404dc989e7-credential-socket" pod "c09c8668-89c0-48ed-86da-fe404dc989e7" (UID: "c09c8668-89c0-48ed-86da-fe404dc989e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:56:18.602458 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:18.602432    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c09c8668-89c0-48ed-86da-fe404dc989e7-istiod-ca-cert" pod "c09c8668-89c0-48ed-86da-fe404dc989e7" (UID: "c09c8668-89c0-48ed-86da-fe404dc989e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:56:18.603504 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:18.603479    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6e524b3-6d28-42a6-975d-d19758cf1378-kube-api-access-l2q6x" pod "a6e524b3-6d28-42a6-975d-d19758cf1378" (UID: "a6e524b3-6d28-42a6-975d-d19758cf1378"). InnerVolumeSpecName "kube-api-access-l2q6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:56:18.603606 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:18.603492    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6e524b3-6d28-42a6-975d-d19758cf1378-istio-envoy" pod "a6e524b3-6d28-42a6-975d-d19758cf1378" (UID: "a6e524b3-6d28-42a6-975d-d19758cf1378"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:56:18.603606 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:18.603512    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c09c8668-89c0-48ed-86da-fe404dc989e7-istio-token" pod "c09c8668-89c0-48ed-86da-fe404dc989e7" (UID: "c09c8668-89c0-48ed-86da-fe404dc989e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:56:18.603850 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:18.603823    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c09c8668-89c0-48ed-86da-fe404dc989e7-kube-api-access-9ggcz" pod "c09c8668-89c0-48ed-86da-fe404dc989e7" (UID: "c09c8668-89c0-48ed-86da-fe404dc989e7"). InnerVolumeSpecName "kube-api-access-9ggcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:56:18.603972 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:18.603937    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6e524b3-6d28-42a6-975d-d19758cf1378-istio-token" pod "a6e524b3-6d28-42a6-975d-d19758cf1378" (UID: "a6e524b3-6d28-42a6-975d-d19758cf1378"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:56:18.604262 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:18.604239    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c09c8668-89c0-48ed-86da-fe404dc989e7-istio-podinfo" pod "c09c8668-89c0-48ed-86da-fe404dc989e7" (UID: "c09c8668-89c0-48ed-86da-fe404dc989e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 15:56:18.604563 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:18.604544    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c09c8668-89c0-48ed-86da-fe404dc989e7-istio-envoy" pod "c09c8668-89c0-48ed-86da-fe404dc989e7" (UID: "c09c8668-89c0-48ed-86da-fe404dc989e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:56:18.604734 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:18.604710    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a6e524b3-6d28-42a6-975d-d19758cf1378-istio-podinfo" pod "a6e524b3-6d28-42a6-975d-d19758cf1378" (UID: "a6e524b3-6d28-42a6-975d-d19758cf1378"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 15:56:18.702352 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:18.702318    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38-workload-socket\") pod \"aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38\" (UID: \"aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38\") "
Nov 01 15:56:18.702533 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:18.702369    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38-workload-certs\") pod \"aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38\" (UID: \"aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38\") "
Nov 01 15:56:18.702533 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:18.702400    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38-istio-podinfo\") pod \"aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38\" (UID: \"aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38\") "
Nov 01 15:56:18.702533 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:18.702458    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38-credential-socket\") pod \"aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38\" (UID: \"aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38\") "
Nov 01 15:56:18.702533 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:18.702494    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38-istio-envoy\") pod \"aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38\" (UID: \"aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38\") "
Nov 01 15:56:18.702776 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:18.702567    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38-istio-token\" (UniqueName: \"kubernetes.io/projected/aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38-istio-token\") pod \"aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38\" (UID: \"aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38\") "
Nov 01 15:56:18.702776 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:18.702604    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38-kube-api-access-m2jr6\" (UniqueName: \"kubernetes.io/projected/aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38-kube-api-access-m2jr6\") pod \"aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38\" (UID: \"aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38\") "
Nov 01 15:56:18.702776 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:18.702641    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38-istio-data\") pod \"aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38\" (UID: \"aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38\") "
Nov 01 15:56:18.702776 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:18.702717    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38-workload-certs" pod "aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38" (UID: "aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:56:18.702776 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:18.702741    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38-istiod-ca-cert\") pod \"aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38\" (UID: \"aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38\") "
Nov 01 15:56:18.703021 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:18.702848    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38-credential-socket" pod "aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38" (UID: "aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:56:18.703078 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:18.703030    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38-workload-socket" pod "aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38" (UID: "aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:56:18.703141 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:18.703076    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38-istiod-ca-cert" pod "aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38" (UID: "aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 15:56:18.703311 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:18.703270    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38-istio-data" pod "aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38" (UID: "aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:56:18.703377 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:18.703339    4993 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6e524b3-6d28-42a6-975d-d19758cf1378-istio-token\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:56:18.703377 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:18.703360    4993 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c09c8668-89c0-48ed-86da-fe404dc989e7-istio-token\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:56:18.703377 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:18.703374    4993 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6e524b3-6d28-42a6-975d-d19758cf1378-istio-envoy\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:56:18.703494 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:18.703392    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2q6x\" (UniqueName: \"kubernetes.io/projected/a6e524b3-6d28-42a6-975d-d19758cf1378-kube-api-access-l2q6x\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:56:18.703494 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:18.703409    4993 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38-istiod-ca-cert\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:56:18.703494 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:18.703422    4993 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6e524b3-6d28-42a6-975d-d19758cf1378-istio-podinfo\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:56:18.703494 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:18.703436    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ggcz\" (UniqueName: \"kubernetes.io/projected/c09c8668-89c0-48ed-86da-fe404dc989e7-kube-api-access-9ggcz\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:56:18.703494 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:18.703451    4993 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6e524b3-6d28-42a6-975d-d19758cf1378-istiod-ca-cert\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:56:18.703494 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:18.703465    4993 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38-workload-socket\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:56:18.703494 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:18.703479    4993 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38-workload-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:56:18.703494 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:18.703491    4993 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c09c8668-89c0-48ed-86da-fe404dc989e7-credential-socket\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:56:18.703745 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:18.703506    4993 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c09c8668-89c0-48ed-86da-fe404dc989e7-istio-podinfo\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:56:18.703745 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:18.703522    4993 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38-credential-socket\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:56:18.703745 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:18.703535    4993 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c09c8668-89c0-48ed-86da-fe404dc989e7-istiod-ca-cert\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:56:18.703745 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:18.703548    4993 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c09c8668-89c0-48ed-86da-fe404dc989e7-istio-data\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:56:18.703745 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:18.703560    4993 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c09c8668-89c0-48ed-86da-fe404dc989e7-workload-socket\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:56:18.703745 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:18.703573    4993 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6e524b3-6d28-42a6-975d-d19758cf1378-workload-socket\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:56:18.703745 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:18.703586    4993 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c09c8668-89c0-48ed-86da-fe404dc989e7-istio-envoy\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:56:18.705280 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:18.705238    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38-istio-podinfo" pod "aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38" (UID: "aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 15:56:18.705280 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:18.705249    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38-kube-api-access-m2jr6" pod "aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38" (UID: "aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38"). InnerVolumeSpecName "kube-api-access-m2jr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:56:18.706508 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:18.706479    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38-istio-envoy" pod "aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38" (UID: "aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 15:56:18.706601 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:18.706515    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38-istio-token" pod "aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38" (UID: "aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 15:56:18.804315 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:18.804221    4993 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38-istio-podinfo\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:56:18.804315 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:18.804257    4993 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38-istio-envoy\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:56:18.804315 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:18.804272    4993 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38-istio-token\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:56:18.804315 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:18.804287    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2jr6\" (UniqueName: \"kubernetes.io/projected/aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38-kube-api-access-m2jr6\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:56:18.804315 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:18.804301    4993 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38-istio-data\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 15:56:19.150245 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:19.150141    4993 generic.go:356] "Generic (PLEG): container finished" podID="a6e524b3-6d28-42a6-975d-d19758cf1378" containerID="1fe78710aa2a3ed537b4c4edc49a10ad52b181820d14d8bee3d6d4664a307fab" exitCode=0
Nov 01 15:56:19.150245 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:19.150230    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-h7dbs"
Nov 01 15:56:19.150481 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:19.150229    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-h7dbs" event={"ID":"a6e524b3-6d28-42a6-975d-d19758cf1378","Type":"ContainerDied","Data":"1fe78710aa2a3ed537b4c4edc49a10ad52b181820d14d8bee3d6d4664a307fab"}
Nov 01 15:56:19.150481 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:19.150382    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-h7dbs" event={"ID":"a6e524b3-6d28-42a6-975d-d19758cf1378","Type":"ContainerDied","Data":"893376d71c4b04f74806235458d6227c2210338e1e9dbdb099d16d7d1ab59c08"}
Nov 01 15:56:19.150481 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:19.150412    4993 scope.go:122] "RemoveContainer" containerID="1fe78710aa2a3ed537b4c4edc49a10ad52b181820d14d8bee3d6d4664a307fab"
Nov 01 15:56:19.151796 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:19.151770    4993 generic.go:356] "Generic (PLEG): container finished" podID="aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38" containerID="76accbdb06292d1a97ff24d91b26149c711c992a2b893e34336a0ec4d315b666" exitCode=0
Nov 01 15:56:19.151935 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:19.151834    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-pnd69"
Nov 01 15:56:19.151935 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:19.151856    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-pnd69" event={"ID":"aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38","Type":"ContainerDied","Data":"76accbdb06292d1a97ff24d91b26149c711c992a2b893e34336a0ec4d315b666"}
Nov 01 15:56:19.152522 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:19.151911    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-pnd69" event={"ID":"aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38","Type":"ContainerDied","Data":"b0c4a45ff61fe4c3f9c67071bc5af5d508b8c1fd29ad78f64a8a2ff6c4e07e39"}
Nov 01 15:56:19.153495 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:19.153469    4993 generic.go:356] "Generic (PLEG): container finished" podID="c09c8668-89c0-48ed-86da-fe404dc989e7" containerID="6fddad4873ecef8a5eb635d6f52310a58c58332ece82ce156f85653fef172fe3" exitCode=0
Nov 01 15:56:19.153617 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:19.153507    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-5pw89"
Nov 01 15:56:19.153617 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:19.153559    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-5pw89" event={"ID":"c09c8668-89c0-48ed-86da-fe404dc989e7","Type":"ContainerDied","Data":"6fddad4873ecef8a5eb635d6f52310a58c58332ece82ce156f85653fef172fe3"}
Nov 01 15:56:19.153617 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:19.153598    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-5pw89" event={"ID":"c09c8668-89c0-48ed-86da-fe404dc989e7","Type":"ContainerDied","Data":"f0c8172474a20e2246b3cfacac48622a9263bd25cdcb0d6b271482cde8ff8baf"}
Nov 01 15:56:19.161034 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:19.161016    4993 scope.go:122] "RemoveContainer" containerID="1fe78710aa2a3ed537b4c4edc49a10ad52b181820d14d8bee3d6d4664a307fab"
Nov 01 15:56:19.161396 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:56:19.161377    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fe78710aa2a3ed537b4c4edc49a10ad52b181820d14d8bee3d6d4664a307fab\": container with ID starting with 1fe78710aa2a3ed537b4c4edc49a10ad52b181820d14d8bee3d6d4664a307fab not found: ID does not exist" containerID="1fe78710aa2a3ed537b4c4edc49a10ad52b181820d14d8bee3d6d4664a307fab"
Nov 01 15:56:19.161450 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:19.161403    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fe78710aa2a3ed537b4c4edc49a10ad52b181820d14d8bee3d6d4664a307fab"} err="failed to get container status \"1fe78710aa2a3ed537b4c4edc49a10ad52b181820d14d8bee3d6d4664a307fab\": rpc error: code = NotFound desc = could not find container \"1fe78710aa2a3ed537b4c4edc49a10ad52b181820d14d8bee3d6d4664a307fab\": container with ID starting with 1fe78710aa2a3ed537b4c4edc49a10ad52b181820d14d8bee3d6d4664a307fab not found: ID does not exist"
Nov 01 15:56:19.161450 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:19.161419    4993 scope.go:122] "RemoveContainer" containerID="76accbdb06292d1a97ff24d91b26149c711c992a2b893e34336a0ec4d315b666"
Nov 01 15:56:19.169423 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:19.169407    4993 scope.go:122] "RemoveContainer" containerID="76accbdb06292d1a97ff24d91b26149c711c992a2b893e34336a0ec4d315b666"
Nov 01 15:56:19.169677 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:56:19.169659    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76accbdb06292d1a97ff24d91b26149c711c992a2b893e34336a0ec4d315b666\": container with ID starting with 76accbdb06292d1a97ff24d91b26149c711c992a2b893e34336a0ec4d315b666 not found: ID does not exist" containerID="76accbdb06292d1a97ff24d91b26149c711c992a2b893e34336a0ec4d315b666"
Nov 01 15:56:19.169725 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:19.169683    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76accbdb06292d1a97ff24d91b26149c711c992a2b893e34336a0ec4d315b666"} err="failed to get container status \"76accbdb06292d1a97ff24d91b26149c711c992a2b893e34336a0ec4d315b666\": rpc error: code = NotFound desc = could not find container \"76accbdb06292d1a97ff24d91b26149c711c992a2b893e34336a0ec4d315b666\": container with ID starting with 76accbdb06292d1a97ff24d91b26149c711c992a2b893e34336a0ec4d315b666 not found: ID does not exist"
Nov 01 15:56:19.169725 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:19.169699    4993 scope.go:122] "RemoveContainer" containerID="6fddad4873ecef8a5eb635d6f52310a58c58332ece82ce156f85653fef172fe3"
Nov 01 15:56:19.177782 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:19.177762    4993 scope.go:122] "RemoveContainer" containerID="6fddad4873ecef8a5eb635d6f52310a58c58332ece82ce156f85653fef172fe3"
Nov 01 15:56:19.178028 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:56:19.178010    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fddad4873ecef8a5eb635d6f52310a58c58332ece82ce156f85653fef172fe3\": container with ID starting with 6fddad4873ecef8a5eb635d6f52310a58c58332ece82ce156f85653fef172fe3 not found: ID does not exist" containerID="6fddad4873ecef8a5eb635d6f52310a58c58332ece82ce156f85653fef172fe3"
Nov 01 15:56:19.178085 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:19.178035    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fddad4873ecef8a5eb635d6f52310a58c58332ece82ce156f85653fef172fe3"} err="failed to get container status \"6fddad4873ecef8a5eb635d6f52310a58c58332ece82ce156f85653fef172fe3\": rpc error: code = NotFound desc = could not find container \"6fddad4873ecef8a5eb635d6f52310a58c58332ece82ce156f85653fef172fe3\": container with ID starting with 6fddad4873ecef8a5eb635d6f52310a58c58332ece82ce156f85653fef172fe3 not found: ID does not exist"
Nov 01 15:56:19.302852 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:19.302816    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-5pw89"]
Nov 01 15:56:19.349072 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:19.349037    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-5pw89"]
Nov 01 15:56:19.449978 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:19.449895    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-h7dbs"]
Nov 01 15:56:19.495946 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:19.495915    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-h7dbs"]
Nov 01 15:56:19.554724 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:19.554689    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-pnd69"]
Nov 01 15:56:19.602196 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:19.602129    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-pnd69"]
Nov 01 15:56:20.796426 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:20.796392    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6e524b3-6d28-42a6-975d-d19758cf1378" path="/var/lib/kubelet/pods/a6e524b3-6d28-42a6-975d-d19758cf1378/volumes"
Nov 01 15:56:20.796922 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:20.796906    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38" path="/var/lib/kubelet/pods/aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38/volumes"
Nov 01 15:56:20.797389 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:20.797374    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c09c8668-89c0-48ed-86da-fe404dc989e7" path="/var/lib/kubelet/pods/c09c8668-89c0-48ed-86da-fe404dc989e7/volumes"
Nov 01 15:56:27.500193 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:56:27.500140    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:56:53.639954 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:53.639922    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xmwtp_bb59051d-c2df-4362-a157-00f14f915e79/console-operator/3.log"
Nov 01 15:56:53.650840 ip-10-0-134-37 kubenswrapper[4993]: I1101 15:56:53.650816    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xmwtp_bb59051d-c2df-4362-a157-00f14f915e79/console-operator/3.log"
Nov 01 15:56:57.498894 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:56:57.498859    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:57:27.499819 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:57:27.499781    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:57:57.499949 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:57:57.499911    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:58:27.500157 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:58:27.500076    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:58:57.500278 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:58:57.500239    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:59:27.545980 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:59:27.545939    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 15:59:57.499385 ip-10-0-134-37 kubenswrapper[4993]: E1101 15:59:57.499348    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 16:00:27.499518 ip-10-0-134-37 kubenswrapper[4993]: E1101 16:00:27.499481    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 16:00:57.500091 ip-10-0-134-37 kubenswrapper[4993]: E1101 16:00:57.500050    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 16:01:25.173432 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:01:25.173326    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-q9j4g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 16:01:25.173432 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:01:25.173402    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q9j4g" podUID="29c351ad-629b-418c-b64d-9ff0e52ccdcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 16:01:27.499915 ip-10-0-134-37 kubenswrapper[4993]: E1101 16:01:27.499878    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 16:01:53.672803 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:01:53.672767    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xmwtp_bb59051d-c2df-4362-a157-00f14f915e79/console-operator/3.log"
Nov 01 16:01:53.685725 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:01:53.685699    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xmwtp_bb59051d-c2df-4362-a157-00f14f915e79/console-operator/3.log"
Nov 01 16:01:57.499430 ip-10-0-134-37 kubenswrapper[4993]: E1101 16:01:57.499389    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 16:02:27.502982 ip-10-0-134-37 kubenswrapper[4993]: E1101 16:02:27.502935    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 16:02:57.556029 ip-10-0-134-37 kubenswrapper[4993]: E1101 16:02:57.555974    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 16:02:58.859192 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:02:58.855350    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-4pmtn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 16:02:58.859192 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:02:58.855422    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4pmtn" podUID="306b0405-fb73-435a-ae73-42a90cd2d2f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 16:03:27.500131 ip-10-0-134-37 kubenswrapper[4993]: E1101 16:03:27.500092    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 16:03:57.500062 ip-10-0-134-37 kubenswrapper[4993]: E1101 16:03:57.500024    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 16:04:21.036478 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:04:21.036396    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-9ebec3/simulator-single-main-9ebec3-kserve-67c659b995-4mjdf"]
Nov 01 16:04:21.037068 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:04:21.037004    4993 cpu_manager.go:414] "containerMap: removing container" podUID="c09c8668-89c0-48ed-86da-fe404dc989e7" containerName="istio-proxy"
Nov 01 16:04:21.037068 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:04:21.037028    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c09c8668-89c0-48ed-86da-fe404dc989e7" containerName="istio-proxy"
Nov 01 16:04:21.037068 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:04:21.037058    4993 cpu_manager.go:414] "containerMap: removing container" podUID="aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38" containerName="istio-proxy"
Nov 01 16:04:21.037068 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:04:21.037066    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38" containerName="istio-proxy"
Nov 01 16:04:21.037333 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:04:21.037083    4993 cpu_manager.go:414] "containerMap: removing container" podUID="a6e524b3-6d28-42a6-975d-d19758cf1378" containerName="istio-proxy"
Nov 01 16:04:21.037333 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:04:21.037111    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6e524b3-6d28-42a6-975d-d19758cf1378" containerName="istio-proxy"
Nov 01 16:04:21.037333 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:04:21.037277    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="c09c8668-89c0-48ed-86da-fe404dc989e7" containerName="istio-proxy"
Nov 01 16:04:21.037333 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:04:21.037301    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="aae6e7f9-ab68-4cb7-8c3b-4c372e0f5b38" containerName="istio-proxy"
Nov 01 16:04:21.037333 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:04:21.037313    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6e524b3-6d28-42a6-975d-d19758cf1378" containerName="istio-proxy"
Nov 01 16:04:21.040572 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:04:21.040550    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-9ebec3/simulator-single-main-9ebec3-kserve-67c659b995-4mjdf"
Nov 01 16:04:21.135087 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:04:21.135053    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-9ebec3\"/\"kube-root-ca.crt\""
Nov 01 16:04:21.135358 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:04:21.135341    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-9ebec3\"/\"openshift-service-ca.crt\""
Nov 01 16:04:21.135444 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:04:21.135393    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-9ebec3\"/\"simulator-single-main-9ebec3-kserve-self-signed-certs\""
Nov 01 16:04:21.135444 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:04:21.135438    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-9ebec3\"/\"default-dockercfg-2kznr\""
Nov 01 16:04:21.137936 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:04:21.137910    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-9ebec3/simulator-single-main-9ebec3-kserve-67c659b995-4mjdf"]
Nov 01 16:04:21.225082 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:04:21.225040    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff-tls-certs\") pod \"simulator-single-main-9ebec3-kserve-67c659b995-4mjdf\" (UID: \"dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff\") " pod="e2e-models-e2e-worker-single-main-9ebec3/simulator-single-main-9ebec3-kserve-67c659b995-4mjdf"
Nov 01 16:04:21.225082 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:04:21.225084    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff-home\") pod \"simulator-single-main-9ebec3-kserve-67c659b995-4mjdf\" (UID: \"dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff\") " pod="e2e-models-e2e-worker-single-main-9ebec3/simulator-single-main-9ebec3-kserve-67c659b995-4mjdf"
Nov 01 16:04:21.225322 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:04:21.225235    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff-model-cache\") pod \"simulator-single-main-9ebec3-kserve-67c659b995-4mjdf\" (UID: \"dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff\") " pod="e2e-models-e2e-worker-single-main-9ebec3/simulator-single-main-9ebec3-kserve-67c659b995-4mjdf"
Nov 01 16:04:21.225322 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:04:21.225266    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgc9q\" (UniqueName: \"kubernetes.io/projected/dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff-kube-api-access-pgc9q\") pod \"simulator-single-main-9ebec3-kserve-67c659b995-4mjdf\" (UID: \"dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff\") " pod="e2e-models-e2e-worker-single-main-9ebec3/simulator-single-main-9ebec3-kserve-67c659b995-4mjdf"
Nov 01 16:04:21.225395 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:04:21.225365    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff-dshm\") pod \"simulator-single-main-9ebec3-kserve-67c659b995-4mjdf\" (UID: \"dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff\") " pod="e2e-models-e2e-worker-single-main-9ebec3/simulator-single-main-9ebec3-kserve-67c659b995-4mjdf"
Nov 01 16:04:21.225433 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:04:21.225407    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff-tmp-dir\") pod \"simulator-single-main-9ebec3-kserve-67c659b995-4mjdf\" (UID: \"dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff\") " pod="e2e-models-e2e-worker-single-main-9ebec3/simulator-single-main-9ebec3-kserve-67c659b995-4mjdf"
Nov 01 16:04:21.326864 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:04:21.326773    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff-dshm\") pod \"simulator-single-main-9ebec3-kserve-67c659b995-4mjdf\" (UID: \"dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff\") " pod="e2e-models-e2e-worker-single-main-9ebec3/simulator-single-main-9ebec3-kserve-67c659b995-4mjdf"
Nov 01 16:04:21.326864 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:04:21.326834    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff-tmp-dir\") pod \"simulator-single-main-9ebec3-kserve-67c659b995-4mjdf\" (UID: \"dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff\") " pod="e2e-models-e2e-worker-single-main-9ebec3/simulator-single-main-9ebec3-kserve-67c659b995-4mjdf"
Nov 01 16:04:21.326864 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:04:21.326862    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff-tls-certs\") pod \"simulator-single-main-9ebec3-kserve-67c659b995-4mjdf\" (UID: \"dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff\") " pod="e2e-models-e2e-worker-single-main-9ebec3/simulator-single-main-9ebec3-kserve-67c659b995-4mjdf"
Nov 01 16:04:21.327148 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:04:21.326884    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff-home\") pod \"simulator-single-main-9ebec3-kserve-67c659b995-4mjdf\" (UID: \"dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff\") " pod="e2e-models-e2e-worker-single-main-9ebec3/simulator-single-main-9ebec3-kserve-67c659b995-4mjdf"
Nov 01 16:04:21.327148 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:04:21.326924    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff-model-cache\") pod \"simulator-single-main-9ebec3-kserve-67c659b995-4mjdf\" (UID: \"dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff\") " pod="e2e-models-e2e-worker-single-main-9ebec3/simulator-single-main-9ebec3-kserve-67c659b995-4mjdf"
Nov 01 16:04:21.327148 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:04:21.326950    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgc9q\" (UniqueName: \"kubernetes.io/projected/dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff-kube-api-access-pgc9q\") pod \"simulator-single-main-9ebec3-kserve-67c659b995-4mjdf\" (UID: \"dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff\") " pod="e2e-models-e2e-worker-single-main-9ebec3/simulator-single-main-9ebec3-kserve-67c659b995-4mjdf"
Nov 01 16:04:21.327425 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:04:21.327394    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff-tmp-dir\") pod \"simulator-single-main-9ebec3-kserve-67c659b995-4mjdf\" (UID: \"dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff\") " pod="e2e-models-e2e-worker-single-main-9ebec3/simulator-single-main-9ebec3-kserve-67c659b995-4mjdf"
Nov 01 16:04:21.327603 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:04:21.327582    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff-model-cache\") pod \"simulator-single-main-9ebec3-kserve-67c659b995-4mjdf\" (UID: \"dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff\") " pod="e2e-models-e2e-worker-single-main-9ebec3/simulator-single-main-9ebec3-kserve-67c659b995-4mjdf"
Nov 01 16:04:21.327683 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:04:21.327613    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff-home\") pod \"simulator-single-main-9ebec3-kserve-67c659b995-4mjdf\" (UID: \"dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff\") " pod="e2e-models-e2e-worker-single-main-9ebec3/simulator-single-main-9ebec3-kserve-67c659b995-4mjdf"
Nov 01 16:04:21.329218 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:04:21.329193    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff-dshm\") pod \"simulator-single-main-9ebec3-kserve-67c659b995-4mjdf\" (UID: \"dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff\") " pod="e2e-models-e2e-worker-single-main-9ebec3/simulator-single-main-9ebec3-kserve-67c659b995-4mjdf"
Nov 01 16:04:21.329788 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:04:21.329764    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff-tls-certs\") pod \"simulator-single-main-9ebec3-kserve-67c659b995-4mjdf\" (UID: \"dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff\") " pod="e2e-models-e2e-worker-single-main-9ebec3/simulator-single-main-9ebec3-kserve-67c659b995-4mjdf"
Nov 01 16:04:21.379860 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:04:21.379820    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgc9q\" (UniqueName: \"kubernetes.io/projected/dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff-kube-api-access-pgc9q\") pod \"simulator-single-main-9ebec3-kserve-67c659b995-4mjdf\" (UID: \"dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff\") " pod="e2e-models-e2e-worker-single-main-9ebec3/simulator-single-main-9ebec3-kserve-67c659b995-4mjdf"
Nov 01 16:04:21.655948 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:04:21.655857    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-9ebec3/simulator-single-main-9ebec3-kserve-67c659b995-4mjdf"
Nov 01 16:04:22.028542 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:04:22.028507    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-9ebec3/simulator-single-main-9ebec3-kserve-67c659b995-4mjdf"]
Nov 01 16:04:22.036380 ip-10-0-134-37 kubenswrapper[4993]: W1101 16:04:22.036352    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddc73d7f8_e7ec_4f3a_9a6f_26bea3ef80ff.slice/crio-a452edd73153ef7ba260b89d5a9dbceb9f15b1dc8e4bc80a73d30073a7140ffb WatchSource:0}: Error finding container a452edd73153ef7ba260b89d5a9dbceb9f15b1dc8e4bc80a73d30073a7140ffb: Status 404 returned error can't find the container with id a452edd73153ef7ba260b89d5a9dbceb9f15b1dc8e4bc80a73d30073a7140ffb
Nov 01 16:04:22.761706 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:04:22.761670    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-9ebec3/simulator-single-main-9ebec3-kserve-67c659b995-4mjdf" event={"ID":"dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff","Type":"ContainerStarted","Data":"4f1c88d6e1dd41cd3037a04fab3d0e66f468739fa55a23ad64a5b866e394ab47"}
Nov 01 16:04:22.762119 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:04:22.761716    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-9ebec3/simulator-single-main-9ebec3-kserve-67c659b995-4mjdf" event={"ID":"dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff","Type":"ContainerStarted","Data":"a452edd73153ef7ba260b89d5a9dbceb9f15b1dc8e4bc80a73d30073a7140ffb"}
Nov 01 16:04:22.867319 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:04:22.867264    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-9ebec3/simulator-single-main-9ebec3-kserve-67c659b995-4mjdf" podStartSLOduration=2.867246559 podStartE2EDuration="2.867246559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:04:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:04:22.866409829 +0000 UTC m=+6450.196272567" watchObservedRunningTime="2026-11-01 16:04:22.867246559 +0000 UTC m=+6450.197109298"
Nov 01 16:04:23.656856 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:04:23.656825    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-9ebec3/simulator-single-main-9ebec3-kserve-67c659b995-4mjdf"
Nov 01 16:04:27.500557 ip-10-0-134-37 kubenswrapper[4993]: E1101 16:04:27.500520    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 16:04:31.656908 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:04:31.656861    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-9ebec3/simulator-single-main-9ebec3-kserve-67c659b995-4mjdf"
Nov 01 16:04:31.669768 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:04:31.669735    4993 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-9ebec3/simulator-single-main-9ebec3-kserve-67c659b995-4mjdf"
Nov 01 16:04:31.816116 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:04:31.816087    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-9ebec3/simulator-single-main-9ebec3-kserve-67c659b995-4mjdf"
Nov 01 16:04:57.499870 ip-10-0-134-37 kubenswrapper[4993]: E1101 16:04:57.499832    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 16:05:21.389778 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:05:21.389740    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-9ebec3/simulator-single-main-9ebec3-kserve-67c659b995-4mjdf"]
Nov 01 16:05:27.500573 ip-10-0-134-37 kubenswrapper[4993]: E1101 16:05:27.500532    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 16:05:36.410210 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:05:36.410101    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-9ebec3/simulator-single-main-9ebec3-kserve-67c659b995-4mjdf" podUID="dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff" containerName="main" containerID="cri-o://4f1c88d6e1dd41cd3037a04fab3d0e66f468739fa55a23ad64a5b866e394ab47" gracePeriod=45
Nov 01 16:05:36.665498 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:05:36.665442    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-9ebec3/simulator-single-main-9ebec3-kserve-67c659b995-4mjdf"
Nov 01 16:05:36.860983 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:05:36.860951    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff-kube-api-access-pgc9q\" (UniqueName: \"kubernetes.io/projected/dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff-kube-api-access-pgc9q\") pod \"dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff\" (UID: \"dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff\") "
Nov 01 16:05:36.861163 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:05:36.860999    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff-home\" (UniqueName: \"kubernetes.io/empty-dir/dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff-home\") pod \"dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff\" (UID: \"dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff\") "
Nov 01 16:05:36.861163 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:05:36.861076    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff-tls-certs\" (UniqueName: \"kubernetes.io/secret/dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff-tls-certs\") pod \"dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff\" (UID: \"dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff\") "
Nov 01 16:05:36.861163 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:05:36.861095    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff-model-cache\") pod \"dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff\" (UID: \"dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff\") "
Nov 01 16:05:36.861163 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:05:36.861140    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff-tmp-dir\") pod \"dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff\" (UID: \"dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff\") "
Nov 01 16:05:36.861418 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:05:36.861215    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff-dshm\") pod \"dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff\" (UID: \"dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff\") "
Nov 01 16:05:36.861418 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:05:36.861359    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff-home" pod "dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff" (UID: "dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 16:05:36.861523 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:05:36.861411    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff-model-cache" pod "dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff" (UID: "dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 16:05:36.861630 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:05:36.861610    4993 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff-model-cache\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 16:05:36.861695 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:05:36.861637    4993 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff-home\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 16:05:36.861756 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:05:36.861698    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff-tmp-dir" pod "dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff" (UID: "dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 16:05:36.863307 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:05:36.863277    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff-tls-certs" pod "dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff" (UID: "dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 16:05:36.863416 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:05:36.863340    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff-dshm" pod "dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff" (UID: "dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 16:05:36.863416 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:05:36.863361    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff-kube-api-access-pgc9q" pod "dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff" (UID: "dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff"). InnerVolumeSpecName "kube-api-access-pgc9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 16:05:36.962834 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:05:36.962749    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgc9q\" (UniqueName: \"kubernetes.io/projected/dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff-kube-api-access-pgc9q\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 16:05:36.962834 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:05:36.962782    4993 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff-tls-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 16:05:36.962834 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:05:36.962792    4993 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff-tmp-dir\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 16:05:36.962834 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:05:36.962799    4993 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff-dshm\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 16:05:37.128609 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:05:37.128576    4993 generic.go:356] "Generic (PLEG): container finished" podID="dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff" containerID="4f1c88d6e1dd41cd3037a04fab3d0e66f468739fa55a23ad64a5b866e394ab47" exitCode=0
Nov 01 16:05:37.128800 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:05:37.128623    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-9ebec3/simulator-single-main-9ebec3-kserve-67c659b995-4mjdf"
Nov 01 16:05:37.128800 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:05:37.128623    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-9ebec3/simulator-single-main-9ebec3-kserve-67c659b995-4mjdf" event={"ID":"dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff","Type":"ContainerDied","Data":"4f1c88d6e1dd41cd3037a04fab3d0e66f468739fa55a23ad64a5b866e394ab47"}
Nov 01 16:05:37.128800 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:05:37.128740    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-9ebec3/simulator-single-main-9ebec3-kserve-67c659b995-4mjdf" event={"ID":"dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff","Type":"ContainerDied","Data":"a452edd73153ef7ba260b89d5a9dbceb9f15b1dc8e4bc80a73d30073a7140ffb"}
Nov 01 16:05:37.128800 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:05:37.128765    4993 scope.go:122] "RemoveContainer" containerID="4f1c88d6e1dd41cd3037a04fab3d0e66f468739fa55a23ad64a5b866e394ab47"
Nov 01 16:05:37.140709 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:05:37.140363    4993 scope.go:122] "RemoveContainer" containerID="4f1c88d6e1dd41cd3037a04fab3d0e66f468739fa55a23ad64a5b866e394ab47"
Nov 01 16:05:37.140709 ip-10-0-134-37 kubenswrapper[4993]: E1101 16:05:37.140686    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f1c88d6e1dd41cd3037a04fab3d0e66f468739fa55a23ad64a5b866e394ab47\": container with ID starting with 4f1c88d6e1dd41cd3037a04fab3d0e66f468739fa55a23ad64a5b866e394ab47 not found: ID does not exist" containerID="4f1c88d6e1dd41cd3037a04fab3d0e66f468739fa55a23ad64a5b866e394ab47"
Nov 01 16:05:37.140961 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:05:37.140720    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f1c88d6e1dd41cd3037a04fab3d0e66f468739fa55a23ad64a5b866e394ab47"} err="failed to get container status \"4f1c88d6e1dd41cd3037a04fab3d0e66f468739fa55a23ad64a5b866e394ab47\": rpc error: code = NotFound desc = could not find container \"4f1c88d6e1dd41cd3037a04fab3d0e66f468739fa55a23ad64a5b866e394ab47\": container with ID starting with 4f1c88d6e1dd41cd3037a04fab3d0e66f468739fa55a23ad64a5b866e394ab47 not found: ID does not exist"
Nov 01 16:05:37.279868 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:05:37.279832    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-9ebec3/simulator-single-main-9ebec3-kserve-67c659b995-4mjdf"]
Nov 01 16:05:37.282078 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:05:37.282048    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-9ebec3/simulator-single-main-9ebec3-kserve-67c659b995-4mjdf"]
Nov 01 16:05:38.793302 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:05:38.793264    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff" path="/var/lib/kubelet/pods/dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff/volumes"
Nov 01 16:05:57.499597 ip-10-0-134-37 kubenswrapper[4993]: E1101 16:05:57.499560    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 16:06:21.112045 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:06:21.112008    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cbwwd"]
Nov 01 16:06:21.112915 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:06:21.112882    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cbwwd" podUID="c535f9df-00b8-4b69-91b7-e38837ab6802" containerName="manager" containerID="cri-o://3a6f577fa91ede25dd294b7033550c478d16d3cf6580e736ac11600d32bcfb84" gracePeriod=10
Nov 01 16:06:21.354264 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:06:21.354234    4993 generic.go:356] "Generic (PLEG): container finished" podID="c535f9df-00b8-4b69-91b7-e38837ab6802" containerID="3a6f577fa91ede25dd294b7033550c478d16d3cf6580e736ac11600d32bcfb84" exitCode=0
Nov 01 16:06:21.354390 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:06:21.354308    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cbwwd" event={"ID":"c535f9df-00b8-4b69-91b7-e38837ab6802","Type":"ContainerDied","Data":"3a6f577fa91ede25dd294b7033550c478d16d3cf6580e736ac11600d32bcfb84"}
Nov 01 16:06:21.354390 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:06:21.354373    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cbwwd" event={"ID":"c535f9df-00b8-4b69-91b7-e38837ab6802","Type":"ContainerDied","Data":"c331d43cbbaef25c26c99a730c954ab052751db30a6ce71a0fa8d18e99df05e7"}
Nov 01 16:06:21.354466 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:06:21.354397    4993 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c331d43cbbaef25c26c99a730c954ab052751db30a6ce71a0fa8d18e99df05e7"
Nov 01 16:06:21.366212 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:06:21.366133    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cbwwd"
Nov 01 16:06:21.489973 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:06:21.489937    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c535f9df-00b8-4b69-91b7-e38837ab6802-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c535f9df-00b8-4b69-91b7-e38837ab6802-extensions-socket-volume\") pod \"c535f9df-00b8-4b69-91b7-e38837ab6802\" (UID: \"c535f9df-00b8-4b69-91b7-e38837ab6802\") "
Nov 01 16:06:21.490164 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:06:21.490148    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c535f9df-00b8-4b69-91b7-e38837ab6802-kube-api-access-xvvbp\" (UniqueName: \"kubernetes.io/projected/c535f9df-00b8-4b69-91b7-e38837ab6802-kube-api-access-xvvbp\") pod \"c535f9df-00b8-4b69-91b7-e38837ab6802\" (UID: \"c535f9df-00b8-4b69-91b7-e38837ab6802\") "
Nov 01 16:06:21.490399 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:06:21.490376    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c535f9df-00b8-4b69-91b7-e38837ab6802-extensions-socket-volume" pod "c535f9df-00b8-4b69-91b7-e38837ab6802" (UID: "c535f9df-00b8-4b69-91b7-e38837ab6802"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 16:06:21.490508 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:06:21.490491    4993 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c535f9df-00b8-4b69-91b7-e38837ab6802-extensions-socket-volume\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 16:06:21.492516 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:06:21.492481    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c535f9df-00b8-4b69-91b7-e38837ab6802-kube-api-access-xvvbp" pod "c535f9df-00b8-4b69-91b7-e38837ab6802" (UID: "c535f9df-00b8-4b69-91b7-e38837ab6802"). InnerVolumeSpecName "kube-api-access-xvvbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 16:06:21.591331 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:06:21.591292    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvvbp\" (UniqueName: \"kubernetes.io/projected/c535f9df-00b8-4b69-91b7-e38837ab6802-kube-api-access-xvvbp\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 16:06:22.359042 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:06:22.359008    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cbwwd"
Nov 01 16:06:22.557651 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:06:22.557620    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cbwwd"]
Nov 01 16:06:22.604368 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:06:22.604331    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cbwwd"]
Nov 01 16:06:22.793994 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:06:22.793953    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c535f9df-00b8-4b69-91b7-e38837ab6802" path="/var/lib/kubelet/pods/c535f9df-00b8-4b69-91b7-e38837ab6802/volumes"
Nov 01 16:06:27.500148 ip-10-0-134-37 kubenswrapper[4993]: E1101 16:06:27.500114    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 16:06:53.705438 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:06:53.705405    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xmwtp_bb59051d-c2df-4362-a157-00f14f915e79/console-operator/3.log"
Nov 01 16:06:53.719363 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:06:53.719337    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xmwtp_bb59051d-c2df-4362-a157-00f14f915e79/console-operator/3.log"
Nov 01 16:06:54.553805 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:06:54.553776    4993 scope.go:122] "RemoveContainer" containerID="3a6f577fa91ede25dd294b7033550c478d16d3cf6580e736ac11600d32bcfb84"
Nov 01 16:06:57.499857 ip-10-0-134-37 kubenswrapper[4993]: E1101 16:06:57.499821    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 16:07:24.754310 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:07:24.754223    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jrbld"]
Nov 01 16:07:24.754791 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:07:24.754671    4993 cpu_manager.go:414] "containerMap: removing container" podUID="c535f9df-00b8-4b69-91b7-e38837ab6802" containerName="manager"
Nov 01 16:07:24.754791 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:07:24.754685    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c535f9df-00b8-4b69-91b7-e38837ab6802" containerName="manager"
Nov 01 16:07:24.754791 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:07:24.754708    4993 cpu_manager.go:414] "containerMap: removing container" podUID="dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff" containerName="main"
Nov 01 16:07:24.754791 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:07:24.754713    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff" containerName="main"
Nov 01 16:07:24.754923 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:07:24.754813    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="c535f9df-00b8-4b69-91b7-e38837ab6802" containerName="manager"
Nov 01 16:07:24.754923 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:07:24.754824    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc73d7f8-e7ec-4f3a-9a6f-26bea3ef80ff" containerName="main"
Nov 01 16:07:24.757874 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:07:24.757858    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jrbld"
Nov 01 16:07:24.810357 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:07:24.810305    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-wpjdj\""
Nov 01 16:07:24.858548 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:07:24.858508    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jrbld"]
Nov 01 16:07:24.883774 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:07:24.883738    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4ef10f88-3954-434c-8ffa-f2ad810e75af-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-jrbld\" (UID: \"4ef10f88-3954-434c-8ffa-f2ad810e75af\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jrbld"
Nov 01 16:07:24.883901 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:07:24.883789    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fklqn\" (UniqueName: \"kubernetes.io/projected/4ef10f88-3954-434c-8ffa-f2ad810e75af-kube-api-access-fklqn\") pod \"kuadrant-operator-controller-manager-678d87766f-jrbld\" (UID: \"4ef10f88-3954-434c-8ffa-f2ad810e75af\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jrbld"
Nov 01 16:07:24.985315 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:07:24.985279    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4ef10f88-3954-434c-8ffa-f2ad810e75af-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-jrbld\" (UID: \"4ef10f88-3954-434c-8ffa-f2ad810e75af\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jrbld"
Nov 01 16:07:24.985477 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:07:24.985323    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fklqn\" (UniqueName: \"kubernetes.io/projected/4ef10f88-3954-434c-8ffa-f2ad810e75af-kube-api-access-fklqn\") pod \"kuadrant-operator-controller-manager-678d87766f-jrbld\" (UID: \"4ef10f88-3954-434c-8ffa-f2ad810e75af\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jrbld"
Nov 01 16:07:24.985715 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:07:24.985691    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4ef10f88-3954-434c-8ffa-f2ad810e75af-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-jrbld\" (UID: \"4ef10f88-3954-434c-8ffa-f2ad810e75af\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jrbld"
Nov 01 16:07:25.039149 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:07:25.039078    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fklqn\" (UniqueName: \"kubernetes.io/projected/4ef10f88-3954-434c-8ffa-f2ad810e75af-kube-api-access-fklqn\") pod \"kuadrant-operator-controller-manager-678d87766f-jrbld\" (UID: \"4ef10f88-3954-434c-8ffa-f2ad810e75af\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jrbld"
Nov 01 16:07:25.071031 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:07:25.071006    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jrbld"
Nov 01 16:07:25.552847 ip-10-0-134-37 kubenswrapper[4993]: W1101 16:07:25.552816    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ef10f88_3954_434c_8ffa_f2ad810e75af.slice/crio-f2851389d08567792a81a42798a4f066d350a531b9452d3571f4bf1740ade036 WatchSource:0}: Error finding container f2851389d08567792a81a42798a4f066d350a531b9452d3571f4bf1740ade036: Status 404 returned error can't find the container with id f2851389d08567792a81a42798a4f066d350a531b9452d3571f4bf1740ade036
Nov 01 16:07:25.590934 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:07:25.590903    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jrbld"]
Nov 01 16:07:25.664194 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:07:25.664133    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jrbld" event={"ID":"4ef10f88-3954-434c-8ffa-f2ad810e75af","Type":"ContainerStarted","Data":"80ec068378393c15610f318010bb33468fe3e12c9a9a99faeeff1d49b60f14af"}
Nov 01 16:07:25.664389 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:07:25.664221    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jrbld" event={"ID":"4ef10f88-3954-434c-8ffa-f2ad810e75af","Type":"ContainerStarted","Data":"f2851389d08567792a81a42798a4f066d350a531b9452d3571f4bf1740ade036"}
Nov 01 16:07:25.664389 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:07:25.664336    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jrbld"
Nov 01 16:07:25.775660 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:07:25.775601    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jrbld" podStartSLOduration=1.7755856140000001 podStartE2EDuration="1.775585614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:07:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:07:25.774580924 +0000 UTC m=+6633.104443661" watchObservedRunningTime="2026-11-01 16:07:25.775585614 +0000 UTC m=+6633.105448351"
Nov 01 16:07:27.499918 ip-10-0-134-37 kubenswrapper[4993]: E1101 16:07:27.499883    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 16:07:36.673235 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:07:36.673195    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jrbld"
Nov 01 16:07:57.500090 ip-10-0-134-37 kubenswrapper[4993]: E1101 16:07:57.500050    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 16:08:13.868193 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:08:13.867551    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-4pmtn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 16:08:13.868193 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:08:13.867648    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-4pmtn" podUID="306b0405-fb73-435a-ae73-42a90cd2d2f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 16:08:27.499485 ip-10-0-134-37 kubenswrapper[4993]: E1101 16:08:27.499447    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 16:08:28.085053 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:08:28.085010    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-74dfdc6bb5-gfd62"]
Nov 01 16:08:28.088732 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:08:28.088703    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-74dfdc6bb5-gfd62"
Nov 01 16:08:28.180722 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:08:28.180684    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4095b8a6-101a-4426-9309-e49abb22203d-metrics-certs\") pod \"maas-api-74dfdc6bb5-gfd62\" (UID: \"4095b8a6-101a-4426-9309-e49abb22203d\") " pod="odh-ai-gateway-infra/maas-api-74dfdc6bb5-gfd62"
Nov 01 16:08:28.180903 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:08:28.180735    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4095b8a6-101a-4426-9309-e49abb22203d-openshift-service-ca\") pod \"maas-api-74dfdc6bb5-gfd62\" (UID: \"4095b8a6-101a-4426-9309-e49abb22203d\") " pod="odh-ai-gateway-infra/maas-api-74dfdc6bb5-gfd62"
Nov 01 16:08:28.180903 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:08:28.180814    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km94f\" (UniqueName: \"kubernetes.io/projected/4095b8a6-101a-4426-9309-e49abb22203d-kube-api-access-km94f\") pod \"maas-api-74dfdc6bb5-gfd62\" (UID: \"4095b8a6-101a-4426-9309-e49abb22203d\") " pod="odh-ai-gateway-infra/maas-api-74dfdc6bb5-gfd62"
Nov 01 16:08:28.180903 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:08:28.180869    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4095b8a6-101a-4426-9309-e49abb22203d-maas-api-tls\") pod \"maas-api-74dfdc6bb5-gfd62\" (UID: \"4095b8a6-101a-4426-9309-e49abb22203d\") " pod="odh-ai-gateway-infra/maas-api-74dfdc6bb5-gfd62"
Nov 01 16:08:28.186422 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:08:28.186393    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 16:08:28.186552 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:08:28.186394    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 16:08:28.186552 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:08:28.186540    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Nov 01 16:08:28.186671 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:08:28.186567    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 01 16:08:28.186896 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:08:28.186872    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7t7jm\""
Nov 01 16:08:28.189367 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:08:28.189344    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-74dfdc6bb5-gfd62"]
Nov 01 16:08:28.282089 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:08:28.282051    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4095b8a6-101a-4426-9309-e49abb22203d-openshift-service-ca\") pod \"maas-api-74dfdc6bb5-gfd62\" (UID: \"4095b8a6-101a-4426-9309-e49abb22203d\") " pod="odh-ai-gateway-infra/maas-api-74dfdc6bb5-gfd62"
Nov 01 16:08:28.282290 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:08:28.282115    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km94f\" (UniqueName: \"kubernetes.io/projected/4095b8a6-101a-4426-9309-e49abb22203d-kube-api-access-km94f\") pod \"maas-api-74dfdc6bb5-gfd62\" (UID: \"4095b8a6-101a-4426-9309-e49abb22203d\") " pod="odh-ai-gateway-infra/maas-api-74dfdc6bb5-gfd62"
Nov 01 16:08:28.282290 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:08:28.282150    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4095b8a6-101a-4426-9309-e49abb22203d-maas-api-tls\") pod \"maas-api-74dfdc6bb5-gfd62\" (UID: \"4095b8a6-101a-4426-9309-e49abb22203d\") " pod="odh-ai-gateway-infra/maas-api-74dfdc6bb5-gfd62"
Nov 01 16:08:28.282290 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:08:28.282235    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4095b8a6-101a-4426-9309-e49abb22203d-metrics-certs\") pod \"maas-api-74dfdc6bb5-gfd62\" (UID: \"4095b8a6-101a-4426-9309-e49abb22203d\") " pod="odh-ai-gateway-infra/maas-api-74dfdc6bb5-gfd62"
Nov 01 16:08:28.282774 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:08:28.282748    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4095b8a6-101a-4426-9309-e49abb22203d-openshift-service-ca\") pod \"maas-api-74dfdc6bb5-gfd62\" (UID: \"4095b8a6-101a-4426-9309-e49abb22203d\") " pod="odh-ai-gateway-infra/maas-api-74dfdc6bb5-gfd62"
Nov 01 16:08:28.284676 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:08:28.284653    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4095b8a6-101a-4426-9309-e49abb22203d-maas-api-tls\") pod \"maas-api-74dfdc6bb5-gfd62\" (UID: \"4095b8a6-101a-4426-9309-e49abb22203d\") " pod="odh-ai-gateway-infra/maas-api-74dfdc6bb5-gfd62"
Nov 01 16:08:28.284779 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:08:28.284697    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4095b8a6-101a-4426-9309-e49abb22203d-metrics-certs\") pod \"maas-api-74dfdc6bb5-gfd62\" (UID: \"4095b8a6-101a-4426-9309-e49abb22203d\") " pod="odh-ai-gateway-infra/maas-api-74dfdc6bb5-gfd62"
Nov 01 16:08:28.334665 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:08:28.334632    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-km94f\" (UniqueName: \"kubernetes.io/projected/4095b8a6-101a-4426-9309-e49abb22203d-kube-api-access-km94f\") pod \"maas-api-74dfdc6bb5-gfd62\" (UID: \"4095b8a6-101a-4426-9309-e49abb22203d\") " pod="odh-ai-gateway-infra/maas-api-74dfdc6bb5-gfd62"
Nov 01 16:08:28.401631 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:08:28.401551    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-74dfdc6bb5-gfd62"
Nov 01 16:08:28.769832 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:08:28.769794    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-74dfdc6bb5-gfd62"]
Nov 01 16:08:28.776808 ip-10-0-134-37 kubenswrapper[4993]: W1101 16:08:28.776780    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4095b8a6_101a_4426_9309_e49abb22203d.slice/crio-4f535073f97be14dee74609005112c8ef7c7410faa0e4565e55db17c8536dd8e WatchSource:0}: Error finding container 4f535073f97be14dee74609005112c8ef7c7410faa0e4565e55db17c8536dd8e: Status 404 returned error can't find the container with id 4f535073f97be14dee74609005112c8ef7c7410faa0e4565e55db17c8536dd8e
Nov 01 16:08:28.778224 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:08:28.778200    4993 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 16:08:29.008453 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:08:29.008411    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-74dfdc6bb5-gfd62" event={"ID":"4095b8a6-101a-4426-9309-e49abb22203d","Type":"ContainerStarted","Data":"4f535073f97be14dee74609005112c8ef7c7410faa0e4565e55db17c8536dd8e"}
Nov 01 16:08:30.015560 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:08:30.015460    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-74dfdc6bb5-gfd62" event={"ID":"4095b8a6-101a-4426-9309-e49abb22203d","Type":"ContainerStarted","Data":"22e13e4a16e281f9785936b5c6e1ca77e3167e4b41c8befc0d46e4d4028e43bf"}
Nov 01 16:08:30.015560 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:08:30.015536    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-74dfdc6bb5-gfd62"
Nov 01 16:08:30.170019 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:08:30.169962    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-74dfdc6bb5-gfd62" podStartSLOduration=1.20525603 podStartE2EDuration="2.16994699s" totalImagesPullingTime="964.69096ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:08:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:08:30.167829583 +0000 UTC m=+6697.497692320" watchObservedRunningTime="2026-11-01 16:08:30.16994699 +0000 UTC m=+6697.499809727"
Nov 01 16:08:36.026351 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:08:36.026265    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-74dfdc6bb5-gfd62"
Nov 01 16:08:36.942099 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:08:36.942054    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-58788d547c-cprcs"]
Nov 01 16:08:36.945848 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:08:36.945824    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-58788d547c-cprcs"
Nov 01 16:08:37.033058 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:08:37.033022    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-58788d547c-cprcs"]
Nov 01 16:08:37.055553 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:08:37.055511    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2995e7ce-ec0a-4507-9736-af1ec51ca2a5-metrics-certs\") pod \"maas-api-58788d547c-cprcs\" (UID: \"2995e7ce-ec0a-4507-9736-af1ec51ca2a5\") " pod="odh-ai-gateway-infra/maas-api-58788d547c-cprcs"
Nov 01 16:08:37.055738 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:08:37.055605    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2995e7ce-ec0a-4507-9736-af1ec51ca2a5-openshift-service-ca\") pod \"maas-api-58788d547c-cprcs\" (UID: \"2995e7ce-ec0a-4507-9736-af1ec51ca2a5\") " pod="odh-ai-gateway-infra/maas-api-58788d547c-cprcs"
Nov 01 16:08:37.055738 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:08:37.055671    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2995e7ce-ec0a-4507-9736-af1ec51ca2a5-maas-api-tls\") pod \"maas-api-58788d547c-cprcs\" (UID: \"2995e7ce-ec0a-4507-9736-af1ec51ca2a5\") " pod="odh-ai-gateway-infra/maas-api-58788d547c-cprcs"
Nov 01 16:08:37.055816 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:08:37.055740    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqdc5\" (UniqueName: \"kubernetes.io/projected/2995e7ce-ec0a-4507-9736-af1ec51ca2a5-kube-api-access-cqdc5\") pod \"maas-api-58788d547c-cprcs\" (UID: \"2995e7ce-ec0a-4507-9736-af1ec51ca2a5\") " pod="odh-ai-gateway-infra/maas-api-58788d547c-cprcs"
Nov 01 16:08:37.156442 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:08:37.156402    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2995e7ce-ec0a-4507-9736-af1ec51ca2a5-metrics-certs\") pod \"maas-api-58788d547c-cprcs\" (UID: \"2995e7ce-ec0a-4507-9736-af1ec51ca2a5\") " pod="odh-ai-gateway-infra/maas-api-58788d547c-cprcs"
Nov 01 16:08:37.156619 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:08:37.156451    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2995e7ce-ec0a-4507-9736-af1ec51ca2a5-openshift-service-ca\") pod \"maas-api-58788d547c-cprcs\" (UID: \"2995e7ce-ec0a-4507-9736-af1ec51ca2a5\") " pod="odh-ai-gateway-infra/maas-api-58788d547c-cprcs"
Nov 01 16:08:37.156619 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:08:37.156472    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2995e7ce-ec0a-4507-9736-af1ec51ca2a5-maas-api-tls\") pod \"maas-api-58788d547c-cprcs\" (UID: \"2995e7ce-ec0a-4507-9736-af1ec51ca2a5\") " pod="odh-ai-gateway-infra/maas-api-58788d547c-cprcs"
Nov 01 16:08:37.156619 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:08:37.156538    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqdc5\" (UniqueName: \"kubernetes.io/projected/2995e7ce-ec0a-4507-9736-af1ec51ca2a5-kube-api-access-cqdc5\") pod \"maas-api-58788d547c-cprcs\" (UID: \"2995e7ce-ec0a-4507-9736-af1ec51ca2a5\") " pod="odh-ai-gateway-infra/maas-api-58788d547c-cprcs"
Nov 01 16:08:37.157301 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:08:37.157274    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2995e7ce-ec0a-4507-9736-af1ec51ca2a5-openshift-service-ca\") pod \"maas-api-58788d547c-cprcs\" (UID: \"2995e7ce-ec0a-4507-9736-af1ec51ca2a5\") " pod="odh-ai-gateway-infra/maas-api-58788d547c-cprcs"
Nov 01 16:08:37.159084 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:08:37.159059    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2995e7ce-ec0a-4507-9736-af1ec51ca2a5-maas-api-tls\") pod \"maas-api-58788d547c-cprcs\" (UID: \"2995e7ce-ec0a-4507-9736-af1ec51ca2a5\") " pod="odh-ai-gateway-infra/maas-api-58788d547c-cprcs"
Nov 01 16:08:37.159191 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:08:37.159096    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2995e7ce-ec0a-4507-9736-af1ec51ca2a5-metrics-certs\") pod \"maas-api-58788d547c-cprcs\" (UID: \"2995e7ce-ec0a-4507-9736-af1ec51ca2a5\") " pod="odh-ai-gateway-infra/maas-api-58788d547c-cprcs"
Nov 01 16:08:37.210250 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:08:37.210163    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqdc5\" (UniqueName: \"kubernetes.io/projected/2995e7ce-ec0a-4507-9736-af1ec51ca2a5-kube-api-access-cqdc5\") pod \"maas-api-58788d547c-cprcs\" (UID: \"2995e7ce-ec0a-4507-9736-af1ec51ca2a5\") " pod="odh-ai-gateway-infra/maas-api-58788d547c-cprcs"
Nov 01 16:08:37.259084 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:08:37.259055    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-58788d547c-cprcs"
Nov 01 16:08:37.625861 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:08:37.625825    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-58788d547c-cprcs"]
Nov 01 16:08:37.633357 ip-10-0-134-37 kubenswrapper[4993]: W1101 16:08:37.633328    4993 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2995e7ce_ec0a_4507_9736_af1ec51ca2a5.slice/crio-40d8bcea1b1c464f7e479709efcf67e146995a7c65f3da61b8d00a769be5e850 WatchSource:0}: Error finding container 40d8bcea1b1c464f7e479709efcf67e146995a7c65f3da61b8d00a769be5e850: Status 404 returned error can't find the container with id 40d8bcea1b1c464f7e479709efcf67e146995a7c65f3da61b8d00a769be5e850
Nov 01 16:08:38.057931 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:08:38.057895    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-58788d547c-cprcs" event={"ID":"2995e7ce-ec0a-4507-9736-af1ec51ca2a5","Type":"ContainerStarted","Data":"40d8bcea1b1c464f7e479709efcf67e146995a7c65f3da61b8d00a769be5e850"}
Nov 01 16:08:39.064893 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:08:39.064860    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-58788d547c-cprcs" event={"ID":"2995e7ce-ec0a-4507-9736-af1ec51ca2a5","Type":"ContainerStarted","Data":"dd0460bf48dde103b3794cf6023f73941d0a9d6aae68b0a4f9c6b331f3e4788b"}
Nov 01 16:08:39.065293 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:08:39.064996    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-58788d547c-cprcs"
Nov 01 16:08:39.173889 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:08:39.173833    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-58788d547c-cprcs" podStartSLOduration=2.311643152 podStartE2EDuration="3.173817866s" totalImagesPullingTime="862.174714ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:08:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:08:39.171054061 +0000 UTC m=+6706.500916798" watchObservedRunningTime="2026-11-01 16:08:39.173817866 +0000 UTC m=+6706.503680651"
Nov 01 16:08:45.075927 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:08:45.075892    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-58788d547c-cprcs"
Nov 01 16:08:45.231377 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:08:45.231336    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-74dfdc6bb5-gfd62"]
Nov 01 16:08:45.231650 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:08:45.231594    4993 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-74dfdc6bb5-gfd62" podUID="4095b8a6-101a-4426-9309-e49abb22203d" containerName="maas-api" containerID="cri-o://22e13e4a16e281f9785936b5c6e1ca77e3167e4b41c8befc0d46e4d4028e43bf" gracePeriod=30
Nov 01 16:08:45.483955 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:08:45.483930    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-74dfdc6bb5-gfd62"
Nov 01 16:08:45.559215 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:08:45.559161    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4095b8a6-101a-4426-9309-e49abb22203d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4095b8a6-101a-4426-9309-e49abb22203d-openshift-service-ca\") pod \"4095b8a6-101a-4426-9309-e49abb22203d\" (UID: \"4095b8a6-101a-4426-9309-e49abb22203d\") "
Nov 01 16:08:45.559381 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:08:45.559228    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4095b8a6-101a-4426-9309-e49abb22203d-kube-api-access-km94f\" (UniqueName: \"kubernetes.io/projected/4095b8a6-101a-4426-9309-e49abb22203d-kube-api-access-km94f\") pod \"4095b8a6-101a-4426-9309-e49abb22203d\" (UID: \"4095b8a6-101a-4426-9309-e49abb22203d\") "
Nov 01 16:08:45.559381 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:08:45.559252    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4095b8a6-101a-4426-9309-e49abb22203d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4095b8a6-101a-4426-9309-e49abb22203d-maas-api-tls\") pod \"4095b8a6-101a-4426-9309-e49abb22203d\" (UID: \"4095b8a6-101a-4426-9309-e49abb22203d\") "
Nov 01 16:08:45.559381 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:08:45.559364    4993 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4095b8a6-101a-4426-9309-e49abb22203d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4095b8a6-101a-4426-9309-e49abb22203d-metrics-certs\") pod \"4095b8a6-101a-4426-9309-e49abb22203d\" (UID: \"4095b8a6-101a-4426-9309-e49abb22203d\") "
Nov 01 16:08:45.559557 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:08:45.559514    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4095b8a6-101a-4426-9309-e49abb22203d-openshift-service-ca" pod "4095b8a6-101a-4426-9309-e49abb22203d" (UID: "4095b8a6-101a-4426-9309-e49abb22203d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 16:08:45.561435 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:08:45.561404    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4095b8a6-101a-4426-9309-e49abb22203d-kube-api-access-km94f" pod "4095b8a6-101a-4426-9309-e49abb22203d" (UID: "4095b8a6-101a-4426-9309-e49abb22203d"). InnerVolumeSpecName "kube-api-access-km94f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 16:08:45.561538 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:08:45.561430    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4095b8a6-101a-4426-9309-e49abb22203d-metrics-certs" pod "4095b8a6-101a-4426-9309-e49abb22203d" (UID: "4095b8a6-101a-4426-9309-e49abb22203d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 16:08:45.561538 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:08:45.561441    4993 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4095b8a6-101a-4426-9309-e49abb22203d-maas-api-tls" pod "4095b8a6-101a-4426-9309-e49abb22203d" (UID: "4095b8a6-101a-4426-9309-e49abb22203d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 16:08:45.660420 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:08:45.660341    4993 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4095b8a6-101a-4426-9309-e49abb22203d-metrics-certs\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 16:08:45.660420 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:08:45.660369    4993 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4095b8a6-101a-4426-9309-e49abb22203d-openshift-service-ca\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 16:08:45.660420 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:08:45.660379    4993 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-km94f\" (UniqueName: \"kubernetes.io/projected/4095b8a6-101a-4426-9309-e49abb22203d-kube-api-access-km94f\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 16:08:45.660420 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:08:45.660388    4993 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4095b8a6-101a-4426-9309-e49abb22203d-maas-api-tls\") on node \"ip-10-0-134-37.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 16:08:46.116092 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:08:46.116044    4993 generic.go:356] "Generic (PLEG): container finished" podID="4095b8a6-101a-4426-9309-e49abb22203d" containerID="22e13e4a16e281f9785936b5c6e1ca77e3167e4b41c8befc0d46e4d4028e43bf" exitCode=0
Nov 01 16:08:46.116623 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:08:46.116124    4993 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-74dfdc6bb5-gfd62"
Nov 01 16:08:46.116623 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:08:46.116227    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-74dfdc6bb5-gfd62" event={"ID":"4095b8a6-101a-4426-9309-e49abb22203d","Type":"ContainerDied","Data":"22e13e4a16e281f9785936b5c6e1ca77e3167e4b41c8befc0d46e4d4028e43bf"}
Nov 01 16:08:46.116623 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:08:46.116312    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-74dfdc6bb5-gfd62" event={"ID":"4095b8a6-101a-4426-9309-e49abb22203d","Type":"ContainerDied","Data":"4f535073f97be14dee74609005112c8ef7c7410faa0e4565e55db17c8536dd8e"}
Nov 01 16:08:46.116623 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:08:46.116341    4993 scope.go:122] "RemoveContainer" containerID="22e13e4a16e281f9785936b5c6e1ca77e3167e4b41c8befc0d46e4d4028e43bf"
Nov 01 16:08:46.126957 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:08:46.126937    4993 scope.go:122] "RemoveContainer" containerID="22e13e4a16e281f9785936b5c6e1ca77e3167e4b41c8befc0d46e4d4028e43bf"
Nov 01 16:08:46.127355 ip-10-0-134-37 kubenswrapper[4993]: E1101 16:08:46.127295    4993 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22e13e4a16e281f9785936b5c6e1ca77e3167e4b41c8befc0d46e4d4028e43bf\": container with ID starting with 22e13e4a16e281f9785936b5c6e1ca77e3167e4b41c8befc0d46e4d4028e43bf not found: ID does not exist" containerID="22e13e4a16e281f9785936b5c6e1ca77e3167e4b41c8befc0d46e4d4028e43bf"
Nov 01 16:08:46.127355 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:08:46.127327    4993 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22e13e4a16e281f9785936b5c6e1ca77e3167e4b41c8befc0d46e4d4028e43bf"} err="failed to get container status \"22e13e4a16e281f9785936b5c6e1ca77e3167e4b41c8befc0d46e4d4028e43bf\": rpc error: code = NotFound desc = could not find container \"22e13e4a16e281f9785936b5c6e1ca77e3167e4b41c8befc0d46e4d4028e43bf\": container with ID starting with 22e13e4a16e281f9785936b5c6e1ca77e3167e4b41c8befc0d46e4d4028e43bf not found: ID does not exist"
Nov 01 16:08:46.268240 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:08:46.268197    4993 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-74dfdc6bb5-gfd62"]
Nov 01 16:08:46.314212 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:08:46.314158    4993 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-74dfdc6bb5-gfd62"]
Nov 01 16:08:46.794333 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:08:46.794290    4993 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4095b8a6-101a-4426-9309-e49abb22203d" path="/var/lib/kubelet/pods/4095b8a6-101a-4426-9309-e49abb22203d/volumes"
Nov 01 16:08:57.499809 ip-10-0-134-37 kubenswrapper[4993]: E1101 16:08:57.499774    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 16:09:25.176192 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:09:25.173285    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-q9j4g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 16:09:25.176192 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:09:25.173368    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q9j4g" podUID="29c351ad-629b-418c-b64d-9ff0e52ccdcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 16:09:27.499148 ip-10-0-134-37 kubenswrapper[4993]: E1101 16:09:27.499114    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 16:09:57.550737 ip-10-0-134-37 kubenswrapper[4993]: E1101 16:09:57.550695    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 16:10:27.499377 ip-10-0-134-37 kubenswrapper[4993]: E1101 16:10:27.499288    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 16:10:57.415393 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:10:57.415335    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-flzf5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 16:10:57.417826 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:10:57.415437    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-flzf5" podUID="e3d2334c-8e83-4a16-a126-cbe45120efa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 16:10:57.506020 ip-10-0-134-37 kubenswrapper[4993]: E1101 16:10:57.505974    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 16:11:27.499664 ip-10-0-134-37 kubenswrapper[4993]: E1101 16:11:27.499627    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 16:11:53.747060 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:11:53.747028    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xmwtp_bb59051d-c2df-4362-a157-00f14f915e79/console-operator/3.log"
Nov 01 16:11:53.759857 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:11:53.759829    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xmwtp_bb59051d-c2df-4362-a157-00f14f915e79/console-operator/3.log"
Nov 01 16:11:57.500261 ip-10-0-134-37 kubenswrapper[4993]: E1101 16:11:57.500227    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 16:12:10.176268 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:12:10.176214    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-q9j4g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 16:12:10.177033 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:12:10.176979    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-q9j4g" podUID="29c351ad-629b-418c-b64d-9ff0e52ccdcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 16:12:27.499427 ip-10-0-134-37 kubenswrapper[4993]: E1101 16:12:27.499391    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 16:12:57.500227 ip-10-0-134-37 kubenswrapper[4993]: E1101 16:12:57.500189    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 16:13:12.421430 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:13:12.421303    4993 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-flzf5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 16:13:12.422101 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:13:12.422055    4993 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-flzf5" podUID="e3d2334c-8e83-4a16-a126-cbe45120efa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 16:13:27.500279 ip-10-0-134-37 kubenswrapper[4993]: E1101 16:13:27.500240    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 16:13:57.499535 ip-10-0-134-37 kubenswrapper[4993]: E1101 16:13:57.499495    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 16:14:27.499269 ip-10-0-134-37 kubenswrapper[4993]: E1101 16:14:27.499226    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 16:14:52.074415 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:14:52.074375    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-hg8lm_4a9c6ca9-d1f4-41b1-9496-2e32c99edfce/manager/0.log"
Nov 01 16:14:52.342932 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:14:52.342853    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-2chcp_31f198c9-adac-4cd9-8cd9-2551a89aad9c/manager/0.log"
Nov 01 16:14:52.638792 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:14:52.638706    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-5csbz_93dc6178-9868-4299-bad6-0ee49fc5d2ad/manager/0.log"
Nov 01 16:14:53.478423 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:14:53.478391    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-lr4hb_6c32c11d-d77f-458d-9cf0-fe262d31aeeb/server/0.log"
Nov 01 16:14:54.116575 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:14:54.116540    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-cmbwq_6489d076-e1bd-402d-9d85-4e3bce4123b0/proxy/0.log"
Nov 01 16:14:55.099660 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:14:55.099611    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-58788d547c-cprcs_2995e7ce-ec0a-4507-9736-af1ec51ca2a5/maas-api/0.log"
Nov 01 16:14:57.499822 ip-10-0-134-37 kubenswrapper[4993]: E1101 16:14:57.499781    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 16:15:01.492852 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:15:01.492819    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-8dvhs_9c1bd12c-d218-4715-9239-1c18ce556fe9/manager/0.log"
Nov 01 16:15:01.790140 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:15:01.790058    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-d8qkk_e5667cdf-da6e-4930-93c0-f6dfc6910d45/manager/0.log"
Nov 01 16:15:02.464800 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:15:02.464769    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-jrbld_4ef10f88-3954-434c-8ffa-f2ad810e75af/manager/0.log"
Nov 01 16:15:04.168977 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:15:04.168934    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7cfc97f8f7nb8nm_07f0043b-a848-44d4-9cc2-d6de906d2c01/istio-proxy/0.log"
Nov 01 16:15:05.156819 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:15:05.156776    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-4pmtn_306b0405-fb73-435a-ae73-42a90cd2d2f9/istio-proxy/1.log"
Nov 01 16:15:05.789374 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:15:05.789344    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-flzf5_e3d2334c-8e83-4a16-a126-cbe45120efa7/istio-proxy/0.log"
Nov 01 16:15:07.168352 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:15:07.168318    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-q9j4g_29c351ad-629b-418c-b64d-9ff0e52ccdcb/istio-proxy/0.log"
Nov 01 16:15:08.059185 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:15:08.059144    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-65dccbd94c-bcw96_4dacd9b6-f9a3-4ae5-969b-c374a28c242a/istio-proxy/0.log"
Nov 01 16:15:08.762140 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:15:08.762107    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-b9d995dcc-lnk2n_c27ff402-235a-4d73-a139-cf928a6d5cbe/router/0.log"
Nov 01 16:15:09.561493 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:15:09.561464    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-4pmtn_306b0405-fb73-435a-ae73-42a90cd2d2f9/istio-proxy/0.log"
Nov 01 16:15:27.499492 ip-10-0-134-37 kubenswrapper[4993]: E1101 16:15:27.499455    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 16:15:30.953308 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:15:30.953266    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-sc9jq_8b1eb53c-8c9a-46c4-b654-2aef42491b14/global-pull-secret-syncer/0.log"
Nov 01 16:15:32.134045 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:15:32.134018    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-4lzxm_2460be58-d51b-4876-bf9b-2a0e57901386/connection-checker/0.log"
Nov 01 16:15:32.543050 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:15:32.543020    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-xmfr4_6a8f947a-0f8f-41aa-b646-4459fa8126a3/connection-checker/0.log"
Nov 01 16:15:32.950028 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:15:32.949955    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-zfqjv_d8a5ccab-4d5e-4aaf-ad08-766238eca05f/connection-checker/0.log"
Nov 01 16:15:34.413490 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:15:34.413458    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-qzf26_ccc5b32b-07bc-4b3c-91ef-b184b0a9afe3/konnectivity-agent/0.log"
Nov 01 16:15:35.170272 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:15:35.170238    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-hg8lm_4a9c6ca9-d1f4-41b1-9496-2e32c99edfce/manager/0.log"
Nov 01 16:15:35.311070 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:15:35.311037    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-37.us-west-2.compute.internal_f4211ca6cf8dc522c889bb47e17f738b/haproxy/0.log"
Nov 01 16:15:35.424087 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:15:35.423982    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-2chcp_31f198c9-adac-4cd9-8cd9-2551a89aad9c/manager/0.log"
Nov 01 16:15:35.948817 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:15:35.948785    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-5csbz_93dc6178-9868-4299-bad6-0ee49fc5d2ad/manager/0.log"
Nov 01 16:15:36.274957 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:15:36.274927    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-lr4hb_6c32c11d-d77f-458d-9cf0-fe262d31aeeb/server/0.log"
Nov 01 16:15:38.274339 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:15:38.274300    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-cmbwq_6489d076-e1bd-402d-9d85-4e3bce4123b0/proxy/0.log"
Nov 01 16:15:40.973524 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:15:40.973486    4993 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8k5v4/perf-node-gather-daemonset-5qv9x"]
Nov 01 16:15:40.974033 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:15:40.973955    4993 cpu_manager.go:414] "containerMap: removing container" podUID="4095b8a6-101a-4426-9309-e49abb22203d" containerName="maas-api"
Nov 01 16:15:40.974033 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:15:40.973969    4993 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4095b8a6-101a-4426-9309-e49abb22203d" containerName="maas-api"
Nov 01 16:15:40.974164 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:15:40.974153    4993 memory_manager.go:365] "RemoveStaleState removing state" podUID="4095b8a6-101a-4426-9309-e49abb22203d" containerName="maas-api"
Nov 01 16:15:40.977758 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:15:40.977731    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8k5v4/perf-node-gather-daemonset-5qv9x"
Nov 01 16:15:41.073042 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:15:41.073008    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8k5v4\"/\"kube-root-ca.crt\""
Nov 01 16:15:41.073042 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:15:41.073012    4993 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8k5v4\"/\"openshift-service-ca.crt\""
Nov 01 16:15:41.073042 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:15:41.073019    4993 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-8k5v4\"/\"default-dockercfg-tf4nc\""
Nov 01 16:15:41.075291 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:15:41.075264    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8k5v4/perf-node-gather-daemonset-5qv9x"]
Nov 01 16:15:41.136321 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:15:41.136278    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/10533f79-beca-4101-b3b2-b60055f7c6ca-lib-modules\") pod \"perf-node-gather-daemonset-5qv9x\" (UID: \"10533f79-beca-4101-b3b2-b60055f7c6ca\") " pod="openshift-must-gather-8k5v4/perf-node-gather-daemonset-5qv9x"
Nov 01 16:15:41.136516 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:15:41.136377    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shwpl\" (UniqueName: \"kubernetes.io/projected/10533f79-beca-4101-b3b2-b60055f7c6ca-kube-api-access-shwpl\") pod \"perf-node-gather-daemonset-5qv9x\" (UID: \"10533f79-beca-4101-b3b2-b60055f7c6ca\") " pod="openshift-must-gather-8k5v4/perf-node-gather-daemonset-5qv9x"
Nov 01 16:15:41.136516 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:15:41.136475    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/10533f79-beca-4101-b3b2-b60055f7c6ca-podres\") pod \"perf-node-gather-daemonset-5qv9x\" (UID: \"10533f79-beca-4101-b3b2-b60055f7c6ca\") " pod="openshift-must-gather-8k5v4/perf-node-gather-daemonset-5qv9x"
Nov 01 16:15:41.136709 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:15:41.136530    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/10533f79-beca-4101-b3b2-b60055f7c6ca-sys\") pod \"perf-node-gather-daemonset-5qv9x\" (UID: \"10533f79-beca-4101-b3b2-b60055f7c6ca\") " pod="openshift-must-gather-8k5v4/perf-node-gather-daemonset-5qv9x"
Nov 01 16:15:41.136709 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:15:41.136630    4993 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/10533f79-beca-4101-b3b2-b60055f7c6ca-proc\") pod \"perf-node-gather-daemonset-5qv9x\" (UID: \"10533f79-beca-4101-b3b2-b60055f7c6ca\") " pod="openshift-must-gather-8k5v4/perf-node-gather-daemonset-5qv9x"
Nov 01 16:15:41.237346 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:15:41.237313    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/10533f79-beca-4101-b3b2-b60055f7c6ca-lib-modules\") pod \"perf-node-gather-daemonset-5qv9x\" (UID: \"10533f79-beca-4101-b3b2-b60055f7c6ca\") " pod="openshift-must-gather-8k5v4/perf-node-gather-daemonset-5qv9x"
Nov 01 16:15:41.237515 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:15:41.237357    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shwpl\" (UniqueName: \"kubernetes.io/projected/10533f79-beca-4101-b3b2-b60055f7c6ca-kube-api-access-shwpl\") pod \"perf-node-gather-daemonset-5qv9x\" (UID: \"10533f79-beca-4101-b3b2-b60055f7c6ca\") " pod="openshift-must-gather-8k5v4/perf-node-gather-daemonset-5qv9x"
Nov 01 16:15:41.237515 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:15:41.237387    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/10533f79-beca-4101-b3b2-b60055f7c6ca-podres\") pod \"perf-node-gather-daemonset-5qv9x\" (UID: \"10533f79-beca-4101-b3b2-b60055f7c6ca\") " pod="openshift-must-gather-8k5v4/perf-node-gather-daemonset-5qv9x"
Nov 01 16:15:41.237515 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:15:41.237413    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/10533f79-beca-4101-b3b2-b60055f7c6ca-sys\") pod \"perf-node-gather-daemonset-5qv9x\" (UID: \"10533f79-beca-4101-b3b2-b60055f7c6ca\") " pod="openshift-must-gather-8k5v4/perf-node-gather-daemonset-5qv9x"
Nov 01 16:15:41.237515 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:15:41.237449    4993 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/10533f79-beca-4101-b3b2-b60055f7c6ca-proc\") pod \"perf-node-gather-daemonset-5qv9x\" (UID: \"10533f79-beca-4101-b3b2-b60055f7c6ca\") " pod="openshift-must-gather-8k5v4/perf-node-gather-daemonset-5qv9x"
Nov 01 16:15:41.237515 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:15:41.237469    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/10533f79-beca-4101-b3b2-b60055f7c6ca-lib-modules\") pod \"perf-node-gather-daemonset-5qv9x\" (UID: \"10533f79-beca-4101-b3b2-b60055f7c6ca\") " pod="openshift-must-gather-8k5v4/perf-node-gather-daemonset-5qv9x"
Nov 01 16:15:41.237701 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:15:41.237520    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/10533f79-beca-4101-b3b2-b60055f7c6ca-sys\") pod \"perf-node-gather-daemonset-5qv9x\" (UID: \"10533f79-beca-4101-b3b2-b60055f7c6ca\") " pod="openshift-must-gather-8k5v4/perf-node-gather-daemonset-5qv9x"
Nov 01 16:15:41.237701 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:15:41.237532    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/10533f79-beca-4101-b3b2-b60055f7c6ca-proc\") pod \"perf-node-gather-daemonset-5qv9x\" (UID: \"10533f79-beca-4101-b3b2-b60055f7c6ca\") " pod="openshift-must-gather-8k5v4/perf-node-gather-daemonset-5qv9x"
Nov 01 16:15:41.237701 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:15:41.237537    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/10533f79-beca-4101-b3b2-b60055f7c6ca-podres\") pod \"perf-node-gather-daemonset-5qv9x\" (UID: \"10533f79-beca-4101-b3b2-b60055f7c6ca\") " pod="openshift-must-gather-8k5v4/perf-node-gather-daemonset-5qv9x"
Nov 01 16:15:41.277767 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:15:41.277732    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-8dvhs_9c1bd12c-d218-4715-9239-1c18ce556fe9/manager/0.log"
Nov 01 16:15:41.290660 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:15:41.290633    4993 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shwpl\" (UniqueName: \"kubernetes.io/projected/10533f79-beca-4101-b3b2-b60055f7c6ca-kube-api-access-shwpl\") pod \"perf-node-gather-daemonset-5qv9x\" (UID: \"10533f79-beca-4101-b3b2-b60055f7c6ca\") " pod="openshift-must-gather-8k5v4/perf-node-gather-daemonset-5qv9x"
Nov 01 16:15:41.293484 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:15:41.293465    4993 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8k5v4/perf-node-gather-daemonset-5qv9x"
Nov 01 16:15:41.345357 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:15:41.345330    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-hg8lm_4a9c6ca9-d1f4-41b1-9496-2e32c99edfce/manager/0.log"
Nov 01 16:15:41.600584 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:15:41.600490    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-2chcp_31f198c9-adac-4cd9-8cd9-2551a89aad9c/manager/0.log"
Nov 01 16:15:41.663235 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:15:41.663198    4993 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8k5v4/perf-node-gather-daemonset-5qv9x"]
Nov 01 16:15:41.758716 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:15:41.758682    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-d8qkk_e5667cdf-da6e-4930-93c0-f6dfc6910d45/manager/0.log"
Nov 01 16:15:42.192474 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:15:42.192435    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-5csbz_93dc6178-9868-4299-bad6-0ee49fc5d2ad/manager/0.log"
Nov 01 16:15:42.523452 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:15:42.523415    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8k5v4/perf-node-gather-daemonset-5qv9x" event={"ID":"10533f79-beca-4101-b3b2-b60055f7c6ca","Type":"ContainerStarted","Data":"6c09dd465d6718d76f4a6b61fcccadb847a97fdd361ec4a34136bb66e6433e23"}
Nov 01 16:15:42.523623 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:15:42.523497    4993 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8k5v4/perf-node-gather-daemonset-5qv9x" event={"ID":"10533f79-beca-4101-b3b2-b60055f7c6ca","Type":"ContainerStarted","Data":"bc1127244ed122caedfe9a7fa8a473c1aee1dcd650ed2eec173ef222f2bdb49c"}
Nov 01 16:15:42.523623 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:15:42.523555    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-8k5v4/perf-node-gather-daemonset-5qv9x"
Nov 01 16:15:42.532138 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:15:42.532115    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-lr4hb_6c32c11d-d77f-458d-9cf0-fe262d31aeeb/server/0.log"
Nov 01 16:15:42.629594 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:15:42.629541    4993 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-8k5v4/perf-node-gather-daemonset-5qv9x" podStartSLOduration=2.62952742 podStartE2EDuration="2.62952742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 16:15:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:15:42.627144737 +0000 UTC m=+7129.957007474" watchObservedRunningTime="2026-11-01 16:15:42.62952742 +0000 UTC m=+7129.959390156"
Nov 01 16:15:42.658405 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:15:42.658371    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-jrbld_4ef10f88-3954-434c-8ffa-f2ad810e75af/manager/0.log"
Nov 01 16:15:44.718581 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:15:44.718534    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-cmbwq_6489d076-e1bd-402d-9d85-4e3bce4123b0/proxy/0.log"
Nov 01 16:15:47.946852 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:15:47.946803    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-hg8lm_4a9c6ca9-d1f4-41b1-9496-2e32c99edfce/manager/0.log"
Nov 01 16:15:48.204269 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:15:48.204188    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-2chcp_31f198c9-adac-4cd9-8cd9-2551a89aad9c/manager/0.log"
Nov 01 16:15:48.539895 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:15:48.539860    4993 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-8k5v4/perf-node-gather-daemonset-5qv9x"
Nov 01 16:15:48.904505 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:15:48.904412    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-5csbz_93dc6178-9868-4299-bad6-0ee49fc5d2ad/manager/0.log"
Nov 01 16:15:49.021366 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:15:49.021329    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-bgmv8_037ca099-5ad7-4874-a984-462c07801e70/cluster-monitoring-operator/0.log"
Nov 01 16:15:49.657293 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:15:49.657268    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-lr4hb_6c32c11d-d77f-458d-9cf0-fe262d31aeeb/server/0.log"
Nov 01 16:15:51.621775 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:15:51.621741    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bb9q4_503865ae-b17e-4535-b61e-29df5033497f/node-exporter/0.log"
Nov 01 16:15:51.752108 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:15:51.752072    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-cmbwq_6489d076-e1bd-402d-9d85-4e3bce4123b0/proxy/0.log"
Nov 01 16:15:52.022885 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:15:52.022854    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bb9q4_503865ae-b17e-4535-b61e-29df5033497f/kube-rbac-proxy/0.log"
Nov 01 16:15:52.424765 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:15:52.424679    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bb9q4_503865ae-b17e-4535-b61e-29df5033497f/init-textfile/0.log"
Nov 01 16:15:55.170026 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:15:55.169992    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7cfc97f8f7nb8nm_07f0043b-a848-44d4-9cc2-d6de906d2c01/istio-proxy/0.log"
Nov 01 16:15:57.504568 ip-10-0-134-37 kubenswrapper[4993]: E1101 16:15:57.504527    4993 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zq4pm" containerName="extractor"
Nov 01 16:15:58.147985 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:15:58.147950    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-4pmtn_306b0405-fb73-435a-ae73-42a90cd2d2f9/istio-proxy/0.log"
Nov 01 16:15:58.151667 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:15:58.151640    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-4pmtn_306b0405-fb73-435a-ae73-42a90cd2d2f9/istio-proxy/1.log"
Nov 01 16:15:58.572141 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:15:58.572113    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-flzf5_e3d2334c-8e83-4a16-a126-cbe45120efa7/istio-proxy/0.log"
Nov 01 16:15:59.662044 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:15:59.662009    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-q9j4g_29c351ad-629b-418c-b64d-9ff0e52ccdcb/istio-proxy/0.log"
Nov 01 16:16:00.306664 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:16:00.306630    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-65dccbd94c-bcw96_4dacd9b6-f9a3-4ae5-969b-c374a28c242a/istio-proxy/0.log"
Nov 01 16:16:00.331142 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:16:00.331102    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-nrd87_46a13905-2baf-4486-b3d7-ba080d5657cb/prometheus-operator-admission-webhook/0.log"
Nov 01 16:16:00.949096 ip-10-0-134-37 kubenswrapper[4993]: I1101 16:16:00.949063    4993 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-b9d995dcc-lnk2n_c27ff402-235a-4d73-a139-cf928a6d5cbe/router/0.log"