Nov 03 19:30:42.102706 ip-10-0-134-222 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 19:30:42.498947 ip-10-0-134-222 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 19:30:42.498947 ip-10-0-134-222 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 19:30:42.498947 ip-10-0-134-222 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 19:30:42.498947 ip-10-0-134-222 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 19:30:42.503157 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503063    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:30:42.503157 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503155    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:30:42.503157 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503160    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:30:42.503157 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503163    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:30:42.503293 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503167    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:30:42.503293 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503170    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:30:42.503293 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503173    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:30:42.503293 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503176    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:30:42.503293 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503179    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:30:42.503293 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503182    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:30:42.503293 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503184    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:30:42.503293 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503188    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:30:42.503293 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503190    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:30:42.503293 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503193    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:30:42.503293 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503196    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:30:42.503293 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503198    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:30:42.503293 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503201    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:30:42.503293 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503204    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:30:42.503293 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503206    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:30:42.503293 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503210    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:30:42.503293 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503213    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:30:42.503293 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503216    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:30:42.503293 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503218    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:30:42.503293 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503221    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:30:42.503772 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503223    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:30:42.503772 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503226    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:30:42.503772 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503229    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:30:42.503772 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503232    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:30:42.503772 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503236    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:30:42.503772 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503247    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:30:42.503772 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503251    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:30:42.503772 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503254    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:30:42.503772 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503258    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:30:42.503772 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503261    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:30:42.503772 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503264    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:30:42.503772 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503267    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:30:42.503772 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503269    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:30:42.503772 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503273    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:30:42.503772 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503276    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:30:42.503772 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503279    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:30:42.503772 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503282    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:30:42.503772 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503284    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:30:42.503772 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503287    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:30:42.504256 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503290    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:30:42.504256 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503293    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:30:42.504256 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503295    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:30:42.504256 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503298    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:30:42.504256 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503305    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:30:42.504256 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503307    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:30:42.504256 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503310    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:30:42.504256 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503313    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:30:42.504256 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503315    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:30:42.504256 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503318    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:30:42.504256 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503321    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:30:42.504256 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503324    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:30:42.504256 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503326    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:30:42.504256 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503329    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:30:42.504256 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503331    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:30:42.504256 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503334    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:30:42.504256 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503338    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:30:42.504256 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503360    2575 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:30:42.504256 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503364    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:30:42.504728 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503368    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:30:42.504728 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503371    2575 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:30:42.504728 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503374    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:30:42.504728 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503390    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:30:42.504728 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503393    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:30:42.504728 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503396    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:30:42.504728 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503399    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:30:42.504728 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503402    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:30:42.504728 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503405    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:30:42.504728 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503407    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:30:42.504728 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503410    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:30:42.504728 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503413    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:30:42.504728 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503415    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:30:42.504728 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503418    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:30:42.504728 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503421    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:30:42.504728 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503423    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:30:42.504728 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503426    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:30:42.504728 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503428    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:30:42.504728 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503432    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:30:42.505187 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503435    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:30:42.505187 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503438    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:30:42.505187 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503440    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:30:42.505187 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503443    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:30:42.505187 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503446    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:30:42.505187 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503448    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:30:42.505187 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503451    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:30:42.505187 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503453    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:30:42.505187 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503455    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:30:42.505187 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503458    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:30:42.505187 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503460    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:30:42.505187 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503463    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:30:42.505187 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503466    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:30:42.505187 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503468    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:30:42.505187 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503470    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:30:42.505187 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503473    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:30:42.505187 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503475    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:30:42.505187 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503479    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:30:42.505187 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503482    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:30:42.505649 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503484    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:30:42.505649 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503487    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:30:42.505649 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503490    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:30:42.505649 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503492    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:30:42.505649 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503497    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:30:42.505649 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.503500    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:30:42.505649 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504299    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:30:42.505649 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504306    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:30:42.505649 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504309    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:30:42.505649 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504312    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:30:42.505649 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504314    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:30:42.505649 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504317    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:30:42.505649 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504320    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:30:42.505649 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504324    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:30:42.505649 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504326    2575 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:30:42.505649 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504330    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:30:42.505649 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504332    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:30:42.505649 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504335    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:30:42.505649 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504338    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:30:42.506160 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504340    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:30:42.506160 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504343    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:30:42.506160 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504345    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:30:42.506160 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504348    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:30:42.506160 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504351    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:30:42.506160 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504353    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:30:42.506160 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504356    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:30:42.506160 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504358    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:30:42.506160 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504361    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:30:42.506160 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504363    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:30:42.506160 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504366    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:30:42.506160 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504369    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:30:42.506160 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504372    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:30:42.506160 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504374    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:30:42.506160 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504393    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:30:42.506160 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504396    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:30:42.506160 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504399    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:30:42.506160 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504402    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:30:42.506160 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504404    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:30:42.506160 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504407    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:30:42.506686 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504410    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:30:42.506686 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504413    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:30:42.506686 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504415    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:30:42.506686 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504418    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:30:42.506686 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504421    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:30:42.506686 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504423    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:30:42.506686 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504426    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:30:42.506686 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504431    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:30:42.506686 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504435    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:30:42.506686 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504438    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:30:42.506686 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504441    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:30:42.506686 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504444    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:30:42.506686 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504446    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:30:42.506686 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504449    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:30:42.506686 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504453    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:30:42.506686 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504455    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:30:42.506686 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504458    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:30:42.506686 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504461    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:30:42.506686 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504463    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:30:42.506686 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504466    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:30:42.507169 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504469    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:30:42.507169 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504471    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:30:42.507169 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504474    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:30:42.507169 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504477    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:30:42.507169 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504480    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:30:42.507169 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504483    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:30:42.507169 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504486    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:30:42.507169 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504489    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:30:42.507169 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504492    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:30:42.507169 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504494    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:30:42.507169 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504497    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:30:42.507169 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504500    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:30:42.507169 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504502    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:30:42.507169 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504505    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:30:42.507169 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504508    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:30:42.507169 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504511    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:30:42.507169 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504513    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:30:42.507169 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504516    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:30:42.507169 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504518    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:30:42.507169 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504521    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:30:42.507711 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504531    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:30:42.507711 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504533    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:30:42.507711 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504536    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:30:42.507711 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504539    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:30:42.507711 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504541    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:30:42.507711 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504544    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:30:42.507711 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504546    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:30:42.507711 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504549    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:30:42.507711 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504552    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:30:42.507711 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504554    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:30:42.507711 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504557    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:30:42.507711 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504559    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:30:42.507711 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504562    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:30:42.507711 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504565    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:30:42.507711 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504568    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:30:42.507711 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504570    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:30:42.507711 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504573    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:30:42.507711 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504576    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:30:42.507711 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504579    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:30:42.508200 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504582    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:30:42.508200 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504584    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:30:42.508200 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504587    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:30:42.508200 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504590    2575 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:30:42.508200 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504592    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:30:42.508200 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504595    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:30:42.508200 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504598    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:30:42.508200 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504601    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:30:42.508200 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504605    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:30:42.508200 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504609    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:30:42.508200 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504612    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:30:42.508200 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504614    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:30:42.508200 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504617    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:30:42.508200 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.504619    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:30:42.508200 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505157    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:30:42.508200 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505162    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:30:42.508200 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505166    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:30:42.508200 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505169    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:30:42.508200 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505171    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:30:42.508676 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505174    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:30:42.508676 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505177    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:30:42.508676 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505179    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:30:42.508676 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505182    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:30:42.508676 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505185    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:30:42.508676 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505187    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:30:42.508676 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505190    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:30:42.508676 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505193    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:30:42.508676 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505196    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:30:42.508676 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505199    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:30:42.508676 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505201    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:30:42.508676 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505204    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:30:42.508676 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505207    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:30:42.508676 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505210    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:30:42.508676 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505212    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:30:42.508676 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505215    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:30:42.508676 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505218    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:30:42.508676 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505220    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:30:42.508676 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505223    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:30:42.509173 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505226    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:30:42.509173 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505228    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:30:42.509173 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505231    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:30:42.509173 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505233    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:30:42.509173 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505236    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:30:42.509173 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505239    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:30:42.509173 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505242    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:30:42.509173 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505246    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:30:42.509173 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505251    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:30:42.509173 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505254    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:30:42.509173 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505257    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:30:42.509173 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505260    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:30:42.509173 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505262    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:30:42.509173 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505265    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:30:42.509173 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505267    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:30:42.509173 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505269    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:30:42.509173 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505272    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:30:42.509173 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505275    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:30:42.509173 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505277    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:30:42.509723 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505280    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:30:42.509723 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505282    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:30:42.509723 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505285    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:30:42.509723 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505288    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:30:42.509723 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505290    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:30:42.509723 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505293    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:30:42.509723 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505295    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:30:42.509723 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505298    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:30:42.509723 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505301    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:30:42.509723 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505303    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:30:42.509723 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505306    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:30:42.509723 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505309    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:30:42.509723 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505312    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:30:42.509723 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505316    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:30:42.509723 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505320    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:30:42.509723 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505324    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:30:42.509723 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505327    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:30:42.509723 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505329    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:30:42.509723 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505332    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:30:42.510220 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505335    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:30:42.510220 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505337    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:30:42.510220 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505339    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:30:42.510220 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505343    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:30:42.510220 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505345    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:30:42.510220 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505348    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:30:42.510220 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505350    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:30:42.510220 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505354    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:30:42.510220 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505356    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:30:42.510220 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505359    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:30:42.510220 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505361    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:30:42.510220 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505364    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:30:42.510220 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505366    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:30:42.510220 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505368    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:30:42.510220 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505371    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:30:42.510220 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505374    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:30:42.510220 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505391    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:30:42.510220 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505393    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:30:42.510220 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505396    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:30:42.510220 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505399    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:30:42.510741 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505401    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:30:42.510741 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505404    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:30:42.510741 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505407    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:30:42.510741 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505409    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:30:42.510741 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505412    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:30:42.510741 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505414    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:30:42.510741 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505417    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:30:42.510741 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505420    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:30:42.510741 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505422    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:30:42.510741 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505424    2575 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:30:42.510741 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505427    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:30:42.510741 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505430    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:30:42.510741 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505432    2575 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:30:42.510741 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505435    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:30:42.510741 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505437    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:30:42.510741 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505439    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:30:42.510741 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505442    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:30:42.510741 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505445    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:30:42.510741 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505448    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:30:42.510741 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505450    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:30:42.511252 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505453    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:30:42.511252 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505456    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:30:42.511252 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505459    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:30:42.511252 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.505461    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:30:42.511252 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506478    2575 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 19:30:42.511252 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506489    2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 19:30:42.511252 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506497    2575 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 19:30:42.511252 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506502    2575 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 19:30:42.511252 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506507    2575 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 19:30:42.511252 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506511    2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 19:30:42.511252 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506515    2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 19:30:42.511252 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506520    2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 19:30:42.511252 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506524    2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 19:30:42.511252 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506527    2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 19:30:42.511252 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506531    2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 19:30:42.511252 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506535    2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 19:30:42.511252 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506538    2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 19:30:42.511252 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506541    2575 flags.go:64] FLAG: --cgroup-root=""
Nov 03 19:30:42.511252 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506544    2575 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 19:30:42.511252 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506547    2575 flags.go:64] FLAG: --client-ca-file=""
Nov 03 19:30:42.511252 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506550    2575 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 19:30:42.511252 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506553    2575 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 19:30:42.511252 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506558    2575 flags.go:64] FLAG: --cluster-domain=""
Nov 03 19:30:42.511252 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506561    2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 19:30:42.511845 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506564    2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 19:30:42.511845 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506568    2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 19:30:42.511845 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506571    2575 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 19:30:42.511845 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506576    2575 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 19:30:42.511845 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506579    2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 19:30:42.511845 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506583    2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 19:30:42.511845 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506587    2575 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 19:30:42.511845 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506590    2575 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 19:30:42.511845 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506593    2575 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 19:30:42.511845 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506597    2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 19:30:42.511845 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506600    2575 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 19:30:42.511845 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506604    2575 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 19:30:42.511845 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506609    2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 19:30:42.511845 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506612    2575 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 19:30:42.511845 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506615    2575 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 19:30:42.511845 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506618    2575 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 19:30:42.511845 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506622    2575 flags.go:64] FLAG: --enable-server="true"
Nov 03 19:30:42.511845 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506625    2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 19:30:42.511845 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506630    2575 flags.go:64] FLAG: --event-burst="100"
Nov 03 19:30:42.511845 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506633    2575 flags.go:64] FLAG: --event-qps="50"
Nov 03 19:30:42.511845 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506636    2575 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 19:30:42.511845 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506640    2575 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 19:30:42.511845 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506643    2575 flags.go:64] FLAG: --eviction-hard=""
Nov 03 19:30:42.511845 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506647    2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 19:30:42.512496 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506651    2575 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 19:30:42.512496 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506654    2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 19:30:42.512496 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506657    2575 flags.go:64] FLAG: --eviction-soft=""
Nov 03 19:30:42.512496 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506660    2575 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 19:30:42.512496 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506663    2575 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 19:30:42.512496 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506666    2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 19:30:42.512496 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506672    2575 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 19:30:42.512496 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506675    2575 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 19:30:42.512496 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506678    2575 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 19:30:42.512496 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506681    2575 flags.go:64] FLAG: --feature-gates=""
Nov 03 19:30:42.512496 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506685    2575 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 19:30:42.512496 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506689    2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 19:30:42.512496 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506692    2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 19:30:42.512496 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506696    2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 19:30:42.512496 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506699    2575 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 19:30:42.512496 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506703    2575 flags.go:64] FLAG: --help="false"
Nov 03 19:30:42.512496 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506706    2575 flags.go:64] FLAG: --hostname-override="ip-10-0-134-222.us-east-2.compute.internal"
Nov 03 19:30:42.512496 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506710    2575 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 19:30:42.512496 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506713    2575 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 19:30:42.512496 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506716    2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 19:30:42.512496 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506719    2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 19:30:42.512496 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506723    2575 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 19:30:42.512496 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506726    2575 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 19:30:42.513040 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506729    2575 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 19:30:42.513040 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506732    2575 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 19:30:42.513040 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506735    2575 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 19:30:42.513040 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506738    2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 19:30:42.513040 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506742    2575 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 19:30:42.513040 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506745    2575 flags.go:64] FLAG: --kube-reserved=""
Nov 03 19:30:42.513040 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506748    2575 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 19:30:42.513040 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506751    2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 19:30:42.513040 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506754    2575 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 19:30:42.513040 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506757    2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 19:30:42.513040 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506760    2575 flags.go:64] FLAG: --lock-file=""
Nov 03 19:30:42.513040 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506763    2575 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 19:30:42.513040 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506766    2575 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 19:30:42.513040 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506769    2575 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 19:30:42.513040 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506776    2575 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 19:30:42.513040 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506782    2575 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 19:30:42.513040 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506785    2575 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 19:30:42.513040 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506788    2575 flags.go:64] FLAG: --logging-format="text"
Nov 03 19:30:42.513040 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506791    2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 19:30:42.513040 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506794    2575 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 19:30:42.513040 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506798    2575 flags.go:64] FLAG: --manifest-url=""
Nov 03 19:30:42.513040 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506801    2575 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 19:30:42.513040 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506806    2575 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 19:30:42.513040 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506809    2575 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 19:30:42.513040 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506813    2575 flags.go:64] FLAG: --max-pods="110"
Nov 03 19:30:42.513654 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506817    2575 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 19:30:42.513654 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506820    2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 19:30:42.513654 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506823    2575 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 19:30:42.513654 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506826    2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 19:30:42.513654 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506830    2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 19:30:42.513654 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506834    2575 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 19:30:42.513654 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506837    2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 19:30:42.513654 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506846    2575 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 19:30:42.513654 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506849    2575 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 19:30:42.513654 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506852    2575 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 19:30:42.513654 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506855    2575 flags.go:64] FLAG: --pod-cidr=""
Nov 03 19:30:42.513654 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506859    2575 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 19:30:42.513654 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506862    2575 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 19:30:42.513654 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506865    2575 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 19:30:42.513654 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506868    2575 flags.go:64] FLAG: --port="10250"
Nov 03 19:30:42.513654 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506871    2575 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 19:30:42.513654 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506874    2575 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0e99d08dc592670ed"
Nov 03 19:30:42.513654 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506877    2575 flags.go:64] FLAG: --qos-reserved=""
Nov 03 19:30:42.513654 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506880    2575 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 19:30:42.513654 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506884    2575 flags.go:64] FLAG: --register-node="true"
Nov 03 19:30:42.513654 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506887    2575 flags.go:64] FLAG: --register-with-taints=""
Nov 03 19:30:42.513654 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506891    2575 flags.go:64] FLAG: --registry-burst="10"
Nov 03 19:30:42.513654 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506895    2575 flags.go:64] FLAG: --registry-qps="5"
Nov 03 19:30:42.513654 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506898    2575 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 19:30:42.513654 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506901    2575 flags.go:64] FLAG: --reserved-memory=""
Nov 03 19:30:42.513654 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506905    2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 19:30:42.514276 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506908    2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 19:30:42.514276 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506911    2575 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 19:30:42.514276 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506914    2575 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 19:30:42.514276 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506917    2575 flags.go:64] FLAG: --runonce="false"
Nov 03 19:30:42.514276 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506920    2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 19:30:42.514276 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506923    2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 19:30:42.514276 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506927    2575 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 19:30:42.514276 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506930    2575 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 19:30:42.514276 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506933    2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 19:30:42.514276 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506936    2575 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 19:30:42.514276 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506941    2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 19:30:42.514276 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506944    2575 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 19:30:42.514276 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506947    2575 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 19:30:42.514276 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506951    2575 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 19:30:42.514276 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506954    2575 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 19:30:42.514276 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506957    2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 19:30:42.514276 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506960    2575 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 19:30:42.514276 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506963    2575 flags.go:64] FLAG: --system-cgroups=""
Nov 03 19:30:42.514276 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506966    2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 19:30:42.514276 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506972    2575 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 19:30:42.514276 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506975    2575 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 19:30:42.514276 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506978    2575 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 19:30:42.514276 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506983    2575 flags.go:64] FLAG: --tls-min-version=""
Nov 03 19:30:42.514276 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506985    2575 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 19:30:42.514276 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506988    2575 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 19:30:42.514885 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506992    2575 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 19:30:42.514885 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506994    2575 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 19:30:42.514885 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.506997    2575 flags.go:64] FLAG: --v="2"
Nov 03 19:30:42.514885 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.507004    2575 flags.go:64] FLAG: --version="false"
Nov 03 19:30:42.514885 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.507009    2575 flags.go:64] FLAG: --vmodule=""
Nov 03 19:30:42.514885 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.507013    2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 19:30:42.514885 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.507016    2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 19:30:42.514885 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507175    2575 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:30:42.514885 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507179    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:30:42.514885 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507183    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:30:42.514885 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507186    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:30:42.514885 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507189    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:30:42.514885 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507191    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:30:42.514885 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507194    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:30:42.514885 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507197    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:30:42.514885 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507200    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:30:42.514885 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507203    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:30:42.514885 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507207    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:30:42.514885 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507209    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:30:42.514885 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507212    2575 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:30:42.514885 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507215    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:30:42.514885 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507218    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:30:42.515444 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507221    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:30:42.515444 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507224    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:30:42.515444 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507227    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:30:42.515444 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507229    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:30:42.515444 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507232    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:30:42.515444 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507236    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:30:42.515444 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507240    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:30:42.515444 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507243    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:30:42.515444 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507246    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:30:42.515444 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507249    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:30:42.515444 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507252    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:30:42.515444 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507255    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:30:42.515444 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507258    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:30:42.515444 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507261    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:30:42.515444 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507264    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:30:42.515444 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507267    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:30:42.515444 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507269    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:30:42.515444 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507272    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:30:42.515444 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507274    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:30:42.515961 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507277    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:30:42.515961 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507280    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:30:42.515961 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507282    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:30:42.515961 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507285    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:30:42.515961 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507287    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:30:42.515961 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507290    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:30:42.515961 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507293    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:30:42.515961 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507296    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:30:42.515961 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507300    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:30:42.515961 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507302    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:30:42.515961 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507305    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:30:42.515961 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507314    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:30:42.515961 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507317    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:30:42.515961 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507320    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:30:42.515961 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507322    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:30:42.515961 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507325    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:30:42.515961 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507327    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:30:42.515961 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507330    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:30:42.515961 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507333    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:30:42.515961 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507335    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:30:42.516525 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507338    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:30:42.516525 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507341    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:30:42.516525 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507343    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:30:42.516525 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507346    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:30:42.516525 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507348    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:30:42.516525 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507351    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:30:42.516525 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507354    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:30:42.516525 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507356    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:30:42.516525 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507359    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:30:42.516525 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507361    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:30:42.516525 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507364    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:30:42.516525 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507367    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:30:42.516525 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507369    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:30:42.516525 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507372    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:30:42.516525 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507374    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:30:42.516525 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507387    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:30:42.516525 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507390    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:30:42.516525 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507393    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:30:42.516525 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507395    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:30:42.517025 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507406    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:30:42.517025 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507410    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:30:42.517025 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507413    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:30:42.517025 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507416    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:30:42.517025 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507418    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:30:42.517025 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507421    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:30:42.517025 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507424    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:30:42.517025 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507426    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:30:42.517025 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507429    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:30:42.517025 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507431    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:30:42.517025 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507434    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:30:42.517025 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507436    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:30:42.517025 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507439    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:30:42.517025 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507441    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:30:42.517025 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507444    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:30:42.517025 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507447    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:30:42.517025 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507449    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:30:42.517025 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507452    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:30:42.517025 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507454    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:30:42.517025 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507457    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:30:42.517529 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507460    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:30:42.517529 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507463    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:30:42.517529 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507465    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:30:42.517529 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507469    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:30:42.517529 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507472    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:30:42.517529 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507475    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:30:42.517529 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507478    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:30:42.517529 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507481    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:30:42.517529 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507483    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:30:42.517529 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507485    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:30:42.517529 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507488    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:30:42.517529 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507491    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:30:42.517529 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507493    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:30:42.517529 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.507529    2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 19:30:42.517529 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507572    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:30:42.517529 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507577    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:30:42.517912 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507580    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:30:42.517912 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507583    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:30:42.517912 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507585    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:30:42.517912 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507588    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:30:42.517912 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507590    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:30:42.517912 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507593    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:30:42.517912 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507596    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:30:42.517912 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507598    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:30:42.517912 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507601    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:30:42.517912 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507604    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:30:42.517912 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507606    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:30:42.517912 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507609    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:30:42.517912 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507611    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:30:42.517912 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507614    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:30:42.517912 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507616    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:30:42.517912 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507619    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:30:42.517912 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507622    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:30:42.517912 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507624    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:30:42.517912 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507627    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:30:42.517912 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507629    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:30:42.518394 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507632    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:30:42.518394 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507635    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:30:42.518394 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507637    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:30:42.518394 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507639    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:30:42.518394 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507643    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:30:42.518394 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507646    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:30:42.518394 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507649    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:30:42.518394 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507652    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:30:42.518394 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507654    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:30:42.518394 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507658    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:30:42.518394 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507666    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:30:42.518394 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507669    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:30:42.518394 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507672    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:30:42.518394 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507675    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:30:42.518394 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507677    2575 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:30:42.518394 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507680    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:30:42.518394 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507682    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:30:42.518394 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507685    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:30:42.518394 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507687    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:30:42.518835 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507690    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:30:42.518835 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507692    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:30:42.518835 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507695    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:30:42.518835 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507697    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:30:42.518835 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507700    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:30:42.518835 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507703    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:30:42.518835 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507705    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:30:42.518835 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507708    2575 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:30:42.518835 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507711    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:30:42.518835 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507715    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:30:42.518835 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507718    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:30:42.518835 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507721    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:30:42.518835 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507730    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:30:42.518835 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507733    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:30:42.518835 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507735    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:30:42.518835 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507740    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:30:42.518835 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507744    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:30:42.518835 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507748    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:30:42.518835 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507751    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:30:42.519286 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507755    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:30:42.519286 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507758    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:30:42.519286 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507761    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:30:42.519286 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507765    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:30:42.519286 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507767    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:30:42.519286 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507775    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:30:42.519286 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507778    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:30:42.519286 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507781    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:30:42.519286 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507784    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:30:42.519286 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507787    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:30:42.519286 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507789    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:30:42.519286 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507792    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:30:42.519286 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507794    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:30:42.519286 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507797    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:30:42.519286 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507800    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:30:42.519286 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507802    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:30:42.519286 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507805    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:30:42.519286 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507807    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:30:42.519286 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507810    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:30:42.519286 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507813    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:30:42.519777 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507815    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:30:42.519777 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507817    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:30:42.519777 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507821    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:30:42.519777 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507824    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:30:42.519777 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507826    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:30:42.519777 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507829    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:30:42.519777 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507831    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:30:42.519777 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507834    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:30:42.519777 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507837    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:30:42.519777 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507839    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:30:42.519777 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507842    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:30:42.519777 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507845    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:30:42.519777 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507848    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:30:42.519777 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507850    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:30:42.519777 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507856    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:30:42.519777 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507861    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:30:42.519777 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507864    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:30:42.519777 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507866    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:30:42.519777 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507874    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:30:42.520218 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507877    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:30:42.520218 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507880    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:30:42.520218 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507882    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:30:42.520218 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507885    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:30:42.520218 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507887    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:30:42.520218 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507890    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:30:42.520218 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.507892    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:30:42.520218 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.514837    2575 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 19:30:42.520218 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.514854    2575 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 19:30:42.520218 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.515446    2575 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 19:30:42.520218 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.515455    2575 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 19:30:42.520218 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515528    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:30:42.520218 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515534    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:30:42.520218 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515537    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:30:42.520218 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515541    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:30:42.520218 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515546    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:30:42.520218 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515550    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:30:42.520218 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515554    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:30:42.520218 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515556    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:30:42.520736 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515559    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:30:42.520736 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515563    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:30:42.520736 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515566    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:30:42.520736 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515568    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:30:42.520736 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515574    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:30:42.520736 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515578    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:30:42.520736 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515580    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:30:42.520736 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515583    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:30:42.520736 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515586    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:30:42.520736 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515589    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:30:42.520736 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515591    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:30:42.520736 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515594    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:30:42.520736 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515598    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:30:42.520736 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515601    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:30:42.520736 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515604    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:30:42.520736 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515607    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:30:42.520736 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515609    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:30:42.520736 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515612    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:30:42.520736 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515615    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:30:42.520736 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515617    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:30:42.521228 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515620    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:30:42.521228 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515623    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:30:42.521228 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515626    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:30:42.521228 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515629    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:30:42.521228 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515631    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:30:42.521228 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515634    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:30:42.521228 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515638    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:30:42.521228 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515642    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:30:42.521228 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515645    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:30:42.521228 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515648    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:30:42.521228 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515650    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:30:42.521228 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515653    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:30:42.521228 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515656    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:30:42.521228 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515658    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:30:42.521228 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515661    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:30:42.521228 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515663    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:30:42.521228 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515666    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:30:42.521228 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515668    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:30:42.521228 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515671    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:30:42.521707 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515674    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:30:42.521707 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515676    2575 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:30:42.521707 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515679    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:30:42.521707 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515681    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:30:42.521707 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515684    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:30:42.521707 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515686    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:30:42.521707 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515689    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:30:42.521707 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515692    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:30:42.521707 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515695    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:30:42.521707 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515698    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:30:42.521707 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515701    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:30:42.521707 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515703    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:30:42.521707 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515706    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:30:42.521707 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515709    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:30:42.521707 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515712    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:30:42.521707 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515714    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:30:42.521707 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515717    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:30:42.521707 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515720    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:30:42.521707 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515722    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:30:42.521707 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515725    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:30:42.522168 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515727    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:30:42.522168 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515730    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:30:42.522168 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515732    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:30:42.522168 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515735    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:30:42.522168 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515738    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:30:42.522168 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515740    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:30:42.522168 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515743    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:30:42.522168 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515746    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:30:42.522168 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515748    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:30:42.522168 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515751    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:30:42.522168 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515753    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:30:42.522168 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515756    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:30:42.522168 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515758    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:30:42.522168 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515761    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:30:42.522168 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515763    2575 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:30:42.522168 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515766    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:30:42.522168 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515768    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:30:42.522168 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515770    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:30:42.522168 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515773    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:30:42.522168 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515776    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:30:42.522645 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515779    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:30:42.522645 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515782    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:30:42.522645 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515784    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:30:42.522645 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515787    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:30:42.522645 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515789    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:30:42.522645 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515792    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:30:42.522645 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515795    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:30:42.522645 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515798    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:30:42.522645 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515800    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:30:42.522645 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515803    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:30:42.522645 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515806    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:30:42.522645 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515808    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:30:42.522645 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515811    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:30:42.522645 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515813    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:30:42.522645 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515816    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:30:42.522645 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515818    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:30:42.522645 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515821    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:30:42.522645 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515823    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:30:42.522645 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.515826    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:30:42.523243 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.515857    2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 19:30:42.523243 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516002    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:30:42.523243 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516007    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:30:42.523243 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516009    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:30:42.523243 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516012    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:30:42.523243 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516015    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:30:42.523243 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516018    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:30:42.523243 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516020    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:30:42.523243 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516023    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:30:42.523243 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516026    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:30:42.523243 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516028    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:30:42.523243 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516031    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:30:42.523243 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516033    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:30:42.523243 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516037    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:30:42.523243 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516041    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:30:42.523632 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516045    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:30:42.523632 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516049    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:30:42.523632 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516053    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:30:42.523632 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516056    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:30:42.523632 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516059    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:30:42.523632 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516062    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:30:42.523632 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516064    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:30:42.523632 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516066    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:30:42.523632 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516069    2575 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:30:42.523632 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516072    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:30:42.523632 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516074    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:30:42.523632 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516077    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:30:42.523632 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516079    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:30:42.523632 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516082    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:30:42.523632 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516085    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:30:42.523632 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516087    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:30:42.523632 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516090    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:30:42.523632 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516093    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:30:42.523632 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516097    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:30:42.524138 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516100    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:30:42.524138 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516103    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:30:42.524138 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516105    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:30:42.524138 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516108    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:30:42.524138 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516111    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:30:42.524138 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516113    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:30:42.524138 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516116    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:30:42.524138 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516119    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:30:42.524138 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516121    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:30:42.524138 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516124    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:30:42.524138 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516126    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:30:42.524138 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516129    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:30:42.524138 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516132    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:30:42.524138 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516135    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:30:42.524138 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516138    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:30:42.524138 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516140    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:30:42.524138 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516143    2575 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:30:42.524138 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516146    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:30:42.524138 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516149    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:30:42.524683 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516151    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:30:42.524683 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516154    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:30:42.524683 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516156    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:30:42.524683 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516159    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:30:42.524683 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516162    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:30:42.524683 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516164    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:30:42.524683 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516167    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:30:42.524683 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516169    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:30:42.524683 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516172    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:30:42.524683 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516174    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:30:42.524683 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516177    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:30:42.524683 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516179    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:30:42.524683 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516182    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:30:42.524683 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516184    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:30:42.524683 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516187    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:30:42.524683 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516189    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:30:42.524683 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516192    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:30:42.524683 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516194    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:30:42.524683 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516197    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:30:42.524683 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516200    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:30:42.525517 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516202    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:30:42.525517 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516205    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:30:42.525517 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516207    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:30:42.525517 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516210    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:30:42.525517 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516212    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:30:42.525517 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516215    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:30:42.525517 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516218    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:30:42.525517 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516221    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:30:42.525517 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516223    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:30:42.525517 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516226    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:30:42.525517 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516229    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:30:42.525517 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516232    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:30:42.525517 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516235    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:30:42.525517 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516237    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:30:42.525517 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516240    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:30:42.525517 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516242    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:30:42.525517 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516244    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:30:42.525517 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516247    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:30:42.525517 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516249    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:30:42.526140 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516252    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:30:42.526140 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516254    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:30:42.526140 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516257    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:30:42.526140 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516259    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:30:42.526140 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516262    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:30:42.526140 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516264    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:30:42.526140 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516267    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:30:42.526140 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516269    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:30:42.526140 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516272    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:30:42.526140 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516274    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:30:42.526140 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516277    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:30:42.526140 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516279    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:30:42.526140 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516282    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:30:42.526140 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516284    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:30:42.526140 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516287    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:30:42.526140 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.516309    2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 19:30:42.526563 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516340    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:30:42.526563 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516345    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:30:42.526563 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516349    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:30:42.526563 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516353    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:30:42.526563 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516356    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:30:42.526563 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516359    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:30:42.526563 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516362    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:30:42.526563 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516365    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:30:42.526563 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516367    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:30:42.526563 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516370    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:30:42.526563 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516373    2575 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:30:42.526563 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516389    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:30:42.526563 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516394    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:30:42.526563 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516398    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:30:42.526563 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516402    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:30:42.526563 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516405    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:30:42.526563 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516408    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:30:42.526563 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516411    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:30:42.526563 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516413    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:30:42.527028 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516416    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:30:42.527028 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516420    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:30:42.527028 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516423    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:30:42.527028 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516426    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:30:42.527028 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516430    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:30:42.527028 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516433    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:30:42.527028 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516436    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:30:42.527028 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516438    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:30:42.527028 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516441    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:30:42.527028 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516443    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:30:42.527028 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516446    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:30:42.527028 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516448    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:30:42.527028 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516451    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:30:42.527028 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516453    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:30:42.527028 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516456    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:30:42.527028 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516459    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:30:42.527028 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516461    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:30:42.527028 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516464    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:30:42.527028 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516467    2575 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:30:42.527566 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516470    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:30:42.527566 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516473    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:30:42.527566 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516475    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:30:42.527566 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516478    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:30:42.527566 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516480    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:30:42.527566 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516483    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:30:42.527566 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516485    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:30:42.527566 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516488    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:30:42.527566 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516491    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:30:42.527566 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516494    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:30:42.527566 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516497    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:30:42.527566 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516499    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:30:42.527566 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516502    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:30:42.527566 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516504    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:30:42.527566 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516507    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:30:42.527566 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516510    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:30:42.527566 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516512    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:30:42.527566 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516515    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:30:42.527566 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516517    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:30:42.527566 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516519    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:30:42.528051 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516522    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:30:42.528051 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516525    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:30:42.528051 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516528    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:30:42.528051 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516531    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:30:42.528051 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516533    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:30:42.528051 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516536    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:30:42.528051 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516539    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:30:42.528051 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516541    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:30:42.528051 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516544    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:30:42.528051 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516549    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:30:42.528051 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516552    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:30:42.528051 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516554    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:30:42.528051 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516557    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:30:42.528051 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516560    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:30:42.528051 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516563    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:30:42.528051 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516565    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:30:42.528051 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516568    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:30:42.528051 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516570    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:30:42.528051 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516573    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:30:42.528051 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516575    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:30:42.528536 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516578    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:30:42.528536 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516581    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:30:42.528536 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516584    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:30:42.528536 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516586    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:30:42.528536 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516589    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:30:42.528536 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516591    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:30:42.528536 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516594    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:30:42.528536 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516596    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:30:42.528536 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516599    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:30:42.528536 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516601    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:30:42.528536 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516604    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:30:42.528536 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516606    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:30:42.528536 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516609    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:30:42.528536 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516611    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:30:42.528536 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516614    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:30:42.528536 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516616    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:30:42.528536 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516619    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:30:42.528536 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516621    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:30:42.528536 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516624    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:30:42.529018 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516627    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:30:42.529018 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516630    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:30:42.529018 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516634    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:30:42.529018 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516637    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:30:42.529018 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516639    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:30:42.529018 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516642    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:30:42.529018 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516645    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:30:42.529018 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516648    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:30:42.529018 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:42.516650    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:30:42.529018 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.516708    2575 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 19:30:42.529018 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.519081    2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 03 19:30:42.529018 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.520155    2575 server.go:1014] "Starting client certificate rotation"
Nov 03 19:30:42.529018 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.520262    2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 19:30:42.529018 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.520305    2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 19:30:42.529018 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.522389    2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 19:30:42.529018 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.526029    2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 19:30:42.538729 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.538708    2575 log.go:25] "Validated CRI v1 runtime API"
Nov 03 19:30:42.541874 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.541856    2575 log.go:25] "Validated CRI v1 image API"
Nov 03 19:30:42.542733 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.542710    2575 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 19:30:42.547612 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.547582    2575 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 19:30:42.547914 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.547891    2575 fs.go:135] Filesystem UUIDs: map[2ac96c12-af93-455d-b27a-5d1a5d26e1de:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 a137b2b5-e435-49c9-b1c4-e835406e68cf:/dev/nvme0n1p3]
Nov 03 19:30:42.547992 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.547915    2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 03 19:30:42.553474 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.553327    2575 manager.go:217] Machine: {Timestamp:2026-11-03 19:30:42.55194985 +0000 UTC m=+0.329683329 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3083634 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24b01d75922ceb6cac702efe3d6796 SystemUUID:ec24b01d-7592-2ceb-6cac-702efe3d6796 BootID:6155d873-3697-438a-a3c0-03814e53fc30 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ff:bc:59:77 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ff:bc:59:77 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1a:b6:0e:89:bb:b2 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 19:30:42.554014 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.554000    2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 19:30:42.554137 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.554120    2575 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 19:30:42.556924 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.556891    2575 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 19:30:42.557069 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.556926    2575 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-222.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 19:30:42.557568 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.557557    2575 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 19:30:42.557604 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.557570    2575 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 19:30:42.557604 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.557597    2575 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 19:30:42.557658 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.557611    2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 19:30:42.558641 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.558630    2575 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 19:30:42.562270 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.562254    2575 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 19:30:42.562420 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.562409    2575 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 19:30:42.562473 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.562448    2575 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 19:30:42.562473 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.562459    2575 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 19:30:42.562473 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.562472    2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 19:30:42.562612 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.562483    2575 kubelet.go:405] "Adding apiserver pod source"
Nov 03 19:30:42.562612 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.562497    2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 19:30:42.562982 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.562965    2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 19:30:42.567935 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.567692    2575 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 19:30:42.572872 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.572855    2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 19:30:42.573690 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.573669    2575 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 19:30:42.573729 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.573708    2575 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 19:30:42.573904 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.573894    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 19:30:42.573939 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.573911    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 19:30:42.573939 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.573917    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 19:30:42.573939 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.573922    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 19:30:42.573939 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.573929    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 19:30:42.573939 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.573935    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 19:30:42.574431 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.574421    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 19:30:42.574469 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.574435    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 19:30:42.574469 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.574445    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 19:30:42.574469 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.574454    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 19:30:42.574469 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.574465    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 19:30:42.574609 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.574479    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 19:30:42.575165 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.575149    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 19:30:42.575226 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.575169    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 19:30:42.578602 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.578582    2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-222.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 03 19:30:42.579040 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:42.579006    2575 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Nov 03 19:30:42.579075 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:42.579011    2575 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-222.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Nov 03 19:30:42.580288 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.580276    2575 server.go:1263] "Started kubelet"
Nov 03 19:30:42.580415 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.580359    2575 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 19:30:42.580519 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.580399    2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 19:30:42.580519 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.580490    2575 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 19:30:42.581215 ip-10-0-134-222 systemd[1]: Started Kubernetes Kubelet.
Nov 03 19:30:42.581752 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.581735    2575 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 19:30:42.586433 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.585655    2575 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 19:30:42.586771 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.586741    2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ngjq7"
Nov 03 19:30:42.588120 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:42.588103    2575 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 03 19:30:42.589655 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.589637    2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 19:30:42.590591 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.590575    2575 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 19:30:42.590678 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:42.590659    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-222.us-east-2.compute.internal\" not found"
Nov 03 19:30:42.590921 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:42.590002    2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-222.us-east-2.compute.internal.18e49fca3e733a3e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-222.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-222.us-east-2.compute.internal,},FirstTimestamp:2026-11-03 19:30:42.58024915 +0000 UTC m=+0.357982629,LastTimestamp:2026-11-03 19:30:42.58024915 +0000 UTC m=+0.357982629,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-222.us-east-2.compute.internal,}"
Nov 03 19:30:42.591158 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.591142    2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 19:30:42.591269 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.591144    2575 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 19:30:42.591269 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.591261    2575 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 19:30:42.591406 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.591366    2575 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 19:30:42.591452 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.591418    2575 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 19:30:42.591518 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.591504    2575 factory.go:55] Registering systemd factory
Nov 03 19:30:42.591546 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.591528    2575 factory.go:223] Registration of the systemd container factory successfully
Nov 03 19:30:42.591806 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.591790    2575 factory.go:153] Registering CRI-O factory
Nov 03 19:30:42.591852 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.591810    2575 factory.go:223] Registration of the crio container factory successfully
Nov 03 19:30:42.591954 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.591942    2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 19:30:42.592028 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.592017    2575 factory.go:103] Registering Raw factory
Nov 03 19:30:42.592059 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.592036    2575 manager.go:1199] Started watching for new ooms in manager
Nov 03 19:30:42.593170 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.593152    2575 manager.go:319] Starting recovery of all containers
Nov 03 19:30:42.593779 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.593756    2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ngjq7"
Nov 03 19:30:42.603563 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:42.603509    2575 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-222.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 03 19:30:42.608791 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.608772    2575 manager.go:324] Recovery completed
Nov 03 19:30:42.608906 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.608797    2575 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 19:30:42.613530 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.613516    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 19:30:42.615312 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.615293    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-222.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 19:30:42.615421 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.615332    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-222.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 19:30:42.615421 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.615348    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-222.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 19:30:42.616068 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.616055    2575 cpu_manager.go:226] "Starting" policy="none"
Nov 03 19:30:42.616068 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.616064    2575 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 19:30:42.616175 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.616081    2575 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 19:30:42.616175 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.616113    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 19:30:42.616918 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.616903    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-222.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 19:30:42.616998 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.616936    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-222.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 19:30:42.616998 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.616951    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-222.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 19:30:42.618274 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.618258    2575 policy_none.go:50] "Start"
Nov 03 19:30:42.618320 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.618283    2575 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 19:30:42.618320 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.618300    2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 19:30:42.620538 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.620526    2575 policy_none.go:44] "Start"
Nov 03 19:30:42.670936 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.662857    2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 19:30:42.670936 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.663168    2575 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 19:30:42.670936 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.664215    2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 19:30:42.670936 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.664249    2575 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 19:30:42.670936 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.664269    2575 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 19:30:42.670936 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:42.664302    2575 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 03 19:30:42.670936 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.666815    2575 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 19:30:42.670936 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.668524    2575 manager.go:341] "Starting Device Plugin manager"
Nov 03 19:30:42.670936 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:42.669163    2575 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 19:30:42.670936 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.669182    2575 server.go:87] "Starting device plugin registration server"
Nov 03 19:30:42.670936 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.669521    2575 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 19:30:42.670936 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.669570    2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 19:30:42.670936 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.669873    2575 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 19:30:42.670936 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.669964    2575 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 19:30:42.670936 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.669976    2575 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 19:30:42.670936 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:42.670551    2575 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 19:30:42.670936 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:42.670591    2575 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-222.us-east-2.compute.internal\" not found"
Nov 03 19:30:42.764700 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.764627    2575 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-222.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-222.us-east-2.compute.internal"]
Nov 03 19:30:42.764785 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.764700    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 19:30:42.765751 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.765734    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-222.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 19:30:42.765843 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.765761    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-222.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 19:30:42.765843 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.765771    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-222.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 19:30:42.767920 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.767904    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 19:30:42.768077 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.768062    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-222.us-east-2.compute.internal"
Nov 03 19:30:42.768129 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.768092    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 19:30:42.768703 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.768688    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-222.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 19:30:42.768782 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.768718    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-222.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 19:30:42.768782 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.768733    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-222.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 19:30:42.768782 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.768724    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-222.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 19:30:42.768782 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.768758    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-222.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 19:30:42.768782 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.768780    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-222.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 19:30:42.769703 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.769688    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 19:30:42.770610 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.770590    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-222.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 19:30:42.770681 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.770621    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-222.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 19:30:42.770681 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.770647    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-222.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 19:30:42.770681 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.770679    2575 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-222.us-east-2.compute.internal"
Nov 03 19:30:42.770987 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.770970    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-222.us-east-2.compute.internal"
Nov 03 19:30:42.771069 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.770996    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 19:30:42.771804 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.771789    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-222.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 19:30:42.771879 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.771819    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-222.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 19:30:42.771879 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.771836    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-222.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 19:30:42.778762 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.778745    2575 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-222.us-east-2.compute.internal"
Nov 03 19:30:42.778847 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:42.778768    2575 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-222.us-east-2.compute.internal\": node \"ip-10-0-134-222.us-east-2.compute.internal\" not found"
Nov 03 19:30:42.792897 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.792870    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c5e667f56d4cbe64bbf7db6ca2bb9e9c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-222.us-east-2.compute.internal\" (UID: \"c5e667f56d4cbe64bbf7db6ca2bb9e9c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-222.us-east-2.compute.internal"
Nov 03 19:30:42.793040 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.792911    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5e667f56d4cbe64bbf7db6ca2bb9e9c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-222.us-east-2.compute.internal\" (UID: \"c5e667f56d4cbe64bbf7db6ca2bb9e9c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-222.us-east-2.compute.internal"
Nov 03 19:30:42.793040 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.792931    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/66cbb2b4d2abe69f7cf665c408777b87-config\") pod \"kube-apiserver-proxy-ip-10-0-134-222.us-east-2.compute.internal\" (UID: \"66cbb2b4d2abe69f7cf665c408777b87\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-222.us-east-2.compute.internal"
Nov 03 19:30:42.795234 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:42.795218    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-222.us-east-2.compute.internal\" not found"
Nov 03 19:30:42.801130 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:42.801103    2575 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-222.us-east-2.compute.internal\" not found" node="ip-10-0-134-222.us-east-2.compute.internal"
Nov 03 19:30:42.804155 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:42.804138    2575 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-222.us-east-2.compute.internal\" not found" node="ip-10-0-134-222.us-east-2.compute.internal"
Nov 03 19:30:42.893322 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.893271    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c5e667f56d4cbe64bbf7db6ca2bb9e9c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-222.us-east-2.compute.internal\" (UID: \"c5e667f56d4cbe64bbf7db6ca2bb9e9c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-222.us-east-2.compute.internal"
Nov 03 19:30:42.893322 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.893318    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5e667f56d4cbe64bbf7db6ca2bb9e9c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-222.us-east-2.compute.internal\" (UID: \"c5e667f56d4cbe64bbf7db6ca2bb9e9c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-222.us-east-2.compute.internal"
Nov 03 19:30:42.893549 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.893345    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/66cbb2b4d2abe69f7cf665c408777b87-config\") pod \"kube-apiserver-proxy-ip-10-0-134-222.us-east-2.compute.internal\" (UID: \"66cbb2b4d2abe69f7cf665c408777b87\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-222.us-east-2.compute.internal"
Nov 03 19:30:42.893549 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.893366    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c5e667f56d4cbe64bbf7db6ca2bb9e9c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-222.us-east-2.compute.internal\" (UID: \"c5e667f56d4cbe64bbf7db6ca2bb9e9c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-222.us-east-2.compute.internal"
Nov 03 19:30:42.893549 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.893401    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5e667f56d4cbe64bbf7db6ca2bb9e9c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-222.us-east-2.compute.internal\" (UID: \"c5e667f56d4cbe64bbf7db6ca2bb9e9c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-222.us-east-2.compute.internal"
Nov 03 19:30:42.893549 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:42.893433    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/66cbb2b4d2abe69f7cf665c408777b87-config\") pod \"kube-apiserver-proxy-ip-10-0-134-222.us-east-2.compute.internal\" (UID: \"66cbb2b4d2abe69f7cf665c408777b87\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-222.us-east-2.compute.internal"
Nov 03 19:30:42.895321 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:42.895301    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-222.us-east-2.compute.internal\" not found"
Nov 03 19:30:42.996024 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:42.995833    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-222.us-east-2.compute.internal\" not found"
Nov 03 19:30:43.096684 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:43.096619    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-222.us-east-2.compute.internal\" not found"
Nov 03 19:30:43.104810 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:43.104785    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-222.us-east-2.compute.internal"
Nov 03 19:30:43.107959 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:43.107942    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-222.us-east-2.compute.internal"
Nov 03 19:30:44.282009 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:44.281971    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-222.us-east-2.compute.internal\" not found"
Nov 03 19:30:44.382500 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:44.382467    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-222.us-east-2.compute.internal\" not found"
Nov 03 19:30:44.483042 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:44.482953    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-222.us-east-2.compute.internal\" not found"
Nov 03 19:30:44.583625 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:44.583598    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-222.us-east-2.compute.internal\" not found"
Nov 03 19:30:44.603739 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:44.603717    2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 03 19:30:44.603868 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:44.603850    2575 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 19:30:44.603923 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:44.603891    2575 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 19:30:44.683740 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:44.683713    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-222.us-east-2.compute.internal\" not found"
Nov 03 19:30:44.689200 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:44.689163    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 19:25:43 +0000 UTC" deadline="2028-06-29 07:30:01.136262024 +0000 UTC"
Nov 03 19:30:44.689200 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:44.689198    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14483h59m16.447067994s"
Nov 03 19:30:44.784648 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:44.784609    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-222.us-east-2.compute.internal\" not found"
Nov 03 19:30:44.847417 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:44.847373    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5e667f56d4cbe64bbf7db6ca2bb9e9c.slice/crio-24683f3bceaa61bf9e0e4d275b35e9a826b29dfed6d50b5f0ca3e5f14ddb40fa WatchSource:0}: Error finding container 24683f3bceaa61bf9e0e4d275b35e9a826b29dfed6d50b5f0ca3e5f14ddb40fa: Status 404 returned error can't find the container with id 24683f3bceaa61bf9e0e4d275b35e9a826b29dfed6d50b5f0ca3e5f14ddb40fa
Nov 03 19:30:44.853991 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:44.853969    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 19:30:44.885194 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:44.885160    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-222.us-east-2.compute.internal\" not found"
Nov 03 19:30:44.892234 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:44.892204    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66cbb2b4d2abe69f7cf665c408777b87.slice/crio-fe3b22311179a16c8c3eeab442b74b9b6046baa06d8bafb32318503a0d2e7908 WatchSource:0}: Error finding container fe3b22311179a16c8c3eeab442b74b9b6046baa06d8bafb32318503a0d2e7908: Status 404 returned error can't find the container with id fe3b22311179a16c8c3eeab442b74b9b6046baa06d8bafb32318503a0d2e7908
Nov 03 19:30:44.986333 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:44.986247    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-222.us-east-2.compute.internal\" not found"
Nov 03 19:30:45.086934 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:45.086904    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-222.us-east-2.compute.internal\" not found"
Nov 03 19:30:45.093145 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.093119    2575 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 19:30:45.187900 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:45.187854    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-222.us-east-2.compute.internal\" not found"
Nov 03 19:30:45.246997 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.246921    2575 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 19:30:45.275971 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.275936    2575 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-222.us-east-2.compute.internal"
Nov 03 19:30:45.287055 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.287030    2575 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 19:30:45.288066 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.288052    2575 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-222.us-east-2.compute.internal"
Nov 03 19:30:45.299415 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.299390    2575 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 19:30:45.591362 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.591277    2575 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 19:30:45.648176 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.648126    2575 apiserver.go:52] "Watching apiserver"
Nov 03 19:30:45.662550 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.660464    2575 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 19:30:45.662550 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.661138    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-l98pn","openshift-multus/multus-9ck9p","openshift-network-operator/iptables-alerter-wkrtt","openshift-ovn-kubernetes/ovnkube-node-k5nbr","kube-system/global-pull-secret-syncer-hrxdt","openshift-image-registry/node-ca-vv2s2","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-222.us-east-2.compute.internal","openshift-multus/network-metrics-daemon-rdr5d","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mcxcv","kube-system/kube-apiserver-proxy-ip-10-0-134-222.us-east-2.compute.internal","openshift-dns/node-resolver-p2p8d","openshift-multus/multus-additional-cni-plugins-g67bt","openshift-network-diagnostics/network-check-target-njg27","kube-system/konnectivity-agent-jwrp9"]
Nov 03 19:30:45.664472 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.664446    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njg27"
Nov 03 19:30:45.664592 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:45.664543    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njg27" podUID="b46f51b2-436b-4967-8879-a16b58930591"
Nov 03 19:30:45.666155 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.666128    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rdr5d"
Nov 03 19:30:45.666259 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:45.666213    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rdr5d" podUID="790607e1-d57c-4622-9a72-1af902ab1d85"
Nov 03 19:30:45.668349 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.668324    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jwrp9"
Nov 03 19:30:45.670625 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.670603    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-p2p8d"
Nov 03 19:30:45.672635 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.672613    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 19:30:45.672785 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.672768    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 19:30:45.672910 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.672883    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-kqxn2\""
Nov 03 19:30:45.673041 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.673025    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9ck9p"
Nov 03 19:30:45.674497 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.674478    2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 03 19:30:45.675195 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.675087    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wkrtt"
Nov 03 19:30:45.677707 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.677510    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:30:45.678662 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.678519    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 19:30:45.681904 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.681883    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrxdt"
Nov 03 19:30:45.681997 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:45.681957    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrxdt" podUID="eee821c4-6fb0-4f84-a58d-23dc2ff77720"
Nov 03 19:30:45.682312 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.682295    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vv2s2"
Nov 03 19:30:45.684615 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.684598    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-l98pn"
Nov 03 19:30:45.687077 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.687058    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mcxcv"
Nov 03 19:30:45.688725 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.688701    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g8fh\" (UniqueName: \"kubernetes.io/projected/8b5e1b09-699d-4db8-9eb0-0b71bdda858f-kube-api-access-2g8fh\") pod \"iptables-alerter-wkrtt\" (UID: \"8b5e1b09-699d-4db8-9eb0-0b71bdda858f\") " pod="openshift-network-operator/iptables-alerter-wkrtt"
Nov 03 19:30:45.688819 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.688739    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-run-systemd\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:30:45.688819 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.688767    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/653db542-8702-4eaa-abc1-a3bb866163c6-cnibin\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:30:45.688819 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.688791    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/653db542-8702-4eaa-abc1-a3bb866163c6-multus-socket-dir-parent\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:30:45.688819 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.688817    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8b5e1b09-699d-4db8-9eb0-0b71bdda858f-iptables-alerter-script\") pod \"iptables-alerter-wkrtt\" (UID: \"8b5e1b09-699d-4db8-9eb0-0b71bdda858f\") " pod="openshift-network-operator/iptables-alerter-wkrtt"
Nov 03 19:30:45.689066 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.688856    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-host-run-ovn-kubernetes\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:30:45.689066 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.688904    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-host-cni-bin\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:30:45.689066 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.688926    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-host-cni-netd\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:30:45.689066 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.688950    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d0ab7c64-a85f-4713-a270-f0cf711fcee5-ovnkube-config\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:30:45.689066 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.688972    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d0ab7c64-a85f-4713-a270-f0cf711fcee5-env-overrides\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:30:45.689066 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.688999    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/653db542-8702-4eaa-abc1-a3bb866163c6-cni-binary-copy\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:30:45.689066 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.689049    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc8v6\" (UniqueName: \"kubernetes.io/projected/d0ab7c64-a85f-4713-a270-f0cf711fcee5-kube-api-access-jc8v6\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:30:45.689419 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.689075    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/653db542-8702-4eaa-abc1-a3bb866163c6-host-var-lib-cni-multus\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:30:45.689419 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.689097    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/653db542-8702-4eaa-abc1-a3bb866163c6-multus-daemon-config\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:30:45.689419 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.689119    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-node-log\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:30:45.689419 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.689153    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:30:45.689419 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.689188    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1c00e787-a35d-4ac4-95a4-2546e29602aa-konnectivity-ca\") pod \"konnectivity-agent-jwrp9\" (UID: \"1c00e787-a35d-4ac4-95a4-2546e29602aa\") " pod="kube-system/konnectivity-agent-jwrp9"
Nov 03 19:30:45.689419 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.689218    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/56e21184-e568-46bc-8e90-204684d6c366-hosts-file\") pod \"node-resolver-p2p8d\" (UID: \"56e21184-e568-46bc-8e90-204684d6c366\") " pod="openshift-dns/node-resolver-p2p8d"
Nov 03 19:30:45.689419 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.689246    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-systemd-units\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:30:45.689419 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.689270    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-host-slash\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:30:45.689419 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.689291    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-host-run-netns\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:30:45.689419 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.689322    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-etc-openvswitch\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:30:45.689419 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.689358    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d0ab7c64-a85f-4713-a270-f0cf711fcee5-ovn-node-metrics-cert\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:30:45.689419 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.689382    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d0ab7c64-a85f-4713-a270-f0cf711fcee5-ovnkube-script-lib\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:30:45.689419 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.689405    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/653db542-8702-4eaa-abc1-a3bb866163c6-etc-kubernetes\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:30:45.690073 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.689428    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-var-lib-openvswitch\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:30:45.690073 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.689452    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/790607e1-d57c-4622-9a72-1af902ab1d85-metrics-certs\") pod \"network-metrics-daemon-rdr5d\" (UID: \"790607e1-d57c-4622-9a72-1af902ab1d85\") " pod="openshift-multus/network-metrics-daemon-rdr5d"
Nov 03 19:30:45.690073 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.689484    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgtvt\" (UniqueName: \"kubernetes.io/projected/790607e1-d57c-4622-9a72-1af902ab1d85-kube-api-access-hgtvt\") pod \"network-metrics-daemon-rdr5d\" (UID: \"790607e1-d57c-4622-9a72-1af902ab1d85\") " pod="openshift-multus/network-metrics-daemon-rdr5d"
Nov 03 19:30:45.690073 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.689512    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/653db542-8702-4eaa-abc1-a3bb866163c6-system-cni-dir\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:30:45.690073 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.689535    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/653db542-8702-4eaa-abc1-a3bb866163c6-host-run-multus-certs\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:30:45.690073 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.689566    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpwxv\" (UniqueName: \"kubernetes.io/projected/653db542-8702-4eaa-abc1-a3bb866163c6-kube-api-access-mpwxv\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:30:45.690073 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.689589    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8b5e1b09-699d-4db8-9eb0-0b71bdda858f-host-slash\") pod \"iptables-alerter-wkrtt\" (UID: \"8b5e1b09-699d-4db8-9eb0-0b71bdda858f\") " pod="openshift-network-operator/iptables-alerter-wkrtt"
Nov 03 19:30:45.690073 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.689612    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-run-openvswitch\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:30:45.690073 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.689634    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-log-socket\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:30:45.690073 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.689686    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1c00e787-a35d-4ac4-95a4-2546e29602aa-agent-certs\") pod \"konnectivity-agent-jwrp9\" (UID: \"1c00e787-a35d-4ac4-95a4-2546e29602aa\") " pod="kube-system/konnectivity-agent-jwrp9"
Nov 03 19:30:45.690073 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.689715    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/653db542-8702-4eaa-abc1-a3bb866163c6-host-run-k8s-cni-cncf-io\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:30:45.690073 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.689737    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/653db542-8702-4eaa-abc1-a3bb866163c6-host-run-netns\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:30:45.690073 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.689760    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/653db542-8702-4eaa-abc1-a3bb866163c6-host-var-lib-cni-bin\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:30:45.690073 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.689781    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/653db542-8702-4eaa-abc1-a3bb866163c6-hostroot\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:30:45.690073 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.689806    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/653db542-8702-4eaa-abc1-a3bb866163c6-multus-conf-dir\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:30:45.690073 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.689829    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-host-kubelet\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:30:45.690073 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.689852    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-run-ovn\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:30:45.690727 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.689879    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f75xn\" (UniqueName: \"kubernetes.io/projected/b46f51b2-436b-4967-8879-a16b58930591-kube-api-access-f75xn\") pod \"network-check-target-njg27\" (UID: \"b46f51b2-436b-4967-8879-a16b58930591\") " pod="openshift-network-diagnostics/network-check-target-njg27"
Nov 03 19:30:45.690727 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.689903    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/653db542-8702-4eaa-abc1-a3bb866163c6-multus-cni-dir\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:30:45.690727 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.689929    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/653db542-8702-4eaa-abc1-a3bb866163c6-host-var-lib-kubelet\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:30:45.690727 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.689953    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56e21184-e568-46bc-8e90-204684d6c366-tmp-dir\") pod \"node-resolver-p2p8d\" (UID: \"56e21184-e568-46bc-8e90-204684d6c366\") " pod="openshift-dns/node-resolver-p2p8d"
Nov 03 19:30:45.690727 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.689978    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wk2r\" (UniqueName: \"kubernetes.io/projected/56e21184-e568-46bc-8e90-204684d6c366-kube-api-access-6wk2r\") pod \"node-resolver-p2p8d\" (UID: \"56e21184-e568-46bc-8e90-204684d6c366\") " pod="openshift-dns/node-resolver-p2p8d"
Nov 03 19:30:45.690727 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.690207    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-g67bt"
Nov 03 19:30:45.694942 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.694901    2575 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 19:30:45.707711 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.707688    2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-9smz7"
Nov 03 19:30:45.714504 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.714476    2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-9smz7"
Nov 03 19:30:45.754375 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.754330    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-222.us-east-2.compute.internal" event={"ID":"66cbb2b4d2abe69f7cf665c408777b87","Type":"ContainerStarted","Data":"fe3b22311179a16c8c3eeab442b74b9b6046baa06d8bafb32318503a0d2e7908"}
Nov 03 19:30:45.755547 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.755517    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-222.us-east-2.compute.internal" event={"ID":"c5e667f56d4cbe64bbf7db6ca2bb9e9c","Type":"ContainerStarted","Data":"24683f3bceaa61bf9e0e4d275b35e9a826b29dfed6d50b5f0ca3e5f14ddb40fa"}
Nov 03 19:30:45.773027 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.772993    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 19:30:45.773249 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.773164    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 19:30:45.773249 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.773194    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 19:30:45.773379 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.773301    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 19:30:45.773379 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.773324    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 19:30:45.773483 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.773397    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 19:30:45.773483 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.773447    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 19:30:45.773578 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.773545    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 19:30:45.773627 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.773611    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 19:30:45.773692 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.773657    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-k6x7r\""
Nov 03 19:30:45.774882 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.774862    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 19:30:45.774986 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.774862    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 19:30:45.774986 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.774899    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-vgldx\""
Nov 03 19:30:45.775477 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.775460    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 19:30:45.775567 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.775499    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 19:30:45.775567 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.775535    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 19:30:45.775683 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.775568    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9h8df\""
Nov 03 19:30:45.775683 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.775630    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 19:30:45.776232 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.775887    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 19:30:45.776232 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.775975    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 19:30:45.776232 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.776023    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 19:30:45.776232 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.776066    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 19:30:45.776232 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.776104    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 19:30:45.776232 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.776138    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 19:30:45.776232 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.776191    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 19:30:45.776576 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.776335    2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 19:30:45.777844 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.777825    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 19:30:45.778083 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.778069    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6p8wh\""
Nov 03 19:30:45.778200 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.778186    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-wllhj\""
Nov 03 19:30:45.778265 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.778224    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pq7nf\""
Nov 03 19:30:45.778265 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.778243    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-bh6bb\""
Nov 03 19:30:45.778358 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.778261    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 19:30:45.778358 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.778300    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-glk7q\""
Nov 03 19:30:45.791239 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.791201    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/653db542-8702-4eaa-abc1-a3bb866163c6-host-var-lib-cni-multus\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:30:45.791239 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.791238    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-node-log\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:30:45.791422 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.791268    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:30:45.791422 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.791297    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b797bdb0-b354-4dfc-a50a-3c60ecacbf71-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g67bt\" (UID: \"b797bdb0-b354-4dfc-a50a-3c60ecacbf71\") " pod="openshift-multus/multus-additional-cni-plugins-g67bt"
Nov 03 19:30:45.791422 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.791324    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25-etc-modprobe-d\") pod \"tuned-l98pn\" (UID: \"0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25\") " pod="openshift-cluster-node-tuning-operator/tuned-l98pn"
Nov 03 19:30:45.791422 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.791330    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-node-log\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:30:45.791422 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.791333    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/653db542-8702-4eaa-abc1-a3bb866163c6-host-var-lib-cni-multus\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:30:45.791422 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.791350    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1c00e787-a35d-4ac4-95a4-2546e29602aa-konnectivity-ca\") pod \"konnectivity-agent-jwrp9\" (UID: \"1c00e787-a35d-4ac4-95a4-2546e29602aa\") " pod="kube-system/konnectivity-agent-jwrp9"
Nov 03 19:30:45.791422 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.791399    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/56e21184-e568-46bc-8e90-204684d6c366-hosts-file\") pod \"node-resolver-p2p8d\" (UID: \"56e21184-e568-46bc-8e90-204684d6c366\") " pod="openshift-dns/node-resolver-p2p8d"
Nov 03 19:30:45.791780 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.791427    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:30:45.791780 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.791451    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-host-slash\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:30:45.791780 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.791467    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d0ab7c64-a85f-4713-a270-f0cf711fcee5-ovn-node-metrics-cert\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:30:45.791780 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.791515    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25-run\") pod \"tuned-l98pn\" (UID: \"0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25\") " pod="openshift-cluster-node-tuning-operator/tuned-l98pn"
Nov 03 19:30:45.791780 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.791530    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/653db542-8702-4eaa-abc1-a3bb866163c6-etc-kubernetes\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:30:45.791780 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.791552    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr684\" (UniqueName: \"kubernetes.io/projected/b797bdb0-b354-4dfc-a50a-3c60ecacbf71-kube-api-access-xr684\") pod \"multus-additional-cni-plugins-g67bt\" (UID: \"b797bdb0-b354-4dfc-a50a-3c60ecacbf71\") " pod="openshift-multus/multus-additional-cni-plugins-g67bt"
Nov 03 19:30:45.791780 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.791606    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/56e21184-e568-46bc-8e90-204684d6c366-hosts-file\") pod \"node-resolver-p2p8d\" (UID: \"56e21184-e568-46bc-8e90-204684d6c366\") " pod="openshift-dns/node-resolver-p2p8d"
Nov 03 19:30:45.791780 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.791695    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25-etc-kubernetes\") pod \"tuned-l98pn\" (UID: \"0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25\") " pod="openshift-cluster-node-tuning-operator/tuned-l98pn"
Nov 03 19:30:45.791780 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.791703    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/653db542-8702-4eaa-abc1-a3bb866163c6-etc-kubernetes\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:30:45.791780 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.791743    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25-var-lib-kubelet\") pod \"tuned-l98pn\" (UID: \"0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25\") " pod="openshift-cluster-node-tuning-operator/tuned-l98pn"
Nov 03 19:30:45.791780 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.791749    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-host-slash\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:30:45.791780 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.791775    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce-registration-dir\") pod \"aws-ebs-csi-driver-node-mcxcv\" (UID: \"8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mcxcv"
Nov 03 19:30:45.792326 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.791820    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce-device-dir\") pod \"aws-ebs-csi-driver-node-mcxcv\" (UID: \"8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mcxcv"
Nov 03 19:30:45.792326 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.791849    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/653db542-8702-4eaa-abc1-a3bb866163c6-system-cni-dir\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:30:45.792326 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.791892    2575 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 19:30:45.792326 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.791976    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8b5e1b09-699d-4db8-9eb0-0b71bdda858f-host-slash\") pod \"iptables-alerter-wkrtt\" (UID: \"8b5e1b09-699d-4db8-9eb0-0b71bdda858f\") " pod="openshift-network-operator/iptables-alerter-wkrtt"
Nov 03 19:30:45.792326 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.792001    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-run-openvswitch\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:30:45.792326 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.792009    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/653db542-8702-4eaa-abc1-a3bb866163c6-system-cni-dir\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:30:45.792326 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.792023    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-log-socket\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:30:45.792326 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.792050    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/545f3c9a-2ac6-44a3-be0b-cf71b1592d42-host\") pod \"node-ca-vv2s2\" (UID: \"545f3c9a-2ac6-44a3-be0b-cf71b1592d42\") " pod="openshift-image-registry/node-ca-vv2s2"
Nov 03 19:30:45.792326 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.792065    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-log-socket\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:30:45.792326 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.792078    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25-etc-systemd\") pod \"tuned-l98pn\" (UID: \"0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25\") " pod="openshift-cluster-node-tuning-operator/tuned-l98pn"
Nov 03 19:30:45.792326 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.792084    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8b5e1b09-699d-4db8-9eb0-0b71bdda858f-host-slash\") pod \"iptables-alerter-wkrtt\" (UID: \"8b5e1b09-699d-4db8-9eb0-0b71bdda858f\") " pod="openshift-network-operator/iptables-alerter-wkrtt"
Nov 03 19:30:45.792326 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.792104    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-run-openvswitch\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:30:45.792326 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.792137    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mcxcv\" (UID: \"8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mcxcv"
Nov 03 19:30:45.792326 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.792178    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/653db542-8702-4eaa-abc1-a3bb866163c6-hostroot\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:30:45.792326 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.792199    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/653db542-8702-4eaa-abc1-a3bb866163c6-multus-conf-dir\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:30:45.792326 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.792217    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-host-kubelet\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:30:45.792326 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.792248    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-run-ovn\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:30:45.792326 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.792264    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/653db542-8702-4eaa-abc1-a3bb866163c6-hostroot\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:30:45.793179 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.792273    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc8v6\" (UniqueName: \"kubernetes.io/projected/d0ab7c64-a85f-4713-a270-f0cf711fcee5-kube-api-access-jc8v6\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:30:45.793179 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.792269    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/653db542-8702-4eaa-abc1-a3bb866163c6-multus-conf-dir\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:30:45.793179 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.792312    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-run-ovn\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:30:45.793179 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.792339    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f75xn\" (UniqueName: \"kubernetes.io/projected/b46f51b2-436b-4967-8879-a16b58930591-kube-api-access-f75xn\") pod \"network-check-target-njg27\" (UID: \"b46f51b2-436b-4967-8879-a16b58930591\") " pod="openshift-network-diagnostics/network-check-target-njg27"
Nov 03 19:30:45.793179 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.792342    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-host-kubelet\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:30:45.793179 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.792375    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/653db542-8702-4eaa-abc1-a3bb866163c6-host-var-lib-kubelet\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:30:45.793179 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.792401    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g8fh\" (UniqueName: \"kubernetes.io/projected/8b5e1b09-699d-4db8-9eb0-0b71bdda858f-kube-api-access-2g8fh\") pod \"iptables-alerter-wkrtt\" (UID: \"8b5e1b09-699d-4db8-9eb0-0b71bdda858f\") " pod="openshift-network-operator/iptables-alerter-wkrtt"
Nov 03 19:30:45.793179 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.792423    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-run-systemd\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:30:45.793179 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.792426    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1c00e787-a35d-4ac4-95a4-2546e29602aa-konnectivity-ca\") pod \"konnectivity-agent-jwrp9\" (UID: \"1c00e787-a35d-4ac4-95a4-2546e29602aa\") " pod="kube-system/konnectivity-agent-jwrp9"
Nov 03 19:30:45.793179 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.792445    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-host-cni-netd\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:30:45.793179 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.792467    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d0ab7c64-a85f-4713-a270-f0cf711fcee5-env-overrides\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:30:45.793179 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.792497    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/eee821c4-6fb0-4f84-a58d-23dc2ff77720-kubelet-config\") pod \"global-pull-secret-syncer-hrxdt\" (UID: \"eee821c4-6fb0-4f84-a58d-23dc2ff77720\") " pod="kube-system/global-pull-secret-syncer-hrxdt"
Nov 03 19:30:45.793179 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.792522    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25-etc-sysconfig\") pod \"tuned-l98pn\" (UID: \"0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25\") " pod="openshift-cluster-node-tuning-operator/tuned-l98pn"
Nov 03 19:30:45.793179 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.792548    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8b5e1b09-699d-4db8-9eb0-0b71bdda858f-iptables-alerter-script\") pod \"iptables-alerter-wkrtt\" (UID: \"8b5e1b09-699d-4db8-9eb0-0b71bdda858f\") " pod="openshift-network-operator/iptables-alerter-wkrtt"
Nov 03 19:30:45.793179 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.792560    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-host-cni-netd\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:30:45.793179 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.792572    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d0ab7c64-a85f-4713-a270-f0cf711fcee5-ovnkube-config\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:30:45.793179 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.792659    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-run-systemd\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:30:45.794039 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.792689    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55hqg\" (UniqueName: \"kubernetes.io/projected/545f3c9a-2ac6-44a3-be0b-cf71b1592d42-kube-api-access-55hqg\") pod \"node-ca-vv2s2\" (UID: \"545f3c9a-2ac6-44a3-be0b-cf71b1592d42\") " pod="openshift-image-registry/node-ca-vv2s2"
Nov 03 19:30:45.794039 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.792584    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/653db542-8702-4eaa-abc1-a3bb866163c6-host-var-lib-kubelet\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:30:45.794039 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.792719    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25-etc-sysctl-d\") pod \"tuned-l98pn\" (UID: \"0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25\") " pod="openshift-cluster-node-tuning-operator/tuned-l98pn"
Nov 03 19:30:45.794039 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.792758    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25-tmp\") pod \"tuned-l98pn\" (UID: \"0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25\") " pod="openshift-cluster-node-tuning-operator/tuned-l98pn"
Nov 03 19:30:45.794039 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.792808    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/653db542-8702-4eaa-abc1-a3bb866163c6-cni-binary-copy\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:30:45.794039 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.793057    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8b5e1b09-699d-4db8-9eb0-0b71bdda858f-iptables-alerter-script\") pod \"iptables-alerter-wkrtt\" (UID: \"8b5e1b09-699d-4db8-9eb0-0b71bdda858f\") " pod="openshift-network-operator/iptables-alerter-wkrtt"
Nov 03 19:30:45.794039 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.793100    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/545f3c9a-2ac6-44a3-be0b-cf71b1592d42-serviceca\") pod \"node-ca-vv2s2\" (UID: \"545f3c9a-2ac6-44a3-be0b-cf71b1592d42\") " pod="openshift-image-registry/node-ca-vv2s2"
Nov 03 19:30:45.794039 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.793120    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d0ab7c64-a85f-4713-a270-f0cf711fcee5-ovnkube-config\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:30:45.794039 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.793140    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25-etc-sysctl-conf\") pod \"tuned-l98pn\" (UID: \"0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25\") " pod="openshift-cluster-node-tuning-operator/tuned-l98pn"
Nov 03 19:30:45.794039 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.793175    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/653db542-8702-4eaa-abc1-a3bb866163c6-multus-daemon-config\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:30:45.794039 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.793217    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eee821c4-6fb0-4f84-a58d-23dc2ff77720-original-pull-secret\") pod \"global-pull-secret-syncer-hrxdt\" (UID: \"eee821c4-6fb0-4f84-a58d-23dc2ff77720\") " pod="kube-system/global-pull-secret-syncer-hrxdt"
Nov 03 19:30:45.794039 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.793242    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b797bdb0-b354-4dfc-a50a-3c60ecacbf71-cnibin\") pod \"multus-additional-cni-plugins-g67bt\" (UID: \"b797bdb0-b354-4dfc-a50a-3c60ecacbf71\") " pod="openshift-multus/multus-additional-cni-plugins-g67bt"
Nov 03 19:30:45.794039 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.793293    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b797bdb0-b354-4dfc-a50a-3c60ecacbf71-cni-binary-copy\") pod \"multus-additional-cni-plugins-g67bt\" (UID: \"b797bdb0-b354-4dfc-a50a-3c60ecacbf71\") " pod="openshift-multus/multus-additional-cni-plugins-g67bt"
Nov 03 19:30:45.794039 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.793362    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-systemd-units\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:30:45.794039 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.793389    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-host-run-netns\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:30:45.794039 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.793416    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-etc-openvswitch\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:30:45.794039 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.793416    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d0ab7c64-a85f-4713-a270-f0cf711fcee5-env-overrides\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:30:45.794838 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.793456    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-etc-openvswitch\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:30:45.794838 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.793486    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-host-run-netns\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:30:45.794838 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.793485    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/653db542-8702-4eaa-abc1-a3bb866163c6-cni-binary-copy\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:30:45.794838 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.793488    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-systemd-units\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:30:45.794838 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.793517    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d0ab7c64-a85f-4713-a270-f0cf711fcee5-ovnkube-script-lib\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:30:45.794838 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.793584    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b797bdb0-b354-4dfc-a50a-3c60ecacbf71-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g67bt\" (UID: \"b797bdb0-b354-4dfc-a50a-3c60ecacbf71\") " pod="openshift-multus/multus-additional-cni-plugins-g67bt"
Nov 03 19:30:45.794838 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.793637    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce-etc-selinux\") pod \"aws-ebs-csi-driver-node-mcxcv\" (UID: \"8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mcxcv"
Nov 03 19:30:45.794838 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.793683    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-var-lib-openvswitch\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:30:45.794838 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.793710    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/eee821c4-6fb0-4f84-a58d-23dc2ff77720-dbus\") pod \"global-pull-secret-syncer-hrxdt\" (UID: \"eee821c4-6fb0-4f84-a58d-23dc2ff77720\") " pod="kube-system/global-pull-secret-syncer-hrxdt"
Nov 03 19:30:45.794838 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.793734    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5pj6\" (UniqueName: \"kubernetes.io/projected/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25-kube-api-access-w5pj6\") pod \"tuned-l98pn\" (UID: \"0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25\") " pod="openshift-cluster-node-tuning-operator/tuned-l98pn"
Nov 03 19:30:45.794838 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.793761    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtt8c\" (UniqueName: \"kubernetes.io/projected/8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce-kube-api-access-mtt8c\") pod \"aws-ebs-csi-driver-node-mcxcv\" (UID: \"8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mcxcv"
Nov 03 19:30:45.794838 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.793796    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/790607e1-d57c-4622-9a72-1af902ab1d85-metrics-certs\") pod \"network-metrics-daemon-rdr5d\" (UID: \"790607e1-d57c-4622-9a72-1af902ab1d85\") " pod="openshift-multus/network-metrics-daemon-rdr5d"
Nov 03 19:30:45.794838 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.793822    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-var-lib-openvswitch\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:30:45.794838 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.793830    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/653db542-8702-4eaa-abc1-a3bb866163c6-multus-daemon-config\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:30:45.794838 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.793831    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgtvt\" (UniqueName: \"kubernetes.io/projected/790607e1-d57c-4622-9a72-1af902ab1d85-kube-api-access-hgtvt\") pod \"network-metrics-daemon-rdr5d\" (UID: \"790607e1-d57c-4622-9a72-1af902ab1d85\") " pod="openshift-multus/network-metrics-daemon-rdr5d"
Nov 03 19:30:45.794838 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:45.794119    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:30:45.794838 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.794157    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/653db542-8702-4eaa-abc1-a3bb866163c6-host-run-multus-certs\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:30:45.795613 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.794184    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpwxv\" (UniqueName: \"kubernetes.io/projected/653db542-8702-4eaa-abc1-a3bb866163c6-kube-api-access-mpwxv\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:30:45.795613 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.794188    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d0ab7c64-a85f-4713-a270-f0cf711fcee5-ovnkube-script-lib\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:30:45.795613 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.794217    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-host-cni-bin\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:30:45.795613 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.794240    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b797bdb0-b354-4dfc-a50a-3c60ecacbf71-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g67bt\" (UID: \"b797bdb0-b354-4dfc-a50a-3c60ecacbf71\") " pod="openshift-multus/multus-additional-cni-plugins-g67bt"
Nov 03 19:30:45.795613 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:45.794279    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/790607e1-d57c-4622-9a72-1af902ab1d85-metrics-certs podName:790607e1-d57c-4622-9a72-1af902ab1d85 nodeName:}" failed. No retries permitted until 2026-11-03 19:30:46.294259055 +0000 UTC m=+2.987703401 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/790607e1-d57c-4622-9a72-1af902ab1d85-metrics-certs") pod "network-metrics-daemon-rdr5d" (UID: "790607e1-d57c-4622-9a72-1af902ab1d85") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:30:45.795613 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.794287    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/653db542-8702-4eaa-abc1-a3bb866163c6-host-run-multus-certs\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:30:45.795613 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.794301    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25-sys\") pod \"tuned-l98pn\" (UID: \"0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25\") " pod="openshift-cluster-node-tuning-operator/tuned-l98pn"
Nov 03 19:30:45.795613 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.794314    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-host-cni-bin\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:30:45.795613 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.794353    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1c00e787-a35d-4ac4-95a4-2546e29602aa-agent-certs\") pod \"konnectivity-agent-jwrp9\" (UID: \"1c00e787-a35d-4ac4-95a4-2546e29602aa\") " pod="kube-system/konnectivity-agent-jwrp9"
Nov 03 19:30:45.795613 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.794378    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/653db542-8702-4eaa-abc1-a3bb866163c6-host-run-k8s-cni-cncf-io\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:30:45.795613 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.794393    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/653db542-8702-4eaa-abc1-a3bb866163c6-host-run-netns\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:30:45.795613 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.794414    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/653db542-8702-4eaa-abc1-a3bb866163c6-host-var-lib-cni-bin\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:30:45.795613 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.794449    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25-lib-modules\") pod \"tuned-l98pn\" (UID: \"0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25\") " pod="openshift-cluster-node-tuning-operator/tuned-l98pn"
Nov 03 19:30:45.795613 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.794475    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25-host\") pod \"tuned-l98pn\" (UID: \"0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25\") " pod="openshift-cluster-node-tuning-operator/tuned-l98pn"
Nov 03 19:30:45.795613 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.794498    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25-etc-tuned\") pod \"tuned-l98pn\" (UID: \"0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25\") " pod="openshift-cluster-node-tuning-operator/tuned-l98pn"
Nov 03 19:30:45.795613 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.794513    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce-sys-fs\") pod \"aws-ebs-csi-driver-node-mcxcv\" (UID: \"8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mcxcv"
Nov 03 19:30:45.796308 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.794530    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/653db542-8702-4eaa-abc1-a3bb866163c6-multus-cni-dir\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:30:45.796308 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.794619    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56e21184-e568-46bc-8e90-204684d6c366-tmp-dir\") pod \"node-resolver-p2p8d\" (UID: \"56e21184-e568-46bc-8e90-204684d6c366\") " pod="openshift-dns/node-resolver-p2p8d"
Nov 03 19:30:45.796308 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.794794    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/653db542-8702-4eaa-abc1-a3bb866163c6-host-var-lib-cni-bin\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:30:45.796308 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.794797    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/653db542-8702-4eaa-abc1-a3bb866163c6-multus-cni-dir\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:30:45.796308 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.795063    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/653db542-8702-4eaa-abc1-a3bb866163c6-host-run-k8s-cni-cncf-io\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:30:45.796308 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.795104    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/653db542-8702-4eaa-abc1-a3bb866163c6-host-run-netns\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:30:45.796308 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.795158    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wk2r\" (UniqueName: \"kubernetes.io/projected/56e21184-e568-46bc-8e90-204684d6c366-kube-api-access-6wk2r\") pod \"node-resolver-p2p8d\" (UID: \"56e21184-e568-46bc-8e90-204684d6c366\") " pod="openshift-dns/node-resolver-p2p8d"
Nov 03 19:30:45.796308 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.795271    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-host-run-ovn-kubernetes\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:30:45.796308 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.795321    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b797bdb0-b354-4dfc-a50a-3c60ecacbf71-system-cni-dir\") pod \"multus-additional-cni-plugins-g67bt\" (UID: \"b797bdb0-b354-4dfc-a50a-3c60ecacbf71\") " pod="openshift-multus/multus-additional-cni-plugins-g67bt"
Nov 03 19:30:45.796308 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.795344    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-host-run-ovn-kubernetes\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:30:45.796308 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.795391    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/653db542-8702-4eaa-abc1-a3bb866163c6-cnibin\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:30:45.796308 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.795401    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56e21184-e568-46bc-8e90-204684d6c366-tmp-dir\") pod \"node-resolver-p2p8d\" (UID: \"56e21184-e568-46bc-8e90-204684d6c366\") " pod="openshift-dns/node-resolver-p2p8d"
Nov 03 19:30:45.796308 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.795421    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/653db542-8702-4eaa-abc1-a3bb866163c6-multus-socket-dir-parent\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:30:45.796308 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.795443    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/653db542-8702-4eaa-abc1-a3bb866163c6-cnibin\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:30:45.796308 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.795475    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/653db542-8702-4eaa-abc1-a3bb866163c6-multus-socket-dir-parent\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:30:45.796308 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.795505    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce-socket-dir\") pod \"aws-ebs-csi-driver-node-mcxcv\" (UID: \"8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mcxcv"
Nov 03 19:30:45.796308 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.796036    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d0ab7c64-a85f-4713-a270-f0cf711fcee5-ovn-node-metrics-cert\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:30:45.797627 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.797605    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1c00e787-a35d-4ac4-95a4-2546e29602aa-agent-certs\") pod \"konnectivity-agent-jwrp9\" (UID: \"1c00e787-a35d-4ac4-95a4-2546e29602aa\") " pod="kube-system/konnectivity-agent-jwrp9"
Nov 03 19:30:45.803153 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:45.803131    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 19:30:45.803269 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:45.803159    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 19:30:45.803269 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:45.803173    2575 projected.go:196] Error preparing data for projected volume kube-api-access-f75xn for pod openshift-network-diagnostics/network-check-target-njg27: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:30:45.803269 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:45.803231    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b46f51b2-436b-4967-8879-a16b58930591-kube-api-access-f75xn podName:b46f51b2-436b-4967-8879-a16b58930591 nodeName:}" failed. No retries permitted until 2026-11-03 19:30:46.303213861 +0000 UTC m=+2.996658195 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-f75xn" (UniqueName: "kubernetes.io/projected/b46f51b2-436b-4967-8879-a16b58930591-kube-api-access-f75xn") pod "network-check-target-njg27" (UID: "b46f51b2-436b-4967-8879-a16b58930591") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:30:45.806935 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.806914    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g8fh\" (UniqueName: \"kubernetes.io/projected/8b5e1b09-699d-4db8-9eb0-0b71bdda858f-kube-api-access-2g8fh\") pod \"iptables-alerter-wkrtt\" (UID: \"8b5e1b09-699d-4db8-9eb0-0b71bdda858f\") " pod="openshift-network-operator/iptables-alerter-wkrtt"
Nov 03 19:30:45.807057 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.807007    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgtvt\" (UniqueName: \"kubernetes.io/projected/790607e1-d57c-4622-9a72-1af902ab1d85-kube-api-access-hgtvt\") pod \"network-metrics-daemon-rdr5d\" (UID: \"790607e1-d57c-4622-9a72-1af902ab1d85\") " pod="openshift-multus/network-metrics-daemon-rdr5d"
Nov 03 19:30:45.807495 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.807474    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc8v6\" (UniqueName: \"kubernetes.io/projected/d0ab7c64-a85f-4713-a270-f0cf711fcee5-kube-api-access-jc8v6\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:30:45.807751 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.807734    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpwxv\" (UniqueName: \"kubernetes.io/projected/653db542-8702-4eaa-abc1-a3bb866163c6-kube-api-access-mpwxv\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:30:45.808342 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.808325    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wk2r\" (UniqueName: \"kubernetes.io/projected/56e21184-e568-46bc-8e90-204684d6c366-kube-api-access-6wk2r\") pod \"node-resolver-p2p8d\" (UID: \"56e21184-e568-46bc-8e90-204684d6c366\") " pod="openshift-dns/node-resolver-p2p8d"
Nov 03 19:30:45.896052 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.895966    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/eee821c4-6fb0-4f84-a58d-23dc2ff77720-kubelet-config\") pod \"global-pull-secret-syncer-hrxdt\" (UID: \"eee821c4-6fb0-4f84-a58d-23dc2ff77720\") " pod="kube-system/global-pull-secret-syncer-hrxdt"
Nov 03 19:30:45.896052 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.896013    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25-etc-sysconfig\") pod \"tuned-l98pn\" (UID: \"0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25\") " pod="openshift-cluster-node-tuning-operator/tuned-l98pn"
Nov 03 19:30:45.896251 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.896090    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25-etc-sysconfig\") pod \"tuned-l98pn\" (UID: \"0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25\") " pod="openshift-cluster-node-tuning-operator/tuned-l98pn"
Nov 03 19:30:45.896251 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.896115    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/eee821c4-6fb0-4f84-a58d-23dc2ff77720-kubelet-config\") pod \"global-pull-secret-syncer-hrxdt\" (UID: \"eee821c4-6fb0-4f84-a58d-23dc2ff77720\") " pod="kube-system/global-pull-secret-syncer-hrxdt"
Nov 03 19:30:45.896251 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.896162    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55hqg\" (UniqueName: \"kubernetes.io/projected/545f3c9a-2ac6-44a3-be0b-cf71b1592d42-kube-api-access-55hqg\") pod \"node-ca-vv2s2\" (UID: \"545f3c9a-2ac6-44a3-be0b-cf71b1592d42\") " pod="openshift-image-registry/node-ca-vv2s2"
Nov 03 19:30:45.896251 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.896183    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25-etc-sysctl-d\") pod \"tuned-l98pn\" (UID: \"0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25\") " pod="openshift-cluster-node-tuning-operator/tuned-l98pn"
Nov 03 19:30:45.896251 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.896200    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25-tmp\") pod \"tuned-l98pn\" (UID: \"0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25\") " pod="openshift-cluster-node-tuning-operator/tuned-l98pn"
Nov 03 19:30:45.896251 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.896216    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/545f3c9a-2ac6-44a3-be0b-cf71b1592d42-serviceca\") pod \"node-ca-vv2s2\" (UID: \"545f3c9a-2ac6-44a3-be0b-cf71b1592d42\") " pod="openshift-image-registry/node-ca-vv2s2"
Nov 03 19:30:45.896251 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.896241    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25-etc-sysctl-conf\") pod \"tuned-l98pn\" (UID: \"0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25\") " pod="openshift-cluster-node-tuning-operator/tuned-l98pn"
Nov 03 19:30:45.896552 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.896264    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eee821c4-6fb0-4f84-a58d-23dc2ff77720-original-pull-secret\") pod \"global-pull-secret-syncer-hrxdt\" (UID: \"eee821c4-6fb0-4f84-a58d-23dc2ff77720\") " pod="kube-system/global-pull-secret-syncer-hrxdt"
Nov 03 19:30:45.896552 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.896280    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b797bdb0-b354-4dfc-a50a-3c60ecacbf71-cnibin\") pod \"multus-additional-cni-plugins-g67bt\" (UID: \"b797bdb0-b354-4dfc-a50a-3c60ecacbf71\") " pod="openshift-multus/multus-additional-cni-plugins-g67bt"
Nov 03 19:30:45.896552 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.896295    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b797bdb0-b354-4dfc-a50a-3c60ecacbf71-cni-binary-copy\") pod \"multus-additional-cni-plugins-g67bt\" (UID: \"b797bdb0-b354-4dfc-a50a-3c60ecacbf71\") " pod="openshift-multus/multus-additional-cni-plugins-g67bt"
Nov 03 19:30:45.896552 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.896313    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b797bdb0-b354-4dfc-a50a-3c60ecacbf71-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g67bt\" (UID: \"b797bdb0-b354-4dfc-a50a-3c60ecacbf71\") " pod="openshift-multus/multus-additional-cni-plugins-g67bt"
Nov 03 19:30:45.896552 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.896329    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce-etc-selinux\") pod \"aws-ebs-csi-driver-node-mcxcv\" (UID: \"8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mcxcv"
Nov 03 19:30:45.896552 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.896343    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/eee821c4-6fb0-4f84-a58d-23dc2ff77720-dbus\") pod \"global-pull-secret-syncer-hrxdt\" (UID: \"eee821c4-6fb0-4f84-a58d-23dc2ff77720\") " pod="kube-system/global-pull-secret-syncer-hrxdt"
Nov 03 19:30:45.896552 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.896357    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5pj6\" (UniqueName: \"kubernetes.io/projected/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25-kube-api-access-w5pj6\") pod \"tuned-l98pn\" (UID: \"0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25\") " pod="openshift-cluster-node-tuning-operator/tuned-l98pn"
Nov 03 19:30:45.896552 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.896374    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtt8c\" (UniqueName: \"kubernetes.io/projected/8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce-kube-api-access-mtt8c\") pod \"aws-ebs-csi-driver-node-mcxcv\" (UID: \"8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mcxcv"
Nov 03 19:30:45.896552 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.896372    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25-etc-sysctl-d\") pod \"tuned-l98pn\" (UID: \"0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25\") " pod="openshift-cluster-node-tuning-operator/tuned-l98pn"
Nov 03 19:30:45.896552 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.896403    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b797bdb0-b354-4dfc-a50a-3c60ecacbf71-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g67bt\" (UID: \"b797bdb0-b354-4dfc-a50a-3c60ecacbf71\") " pod="openshift-multus/multus-additional-cni-plugins-g67bt"
Nov 03 19:30:45.896552 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.896419    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25-sys\") pod \"tuned-l98pn\" (UID: \"0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25\") " pod="openshift-cluster-node-tuning-operator/tuned-l98pn"
Nov 03 19:30:45.896552 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.896436    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25-lib-modules\") pod \"tuned-l98pn\" (UID: \"0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25\") " pod="openshift-cluster-node-tuning-operator/tuned-l98pn"
Nov 03 19:30:45.896552 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.896453    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25-host\") pod \"tuned-l98pn\" (UID: \"0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25\") " pod="openshift-cluster-node-tuning-operator/tuned-l98pn"
Nov 03 19:30:45.896552 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.896475    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25-etc-tuned\") pod \"tuned-l98pn\" (UID: \"0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25\") " pod="openshift-cluster-node-tuning-operator/tuned-l98pn"
Nov 03 19:30:45.896552 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.896495    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce-sys-fs\") pod \"aws-ebs-csi-driver-node-mcxcv\" (UID: \"8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mcxcv"
Nov 03 19:30:45.896552 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.896523    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b797bdb0-b354-4dfc-a50a-3c60ecacbf71-system-cni-dir\") pod \"multus-additional-cni-plugins-g67bt\" (UID: \"b797bdb0-b354-4dfc-a50a-3c60ecacbf71\") " pod="openshift-multus/multus-additional-cni-plugins-g67bt"
Nov 03 19:30:45.896552 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.896541    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce-socket-dir\") pod \"aws-ebs-csi-driver-node-mcxcv\" (UID: \"8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mcxcv"
Nov 03 19:30:45.897276 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.896560    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b797bdb0-b354-4dfc-a50a-3c60ecacbf71-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g67bt\" (UID: \"b797bdb0-b354-4dfc-a50a-3c60ecacbf71\") " pod="openshift-multus/multus-additional-cni-plugins-g67bt"
Nov 03 19:30:45.897276 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.896575    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25-etc-modprobe-d\") pod \"tuned-l98pn\" (UID: \"0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25\") " pod="openshift-cluster-node-tuning-operator/tuned-l98pn"
Nov 03 19:30:45.897276 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.896591    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25-run\") pod \"tuned-l98pn\" (UID: \"0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25\") " pod="openshift-cluster-node-tuning-operator/tuned-l98pn"
Nov 03 19:30:45.897276 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.896608    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr684\" (UniqueName: \"kubernetes.io/projected/b797bdb0-b354-4dfc-a50a-3c60ecacbf71-kube-api-access-xr684\") pod \"multus-additional-cni-plugins-g67bt\" (UID: \"b797bdb0-b354-4dfc-a50a-3c60ecacbf71\") " pod="openshift-multus/multus-additional-cni-plugins-g67bt"
Nov 03 19:30:45.897276 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.896622    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25-etc-kubernetes\") pod \"tuned-l98pn\" (UID: \"0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25\") " pod="openshift-cluster-node-tuning-operator/tuned-l98pn"
Nov 03 19:30:45.897276 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.896661    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25-var-lib-kubelet\") pod \"tuned-l98pn\" (UID: \"0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25\") " pod="openshift-cluster-node-tuning-operator/tuned-l98pn"
Nov 03 19:30:45.897276 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.896694    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce-registration-dir\") pod \"aws-ebs-csi-driver-node-mcxcv\" (UID: \"8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mcxcv"
Nov 03 19:30:45.897276 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.896714    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce-device-dir\") pod \"aws-ebs-csi-driver-node-mcxcv\" (UID: \"8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mcxcv"
Nov 03 19:30:45.897276 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.896731    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/545f3c9a-2ac6-44a3-be0b-cf71b1592d42-host\") pod \"node-ca-vv2s2\" (UID: \"545f3c9a-2ac6-44a3-be0b-cf71b1592d42\") " pod="openshift-image-registry/node-ca-vv2s2"
Nov 03 19:30:45.897276 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.896745    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25-etc-systemd\") pod \"tuned-l98pn\" (UID: \"0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25\") " pod="openshift-cluster-node-tuning-operator/tuned-l98pn"
Nov 03 19:30:45.897276 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.896760    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mcxcv\" (UID: \"8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mcxcv"
Nov 03 19:30:45.897276 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.896837    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mcxcv\" (UID: \"8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mcxcv"
Nov 03 19:30:45.897276 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.897237    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b797bdb0-b354-4dfc-a50a-3c60ecacbf71-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g67bt\" (UID: \"b797bdb0-b354-4dfc-a50a-3c60ecacbf71\") " pod="openshift-multus/multus-additional-cni-plugins-g67bt"
Nov 03 19:30:45.897774 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.897284    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25-sys\") pod \"tuned-l98pn\" (UID: \"0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25\") " pod="openshift-cluster-node-tuning-operator/tuned-l98pn"
Nov 03 19:30:45.897774 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.897362    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25-lib-modules\") pod \"tuned-l98pn\" (UID: \"0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25\") " pod="openshift-cluster-node-tuning-operator/tuned-l98pn"
Nov 03 19:30:45.897774 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.897391    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25-host\") pod \"tuned-l98pn\" (UID: \"0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25\") " pod="openshift-cluster-node-tuning-operator/tuned-l98pn"
Nov 03 19:30:45.897774 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.897462    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b797bdb0-b354-4dfc-a50a-3c60ecacbf71-cni-binary-copy\") pod \"multus-additional-cni-plugins-g67bt\" (UID: \"b797bdb0-b354-4dfc-a50a-3c60ecacbf71\") " pod="openshift-multus/multus-additional-cni-plugins-g67bt"
Nov 03 19:30:45.897977 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.897791    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/545f3c9a-2ac6-44a3-be0b-cf71b1592d42-serviceca\") pod \"node-ca-vv2s2\" (UID: \"545f3c9a-2ac6-44a3-be0b-cf71b1592d42\") " pod="openshift-image-registry/node-ca-vv2s2"
Nov 03 19:30:45.897977 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.897880    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce-sys-fs\") pod \"aws-ebs-csi-driver-node-mcxcv\" (UID: \"8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mcxcv"
Nov 03 19:30:45.897977 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.897916    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25-etc-sysctl-conf\") pod \"tuned-l98pn\" (UID: \"0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25\") " pod="openshift-cluster-node-tuning-operator/tuned-l98pn"
Nov 03 19:30:45.897977 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.897932    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b797bdb0-b354-4dfc-a50a-3c60ecacbf71-system-cni-dir\") pod \"multus-additional-cni-plugins-g67bt\" (UID: \"b797bdb0-b354-4dfc-a50a-3c60ecacbf71\") " pod="openshift-multus/multus-additional-cni-plugins-g67bt"
Nov 03 19:30:45.897977 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.897957    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b797bdb0-b354-4dfc-a50a-3c60ecacbf71-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g67bt\" (UID: \"b797bdb0-b354-4dfc-a50a-3c60ecacbf71\") " pod="openshift-multus/multus-additional-cni-plugins-g67bt"
Nov 03 19:30:45.898209 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:45.897991    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 19:30:45.898209 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.898044    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce-socket-dir\") pod \"aws-ebs-csi-driver-node-mcxcv\" (UID: \"8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mcxcv"
Nov 03 19:30:45.898209 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:45.898052    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eee821c4-6fb0-4f84-a58d-23dc2ff77720-original-pull-secret podName:eee821c4-6fb0-4f84-a58d-23dc2ff77720 nodeName:}" failed. No retries permitted until 2026-11-03 19:30:46.398038309 +0000 UTC m=+3.091482650 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eee821c4-6fb0-4f84-a58d-23dc2ff77720-original-pull-secret") pod "global-pull-secret-syncer-hrxdt" (UID: "eee821c4-6fb0-4f84-a58d-23dc2ff77720") : object "kube-system"/"original-pull-secret" not registered
Nov 03 19:30:45.898209 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.898076    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce-etc-selinux\") pod \"aws-ebs-csi-driver-node-mcxcv\" (UID: \"8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mcxcv"
Nov 03 19:30:45.898209 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.898137    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/eee821c4-6fb0-4f84-a58d-23dc2ff77720-dbus\") pod \"global-pull-secret-syncer-hrxdt\" (UID: \"eee821c4-6fb0-4f84-a58d-23dc2ff77720\") " pod="kube-system/global-pull-secret-syncer-hrxdt"
Nov 03 19:30:45.898209 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.898196    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b797bdb0-b354-4dfc-a50a-3c60ecacbf71-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g67bt\" (UID: \"b797bdb0-b354-4dfc-a50a-3c60ecacbf71\") " pod="openshift-multus/multus-additional-cni-plugins-g67bt"
Nov 03 19:30:45.898500 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.898222    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b797bdb0-b354-4dfc-a50a-3c60ecacbf71-cnibin\") pod \"multus-additional-cni-plugins-g67bt\" (UID: \"b797bdb0-b354-4dfc-a50a-3c60ecacbf71\") " pod="openshift-multus/multus-additional-cni-plugins-g67bt"
Nov 03 19:30:45.898500 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.898276    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25-run\") pod \"tuned-l98pn\" (UID: \"0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25\") " pod="openshift-cluster-node-tuning-operator/tuned-l98pn"
Nov 03 19:30:45.898500 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.898300    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25-etc-modprobe-d\") pod \"tuned-l98pn\" (UID: \"0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25\") " pod="openshift-cluster-node-tuning-operator/tuned-l98pn"
Nov 03 19:30:45.898500 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.898357    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce-registration-dir\") pod \"aws-ebs-csi-driver-node-mcxcv\" (UID: \"8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mcxcv"
Nov 03 19:30:45.898500 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.898401    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25-etc-kubernetes\") pod \"tuned-l98pn\" (UID: \"0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25\") " pod="openshift-cluster-node-tuning-operator/tuned-l98pn"
Nov 03 19:30:45.898500 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.898402    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/545f3c9a-2ac6-44a3-be0b-cf71b1592d42-host\") pod \"node-ca-vv2s2\" (UID: \"545f3c9a-2ac6-44a3-be0b-cf71b1592d42\") " pod="openshift-image-registry/node-ca-vv2s2"
Nov 03 19:30:45.898500 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.898451    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25-var-lib-kubelet\") pod \"tuned-l98pn\" (UID: \"0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25\") " pod="openshift-cluster-node-tuning-operator/tuned-l98pn"
Nov 03 19:30:45.898500 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.898463    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce-device-dir\") pod \"aws-ebs-csi-driver-node-mcxcv\" (UID: \"8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mcxcv"
Nov 03 19:30:45.898500 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.898478    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25-etc-systemd\") pod \"tuned-l98pn\" (UID: \"0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25\") " pod="openshift-cluster-node-tuning-operator/tuned-l98pn"
Nov 03 19:30:45.905027 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.899550    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25-tmp\") pod \"tuned-l98pn\" (UID: \"0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25\") " pod="openshift-cluster-node-tuning-operator/tuned-l98pn"
Nov 03 19:30:45.905027 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.899635    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25-etc-tuned\") pod \"tuned-l98pn\" (UID: \"0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25\") " pod="openshift-cluster-node-tuning-operator/tuned-l98pn"
Nov 03 19:30:45.906655 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.906631    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55hqg\" (UniqueName: \"kubernetes.io/projected/545f3c9a-2ac6-44a3-be0b-cf71b1592d42-kube-api-access-55hqg\") pod \"node-ca-vv2s2\" (UID: \"545f3c9a-2ac6-44a3-be0b-cf71b1592d42\") " pod="openshift-image-registry/node-ca-vv2s2"
Nov 03 19:30:45.908368 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.908298    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr684\" (UniqueName: \"kubernetes.io/projected/b797bdb0-b354-4dfc-a50a-3c60ecacbf71-kube-api-access-xr684\") pod \"multus-additional-cni-plugins-g67bt\" (UID: \"b797bdb0-b354-4dfc-a50a-3c60ecacbf71\") " pod="openshift-multus/multus-additional-cni-plugins-g67bt"
Nov 03 19:30:45.909390 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.909367    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtt8c\" (UniqueName: \"kubernetes.io/projected/8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce-kube-api-access-mtt8c\") pod \"aws-ebs-csi-driver-node-mcxcv\" (UID: \"8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mcxcv"
Nov 03 19:30:45.913029 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.913009    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5pj6\" (UniqueName: \"kubernetes.io/projected/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25-kube-api-access-w5pj6\") pod \"tuned-l98pn\" (UID: \"0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25\") " pod="openshift-cluster-node-tuning-operator/tuned-l98pn"
Nov 03 19:30:45.984314 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.984277    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jwrp9"
Nov 03 19:30:45.992087 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:45.992056    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9ck9p"
Nov 03 19:30:45.995117 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:45.995083    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c00e787_a35d_4ac4_95a4_2546e29602aa.slice/crio-8956f25a94b043ba181c79bc664f46b9ec9492e4feead98a59cff77e662a4726 WatchSource:0}: Error finding container 8956f25a94b043ba181c79bc664f46b9ec9492e4feead98a59cff77e662a4726: Status 404 returned error can't find the container with id 8956f25a94b043ba181c79bc664f46b9ec9492e4feead98a59cff77e662a4726
Nov 03 19:30:46.005880 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:46.005857    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-p2p8d"
Nov 03 19:30:46.013841 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:46.013813    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wkrtt"
Nov 03 19:30:46.017567 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:46.017540    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56e21184_e568_46bc_8e90_204684d6c366.slice/crio-342234a86ae594f8fc680a4b71c5d32cc74809d11193bde259907c98dbdfbe63 WatchSource:0}: Error finding container 342234a86ae594f8fc680a4b71c5d32cc74809d11193bde259907c98dbdfbe63: Status 404 returned error can't find the container with id 342234a86ae594f8fc680a4b71c5d32cc74809d11193bde259907c98dbdfbe63
Nov 03 19:30:46.025753 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:46.025727    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b5e1b09_699d_4db8_9eb0_0b71bdda858f.slice/crio-df61d6453bb2b7ff05aa8df94e63ef5f6d6616ebd51be5be599aa73f996b191d WatchSource:0}: Error finding container df61d6453bb2b7ff05aa8df94e63ef5f6d6616ebd51be5be599aa73f996b191d: Status 404 returned error can't find the container with id df61d6453bb2b7ff05aa8df94e63ef5f6d6616ebd51be5be599aa73f996b191d
Nov 03 19:30:46.037288 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:46.037262    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vv2s2"
Nov 03 19:30:46.041194 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:46.041024    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-l98pn"
Nov 03 19:30:46.042431 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:46.042406    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:30:46.050091 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:46.050059    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod545f3c9a_2ac6_44a3_be0b_cf71b1592d42.slice/crio-f8ab706e01419505523c810aac56fba7d29a8c4703c9b0f23565593f44ef80a7 WatchSource:0}: Error finding container f8ab706e01419505523c810aac56fba7d29a8c4703c9b0f23565593f44ef80a7: Status 404 returned error can't find the container with id f8ab706e01419505523c810aac56fba7d29a8c4703c9b0f23565593f44ef80a7
Nov 03 19:30:46.055781 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:46.055759    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mcxcv"
Nov 03 19:30:46.066370 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:46.066341    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ec67cf7_b089_40ac_9a9e_eb2e2ea400ce.slice/crio-e666c09aad34a256678922fa6e602d5132bb9d5e8d6c6b094083ed27b1f676ce WatchSource:0}: Error finding container e666c09aad34a256678922fa6e602d5132bb9d5e8d6c6b094083ed27b1f676ce: Status 404 returned error can't find the container with id e666c09aad34a256678922fa6e602d5132bb9d5e8d6c6b094083ed27b1f676ce
Nov 03 19:30:46.070161 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:46.070134    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-g67bt"
Nov 03 19:30:46.081239 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:30:46.081203    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb797bdb0_b354_4dfc_a50a_3c60ecacbf71.slice/crio-0ed2a89a99f0a7292450791623764b1318e836e96f0da6d2b1da8e710c76aff0 WatchSource:0}: Error finding container 0ed2a89a99f0a7292450791623764b1318e836e96f0da6d2b1da8e710c76aff0: Status 404 returned error can't find the container with id 0ed2a89a99f0a7292450791623764b1318e836e96f0da6d2b1da8e710c76aff0
Nov 03 19:30:46.096841 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:46.096792    2575 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 19:30:46.299541 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:46.299512    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/790607e1-d57c-4622-9a72-1af902ab1d85-metrics-certs\") pod \"network-metrics-daemon-rdr5d\" (UID: \"790607e1-d57c-4622-9a72-1af902ab1d85\") " pod="openshift-multus/network-metrics-daemon-rdr5d"
Nov 03 19:30:46.299660 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:46.299646    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:30:46.299738 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:46.299724    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/790607e1-d57c-4622-9a72-1af902ab1d85-metrics-certs podName:790607e1-d57c-4622-9a72-1af902ab1d85 nodeName:}" failed. No retries permitted until 2026-11-03 19:30:47.299704859 +0000 UTC m=+3.993149182 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/790607e1-d57c-4622-9a72-1af902ab1d85-metrics-certs") pod "network-metrics-daemon-rdr5d" (UID: "790607e1-d57c-4622-9a72-1af902ab1d85") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:30:46.400806 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:46.400769    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eee821c4-6fb0-4f84-a58d-23dc2ff77720-original-pull-secret\") pod \"global-pull-secret-syncer-hrxdt\" (UID: \"eee821c4-6fb0-4f84-a58d-23dc2ff77720\") " pod="kube-system/global-pull-secret-syncer-hrxdt"
Nov 03 19:30:46.400920 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:46.400859    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f75xn\" (UniqueName: \"kubernetes.io/projected/b46f51b2-436b-4967-8879-a16b58930591-kube-api-access-f75xn\") pod \"network-check-target-njg27\" (UID: \"b46f51b2-436b-4967-8879-a16b58930591\") " pod="openshift-network-diagnostics/network-check-target-njg27"
Nov 03 19:30:46.400996 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:46.400982    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 19:30:46.401031 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:46.401002    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 19:30:46.401031 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:46.401011    2575 projected.go:196] Error preparing data for projected volume kube-api-access-f75xn for pod openshift-network-diagnostics/network-check-target-njg27: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:30:46.401088 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:46.401057    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b46f51b2-436b-4967-8879-a16b58930591-kube-api-access-f75xn podName:b46f51b2-436b-4967-8879-a16b58930591 nodeName:}" failed. No retries permitted until 2026-11-03 19:30:47.401043825 +0000 UTC m=+4.094488155 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-f75xn" (UniqueName: "kubernetes.io/projected/b46f51b2-436b-4967-8879-a16b58930591-kube-api-access-f75xn") pod "network-check-target-njg27" (UID: "b46f51b2-436b-4967-8879-a16b58930591") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:30:46.401457 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:46.401438    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 19:30:46.401514 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:46.401492    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eee821c4-6fb0-4f84-a58d-23dc2ff77720-original-pull-secret podName:eee821c4-6fb0-4f84-a58d-23dc2ff77720 nodeName:}" failed. No retries permitted until 2026-11-03 19:30:47.401477909 +0000 UTC m=+4.094922243 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eee821c4-6fb0-4f84-a58d-23dc2ff77720-original-pull-secret") pod "global-pull-secret-syncer-hrxdt" (UID: "eee821c4-6fb0-4f84-a58d-23dc2ff77720") : object "kube-system"/"original-pull-secret" not registered
Nov 03 19:30:46.717288 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:46.717226    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 19:25:45 +0000 UTC" deadline="2028-04-30 04:36:49.626066878 +0000 UTC"
Nov 03 19:30:46.717288 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:46.717285    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13041h6m2.908786265s"
Nov 03 19:30:46.749025 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:46.748985    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rdr5d"
Nov 03 19:30:46.749232 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:46.749205    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rdr5d" podUID="790607e1-d57c-4622-9a72-1af902ab1d85"
Nov 03 19:30:46.766022 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:46.765984    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g67bt" event={"ID":"b797bdb0-b354-4dfc-a50a-3c60ecacbf71","Type":"ContainerStarted","Data":"0ed2a89a99f0a7292450791623764b1318e836e96f0da6d2b1da8e710c76aff0"}
Nov 03 19:30:46.777197 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:46.777158    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr" event={"ID":"d0ab7c64-a85f-4713-a270-f0cf711fcee5","Type":"ContainerStarted","Data":"e997c0a1e8ead87756bf6ba3e07e296f5523e84fa98d10255af27065d299ac79"}
Nov 03 19:30:46.784849 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:46.783222    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-l98pn" event={"ID":"0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25","Type":"ContainerStarted","Data":"f6314b46be66add9fa5d4a634af327f858fa36fc7644392a4cc481decfbd4f98"}
Nov 03 19:30:46.789218 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:46.786801    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vv2s2" event={"ID":"545f3c9a-2ac6-44a3-be0b-cf71b1592d42","Type":"ContainerStarted","Data":"f8ab706e01419505523c810aac56fba7d29a8c4703c9b0f23565593f44ef80a7"}
Nov 03 19:30:46.789365 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:46.789283    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-p2p8d" event={"ID":"56e21184-e568-46bc-8e90-204684d6c366","Type":"ContainerStarted","Data":"342234a86ae594f8fc680a4b71c5d32cc74809d11193bde259907c98dbdfbe63"}
Nov 03 19:30:46.803621 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:46.803467    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mcxcv" event={"ID":"8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce","Type":"ContainerStarted","Data":"e666c09aad34a256678922fa6e602d5132bb9d5e8d6c6b094083ed27b1f676ce"}
Nov 03 19:30:46.812274 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:46.812154    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wkrtt" event={"ID":"8b5e1b09-699d-4db8-9eb0-0b71bdda858f","Type":"ContainerStarted","Data":"df61d6453bb2b7ff05aa8df94e63ef5f6d6616ebd51be5be599aa73f996b191d"}
Nov 03 19:30:46.817684 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:46.817637    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9ck9p" event={"ID":"653db542-8702-4eaa-abc1-a3bb866163c6","Type":"ContainerStarted","Data":"97697af028d0d1433e902ba6ee03eddadbd380fa856a9761702510919ef2426d"}
Nov 03 19:30:46.822032 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:46.821977    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jwrp9" event={"ID":"1c00e787-a35d-4ac4-95a4-2546e29602aa","Type":"ContainerStarted","Data":"8956f25a94b043ba181c79bc664f46b9ec9492e4feead98a59cff77e662a4726"}
Nov 03 19:30:46.828603 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:46.828555    2575 generic.go:356] "Generic (PLEG): container finished" podID="c5e667f56d4cbe64bbf7db6ca2bb9e9c" containerID="e6bc859a65a6a89039d5735a35c1672bcd570c1617da4abc3ea0c18161f936a3" exitCode=0
Nov 03 19:30:46.828603 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:46.828601    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-222.us-east-2.compute.internal" event={"ID":"c5e667f56d4cbe64bbf7db6ca2bb9e9c","Type":"ContainerDied","Data":"e6bc859a65a6a89039d5735a35c1672bcd570c1617da4abc3ea0c18161f936a3"}
Nov 03 19:30:47.309864 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:47.309554    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/790607e1-d57c-4622-9a72-1af902ab1d85-metrics-certs\") pod \"network-metrics-daemon-rdr5d\" (UID: \"790607e1-d57c-4622-9a72-1af902ab1d85\") " pod="openshift-multus/network-metrics-daemon-rdr5d"
Nov 03 19:30:47.309864 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:47.309747    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:30:47.310154 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:47.309943    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/790607e1-d57c-4622-9a72-1af902ab1d85-metrics-certs podName:790607e1-d57c-4622-9a72-1af902ab1d85 nodeName:}" failed. No retries permitted until 2026-11-03 19:30:49.309921376 +0000 UTC m=+6.003365702 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/790607e1-d57c-4622-9a72-1af902ab1d85-metrics-certs") pod "network-metrics-daemon-rdr5d" (UID: "790607e1-d57c-4622-9a72-1af902ab1d85") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:30:47.411568 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:47.410643    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f75xn\" (UniqueName: \"kubernetes.io/projected/b46f51b2-436b-4967-8879-a16b58930591-kube-api-access-f75xn\") pod \"network-check-target-njg27\" (UID: \"b46f51b2-436b-4967-8879-a16b58930591\") " pod="openshift-network-diagnostics/network-check-target-njg27"
Nov 03 19:30:47.411568 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:47.410720    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eee821c4-6fb0-4f84-a58d-23dc2ff77720-original-pull-secret\") pod \"global-pull-secret-syncer-hrxdt\" (UID: \"eee821c4-6fb0-4f84-a58d-23dc2ff77720\") " pod="kube-system/global-pull-secret-syncer-hrxdt"
Nov 03 19:30:47.411568 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:47.410868    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 19:30:47.411568 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:47.410931    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eee821c4-6fb0-4f84-a58d-23dc2ff77720-original-pull-secret podName:eee821c4-6fb0-4f84-a58d-23dc2ff77720 nodeName:}" failed. No retries permitted until 2026-11-03 19:30:49.410913847 +0000 UTC m=+6.104358167 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eee821c4-6fb0-4f84-a58d-23dc2ff77720-original-pull-secret") pod "global-pull-secret-syncer-hrxdt" (UID: "eee821c4-6fb0-4f84-a58d-23dc2ff77720") : object "kube-system"/"original-pull-secret" not registered
Nov 03 19:30:47.411568 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:47.411393    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 19:30:47.411568 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:47.411412    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 19:30:47.411568 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:47.411425    2575 projected.go:196] Error preparing data for projected volume kube-api-access-f75xn for pod openshift-network-diagnostics/network-check-target-njg27: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:30:47.411568 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:47.411470    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b46f51b2-436b-4967-8879-a16b58930591-kube-api-access-f75xn podName:b46f51b2-436b-4967-8879-a16b58930591 nodeName:}" failed. No retries permitted until 2026-11-03 19:30:49.41145557 +0000 UTC m=+6.104899891 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-f75xn" (UniqueName: "kubernetes.io/projected/b46f51b2-436b-4967-8879-a16b58930591-kube-api-access-f75xn") pod "network-check-target-njg27" (UID: "b46f51b2-436b-4967-8879-a16b58930591") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:30:47.718302 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:47.718210    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 19:25:45 +0000 UTC" deadline="2028-08-14 23:23:06.882113754 +0000 UTC"
Nov 03 19:30:47.718302 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:47.718253    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15603h52m19.163865544s"
Nov 03 19:30:47.749734 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:47.749702    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njg27"
Nov 03 19:30:47.749921 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:47.749833    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njg27" podUID="b46f51b2-436b-4967-8879-a16b58930591"
Nov 03 19:30:47.750412 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:47.750276    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrxdt"
Nov 03 19:30:47.750412 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:47.750376    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrxdt" podUID="eee821c4-6fb0-4f84-a58d-23dc2ff77720"
Nov 03 19:30:47.846779 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:47.846739    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-222.us-east-2.compute.internal" event={"ID":"66cbb2b4d2abe69f7cf665c408777b87","Type":"ContainerStarted","Data":"60a9b190e093f6a47399ce390e94b17004d2297d1f2a0e306d22eb797dab4b3b"}
Nov 03 19:30:47.862694 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:47.862297    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-222.us-east-2.compute.internal" event={"ID":"c5e667f56d4cbe64bbf7db6ca2bb9e9c","Type":"ContainerStarted","Data":"7f332c31d651aa688a23cd0d382ff66341dd6f859152d99940ed4ec353703da7"}
Nov 03 19:30:47.881341 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:47.880624    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-222.us-east-2.compute.internal" podStartSLOduration=2.880611261 podStartE2EDuration="2.880611261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:30:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:30:47.879839864 +0000 UTC m=+4.573284203" watchObservedRunningTime="2026-11-03 19:30:47.880611261 +0000 UTC m=+4.574055598"
Nov 03 19:30:47.881341 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:47.880984    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-222.us-east-2.compute.internal" podStartSLOduration=2.880937462 podStartE2EDuration="2.880937462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:30:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:30:47.86390833 +0000 UTC m=+4.557352669" watchObservedRunningTime="2026-11-03 19:30:47.880937462 +0000 UTC m=+4.574381792"
Nov 03 19:30:48.752998 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:48.752967    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rdr5d"
Nov 03 19:30:48.753443 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:48.753108    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rdr5d" podUID="790607e1-d57c-4622-9a72-1af902ab1d85"
Nov 03 19:30:49.326366 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:49.326287    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/790607e1-d57c-4622-9a72-1af902ab1d85-metrics-certs\") pod \"network-metrics-daemon-rdr5d\" (UID: \"790607e1-d57c-4622-9a72-1af902ab1d85\") " pod="openshift-multus/network-metrics-daemon-rdr5d"
Nov 03 19:30:49.326517 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:49.326429    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:30:49.326517 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:49.326481    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/790607e1-d57c-4622-9a72-1af902ab1d85-metrics-certs podName:790607e1-d57c-4622-9a72-1af902ab1d85 nodeName:}" failed. No retries permitted until 2026-11-03 19:30:53.326466907 +0000 UTC m=+10.019911224 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/790607e1-d57c-4622-9a72-1af902ab1d85-metrics-certs") pod "network-metrics-daemon-rdr5d" (UID: "790607e1-d57c-4622-9a72-1af902ab1d85") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:30:49.427146 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:49.427033    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f75xn\" (UniqueName: \"kubernetes.io/projected/b46f51b2-436b-4967-8879-a16b58930591-kube-api-access-f75xn\") pod \"network-check-target-njg27\" (UID: \"b46f51b2-436b-4967-8879-a16b58930591\") " pod="openshift-network-diagnostics/network-check-target-njg27"
Nov 03 19:30:49.427146 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:49.427090    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eee821c4-6fb0-4f84-a58d-23dc2ff77720-original-pull-secret\") pod \"global-pull-secret-syncer-hrxdt\" (UID: \"eee821c4-6fb0-4f84-a58d-23dc2ff77720\") " pod="kube-system/global-pull-secret-syncer-hrxdt"
Nov 03 19:30:49.427331 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:49.427223    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 19:30:49.427331 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:49.427282    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eee821c4-6fb0-4f84-a58d-23dc2ff77720-original-pull-secret podName:eee821c4-6fb0-4f84-a58d-23dc2ff77720 nodeName:}" failed. No retries permitted until 2026-11-03 19:30:53.42726656 +0000 UTC m=+10.120710889 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eee821c4-6fb0-4f84-a58d-23dc2ff77720-original-pull-secret") pod "global-pull-secret-syncer-hrxdt" (UID: "eee821c4-6fb0-4f84-a58d-23dc2ff77720") : object "kube-system"/"original-pull-secret" not registered
Nov 03 19:30:49.430689 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:49.427693    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 19:30:49.430689 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:49.427716    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 19:30:49.430689 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:49.427729    2575 projected.go:196] Error preparing data for projected volume kube-api-access-f75xn for pod openshift-network-diagnostics/network-check-target-njg27: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:30:49.430689 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:49.427775    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b46f51b2-436b-4967-8879-a16b58930591-kube-api-access-f75xn podName:b46f51b2-436b-4967-8879-a16b58930591 nodeName:}" failed. No retries permitted until 2026-11-03 19:30:53.427760949 +0000 UTC m=+10.121205280 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-f75xn" (UniqueName: "kubernetes.io/projected/b46f51b2-436b-4967-8879-a16b58930591-kube-api-access-f75xn") pod "network-check-target-njg27" (UID: "b46f51b2-436b-4967-8879-a16b58930591") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:30:49.752241 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:49.751498    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njg27"
Nov 03 19:30:49.752241 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:49.751626    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njg27" podUID="b46f51b2-436b-4967-8879-a16b58930591"
Nov 03 19:30:49.752241 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:49.752012    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrxdt"
Nov 03 19:30:49.752241 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:49.752103    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrxdt" podUID="eee821c4-6fb0-4f84-a58d-23dc2ff77720"
Nov 03 19:30:49.885077 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:49.885033    2575 generic.go:356] "Generic (PLEG): container finished" podID="b797bdb0-b354-4dfc-a50a-3c60ecacbf71" containerID="fe5e127fb70e46591cc8c1a265341411f5a2987c182946b6b668ef2f62cdf0df" exitCode=0
Nov 03 19:30:49.885546 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:49.885090    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g67bt" event={"ID":"b797bdb0-b354-4dfc-a50a-3c60ecacbf71","Type":"ContainerDied","Data":"fe5e127fb70e46591cc8c1a265341411f5a2987c182946b6b668ef2f62cdf0df"}
Nov 03 19:30:50.749558 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:50.749031    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rdr5d"
Nov 03 19:30:50.749558 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:50.749178    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rdr5d" podUID="790607e1-d57c-4622-9a72-1af902ab1d85"
Nov 03 19:30:50.895342 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:50.895309    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jwrp9" event={"ID":"1c00e787-a35d-4ac4-95a4-2546e29602aa","Type":"ContainerStarted","Data":"b2f153fddd3107310895f90dfc09ca9da3f1aef59c4deef52dfc906b31e7bac9"}
Nov 03 19:30:50.911877 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:50.911818    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-jwrp9" podStartSLOduration=4.138455832 podStartE2EDuration="7.911797713s" totalImagesPullingTime="3.773341881s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:30:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:30:50.911293092 +0000 UTC m=+7.604737431" watchObservedRunningTime="2026-11-03 19:30:50.911797713 +0000 UTC m=+7.605242051"
Nov 03 19:30:51.749890 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:51.749857    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrxdt"
Nov 03 19:30:51.750035 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:51.749998    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrxdt" podUID="eee821c4-6fb0-4f84-a58d-23dc2ff77720"
Nov 03 19:30:51.750298 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:51.749860    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njg27"
Nov 03 19:30:51.750371 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:51.750309    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njg27" podUID="b46f51b2-436b-4967-8879-a16b58930591"
Nov 03 19:30:51.902516 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:51.902484    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vv2s2" event={"ID":"545f3c9a-2ac6-44a3-be0b-cf71b1592d42","Type":"ContainerStarted","Data":"bae7282481f18ed5c7641ad3abad0b3f4514fa38da0c8f346cca2c69918a82a9"}
Nov 03 19:30:52.750448 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:52.749814    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rdr5d"
Nov 03 19:30:52.750448 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:52.749963    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rdr5d" podUID="790607e1-d57c-4622-9a72-1af902ab1d85"
Nov 03 19:30:52.906449 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:52.906414    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mcxcv" event={"ID":"8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce","Type":"ContainerStarted","Data":"269b94fa0c4193519178e1146e43993e616ef71adb64a92cdc63aa37deafeeda"}
Nov 03 19:30:53.362808 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:53.362777    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/790607e1-d57c-4622-9a72-1af902ab1d85-metrics-certs\") pod \"network-metrics-daemon-rdr5d\" (UID: \"790607e1-d57c-4622-9a72-1af902ab1d85\") " pod="openshift-multus/network-metrics-daemon-rdr5d"
Nov 03 19:30:53.362977 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:53.362960    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:30:53.363038 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:53.363029    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/790607e1-d57c-4622-9a72-1af902ab1d85-metrics-certs podName:790607e1-d57c-4622-9a72-1af902ab1d85 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:01.36301414 +0000 UTC m=+18.056458456 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/790607e1-d57c-4622-9a72-1af902ab1d85-metrics-certs") pod "network-metrics-daemon-rdr5d" (UID: "790607e1-d57c-4622-9a72-1af902ab1d85") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:30:53.463408 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:53.463366    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eee821c4-6fb0-4f84-a58d-23dc2ff77720-original-pull-secret\") pod \"global-pull-secret-syncer-hrxdt\" (UID: \"eee821c4-6fb0-4f84-a58d-23dc2ff77720\") " pod="kube-system/global-pull-secret-syncer-hrxdt"
Nov 03 19:30:53.463607 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:53.463483    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f75xn\" (UniqueName: \"kubernetes.io/projected/b46f51b2-436b-4967-8879-a16b58930591-kube-api-access-f75xn\") pod \"network-check-target-njg27\" (UID: \"b46f51b2-436b-4967-8879-a16b58930591\") " pod="openshift-network-diagnostics/network-check-target-njg27"
Nov 03 19:30:53.463707 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:53.463633    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 19:30:53.463707 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:53.463654    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 19:30:53.463707 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:53.463679    2575 projected.go:196] Error preparing data for projected volume kube-api-access-f75xn for pod openshift-network-diagnostics/network-check-target-njg27: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:30:53.463884 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:53.463726    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b46f51b2-436b-4967-8879-a16b58930591-kube-api-access-f75xn podName:b46f51b2-436b-4967-8879-a16b58930591 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:01.463712526 +0000 UTC m=+18.157156843 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-f75xn" (UniqueName: "kubernetes.io/projected/b46f51b2-436b-4967-8879-a16b58930591-kube-api-access-f75xn") pod "network-check-target-njg27" (UID: "b46f51b2-436b-4967-8879-a16b58930591") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:30:53.464154 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:53.464133    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 19:30:53.464247 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:53.464190    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eee821c4-6fb0-4f84-a58d-23dc2ff77720-original-pull-secret podName:eee821c4-6fb0-4f84-a58d-23dc2ff77720 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:01.46417655 +0000 UTC m=+18.157620868 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eee821c4-6fb0-4f84-a58d-23dc2ff77720-original-pull-secret") pod "global-pull-secret-syncer-hrxdt" (UID: "eee821c4-6fb0-4f84-a58d-23dc2ff77720") : object "kube-system"/"original-pull-secret" not registered
Nov 03 19:30:53.562643 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:53.562612    2575 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 19:30:53.753065 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:53.753023    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njg27"
Nov 03 19:30:53.753223 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:53.753154    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njg27" podUID="b46f51b2-436b-4967-8879-a16b58930591"
Nov 03 19:30:53.753518 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:53.753500    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrxdt"
Nov 03 19:30:53.753607 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:53.753583    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrxdt" podUID="eee821c4-6fb0-4f84-a58d-23dc2ff77720"
Nov 03 19:30:53.760526 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:53.760436    2575 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T19:30:53.562633031Z","UUID":"4b560443-9bf4-4996-b953-10bcfc435db1","Handler":null,"Name":"","Endpoint":""}
Nov 03 19:30:53.762155 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:53.762097    2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 19:30:53.762155 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:53.762128    2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 19:30:53.914026 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:53.913984    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mcxcv" event={"ID":"8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce","Type":"ContainerStarted","Data":"6826de78638eaf10ce497cfc9e1c3ee8324daaf0601e358a0af02b8fa642e83f"}
Nov 03 19:30:53.925695 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:53.920062    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-p2p8d" event={"ID":"56e21184-e568-46bc-8e90-204684d6c366","Type":"ContainerStarted","Data":"d870ccbe3f6aa86bd11110de59cb04fdfc7ac37b8fe3a8036953e1629519fc1f"}
Nov 03 19:30:53.936746 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:53.936664    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-p2p8d" podStartSLOduration=3.318116173 podStartE2EDuration="10.936646725s" totalImagesPullingTime="7.618530552s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:30:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:30:53.93646315 +0000 UTC m=+10.629907492" watchObservedRunningTime="2026-11-03 19:30:53.936646725 +0000 UTC m=+10.630091063"
Nov 03 19:30:53.937367 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:53.937322    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-vv2s2" podStartSLOduration=5.138645342 podStartE2EDuration="9.937313071s" totalImagesPullingTime="4.798667729s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:30:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:30:51.9195835 +0000 UTC m=+8.613027843" watchObservedRunningTime="2026-11-03 19:30:53.937313071 +0000 UTC m=+10.630757410"
Nov 03 19:30:54.749801 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:54.749556    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rdr5d"
Nov 03 19:30:54.749957 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:54.749934    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rdr5d" podUID="790607e1-d57c-4622-9a72-1af902ab1d85"
Nov 03 19:30:54.924950 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:54.924913    2575 generic.go:356] "Generic (PLEG): container finished" podID="b797bdb0-b354-4dfc-a50a-3c60ecacbf71" containerID="6016ea8b4a92ebd0d32f6649bfa7d02e3426bf2f54d88a9b56d2d2de6ca937ce" exitCode=0
Nov 03 19:30:54.925584 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:54.925564    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g67bt" event={"ID":"b797bdb0-b354-4dfc-a50a-3c60ecacbf71","Type":"ContainerDied","Data":"6016ea8b4a92ebd0d32f6649bfa7d02e3426bf2f54d88a9b56d2d2de6ca937ce"}
Nov 03 19:30:54.930718 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:54.930684    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mcxcv" event={"ID":"8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce","Type":"ContainerStarted","Data":"a3eb09f70b2f729d61c2939df70ac9d1dd25eb2237098be541a5fad4a8a2a1a9"}
Nov 03 19:30:54.933572 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:54.933519    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9ck9p" event={"ID":"653db542-8702-4eaa-abc1-a3bb866163c6","Type":"ContainerStarted","Data":"7bd99a72a7716dbbd1a4ec85242a7591344cd48fb3d8e6e86a255b16531ad054"}
Nov 03 19:30:54.987150 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:54.987050    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mcxcv" podStartSLOduration=2.909677194 podStartE2EDuration="10.987032105s" totalImagesPullingTime="8.077354911s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:30:44 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:30:54.986844833 +0000 UTC m=+11.680289175" watchObservedRunningTime="2026-11-03 19:30:54.987032105 +0000 UTC m=+11.680476444"
Nov 03 19:30:54.987494 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:54.987450    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-9ck9p" podStartSLOduration=3.181714383 podStartE2EDuration="10.987444419s" totalImagesPullingTime="7.805730036s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:30:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:30:54.967333345 +0000 UTC m=+11.660777682" watchObservedRunningTime="2026-11-03 19:30:54.987444419 +0000 UTC m=+11.680888757"
Nov 03 19:30:55.017973 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:55.017931    2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-jwrp9"
Nov 03 19:30:55.019137 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:55.019112    2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-jwrp9"
Nov 03 19:30:55.748875 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:55.748840    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrxdt"
Nov 03 19:30:55.749072 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:55.748974    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrxdt" podUID="eee821c4-6fb0-4f84-a58d-23dc2ff77720"
Nov 03 19:30:55.749576 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:55.749552    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njg27"
Nov 03 19:30:55.749694 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:55.749661    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njg27" podUID="b46f51b2-436b-4967-8879-a16b58930591"
Nov 03 19:30:55.936970 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:55.936930    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wkrtt" event={"ID":"8b5e1b09-699d-4db8-9eb0-0b71bdda858f","Type":"ContainerStarted","Data":"1b2fc4643cc7ba557d67d190aea0f5a60f00a8f579fdfb6f2d93021457992b27"}
Nov 03 19:30:55.952882 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:55.952812    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-wkrtt" podStartSLOduration=3.854503398 podStartE2EDuration="11.952793881s" totalImagesPullingTime="8.098290483s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:30:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:30:55.952752188 +0000 UTC m=+12.646196528" watchObservedRunningTime="2026-11-03 19:30:55.952793881 +0000 UTC m=+12.646238219"
Nov 03 19:30:56.749193 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:56.749153    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rdr5d"
Nov 03 19:30:56.749370 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:56.749294    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rdr5d" podUID="790607e1-d57c-4622-9a72-1af902ab1d85"
Nov 03 19:30:56.941260 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:56.941223    2575 generic.go:356] "Generic (PLEG): container finished" podID="b797bdb0-b354-4dfc-a50a-3c60ecacbf71" containerID="2624e372a7710aed510c1f076341cadeafcf07abf8774723ca8bd97ab5da6952" exitCode=0
Nov 03 19:30:56.941832 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:56.941313    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g67bt" event={"ID":"b797bdb0-b354-4dfc-a50a-3c60ecacbf71","Type":"ContainerDied","Data":"2624e372a7710aed510c1f076341cadeafcf07abf8774723ca8bd97ab5da6952"}
Nov 03 19:30:56.943698 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:56.943652    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-l98pn" event={"ID":"0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25","Type":"ContainerStarted","Data":"02a94a51b5c5d371eb59f3e37e9df9f8b1015fa7c1a40a23bbdb22ee010f02a9"}
Nov 03 19:30:57.749365 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:57.749331    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrxdt"
Nov 03 19:30:57.749553 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:57.749476    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrxdt" podUID="eee821c4-6fb0-4f84-a58d-23dc2ff77720"
Nov 03 19:30:57.749828 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:57.749809    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njg27"
Nov 03 19:30:57.749932 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:57.749914    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njg27" podUID="b46f51b2-436b-4967-8879-a16b58930591"
Nov 03 19:30:58.749723 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:58.749663    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rdr5d"
Nov 03 19:30:58.750235 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:58.749801    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rdr5d" podUID="790607e1-d57c-4622-9a72-1af902ab1d85"
Nov 03 19:30:58.950175 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:58.950140    2575 generic.go:356] "Generic (PLEG): container finished" podID="b797bdb0-b354-4dfc-a50a-3c60ecacbf71" containerID="1264fc123e789534b0b6945171592450d6d144610484964413f368b833942c14" exitCode=0
Nov 03 19:30:58.950351 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:58.950184    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g67bt" event={"ID":"b797bdb0-b354-4dfc-a50a-3c60ecacbf71","Type":"ContainerDied","Data":"1264fc123e789534b0b6945171592450d6d144610484964413f368b833942c14"}
Nov 03 19:30:58.974936 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:58.974883    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-l98pn" podStartSLOduration=4.680856719 podStartE2EDuration="14.974867994s" totalImagesPullingTime="10.294011275s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:30:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:30:56.983091458 +0000 UTC m=+13.676535794" watchObservedRunningTime="2026-11-03 19:30:58.974867994 +0000 UTC m=+15.668312333"
Nov 03 19:30:59.040334 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:59.040246    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-jwrp9"
Nov 03 19:30:59.040509 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:59.040403    2575 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 19:30:59.045795 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:59.045761    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-jwrp9"
Nov 03 19:30:59.749338 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:59.749095    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrxdt"
Nov 03 19:30:59.749500 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:59.749442    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrxdt" podUID="eee821c4-6fb0-4f84-a58d-23dc2ff77720"
Nov 03 19:30:59.749874 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:30:59.749854    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njg27"
Nov 03 19:30:59.750239 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:30:59.749953    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njg27" podUID="b46f51b2-436b-4967-8879-a16b58930591"
Nov 03 19:31:00.750235 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:00.749760    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rdr5d"
Nov 03 19:31:00.750235 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:00.749895    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rdr5d" podUID="790607e1-d57c-4622-9a72-1af902ab1d85"
Nov 03 19:31:00.957803 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:00.957771    2575 generic.go:356] "Generic (PLEG): container finished" podID="b797bdb0-b354-4dfc-a50a-3c60ecacbf71" containerID="dbc810b52fc22b02df9247e246c8490e439e08074258cf7ec5cbab44613ba588" exitCode=0
Nov 03 19:31:00.957957 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:00.957847    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g67bt" event={"ID":"b797bdb0-b354-4dfc-a50a-3c60ecacbf71","Type":"ContainerDied","Data":"dbc810b52fc22b02df9247e246c8490e439e08074258cf7ec5cbab44613ba588"}
Nov 03 19:31:00.960213 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:00.960038    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr" event={"ID":"d0ab7c64-a85f-4713-a270-f0cf711fcee5","Type":"ContainerStarted","Data":"7baa887eaa6a072724d8c2b172eca45dbef27e460fb355378f0a3f3b9d7dd602"}
Nov 03 19:31:00.960311 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:00.960255    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr" event={"ID":"d0ab7c64-a85f-4713-a270-f0cf711fcee5","Type":"ContainerStarted","Data":"2627185599d42c65409b6e78f54ad98ca54ae32b21a2194318440acac80d390a"}
Nov 03 19:31:01.428871 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:01.426500    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/790607e1-d57c-4622-9a72-1af902ab1d85-metrics-certs\") pod \"network-metrics-daemon-rdr5d\" (UID: \"790607e1-d57c-4622-9a72-1af902ab1d85\") " pod="openshift-multus/network-metrics-daemon-rdr5d"
Nov 03 19:31:01.428871 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:01.426559    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:31:01.428871 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:01.426630    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/790607e1-d57c-4622-9a72-1af902ab1d85-metrics-certs podName:790607e1-d57c-4622-9a72-1af902ab1d85 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:17.426611275 +0000 UTC m=+34.120055605 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/790607e1-d57c-4622-9a72-1af902ab1d85-metrics-certs") pod "network-metrics-daemon-rdr5d" (UID: "790607e1-d57c-4622-9a72-1af902ab1d85") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:31:01.526999 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:01.526959    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f75xn\" (UniqueName: \"kubernetes.io/projected/b46f51b2-436b-4967-8879-a16b58930591-kube-api-access-f75xn\") pod \"network-check-target-njg27\" (UID: \"b46f51b2-436b-4967-8879-a16b58930591\") " pod="openshift-network-diagnostics/network-check-target-njg27"
Nov 03 19:31:01.527204 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:01.527015    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eee821c4-6fb0-4f84-a58d-23dc2ff77720-original-pull-secret\") pod \"global-pull-secret-syncer-hrxdt\" (UID: \"eee821c4-6fb0-4f84-a58d-23dc2ff77720\") " pod="kube-system/global-pull-secret-syncer-hrxdt"
Nov 03 19:31:01.527204 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:01.527119    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 19:31:01.527204 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:01.527140    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 19:31:01.527204 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:01.527140    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 19:31:01.527204 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:01.527151    2575 projected.go:196] Error preparing data for projected volume kube-api-access-f75xn for pod openshift-network-diagnostics/network-check-target-njg27: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:31:01.527204 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:01.527197    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b46f51b2-436b-4967-8879-a16b58930591-kube-api-access-f75xn podName:b46f51b2-436b-4967-8879-a16b58930591 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:17.527184818 +0000 UTC m=+34.220629133 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-f75xn" (UniqueName: "kubernetes.io/projected/b46f51b2-436b-4967-8879-a16b58930591-kube-api-access-f75xn") pod "network-check-target-njg27" (UID: "b46f51b2-436b-4967-8879-a16b58930591") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:31:01.527416 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:01.527213    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eee821c4-6fb0-4f84-a58d-23dc2ff77720-original-pull-secret podName:eee821c4-6fb0-4f84-a58d-23dc2ff77720 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:17.52720716 +0000 UTC m=+34.220651476 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eee821c4-6fb0-4f84-a58d-23dc2ff77720-original-pull-secret") pod "global-pull-secret-syncer-hrxdt" (UID: "eee821c4-6fb0-4f84-a58d-23dc2ff77720") : object "kube-system"/"original-pull-secret" not registered
Nov 03 19:31:01.748927 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:01.748899    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrxdt"
Nov 03 19:31:01.749055 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:01.749030    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrxdt" podUID="eee821c4-6fb0-4f84-a58d-23dc2ff77720"
Nov 03 19:31:01.749132 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:01.749086    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njg27"
Nov 03 19:31:01.749205 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:01.749186    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njg27" podUID="b46f51b2-436b-4967-8879-a16b58930591"
Nov 03 19:31:01.965546 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:01.965513    2575 generic.go:356] "Generic (PLEG): container finished" podID="b797bdb0-b354-4dfc-a50a-3c60ecacbf71" containerID="0344486317f9cc5b931a45412d2bb159d13c62228ed9187fe6e6cd5696fefc3a" exitCode=0
Nov 03 19:31:01.965999 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:01.965591    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g67bt" event={"ID":"b797bdb0-b354-4dfc-a50a-3c60ecacbf71","Type":"ContainerDied","Data":"0344486317f9cc5b931a45412d2bb159d13c62228ed9187fe6e6cd5696fefc3a"}
Nov 03 19:31:01.968507 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:01.968478    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr" event={"ID":"d0ab7c64-a85f-4713-a270-f0cf711fcee5","Type":"ContainerStarted","Data":"1e696972b63135a9aa3949ed0f6fca93526c1e6101bf5202508c7d1f7a96f1b3"}
Nov 03 19:31:01.968594 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:01.968544    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr" event={"ID":"d0ab7c64-a85f-4713-a270-f0cf711fcee5","Type":"ContainerStarted","Data":"81d15041ce52dee32acb8edf2ffed76ef6d6ada481ed471ba8da2efc59f82007"}
Nov 03 19:31:01.968594 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:01.968572    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr" event={"ID":"d0ab7c64-a85f-4713-a270-f0cf711fcee5","Type":"ContainerStarted","Data":"3b3068e875ab5de7c30281ef2c80af304453d71c632504c54c2491fbd931e6e8"}
Nov 03 19:31:01.968656 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:01.968601    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr" event={"ID":"d0ab7c64-a85f-4713-a270-f0cf711fcee5","Type":"ContainerStarted","Data":"b848942a95ff1dea35f8d2ab68e60784d98fed19911e358faffd2c0c326ae0ad"}
Nov 03 19:31:02.749517 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:02.749480    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rdr5d"
Nov 03 19:31:02.749697 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:02.749592    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rdr5d" podUID="790607e1-d57c-4622-9a72-1af902ab1d85"
Nov 03 19:31:02.974221 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:02.974183    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g67bt" event={"ID":"b797bdb0-b354-4dfc-a50a-3c60ecacbf71","Type":"ContainerStarted","Data":"250dd35ea5f908f539ddb33ddc48f45088b7a24326cb0de31333608d8e0babfc"}
Nov 03 19:31:03.001010 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:03.000925    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-g67bt" podStartSLOduration=7.873530296 podStartE2EDuration="19.000911214s" totalImagesPullingTime="11.127380918s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:30:44 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:31:03.000616491 +0000 UTC m=+19.694060863" watchObservedRunningTime="2026-11-03 19:31:03.000911214 +0000 UTC m=+19.694355553"
Nov 03 19:31:03.749695 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:03.749643    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrxdt"
Nov 03 19:31:03.749860 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:03.749753    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njg27"
Nov 03 19:31:03.749860 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:03.749774    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrxdt" podUID="eee821c4-6fb0-4f84-a58d-23dc2ff77720"
Nov 03 19:31:03.749860 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:03.749833    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njg27" podUID="b46f51b2-436b-4967-8879-a16b58930591"
Nov 03 19:31:03.979097 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:03.979053    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr" event={"ID":"d0ab7c64-a85f-4713-a270-f0cf711fcee5","Type":"ContainerStarted","Data":"addd121125f1a66f1c3a25f9935344b31ce674366000222dfa2ee7ea072b4f07"}
Nov 03 19:31:04.749502 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:04.749465    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rdr5d"
Nov 03 19:31:04.749648 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:04.749572    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rdr5d" podUID="790607e1-d57c-4622-9a72-1af902ab1d85"
Nov 03 19:31:05.749494 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:05.749466    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrxdt"
Nov 03 19:31:05.750083 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:05.749496    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njg27"
Nov 03 19:31:05.750083 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:05.749572    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrxdt" podUID="eee821c4-6fb0-4f84-a58d-23dc2ff77720"
Nov 03 19:31:05.750083 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:05.749702    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njg27" podUID="b46f51b2-436b-4967-8879-a16b58930591"
Nov 03 19:31:05.987504 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:05.987465    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr" event={"ID":"d0ab7c64-a85f-4713-a270-f0cf711fcee5","Type":"ContainerStarted","Data":"29b7a1151bb40a2b0b2d490de53e09389a40e369fe8e0a52bb250496932ec578"}
Nov 03 19:31:05.987799 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:05.987776    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:31:05.987926 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:05.987914    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:31:06.003746 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:06.003514    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:31:06.019621 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:06.019559    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr" podStartSLOduration=7.341418739 podStartE2EDuration="22.019546587s" totalImagesPullingTime="14.678127848s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:30:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:31:06.01936401 +0000 UTC m=+22.712808348" watchObservedRunningTime="2026-11-03 19:31:06.019546587 +0000 UTC m=+22.712990925"
Nov 03 19:31:06.749519 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:06.749482    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rdr5d"
Nov 03 19:31:06.749934 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:06.749593    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rdr5d" podUID="790607e1-d57c-4622-9a72-1af902ab1d85"
Nov 03 19:31:06.990066 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:06.990034    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:31:07.005544 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:07.005434    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:31:07.173301 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:07.173258    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rdr5d"]
Nov 03 19:31:07.173572 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:07.173396    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rdr5d"
Nov 03 19:31:07.173572 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:07.173498    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rdr5d" podUID="790607e1-d57c-4622-9a72-1af902ab1d85"
Nov 03 19:31:07.174087 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:07.174066    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-njg27"]
Nov 03 19:31:07.174162 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:07.174156    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njg27"
Nov 03 19:31:07.174246 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:07.174229    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njg27" podUID="b46f51b2-436b-4967-8879-a16b58930591"
Nov 03 19:31:07.174556 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:07.174533    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hrxdt"]
Nov 03 19:31:07.174624 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:07.174615    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrxdt"
Nov 03 19:31:07.174741 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:07.174720    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrxdt" podUID="eee821c4-6fb0-4f84-a58d-23dc2ff77720"
Nov 03 19:31:08.749724 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:08.749690    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njg27"
Nov 03 19:31:08.749724 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:08.749708    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rdr5d"
Nov 03 19:31:08.750146 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:08.749794    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njg27" podUID="b46f51b2-436b-4967-8879-a16b58930591"
Nov 03 19:31:08.750146 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:08.749880    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rdr5d" podUID="790607e1-d57c-4622-9a72-1af902ab1d85"
Nov 03 19:31:08.750146 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:08.749918    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrxdt"
Nov 03 19:31:08.750146 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:08.749969    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrxdt" podUID="eee821c4-6fb0-4f84-a58d-23dc2ff77720"
Nov 03 19:31:10.748924 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:10.748886    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrxdt"
Nov 03 19:31:10.748924 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:10.748935    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njg27"
Nov 03 19:31:10.749480 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:10.748956    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rdr5d"
Nov 03 19:31:10.749480 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:10.749050    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njg27" podUID="b46f51b2-436b-4967-8879-a16b58930591"
Nov 03 19:31:10.749480 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:10.749113    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrxdt" podUID="eee821c4-6fb0-4f84-a58d-23dc2ff77720"
Nov 03 19:31:10.749480 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:10.749189    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rdr5d" podUID="790607e1-d57c-4622-9a72-1af902ab1d85"
Nov 03 19:31:12.749237 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:12.749201    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrxdt"
Nov 03 19:31:12.749654 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:12.749317    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njg27"
Nov 03 19:31:12.749654 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:12.749430    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njg27" podUID="b46f51b2-436b-4967-8879-a16b58930591"
Nov 03 19:31:12.749654 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:12.749320    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrxdt" podUID="eee821c4-6fb0-4f84-a58d-23dc2ff77720"
Nov 03 19:31:12.749654 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:12.749326    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rdr5d"
Nov 03 19:31:12.749654 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:12.749547    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rdr5d" podUID="790607e1-d57c-4622-9a72-1af902ab1d85"
Nov 03 19:31:13.628070 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.628044    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-222.us-east-2.compute.internal" event="NodeReady"
Nov 03 19:31:13.628236 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.628175    2575 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 03 19:31:13.659793 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.659756    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-42xn2"]
Nov 03 19:31:13.662790 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.662770    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-wtgvw"]
Nov 03 19:31:13.662907 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.662890    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-42xn2"
Nov 03 19:31:13.664185 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.664156    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-xgbkd"]
Nov 03 19:31:13.665365 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.665158    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-wtgvw"
Nov 03 19:31:13.666501 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.666468    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hxlrb"]
Nov 03 19:31:13.666710 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.666596    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-xgbkd"
Nov 03 19:31:13.668339 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.668315    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7hcf8"]
Nov 03 19:31:13.668874 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.668852    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hxlrb"
Nov 03 19:31:13.671111 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.670939    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-tx285"]
Nov 03 19:31:13.671371 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.671350    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7hcf8"
Nov 03 19:31:13.676915 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.676894    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-kfdsx"]
Nov 03 19:31:13.678346 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.678326    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-8kh6f"]
Nov 03 19:31:13.679251 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.679224    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-tx285"
Nov 03 19:31:13.680069 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.680051    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kfdsx"
Nov 03 19:31:13.680250 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.680230    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-d74787d9c-nn7sk"]
Nov 03 19:31:13.680989 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.680972    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-8kh6f"
Nov 03 19:31:13.682032 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.682013    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-zpgl6"]
Nov 03 19:31:13.682234 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.682104    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d74787d9c-nn7sk"
Nov 03 19:31:13.685274 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.685242    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-c5c44f4d-dv455"]
Nov 03 19:31:13.685410 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.685363    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-zpgl6"
Nov 03 19:31:13.687186 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.687126    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-mh2tn"]
Nov 03 19:31:13.687279 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.687259    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c5c44f4d-dv455"
Nov 03 19:31:13.689450 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.689398    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Nov 03 19:31:13.689851 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.689831    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Nov 03 19:31:13.689932 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.689899    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Nov 03 19:31:13.690023 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.690001    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Nov 03 19:31:13.691348 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.691277    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Nov 03 19:31:13.691348 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.691347    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Nov 03 19:31:13.691491 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.691471    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-mxrmf\""
Nov 03 19:31:13.692135 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.691875    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-569d4c45f-7pszr"]
Nov 03 19:31:13.692421 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.692395    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mh2tn"
Nov 03 19:31:13.693782 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.693760    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-cbhrt"]
Nov 03 19:31:13.693900 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.693886    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-569d4c45f-7pszr"
Nov 03 19:31:13.695736 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.695713    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-gzf8k"]
Nov 03 19:31:13.695957 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.695934    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cbhrt"
Nov 03 19:31:13.698042 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.698021    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-wtgvw"]
Nov 03 19:31:13.698162 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.698058    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-wj8mp"]
Nov 03 19:31:13.698222 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.698197    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-gzf8k"
Nov 03 19:31:13.705844 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.700891    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-42xn2"]
Nov 03 19:31:13.705844 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.701078    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wj8mp"
Nov 03 19:31:13.705844 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.701868    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-xgbkd"]
Nov 03 19:31:13.705844 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.701903    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-j7m2z"]
Nov 03 19:31:13.707558 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.707283    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hxlrb"]
Nov 03 19:31:13.707558 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.707331    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7hcf8"]
Nov 03 19:31:13.707558 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.707360    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-8kh6f"]
Nov 03 19:31:13.707558 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.707385    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-gzf8k"]
Nov 03 19:31:13.707558 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.707409    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-569d4c45f-7pszr"]
Nov 03 19:31:13.707558 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.707457    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-c5c44f4d-dv455"]
Nov 03 19:31:13.707558 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.707482    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-j7m2z"]
Nov 03 19:31:13.708144 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.707578    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-j7m2z"
Nov 03 19:31:13.710387 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.709340    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-kfdsx"]
Nov 03 19:31:13.710621 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.710594    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wj8mp"]
Nov 03 19:31:13.711323 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.711307    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-d74787d9c-nn7sk"]
Nov 03 19:31:13.712041 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.712021    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-mh2tn"]
Nov 03 19:31:13.712610 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.712594    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-tx285"]
Nov 03 19:31:13.717590 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.715723    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-zpgl6"]
Nov 03 19:31:13.717590 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.717022    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-cbhrt"]
Nov 03 19:31:13.777342 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.777317    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Nov 03 19:31:13.777342 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.777336    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Nov 03 19:31:13.777773 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.777391    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Nov 03 19:31:13.777773 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.777417    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 19:31:13.777773 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.777730    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Nov 03 19:31:13.777898 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.777774    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 03 19:31:13.777898 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.777779    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Nov 03 19:31:13.777898 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.777812    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 19:31:13.777898 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.777822    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Nov 03 19:31:13.777898 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.777825    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Nov 03 19:31:13.777898 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.777882    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 19:31:13.778659 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.778639    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Nov 03 19:31:13.778853 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.778728    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 19:31:13.778853 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.778737    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Nov 03 19:31:13.778853 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.778739    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Nov 03 19:31:13.778853 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.778787    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Nov 03 19:31:13.779699 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.779682    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 19:31:13.779699 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.779653    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Nov 03 19:31:13.779858 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.779736    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 03 19:31:13.779858 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.779760    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 19:31:13.779858 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.779775    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 19:31:13.779858 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.779849    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Nov 03 19:31:13.781085 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.781049    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 19:31:13.781085 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.781081    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 03 19:31:13.781229 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.781084    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 19:31:13.790342 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.790318    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Nov 03 19:31:13.790342 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.790316    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Nov 03 19:31:13.791404 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.791372    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 19:31:13.791404 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.791384    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Nov 03 19:31:13.791404 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.791400    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 19:31:13.791653 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.791623    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Nov 03 19:31:13.791653 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.791638    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Nov 03 19:31:13.791769 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.791653    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Nov 03 19:31:13.791769 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.791728    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Nov 03 19:31:13.793831 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.793810    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Nov 03 19:31:13.793928 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.793815    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 03 19:31:13.793928 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.793865    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 03 19:31:13.794034 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.794024    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-z89rv\""
Nov 03 19:31:13.794096 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.794063    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-rprdn\""
Nov 03 19:31:13.794096 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.794069    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-b925m\""
Nov 03 19:31:13.794096 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.794075    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-n46sn\""
Nov 03 19:31:13.794096 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.794086    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Nov 03 19:31:13.794208 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.794131    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-skb8s\""
Nov 03 19:31:13.794208 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.794135    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-brhwf\""
Nov 03 19:31:13.794309 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.794292    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Nov 03 19:31:13.796042 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.796027    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-gbtrh\""
Nov 03 19:31:13.796042 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.796038    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-cpz4k\""
Nov 03 19:31:13.796208 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.796102    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-8fvqg\""
Nov 03 19:31:13.796303 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.796287    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-sh2j9\""
Nov 03 19:31:13.796356 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.796291    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-dspcg\""
Nov 03 19:31:13.796356 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.796303    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-wrft7\""
Nov 03 19:31:13.798498 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.798480    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 03 19:31:13.803992 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.803955    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Nov 03 19:31:13.803992 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.803959    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 03 19:31:13.804158 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.804146    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 03 19:31:13.812616 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.812593    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3880f3d3-aad1-43c0-b704-01374aad4f78-service-ca-bundle\") pod \"insights-operator-7c4986d449-8kh6f\" (UID: \"3880f3d3-aad1-43c0-b704-01374aad4f78\") " pod="openshift-insights/insights-operator-7c4986d449-8kh6f"
Nov 03 19:31:13.812747 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.812622    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/96c016f7-9098-4a39-9e72-e54da374d8fc-registry-certificates\") pod \"image-registry-d74787d9c-nn7sk\" (UID: \"96c016f7-9098-4a39-9e72-e54da374d8fc\") " pod="openshift-image-registry/image-registry-d74787d9c-nn7sk"
Nov 03 19:31:13.812747 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.812641    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/1753fd31-8933-4567-8146-07f3fd2d10a5-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-7hcf8\" (UID: \"1753fd31-8933-4567-8146-07f3fd2d10a5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7hcf8"
Nov 03 19:31:13.812747 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.812718    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mh9g\" (UniqueName: \"kubernetes.io/projected/3880f3d3-aad1-43c0-b704-01374aad4f78-kube-api-access-8mh9g\") pod \"insights-operator-7c4986d449-8kh6f\" (UID: \"3880f3d3-aad1-43c0-b704-01374aad4f78\") " pod="openshift-insights/insights-operator-7c4986d449-8kh6f"
Nov 03 19:31:13.812862 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.812752    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj2lf\" (UniqueName: \"kubernetes.io/projected/543b8778-baa4-4e09-b697-68e550e42942-kube-api-access-gj2lf\") pod \"ingress-canary-j7m2z\" (UID: \"543b8778-baa4-4e09-b697-68e550e42942\") " pod="openshift-ingress-canary/ingress-canary-j7m2z"
Nov 03 19:31:13.812862 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.812780    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a023911d-3669-43e1-ac77-8a986f3715d6-metrics-certs\") pod \"router-default-569d4c45f-7pszr\" (UID: \"a023911d-3669-43e1-ac77-8a986f3715d6\") " pod="openshift-ingress/router-default-569d4c45f-7pszr"
Nov 03 19:31:13.812862 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.812851    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e2b752d9-88c4-4dee-8b0c-7e9ac252adb3-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-kfdsx\" (UID: \"e2b752d9-88c4-4dee-8b0c-7e9ac252adb3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kfdsx"
Nov 03 19:31:13.812970 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.812941    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/96c016f7-9098-4a39-9e72-e54da374d8fc-image-registry-private-configuration\") pod \"image-registry-d74787d9c-nn7sk\" (UID: \"96c016f7-9098-4a39-9e72-e54da374d8fc\") " pod="openshift-image-registry/image-registry-d74787d9c-nn7sk"
Nov 03 19:31:13.813050 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.813032    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-installation-pull-secrets\") pod \"image-registry-c5c44f4d-dv455\" (UID: \"d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca\") " pod="openshift-image-registry/image-registry-c5c44f4d-dv455"
Nov 03 19:31:13.813136 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.813108    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/a023911d-3669-43e1-ac77-8a986f3715d6-default-certificate\") pod \"router-default-569d4c45f-7pszr\" (UID: \"a023911d-3669-43e1-ac77-8a986f3715d6\") " pod="openshift-ingress/router-default-569d4c45f-7pszr"
Nov 03 19:31:13.813320 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.813171    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a023911d-3669-43e1-ac77-8a986f3715d6-service-ca-bundle\") pod \"router-default-569d4c45f-7pszr\" (UID: \"a023911d-3669-43e1-ac77-8a986f3715d6\") " pod="openshift-ingress/router-default-569d4c45f-7pszr"
Nov 03 19:31:13.813320 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.813200    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfg8j\" (UniqueName: \"kubernetes.io/projected/e10dc749-bc71-456e-a315-6581add01a7b-kube-api-access-nfg8j\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hxlrb\" (UID: \"e10dc749-bc71-456e-a315-6581add01a7b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hxlrb"
Nov 03 19:31:13.813320 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.813225    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-registry-certificates\") pod \"image-registry-c5c44f4d-dv455\" (UID: \"d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca\") " pod="openshift-image-registry/image-registry-c5c44f4d-dv455"
Nov 03 19:31:13.813320 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.813252    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/96c016f7-9098-4a39-9e72-e54da374d8fc-installation-pull-secrets\") pod \"image-registry-d74787d9c-nn7sk\" (UID: \"96c016f7-9098-4a39-9e72-e54da374d8fc\") " pod="openshift-image-registry/image-registry-d74787d9c-nn7sk"
Nov 03 19:31:13.813320 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.813276    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3880f3d3-aad1-43c0-b704-01374aad4f78-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-8kh6f\" (UID: \"3880f3d3-aad1-43c0-b704-01374aad4f78\") " pod="openshift-insights/insights-operator-7c4986d449-8kh6f"
Nov 03 19:31:13.813320 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.813303    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3880f3d3-aad1-43c0-b704-01374aad4f78-serving-cert\") pod \"insights-operator-7c4986d449-8kh6f\" (UID: \"3880f3d3-aad1-43c0-b704-01374aad4f78\") " pod="openshift-insights/insights-operator-7c4986d449-8kh6f"
Nov 03 19:31:13.813967 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.813326    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngpxm\" (UniqueName: \"kubernetes.io/projected/38a6add1-b051-450b-bb96-8ace8a0ccff3-kube-api-access-ngpxm\") pod \"dns-default-wj8mp\" (UID: \"38a6add1-b051-450b-bb96-8ace8a0ccff3\") " pod="openshift-dns/dns-default-wj8mp"
Nov 03 19:31:13.813967 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.813425    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-image-registry-private-configuration\") pod \"image-registry-c5c44f4d-dv455\" (UID: \"d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca\") " pod="openshift-image-registry/image-registry-c5c44f4d-dv455"
Nov 03 19:31:13.813967 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.813498    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w42k\" (UniqueName: \"kubernetes.io/projected/96c016f7-9098-4a39-9e72-e54da374d8fc-kube-api-access-4w42k\") pod \"image-registry-d74787d9c-nn7sk\" (UID: \"96c016f7-9098-4a39-9e72-e54da374d8fc\") " pod="openshift-image-registry/image-registry-d74787d9c-nn7sk"
Nov 03 19:31:13.813967 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.813529    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gm62\" (UniqueName: \"kubernetes.io/projected/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-kube-api-access-8gm62\") pod \"image-registry-c5c44f4d-dv455\" (UID: \"d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca\") " pod="openshift-image-registry/image-registry-c5c44f4d-dv455"
Nov 03 19:31:13.813967 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.813577    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3880f3d3-aad1-43c0-b704-01374aad4f78-tmp\") pod \"insights-operator-7c4986d449-8kh6f\" (UID: \"3880f3d3-aad1-43c0-b704-01374aad4f78\") " pod="openshift-insights/insights-operator-7c4986d449-8kh6f"
Nov 03 19:31:13.813967 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.813809    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/96c016f7-9098-4a39-9e72-e54da374d8fc-registry-tls\") pod \"image-registry-d74787d9c-nn7sk\" (UID: \"96c016f7-9098-4a39-9e72-e54da374d8fc\") " pod="openshift-image-registry/image-registry-d74787d9c-nn7sk"
Nov 03 19:31:13.813967 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.813832    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/96c016f7-9098-4a39-9e72-e54da374d8fc-ca-trust-extracted\") pod \"image-registry-d74787d9c-nn7sk\" (UID: \"96c016f7-9098-4a39-9e72-e54da374d8fc\") " pod="openshift-image-registry/image-registry-d74787d9c-nn7sk"
Nov 03 19:31:13.813967 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.813899    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9fb8c33c-fe15-4daa-a01b-4cb2fc7eae7b-config\") pod \"service-ca-operator-544d749c8c-mh2tn\" (UID: \"9fb8c33c-fe15-4daa-a01b-4cb2fc7eae7b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mh2tn"
Nov 03 19:31:13.813967 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.813934    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-bound-sa-token\") pod \"image-registry-c5c44f4d-dv455\" (UID: \"d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca\") " pod="openshift-image-registry/image-registry-c5c44f4d-dv455"
Nov 03 19:31:13.814332 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.813983    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q68w5\" (UniqueName: \"kubernetes.io/projected/9fb8c33c-fe15-4daa-a01b-4cb2fc7eae7b-kube-api-access-q68w5\") pod \"service-ca-operator-544d749c8c-mh2tn\" (UID: \"9fb8c33c-fe15-4daa-a01b-4cb2fc7eae7b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mh2tn"
Nov 03 19:31:13.814332 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.814000    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xz8xk\" (UniqueName: \"kubernetes.io/projected/e2b752d9-88c4-4dee-8b0c-7e9ac252adb3-kube-api-access-xz8xk\") pod \"cluster-monitoring-operator-884576c57-kfdsx\" (UID: \"e2b752d9-88c4-4dee-8b0c-7e9ac252adb3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kfdsx"
Nov 03 19:31:13.814332 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.814017    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-registry-tls\") pod \"image-registry-c5c44f4d-dv455\" (UID: \"d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca\") " pod="openshift-image-registry/image-registry-c5c44f4d-dv455"
Nov 03 19:31:13.814332 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.814091    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/38a6add1-b051-450b-bb96-8ace8a0ccff3-config-volume\") pod \"dns-default-wj8mp\" (UID: \"38a6add1-b051-450b-bb96-8ace8a0ccff3\") " pod="openshift-dns/dns-default-wj8mp"
Nov 03 19:31:13.814332 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.814129    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-ca-trust-extracted\") pod \"image-registry-c5c44f4d-dv455\" (UID: \"d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca\") " pod="openshift-image-registry/image-registry-c5c44f4d-dv455"
Nov 03 19:31:13.814332 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.814213    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/38a6add1-b051-450b-bb96-8ace8a0ccff3-metrics-tls\") pod \"dns-default-wj8mp\" (UID: \"38a6add1-b051-450b-bb96-8ace8a0ccff3\") " pod="openshift-dns/dns-default-wj8mp"
Nov 03 19:31:13.814332 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.814254    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtqhk\" (UniqueName: \"kubernetes.io/projected/8fa315db-e48d-4ac5-960a-23750974d431-kube-api-access-jtqhk\") pod \"kas-connection-checker-685895c7c9-42xn2\" (UID: \"8fa315db-e48d-4ac5-960a-23750974d431\") " pod="kube-system/kas-connection-checker-685895c7c9-42xn2"
Nov 03 19:31:13.814332 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.814300    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/543b8778-baa4-4e09-b697-68e550e42942-cert\") pod \"ingress-canary-j7m2z\" (UID: \"543b8778-baa4-4e09-b697-68e550e42942\") " pod="openshift-ingress-canary/ingress-canary-j7m2z"
Nov 03 19:31:13.814593 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.814335    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e10dc749-bc71-456e-a315-6581add01a7b-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hxlrb\" (UID: \"e10dc749-bc71-456e-a315-6581add01a7b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hxlrb"
Nov 03 19:31:13.814593 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.814386    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3880f3d3-aad1-43c0-b704-01374aad4f78-snapshots\") pod \"insights-operator-7c4986d449-8kh6f\" (UID: \"3880f3d3-aad1-43c0-b704-01374aad4f78\") " pod="openshift-insights/insights-operator-7c4986d449-8kh6f"
Nov 03 19:31:13.814593 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.814437    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p288b\" (UniqueName: \"kubernetes.io/projected/f97b063e-8403-46bd-9716-72411589a494-kube-api-access-p288b\") pod \"kas-connection-checker-685895c7c9-wtgvw\" (UID: \"f97b063e-8403-46bd-9716-72411589a494\") " pod="kube-system/kas-connection-checker-685895c7c9-wtgvw"
Nov 03 19:31:13.814593 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.814472    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/a023911d-3669-43e1-ac77-8a986f3715d6-stats-auth\") pod \"router-default-569d4c45f-7pszr\" (UID: \"a023911d-3669-43e1-ac77-8a986f3715d6\") " pod="openshift-ingress/router-default-569d4c45f-7pszr"
Nov 03 19:31:13.814593 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.814529    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e10dc749-bc71-456e-a315-6581add01a7b-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hxlrb\" (UID: \"e10dc749-bc71-456e-a315-6581add01a7b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hxlrb"
Nov 03 19:31:13.814593 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.814567    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdw57\" (UniqueName: \"kubernetes.io/projected/f89ef12d-9c60-4538-af9c-f97453bbc855-kube-api-access-xdw57\") pod \"volume-data-source-validator-58c9f5f46d-tx285\" (UID: \"f89ef12d-9c60-4538-af9c-f97453bbc855\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-tx285"
Nov 03 19:31:13.814792 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.814589    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/eb488ece-2ac3-499f-84c1-73534fcbb20b-nginx-conf\") pod \"networking-console-plugin-86c478bb88-zpgl6\" (UID: \"eb488ece-2ac3-499f-84c1-73534fcbb20b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zpgl6"
Nov 03 19:31:13.814792 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.814644    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38a6add1-b051-450b-bb96-8ace8a0ccff3-tmp-dir\") pod \"dns-default-wj8mp\" (UID: \"38a6add1-b051-450b-bb96-8ace8a0ccff3\") " pod="openshift-dns/dns-default-wj8mp"
Nov 03 19:31:13.814792 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.814699    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1753fd31-8933-4567-8146-07f3fd2d10a5-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-7hcf8\" (UID: \"1753fd31-8933-4567-8146-07f3fd2d10a5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7hcf8"
Nov 03 19:31:13.814792 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.814735    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2287d\" (UniqueName: \"kubernetes.io/projected/1611b730-6163-4776-bd2d-3d10c2cc9a88-kube-api-access-2287d\") pod \"network-check-source-6f78d59f99-cbhrt\" (UID: \"1611b730-6163-4776-bd2d-3d10c2cc9a88\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cbhrt"
Nov 03 19:31:13.814792 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.814751    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6f215b42-0f09-46a0-bc98-392c0fa909c9-serving-cert\") pod \"console-operator-7cd56b8dbc-gzf8k\" (UID: \"6f215b42-0f09-46a0-bc98-392c0fa909c9\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gzf8k"
Nov 03 19:31:13.814792 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.814777    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjtq4\" (UniqueName: \"kubernetes.io/projected/6f215b42-0f09-46a0-bc98-392c0fa909c9-kube-api-access-qjtq4\") pod \"console-operator-7cd56b8dbc-gzf8k\" (UID: \"6f215b42-0f09-46a0-bc98-392c0fa909c9\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gzf8k"
Nov 03 19:31:13.814952 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.814794    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftr5x\" (UniqueName: \"kubernetes.io/projected/a023911d-3669-43e1-ac77-8a986f3715d6-kube-api-access-ftr5x\") pod \"router-default-569d4c45f-7pszr\" (UID: \"a023911d-3669-43e1-ac77-8a986f3715d6\") " pod="openshift-ingress/router-default-569d4c45f-7pszr"
Nov 03 19:31:13.814952 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.814824    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/96c016f7-9098-4a39-9e72-e54da374d8fc-bound-sa-token\") pod \"image-registry-d74787d9c-nn7sk\" (UID: \"96c016f7-9098-4a39-9e72-e54da374d8fc\") " pod="openshift-image-registry/image-registry-d74787d9c-nn7sk"
Nov 03 19:31:13.814952 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.814841    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n7xj\" (UniqueName: \"kubernetes.io/projected/1753fd31-8933-4567-8146-07f3fd2d10a5-kube-api-access-6n7xj\") pod \"cluster-samples-operator-6ddb6c8b94-7hcf8\" (UID: \"1753fd31-8933-4567-8146-07f3fd2d10a5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7hcf8"
Nov 03 19:31:13.814952 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.814878    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6f215b42-0f09-46a0-bc98-392c0fa909c9-trusted-ca\") pod \"console-operator-7cd56b8dbc-gzf8k\" (UID: \"6f215b42-0f09-46a0-bc98-392c0fa909c9\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gzf8k"
Nov 03 19:31:13.814952 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.814917    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6f215b42-0f09-46a0-bc98-392c0fa909c9-config\") pod \"console-operator-7cd56b8dbc-gzf8k\" (UID: \"6f215b42-0f09-46a0-bc98-392c0fa909c9\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gzf8k"
Nov 03 19:31:13.814952 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.814941    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e2b752d9-88c4-4dee-8b0c-7e9ac252adb3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kfdsx\" (UID: \"e2b752d9-88c4-4dee-8b0c-7e9ac252adb3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kfdsx"
Nov 03 19:31:13.815111 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.815032    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/96c016f7-9098-4a39-9e72-e54da374d8fc-trusted-ca\") pod \"image-registry-d74787d9c-nn7sk\" (UID: \"96c016f7-9098-4a39-9e72-e54da374d8fc\") " pod="openshift-image-registry/image-registry-d74787d9c-nn7sk"
Nov 03 19:31:13.815111 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.815065    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktkfm\" (UniqueName: \"kubernetes.io/projected/df9f9c4a-d054-42cf-b17c-0d23acd0459b-kube-api-access-ktkfm\") pod \"kas-connection-checker-685895c7c9-xgbkd\" (UID: \"df9f9c4a-d054-42cf-b17c-0d23acd0459b\") " pod="kube-system/kas-connection-checker-685895c7c9-xgbkd"
Nov 03 19:31:13.815111 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.815094    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/eb488ece-2ac3-499f-84c1-73534fcbb20b-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zpgl6\" (UID: \"eb488ece-2ac3-499f-84c1-73534fcbb20b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zpgl6"
Nov 03 19:31:13.815195 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.815134    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-trusted-ca\") pod \"image-registry-c5c44f4d-dv455\" (UID: \"d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca\") " pod="openshift-image-registry/image-registry-c5c44f4d-dv455"
Nov 03 19:31:13.815195 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.815158    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9fb8c33c-fe15-4daa-a01b-4cb2fc7eae7b-serving-cert\") pod \"service-ca-operator-544d749c8c-mh2tn\" (UID: \"9fb8c33c-fe15-4daa-a01b-4cb2fc7eae7b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mh2tn"
Nov 03 19:31:13.915971 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.915866    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/96c016f7-9098-4a39-9e72-e54da374d8fc-bound-sa-token\") pod \"image-registry-d74787d9c-nn7sk\" (UID: \"96c016f7-9098-4a39-9e72-e54da374d8fc\") " pod="openshift-image-registry/image-registry-d74787d9c-nn7sk"
Nov 03 19:31:13.915971 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.915917    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n7xj\" (UniqueName: \"kubernetes.io/projected/1753fd31-8933-4567-8146-07f3fd2d10a5-kube-api-access-6n7xj\") pod \"cluster-samples-operator-6ddb6c8b94-7hcf8\" (UID: \"1753fd31-8933-4567-8146-07f3fd2d10a5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7hcf8"
Nov 03 19:31:13.915971 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.915950    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6f215b42-0f09-46a0-bc98-392c0fa909c9-trusted-ca\") pod \"console-operator-7cd56b8dbc-gzf8k\" (UID: \"6f215b42-0f09-46a0-bc98-392c0fa909c9\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gzf8k"
Nov 03 19:31:13.916274 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.915978    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6f215b42-0f09-46a0-bc98-392c0fa909c9-config\") pod \"console-operator-7cd56b8dbc-gzf8k\" (UID: \"6f215b42-0f09-46a0-bc98-392c0fa909c9\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gzf8k"
Nov 03 19:31:13.916274 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.916036    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e2b752d9-88c4-4dee-8b0c-7e9ac252adb3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kfdsx\" (UID: \"e2b752d9-88c4-4dee-8b0c-7e9ac252adb3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kfdsx"
Nov 03 19:31:13.916274 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.916099    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/96c016f7-9098-4a39-9e72-e54da374d8fc-trusted-ca\") pod \"image-registry-d74787d9c-nn7sk\" (UID: \"96c016f7-9098-4a39-9e72-e54da374d8fc\") " pod="openshift-image-registry/image-registry-d74787d9c-nn7sk"
Nov 03 19:31:13.916274 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.916152    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktkfm\" (UniqueName: \"kubernetes.io/projected/df9f9c4a-d054-42cf-b17c-0d23acd0459b-kube-api-access-ktkfm\") pod \"kas-connection-checker-685895c7c9-xgbkd\" (UID: \"df9f9c4a-d054-42cf-b17c-0d23acd0459b\") " pod="kube-system/kas-connection-checker-685895c7c9-xgbkd"
Nov 03 19:31:13.916274 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.916183    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/eb488ece-2ac3-499f-84c1-73534fcbb20b-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zpgl6\" (UID: \"eb488ece-2ac3-499f-84c1-73534fcbb20b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zpgl6"
Nov 03 19:31:13.916274 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:13.916206    2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 19:31:13.916274 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.916225    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-trusted-ca\") pod \"image-registry-c5c44f4d-dv455\" (UID: \"d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca\") " pod="openshift-image-registry/image-registry-c5c44f4d-dv455"
Nov 03 19:31:13.916274 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.916251    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9fb8c33c-fe15-4daa-a01b-4cb2fc7eae7b-serving-cert\") pod \"service-ca-operator-544d749c8c-mh2tn\" (UID: \"9fb8c33c-fe15-4daa-a01b-4cb2fc7eae7b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mh2tn"
Nov 03 19:31:13.916621 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:13.916300    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2b752d9-88c4-4dee-8b0c-7e9ac252adb3-cluster-monitoring-operator-tls podName:e2b752d9-88c4-4dee-8b0c-7e9ac252adb3 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:14.416265542 +0000 UTC m=+31.109709875 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e2b752d9-88c4-4dee-8b0c-7e9ac252adb3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-kfdsx" (UID: "e2b752d9-88c4-4dee-8b0c-7e9ac252adb3") : secret "cluster-monitoring-operator-tls" not found
Nov 03 19:31:13.916621 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.916325    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3880f3d3-aad1-43c0-b704-01374aad4f78-service-ca-bundle\") pod \"insights-operator-7c4986d449-8kh6f\" (UID: \"3880f3d3-aad1-43c0-b704-01374aad4f78\") " pod="openshift-insights/insights-operator-7c4986d449-8kh6f"
Nov 03 19:31:13.916621 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.916377    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/96c016f7-9098-4a39-9e72-e54da374d8fc-registry-certificates\") pod \"image-registry-d74787d9c-nn7sk\" (UID: \"96c016f7-9098-4a39-9e72-e54da374d8fc\") " pod="openshift-image-registry/image-registry-d74787d9c-nn7sk"
Nov 03 19:31:13.916621 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.916405    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/1753fd31-8933-4567-8146-07f3fd2d10a5-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-7hcf8\" (UID: \"1753fd31-8933-4567-8146-07f3fd2d10a5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7hcf8"
Nov 03 19:31:13.916621 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.916447    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mh9g\" (UniqueName: \"kubernetes.io/projected/3880f3d3-aad1-43c0-b704-01374aad4f78-kube-api-access-8mh9g\") pod \"insights-operator-7c4986d449-8kh6f\" (UID: \"3880f3d3-aad1-43c0-b704-01374aad4f78\") " pod="openshift-insights/insights-operator-7c4986d449-8kh6f"
Nov 03 19:31:13.916621 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.916470    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj2lf\" (UniqueName: \"kubernetes.io/projected/543b8778-baa4-4e09-b697-68e550e42942-kube-api-access-gj2lf\") pod \"ingress-canary-j7m2z\" (UID: \"543b8778-baa4-4e09-b697-68e550e42942\") " pod="openshift-ingress-canary/ingress-canary-j7m2z"
Nov 03 19:31:13.916621 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.916494    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a023911d-3669-43e1-ac77-8a986f3715d6-metrics-certs\") pod \"router-default-569d4c45f-7pszr\" (UID: \"a023911d-3669-43e1-ac77-8a986f3715d6\") " pod="openshift-ingress/router-default-569d4c45f-7pszr"
Nov 03 19:31:13.916621 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.916558    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e2b752d9-88c4-4dee-8b0c-7e9ac252adb3-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-kfdsx\" (UID: \"e2b752d9-88c4-4dee-8b0c-7e9ac252adb3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kfdsx"
Nov 03 19:31:13.916621 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.916605    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/96c016f7-9098-4a39-9e72-e54da374d8fc-image-registry-private-configuration\") pod \"image-registry-d74787d9c-nn7sk\" (UID: \"96c016f7-9098-4a39-9e72-e54da374d8fc\") " pod="openshift-image-registry/image-registry-d74787d9c-nn7sk"
Nov 03 19:31:13.917062 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.916641    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-installation-pull-secrets\") pod \"image-registry-c5c44f4d-dv455\" (UID: \"d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca\") " pod="openshift-image-registry/image-registry-c5c44f4d-dv455"
Nov 03 19:31:13.917062 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.916711    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/a023911d-3669-43e1-ac77-8a986f3715d6-default-certificate\") pod \"router-default-569d4c45f-7pszr\" (UID: \"a023911d-3669-43e1-ac77-8a986f3715d6\") " pod="openshift-ingress/router-default-569d4c45f-7pszr"
Nov 03 19:31:13.917062 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.916738    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a023911d-3669-43e1-ac77-8a986f3715d6-service-ca-bundle\") pod \"router-default-569d4c45f-7pszr\" (UID: \"a023911d-3669-43e1-ac77-8a986f3715d6\") " pod="openshift-ingress/router-default-569d4c45f-7pszr"
Nov 03 19:31:13.917062 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.916785    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfg8j\" (UniqueName: \"kubernetes.io/projected/e10dc749-bc71-456e-a315-6581add01a7b-kube-api-access-nfg8j\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hxlrb\" (UID: \"e10dc749-bc71-456e-a315-6581add01a7b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hxlrb"
Nov 03 19:31:13.917062 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.916868    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6f215b42-0f09-46a0-bc98-392c0fa909c9-config\") pod \"console-operator-7cd56b8dbc-gzf8k\" (UID: \"6f215b42-0f09-46a0-bc98-392c0fa909c9\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gzf8k"
Nov 03 19:31:13.917062 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.916975    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-registry-certificates\") pod \"image-registry-c5c44f4d-dv455\" (UID: \"d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca\") " pod="openshift-image-registry/image-registry-c5c44f4d-dv455"
Nov 03 19:31:13.917062 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.917017    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/96c016f7-9098-4a39-9e72-e54da374d8fc-installation-pull-secrets\") pod \"image-registry-d74787d9c-nn7sk\" (UID: \"96c016f7-9098-4a39-9e72-e54da374d8fc\") " pod="openshift-image-registry/image-registry-d74787d9c-nn7sk"
Nov 03 19:31:13.917062 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.917045    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3880f3d3-aad1-43c0-b704-01374aad4f78-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-8kh6f\" (UID: \"3880f3d3-aad1-43c0-b704-01374aad4f78\") " pod="openshift-insights/insights-operator-7c4986d449-8kh6f"
Nov 03 19:31:13.917451 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.917070    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3880f3d3-aad1-43c0-b704-01374aad4f78-serving-cert\") pod \"insights-operator-7c4986d449-8kh6f\" (UID: \"3880f3d3-aad1-43c0-b704-01374aad4f78\") " pod="openshift-insights/insights-operator-7c4986d449-8kh6f"
Nov 03 19:31:13.917451 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.917094    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngpxm\" (UniqueName: \"kubernetes.io/projected/38a6add1-b051-450b-bb96-8ace8a0ccff3-kube-api-access-ngpxm\") pod \"dns-default-wj8mp\" (UID: \"38a6add1-b051-450b-bb96-8ace8a0ccff3\") " pod="openshift-dns/dns-default-wj8mp"
Nov 03 19:31:13.917451 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.917126    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-image-registry-private-configuration\") pod \"image-registry-c5c44f4d-dv455\" (UID: \"d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca\") " pod="openshift-image-registry/image-registry-c5c44f4d-dv455"
Nov 03 19:31:13.917451 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.917152    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w42k\" (UniqueName: \"kubernetes.io/projected/96c016f7-9098-4a39-9e72-e54da374d8fc-kube-api-access-4w42k\") pod \"image-registry-d74787d9c-nn7sk\" (UID: \"96c016f7-9098-4a39-9e72-e54da374d8fc\") " pod="openshift-image-registry/image-registry-d74787d9c-nn7sk"
Nov 03 19:31:13.917451 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.917174    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gm62\" (UniqueName: \"kubernetes.io/projected/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-kube-api-access-8gm62\") pod \"image-registry-c5c44f4d-dv455\" (UID: \"d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca\") " pod="openshift-image-registry/image-registry-c5c44f4d-dv455"
Nov 03 19:31:13.917451 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:13.917186    2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 03 19:31:13.917451 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.917198    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3880f3d3-aad1-43c0-b704-01374aad4f78-tmp\") pod \"insights-operator-7c4986d449-8kh6f\" (UID: \"3880f3d3-aad1-43c0-b704-01374aad4f78\") " pod="openshift-insights/insights-operator-7c4986d449-8kh6f"
Nov 03 19:31:13.917451 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.917220    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/96c016f7-9098-4a39-9e72-e54da374d8fc-registry-tls\") pod \"image-registry-d74787d9c-nn7sk\" (UID: \"96c016f7-9098-4a39-9e72-e54da374d8fc\") " pod="openshift-image-registry/image-registry-d74787d9c-nn7sk"
Nov 03 19:31:13.917451 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:13.917257    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a023911d-3669-43e1-ac77-8a986f3715d6-metrics-certs podName:a023911d-3669-43e1-ac77-8a986f3715d6 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:14.41722465 +0000 UTC m=+31.110668965 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a023911d-3669-43e1-ac77-8a986f3715d6-metrics-certs") pod "router-default-569d4c45f-7pszr" (UID: "a023911d-3669-43e1-ac77-8a986f3715d6") : secret "router-metrics-certs-default" not found
Nov 03 19:31:13.917451 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.917268    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/96c016f7-9098-4a39-9e72-e54da374d8fc-trusted-ca\") pod \"image-registry-d74787d9c-nn7sk\" (UID: \"96c016f7-9098-4a39-9e72-e54da374d8fc\") " pod="openshift-image-registry/image-registry-d74787d9c-nn7sk"
Nov 03 19:31:13.917451 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:13.917285    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 19:31:13.917451 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.917292    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/96c016f7-9098-4a39-9e72-e54da374d8fc-ca-trust-extracted\") pod \"image-registry-d74787d9c-nn7sk\" (UID: \"96c016f7-9098-4a39-9e72-e54da374d8fc\") " pod="openshift-image-registry/image-registry-d74787d9c-nn7sk"
Nov 03 19:31:13.917451 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.917432    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6f215b42-0f09-46a0-bc98-392c0fa909c9-trusted-ca\") pod \"console-operator-7cd56b8dbc-gzf8k\" (UID: \"6f215b42-0f09-46a0-bc98-392c0fa909c9\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gzf8k"
Nov 03 19:31:13.918253 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.917564    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9fb8c33c-fe15-4daa-a01b-4cb2fc7eae7b-config\") pod \"service-ca-operator-544d749c8c-mh2tn\" (UID: \"9fb8c33c-fe15-4daa-a01b-4cb2fc7eae7b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mh2tn"
Nov 03 19:31:13.918809 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:13.917298    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d74787d9c-nn7sk: secret "image-registry-tls" not found
Nov 03 19:31:13.918809 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.918380    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3880f3d3-aad1-43c0-b704-01374aad4f78-tmp\") pod \"insights-operator-7c4986d449-8kh6f\" (UID: \"3880f3d3-aad1-43c0-b704-01374aad4f78\") " pod="openshift-insights/insights-operator-7c4986d449-8kh6f"
Nov 03 19:31:13.918809 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.918075    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/96c016f7-9098-4a39-9e72-e54da374d8fc-ca-trust-extracted\") pod \"image-registry-d74787d9c-nn7sk\" (UID: \"96c016f7-9098-4a39-9e72-e54da374d8fc\") " pod="openshift-image-registry/image-registry-d74787d9c-nn7sk"
Nov 03 19:31:13.918809 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.918271    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/96c016f7-9098-4a39-9e72-e54da374d8fc-registry-certificates\") pod \"image-registry-d74787d9c-nn7sk\" (UID: \"96c016f7-9098-4a39-9e72-e54da374d8fc\") " pod="openshift-image-registry/image-registry-d74787d9c-nn7sk"
Nov 03 19:31:13.918809 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.918278    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/1753fd31-8933-4567-8146-07f3fd2d10a5-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-7hcf8\" (UID: \"1753fd31-8933-4567-8146-07f3fd2d10a5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7hcf8"
Nov 03 19:31:13.918809 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.918313    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3880f3d3-aad1-43c0-b704-01374aad4f78-service-ca-bundle\") pod \"insights-operator-7c4986d449-8kh6f\" (UID: \"3880f3d3-aad1-43c0-b704-01374aad4f78\") " pod="openshift-insights/insights-operator-7c4986d449-8kh6f"
Nov 03 19:31:13.918809 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.918763    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-registry-certificates\") pod \"image-registry-c5c44f4d-dv455\" (UID: \"d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca\") " pod="openshift-image-registry/image-registry-c5c44f4d-dv455"
Nov 03 19:31:13.918809 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.917834    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-trusted-ca\") pod \"image-registry-c5c44f4d-dv455\" (UID: \"d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca\") " pod="openshift-image-registry/image-registry-c5c44f4d-dv455"
Nov 03 19:31:13.918809 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:13.917663    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a023911d-3669-43e1-ac77-8a986f3715d6-service-ca-bundle podName:a023911d-3669-43e1-ac77-8a986f3715d6 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:14.417648568 +0000 UTC m=+31.111092925 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a023911d-3669-43e1-ac77-8a986f3715d6-service-ca-bundle") pod "router-default-569d4c45f-7pszr" (UID: "a023911d-3669-43e1-ac77-8a986f3715d6") : configmap references non-existent config key: service-ca.crt
Nov 03 19:31:13.919383 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:13.918145    2575 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 19:31:13.919383 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:13.918874    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/96c016f7-9098-4a39-9e72-e54da374d8fc-registry-tls podName:96c016f7-9098-4a39-9e72-e54da374d8fc nodeName:}" failed. No retries permitted until 2026-11-03 19:31:14.418858692 +0000 UTC m=+31.112303011 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/96c016f7-9098-4a39-9e72-e54da374d8fc-registry-tls") pod "image-registry-d74787d9c-nn7sk" (UID: "96c016f7-9098-4a39-9e72-e54da374d8fc") : secret "image-registry-tls" not found
Nov 03 19:31:13.919383 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:13.918893    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb488ece-2ac3-499f-84c1-73534fcbb20b-networking-console-plugin-cert podName:eb488ece-2ac3-499f-84c1-73534fcbb20b nodeName:}" failed. No retries permitted until 2026-11-03 19:31:14.418883349 +0000 UTC m=+31.112327671 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/eb488ece-2ac3-499f-84c1-73534fcbb20b-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-zpgl6" (UID: "eb488ece-2ac3-499f-84c1-73534fcbb20b") : secret "networking-console-plugin-cert" not found
Nov 03 19:31:13.919383 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.918928    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-bound-sa-token\") pod \"image-registry-c5c44f4d-dv455\" (UID: \"d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca\") " pod="openshift-image-registry/image-registry-c5c44f4d-dv455"
Nov 03 19:31:13.919627 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.919333    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q68w5\" (UniqueName: \"kubernetes.io/projected/9fb8c33c-fe15-4daa-a01b-4cb2fc7eae7b-kube-api-access-q68w5\") pod \"service-ca-operator-544d749c8c-mh2tn\" (UID: \"9fb8c33c-fe15-4daa-a01b-4cb2fc7eae7b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mh2tn"
Nov 03 19:31:13.919627 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.919565    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xz8xk\" (UniqueName: \"kubernetes.io/projected/e2b752d9-88c4-4dee-8b0c-7e9ac252adb3-kube-api-access-xz8xk\") pod \"cluster-monitoring-operator-884576c57-kfdsx\" (UID: \"e2b752d9-88c4-4dee-8b0c-7e9ac252adb3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kfdsx"
Nov 03 19:31:13.919627 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.919596    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-registry-tls\") pod \"image-registry-c5c44f4d-dv455\" (UID: \"d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca\") " pod="openshift-image-registry/image-registry-c5c44f4d-dv455"
Nov 03 19:31:13.919627 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.919624    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/38a6add1-b051-450b-bb96-8ace8a0ccff3-config-volume\") pod \"dns-default-wj8mp\" (UID: \"38a6add1-b051-450b-bb96-8ace8a0ccff3\") " pod="openshift-dns/dns-default-wj8mp"
Nov 03 19:31:13.919876 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.919633    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3880f3d3-aad1-43c0-b704-01374aad4f78-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-8kh6f\" (UID: \"3880f3d3-aad1-43c0-b704-01374aad4f78\") " pod="openshift-insights/insights-operator-7c4986d449-8kh6f"
Nov 03 19:31:13.919876 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.919650    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-ca-trust-extracted\") pod \"image-registry-c5c44f4d-dv455\" (UID: \"d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca\") " pod="openshift-image-registry/image-registry-c5c44f4d-dv455"
Nov 03 19:31:13.919876 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.919532    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e2b752d9-88c4-4dee-8b0c-7e9ac252adb3-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-kfdsx\" (UID: \"e2b752d9-88c4-4dee-8b0c-7e9ac252adb3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kfdsx"
Nov 03 19:31:13.920072 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.919998    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-ca-trust-extracted\") pod \"image-registry-c5c44f4d-dv455\" (UID: \"d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca\") " pod="openshift-image-registry/image-registry-c5c44f4d-dv455"
Nov 03 19:31:13.920783 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.920225    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/38a6add1-b051-450b-bb96-8ace8a0ccff3-config-volume\") pod \"dns-default-wj8mp\" (UID: \"38a6add1-b051-450b-bb96-8ace8a0ccff3\") " pod="openshift-dns/dns-default-wj8mp"
Nov 03 19:31:13.920783 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:13.920285    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 19:31:13.920783 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:13.920299    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c5c44f4d-dv455: secret "image-registry-tls" not found
Nov 03 19:31:13.920783 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:13.920358    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-registry-tls podName:d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca nodeName:}" failed. No retries permitted until 2026-11-03 19:31:14.420343402 +0000 UTC m=+31.113787720 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-registry-tls") pod "image-registry-c5c44f4d-dv455" (UID: "d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca") : secret "image-registry-tls" not found
Nov 03 19:31:13.920783 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.920355    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9fb8c33c-fe15-4daa-a01b-4cb2fc7eae7b-config\") pod \"service-ca-operator-544d749c8c-mh2tn\" (UID: \"9fb8c33c-fe15-4daa-a01b-4cb2fc7eae7b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mh2tn"
Nov 03 19:31:13.920783 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.920380    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/38a6add1-b051-450b-bb96-8ace8a0ccff3-metrics-tls\") pod \"dns-default-wj8mp\" (UID: \"38a6add1-b051-450b-bb96-8ace8a0ccff3\") " pod="openshift-dns/dns-default-wj8mp"
Nov 03 19:31:13.920783 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.920411    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtqhk\" (UniqueName: \"kubernetes.io/projected/8fa315db-e48d-4ac5-960a-23750974d431-kube-api-access-jtqhk\") pod \"kas-connection-checker-685895c7c9-42xn2\" (UID: \"8fa315db-e48d-4ac5-960a-23750974d431\") " pod="kube-system/kas-connection-checker-685895c7c9-42xn2"
Nov 03 19:31:13.920783 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.920439    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/543b8778-baa4-4e09-b697-68e550e42942-cert\") pod \"ingress-canary-j7m2z\" (UID: \"543b8778-baa4-4e09-b697-68e550e42942\") " pod="openshift-ingress-canary/ingress-canary-j7m2z"
Nov 03 19:31:13.920783 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:13.920448    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 19:31:13.920783 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.920467    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e10dc749-bc71-456e-a315-6581add01a7b-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hxlrb\" (UID: \"e10dc749-bc71-456e-a315-6581add01a7b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hxlrb"
Nov 03 19:31:13.920783 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:13.920491    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38a6add1-b051-450b-bb96-8ace8a0ccff3-metrics-tls podName:38a6add1-b051-450b-bb96-8ace8a0ccff3 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:14.420480037 +0000 UTC m=+31.113924358 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/38a6add1-b051-450b-bb96-8ace8a0ccff3-metrics-tls") pod "dns-default-wj8mp" (UID: "38a6add1-b051-450b-bb96-8ace8a0ccff3") : secret "dns-default-metrics-tls" not found
Nov 03 19:31:13.920783 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.920518    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3880f3d3-aad1-43c0-b704-01374aad4f78-snapshots\") pod \"insights-operator-7c4986d449-8kh6f\" (UID: \"3880f3d3-aad1-43c0-b704-01374aad4f78\") " pod="openshift-insights/insights-operator-7c4986d449-8kh6f"
Nov 03 19:31:13.920783 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.920550    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p288b\" (UniqueName: \"kubernetes.io/projected/f97b063e-8403-46bd-9716-72411589a494-kube-api-access-p288b\") pod \"kas-connection-checker-685895c7c9-wtgvw\" (UID: \"f97b063e-8403-46bd-9716-72411589a494\") " pod="kube-system/kas-connection-checker-685895c7c9-wtgvw"
Nov 03 19:31:13.920783 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.920575    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/a023911d-3669-43e1-ac77-8a986f3715d6-stats-auth\") pod \"router-default-569d4c45f-7pszr\" (UID: \"a023911d-3669-43e1-ac77-8a986f3715d6\") " pod="openshift-ingress/router-default-569d4c45f-7pszr"
Nov 03 19:31:13.920783 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.920600    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e10dc749-bc71-456e-a315-6581add01a7b-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hxlrb\" (UID: \"e10dc749-bc71-456e-a315-6581add01a7b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hxlrb"
Nov 03 19:31:13.920783 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.920642    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdw57\" (UniqueName: \"kubernetes.io/projected/f89ef12d-9c60-4538-af9c-f97453bbc855-kube-api-access-xdw57\") pod \"volume-data-source-validator-58c9f5f46d-tx285\" (UID: \"f89ef12d-9c60-4538-af9c-f97453bbc855\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-tx285"
Nov 03 19:31:13.921693 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.920688    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/eb488ece-2ac3-499f-84c1-73534fcbb20b-nginx-conf\") pod \"networking-console-plugin-86c478bb88-zpgl6\" (UID: \"eb488ece-2ac3-499f-84c1-73534fcbb20b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zpgl6"
Nov 03 19:31:13.921693 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.920716    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38a6add1-b051-450b-bb96-8ace8a0ccff3-tmp-dir\") pod \"dns-default-wj8mp\" (UID: \"38a6add1-b051-450b-bb96-8ace8a0ccff3\") " pod="openshift-dns/dns-default-wj8mp"
Nov 03 19:31:13.921693 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.920743    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1753fd31-8933-4567-8146-07f3fd2d10a5-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-7hcf8\" (UID: \"1753fd31-8933-4567-8146-07f3fd2d10a5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7hcf8"
Nov 03 19:31:13.921693 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.920769    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2287d\" (UniqueName: \"kubernetes.io/projected/1611b730-6163-4776-bd2d-3d10c2cc9a88-kube-api-access-2287d\") pod \"network-check-source-6f78d59f99-cbhrt\" (UID: \"1611b730-6163-4776-bd2d-3d10c2cc9a88\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cbhrt"
Nov 03 19:31:13.921693 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.920796    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6f215b42-0f09-46a0-bc98-392c0fa909c9-serving-cert\") pod \"console-operator-7cd56b8dbc-gzf8k\" (UID: \"6f215b42-0f09-46a0-bc98-392c0fa909c9\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gzf8k"
Nov 03 19:31:13.921693 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.920821    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjtq4\" (UniqueName: \"kubernetes.io/projected/6f215b42-0f09-46a0-bc98-392c0fa909c9-kube-api-access-qjtq4\") pod \"console-operator-7cd56b8dbc-gzf8k\" (UID: \"6f215b42-0f09-46a0-bc98-392c0fa909c9\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gzf8k"
Nov 03 19:31:13.921693 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.920848    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftr5x\" (UniqueName: \"kubernetes.io/projected/a023911d-3669-43e1-ac77-8a986f3715d6-kube-api-access-ftr5x\") pod \"router-default-569d4c45f-7pszr\" (UID: \"a023911d-3669-43e1-ac77-8a986f3715d6\") " pod="openshift-ingress/router-default-569d4c45f-7pszr"
Nov 03 19:31:13.921693 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.921274    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e10dc749-bc71-456e-a315-6581add01a7b-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hxlrb\" (UID: \"e10dc749-bc71-456e-a315-6581add01a7b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hxlrb"
Nov 03 19:31:13.921693 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.921560    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3880f3d3-aad1-43c0-b704-01374aad4f78-snapshots\") pod \"insights-operator-7c4986d449-8kh6f\" (UID: \"3880f3d3-aad1-43c0-b704-01374aad4f78\") " pod="openshift-insights/insights-operator-7c4986d449-8kh6f"
Nov 03 19:31:13.921693 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.921642    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38a6add1-b051-450b-bb96-8ace8a0ccff3-tmp-dir\") pod \"dns-default-wj8mp\" (UID: \"38a6add1-b051-450b-bb96-8ace8a0ccff3\") " pod="openshift-dns/dns-default-wj8mp"
Nov 03 19:31:13.922215 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.922158    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/96c016f7-9098-4a39-9e72-e54da374d8fc-installation-pull-secrets\") pod \"image-registry-d74787d9c-nn7sk\" (UID: \"96c016f7-9098-4a39-9e72-e54da374d8fc\") " pod="openshift-image-registry/image-registry-d74787d9c-nn7sk"
Nov 03 19:31:13.922371 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.922343    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3880f3d3-aad1-43c0-b704-01374aad4f78-serving-cert\") pod \"insights-operator-7c4986d449-8kh6f\" (UID: \"3880f3d3-aad1-43c0-b704-01374aad4f78\") " pod="openshift-insights/insights-operator-7c4986d449-8kh6f"
Nov 03 19:31:13.922610 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.922588    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/a023911d-3669-43e1-ac77-8a986f3715d6-default-certificate\") pod \"router-default-569d4c45f-7pszr\" (UID: \"a023911d-3669-43e1-ac77-8a986f3715d6\") " pod="openshift-ingress/router-default-569d4c45f-7pszr"
Nov 03 19:31:13.922744 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.922723    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9fb8c33c-fe15-4daa-a01b-4cb2fc7eae7b-serving-cert\") pod \"service-ca-operator-544d749c8c-mh2tn\" (UID: \"9fb8c33c-fe15-4daa-a01b-4cb2fc7eae7b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mh2tn"
Nov 03 19:31:13.923648 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.923624    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-image-registry-private-configuration\") pod \"image-registry-c5c44f4d-dv455\" (UID: \"d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca\") " pod="openshift-image-registry/image-registry-c5c44f4d-dv455"
Nov 03 19:31:13.924045 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:13.924023    2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 03 19:31:13.924045 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.924040    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/96c016f7-9098-4a39-9e72-e54da374d8fc-image-registry-private-configuration\") pod \"image-registry-d74787d9c-nn7sk\" (UID: \"96c016f7-9098-4a39-9e72-e54da374d8fc\") " pod="openshift-image-registry/image-registry-d74787d9c-nn7sk"
Nov 03 19:31:13.924177 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.924101    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/a023911d-3669-43e1-ac77-8a986f3715d6-stats-auth\") pod \"router-default-569d4c45f-7pszr\" (UID: \"a023911d-3669-43e1-ac77-8a986f3715d6\") " pod="openshift-ingress/router-default-569d4c45f-7pszr"
Nov 03 19:31:13.924293 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:13.924181    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1753fd31-8933-4567-8146-07f3fd2d10a5-samples-operator-tls podName:1753fd31-8933-4567-8146-07f3fd2d10a5 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:14.424165078 +0000 UTC m=+31.117609396 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1753fd31-8933-4567-8146-07f3fd2d10a5-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-7hcf8" (UID: "1753fd31-8933-4567-8146-07f3fd2d10a5") : secret "samples-operator-tls" not found
Nov 03 19:31:13.924681 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.924640    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/eb488ece-2ac3-499f-84c1-73534fcbb20b-nginx-conf\") pod \"networking-console-plugin-86c478bb88-zpgl6\" (UID: \"eb488ece-2ac3-499f-84c1-73534fcbb20b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zpgl6"
Nov 03 19:31:13.925621 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.925599    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-installation-pull-secrets\") pod \"image-registry-c5c44f4d-dv455\" (UID: \"d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca\") " pod="openshift-image-registry/image-registry-c5c44f4d-dv455"
Nov 03 19:31:13.926761 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.926284    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n7xj\" (UniqueName: \"kubernetes.io/projected/1753fd31-8933-4567-8146-07f3fd2d10a5-kube-api-access-6n7xj\") pod \"cluster-samples-operator-6ddb6c8b94-7hcf8\" (UID: \"1753fd31-8933-4567-8146-07f3fd2d10a5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7hcf8"
Nov 03 19:31:13.926881 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.926270    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/543b8778-baa4-4e09-b697-68e550e42942-cert\") pod \"ingress-canary-j7m2z\" (UID: \"543b8778-baa4-4e09-b697-68e550e42942\") " pod="openshift-ingress-canary/ingress-canary-j7m2z"
Nov 03 19:31:13.926983 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.926959    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e10dc749-bc71-456e-a315-6581add01a7b-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hxlrb\" (UID: \"e10dc749-bc71-456e-a315-6581add01a7b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hxlrb"
Nov 03 19:31:13.927097 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.926828    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/96c016f7-9098-4a39-9e72-e54da374d8fc-bound-sa-token\") pod \"image-registry-d74787d9c-nn7sk\" (UID: \"96c016f7-9098-4a39-9e72-e54da374d8fc\") " pod="openshift-image-registry/image-registry-d74787d9c-nn7sk"
Nov 03 19:31:13.927280 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.927258    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6f215b42-0f09-46a0-bc98-392c0fa909c9-serving-cert\") pod \"console-operator-7cd56b8dbc-gzf8k\" (UID: \"6f215b42-0f09-46a0-bc98-392c0fa909c9\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gzf8k"
Nov 03 19:31:13.930026 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.929975    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngpxm\" (UniqueName: \"kubernetes.io/projected/38a6add1-b051-450b-bb96-8ace8a0ccff3-kube-api-access-ngpxm\") pod \"dns-default-wj8mp\" (UID: \"38a6add1-b051-450b-bb96-8ace8a0ccff3\") " pod="openshift-dns/dns-default-wj8mp"
Nov 03 19:31:13.930129 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.930047    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gm62\" (UniqueName: \"kubernetes.io/projected/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-kube-api-access-8gm62\") pod \"image-registry-c5c44f4d-dv455\" (UID: \"d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca\") " pod="openshift-image-registry/image-registry-c5c44f4d-dv455"
Nov 03 19:31:13.930408 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.930343    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtqhk\" (UniqueName: \"kubernetes.io/projected/8fa315db-e48d-4ac5-960a-23750974d431-kube-api-access-jtqhk\") pod \"kas-connection-checker-685895c7c9-42xn2\" (UID: \"8fa315db-e48d-4ac5-960a-23750974d431\") " pod="kube-system/kas-connection-checker-685895c7c9-42xn2"
Nov 03 19:31:13.930408 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.930361    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfg8j\" (UniqueName: \"kubernetes.io/projected/e10dc749-bc71-456e-a315-6581add01a7b-kube-api-access-nfg8j\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hxlrb\" (UID: \"e10dc749-bc71-456e-a315-6581add01a7b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hxlrb"
Nov 03 19:31:13.930408 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.930347    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktkfm\" (UniqueName: \"kubernetes.io/projected/df9f9c4a-d054-42cf-b17c-0d23acd0459b-kube-api-access-ktkfm\") pod \"kas-connection-checker-685895c7c9-xgbkd\" (UID: \"df9f9c4a-d054-42cf-b17c-0d23acd0459b\") " pod="kube-system/kas-connection-checker-685895c7c9-xgbkd"
Nov 03 19:31:13.930408 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.930380    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj2lf\" (UniqueName: \"kubernetes.io/projected/543b8778-baa4-4e09-b697-68e550e42942-kube-api-access-gj2lf\") pod \"ingress-canary-j7m2z\" (UID: \"543b8778-baa4-4e09-b697-68e550e42942\") " pod="openshift-ingress-canary/ingress-canary-j7m2z"
Nov 03 19:31:13.930738 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.930556    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mh9g\" (UniqueName: \"kubernetes.io/projected/3880f3d3-aad1-43c0-b704-01374aad4f78-kube-api-access-8mh9g\") pod \"insights-operator-7c4986d449-8kh6f\" (UID: \"3880f3d3-aad1-43c0-b704-01374aad4f78\") " pod="openshift-insights/insights-operator-7c4986d449-8kh6f"
Nov 03 19:31:13.930738 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.930565    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-bound-sa-token\") pod \"image-registry-c5c44f4d-dv455\" (UID: \"d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca\") " pod="openshift-image-registry/image-registry-c5c44f4d-dv455"
Nov 03 19:31:13.931016 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.930990    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w42k\" (UniqueName: \"kubernetes.io/projected/96c016f7-9098-4a39-9e72-e54da374d8fc-kube-api-access-4w42k\") pod \"image-registry-d74787d9c-nn7sk\" (UID: \"96c016f7-9098-4a39-9e72-e54da374d8fc\") " pod="openshift-image-registry/image-registry-d74787d9c-nn7sk"
Nov 03 19:31:13.931372 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.931351    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz8xk\" (UniqueName: \"kubernetes.io/projected/e2b752d9-88c4-4dee-8b0c-7e9ac252adb3-kube-api-access-xz8xk\") pod \"cluster-monitoring-operator-884576c57-kfdsx\" (UID: \"e2b752d9-88c4-4dee-8b0c-7e9ac252adb3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kfdsx"
Nov 03 19:31:13.931413 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.931392    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q68w5\" (UniqueName: \"kubernetes.io/projected/9fb8c33c-fe15-4daa-a01b-4cb2fc7eae7b-kube-api-access-q68w5\") pod \"service-ca-operator-544d749c8c-mh2tn\" (UID: \"9fb8c33c-fe15-4daa-a01b-4cb2fc7eae7b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mh2tn"
Nov 03 19:31:13.932590 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.932567    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p288b\" (UniqueName: \"kubernetes.io/projected/f97b063e-8403-46bd-9716-72411589a494-kube-api-access-p288b\") pod \"kas-connection-checker-685895c7c9-wtgvw\" (UID: \"f97b063e-8403-46bd-9716-72411589a494\") " pod="kube-system/kas-connection-checker-685895c7c9-wtgvw"
Nov 03 19:31:13.933612 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.933588    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftr5x\" (UniqueName: \"kubernetes.io/projected/a023911d-3669-43e1-ac77-8a986f3715d6-kube-api-access-ftr5x\") pod \"router-default-569d4c45f-7pszr\" (UID: \"a023911d-3669-43e1-ac77-8a986f3715d6\") " pod="openshift-ingress/router-default-569d4c45f-7pszr"
Nov 03 19:31:13.936304 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.936277    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjtq4\" (UniqueName: \"kubernetes.io/projected/6f215b42-0f09-46a0-bc98-392c0fa909c9-kube-api-access-qjtq4\") pod \"console-operator-7cd56b8dbc-gzf8k\" (UID: \"6f215b42-0f09-46a0-bc98-392c0fa909c9\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gzf8k"
Nov 03 19:31:13.936374 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.936301    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2287d\" (UniqueName: \"kubernetes.io/projected/1611b730-6163-4776-bd2d-3d10c2cc9a88-kube-api-access-2287d\") pod \"network-check-source-6f78d59f99-cbhrt\" (UID: \"1611b730-6163-4776-bd2d-3d10c2cc9a88\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cbhrt"
Nov 03 19:31:13.936455 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.936439    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdw57\" (UniqueName: \"kubernetes.io/projected/f89ef12d-9c60-4538-af9c-f97453bbc855-kube-api-access-xdw57\") pod \"volume-data-source-validator-58c9f5f46d-tx285\" (UID: \"f89ef12d-9c60-4538-af9c-f97453bbc855\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-tx285"
Nov 03 19:31:13.995024 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:13.994982    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-42xn2"
Nov 03 19:31:14.009485 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:14.009458    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-wtgvw"
Nov 03 19:31:14.019467 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:14.019430    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-xgbkd"
Nov 03 19:31:14.026715 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:14.026640    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hxlrb"
Nov 03 19:31:14.041138 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:14.041088    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-tx285"
Nov 03 19:31:14.052063 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:14.052031    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-8kh6f"
Nov 03 19:31:14.078488 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:14.078250    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mh2tn"
Nov 03 19:31:14.091296 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:14.091056    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cbhrt"
Nov 03 19:31:14.098470 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:14.098354    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-gzf8k"
Nov 03 19:31:14.113171 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:14.112755    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-j7m2z"
Nov 03 19:31:14.298728 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:14.298164    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-42xn2"]
Nov 03 19:31:14.300295 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:31:14.300267    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fa315db_e48d_4ac5_960a_23750974d431.slice/crio-c058fb511ce4a4026f63bb8f9a0cbc4eccf8ee9fea0ed0ed8993337b4a9e8166 WatchSource:0}: Error finding container c058fb511ce4a4026f63bb8f9a0cbc4eccf8ee9fea0ed0ed8993337b4a9e8166: Status 404 returned error can't find the container with id c058fb511ce4a4026f63bb8f9a0cbc4eccf8ee9fea0ed0ed8993337b4a9e8166
Nov 03 19:31:14.308055 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:14.308029    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hxlrb"]
Nov 03 19:31:14.310859 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:31:14.310833    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode10dc749_bc71_456e_a315_6581add01a7b.slice/crio-417924c6ce13224daa5404d770644a58f2de02ee1c63dc259560510c12e4928c WatchSource:0}: Error finding container 417924c6ce13224daa5404d770644a58f2de02ee1c63dc259560510c12e4928c: Status 404 returned error can't find the container with id 417924c6ce13224daa5404d770644a58f2de02ee1c63dc259560510c12e4928c
Nov 03 19:31:14.325927 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:14.325849    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-wtgvw"]
Nov 03 19:31:14.331528 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:31:14.331482    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf97b063e_8403_46bd_9716_72411589a494.slice/crio-5a72890122c2ad33df5c500a2d864911fa62f753e832a2223fec4e3a9c675e44 WatchSource:0}: Error finding container 5a72890122c2ad33df5c500a2d864911fa62f753e832a2223fec4e3a9c675e44: Status 404 returned error can't find the container with id 5a72890122c2ad33df5c500a2d864911fa62f753e832a2223fec4e3a9c675e44
Nov 03 19:31:14.341466 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:14.341440    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-xgbkd"]
Nov 03 19:31:14.348426 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:31:14.348392    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf9f9c4a_d054_42cf_b17c_0d23acd0459b.slice/crio-9fb06094152142396f464199b328c14b14771278ba0d9d040daab45a8693dcd2 WatchSource:0}: Error finding container 9fb06094152142396f464199b328c14b14771278ba0d9d040daab45a8693dcd2: Status 404 returned error can't find the container with id 9fb06094152142396f464199b328c14b14771278ba0d9d040daab45a8693dcd2
Nov 03 19:31:14.360721 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:14.360687    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-tx285"]
Nov 03 19:31:14.364293 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:31:14.364253    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf89ef12d_9c60_4538_af9c_f97453bbc855.slice/crio-02550913e8d27c7c8560a212e5bd9d530b50751200373c6fac7af5b19bc5eba6 WatchSource:0}: Error finding container 02550913e8d27c7c8560a212e5bd9d530b50751200373c6fac7af5b19bc5eba6: Status 404 returned error can't find the container with id 02550913e8d27c7c8560a212e5bd9d530b50751200373c6fac7af5b19bc5eba6
Nov 03 19:31:14.413927 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:14.413834    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-8kh6f"]
Nov 03 19:31:14.416435 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:14.416356    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-cbhrt"]
Nov 03 19:31:14.421601 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:31:14.421556    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3880f3d3_aad1_43c0_b704_01374aad4f78.slice/crio-e49beb36db5058ff6aafac0435a308b909777900afdd978ebec852e8d0fe35e2 WatchSource:0}: Error finding container e49beb36db5058ff6aafac0435a308b909777900afdd978ebec852e8d0fe35e2: Status 404 returned error can't find the container with id e49beb36db5058ff6aafac0435a308b909777900afdd978ebec852e8d0fe35e2
Nov 03 19:31:14.422380 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:31:14.422339    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1611b730_6163_4776_bd2d_3d10c2cc9a88.slice/crio-03c21a45f12598b3509569e9e31fe33175fe0b2949aa92bf67867698de18dbce WatchSource:0}: Error finding container 03c21a45f12598b3509569e9e31fe33175fe0b2949aa92bf67867698de18dbce: Status 404 returned error can't find the container with id 03c21a45f12598b3509569e9e31fe33175fe0b2949aa92bf67867698de18dbce
Nov 03 19:31:14.441841 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:14.441054    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a023911d-3669-43e1-ac77-8a986f3715d6-service-ca-bundle\") pod \"router-default-569d4c45f-7pszr\" (UID: \"a023911d-3669-43e1-ac77-8a986f3715d6\") " pod="openshift-ingress/router-default-569d4c45f-7pszr"
Nov 03 19:31:14.441841 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:14.441202    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/96c016f7-9098-4a39-9e72-e54da374d8fc-registry-tls\") pod \"image-registry-d74787d9c-nn7sk\" (UID: \"96c016f7-9098-4a39-9e72-e54da374d8fc\") " pod="openshift-image-registry/image-registry-d74787d9c-nn7sk"
Nov 03 19:31:14.441841 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:14.441267    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-registry-tls\") pod \"image-registry-c5c44f4d-dv455\" (UID: \"d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca\") " pod="openshift-image-registry/image-registry-c5c44f4d-dv455"
Nov 03 19:31:14.441841 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:14.441317    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a023911d-3669-43e1-ac77-8a986f3715d6-service-ca-bundle podName:a023911d-3669-43e1-ac77-8a986f3715d6 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:15.441294423 +0000 UTC m=+32.134738752 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a023911d-3669-43e1-ac77-8a986f3715d6-service-ca-bundle") pod "router-default-569d4c45f-7pszr" (UID: "a023911d-3669-43e1-ac77-8a986f3715d6") : configmap references non-existent config key: service-ca.crt
Nov 03 19:31:14.441841 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:14.441347    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/38a6add1-b051-450b-bb96-8ace8a0ccff3-metrics-tls\") pod \"dns-default-wj8mp\" (UID: \"38a6add1-b051-450b-bb96-8ace8a0ccff3\") " pod="openshift-dns/dns-default-wj8mp"
Nov 03 19:31:14.441841 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:14.441377    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 19:31:14.441841 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:14.441392    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c5c44f4d-dv455: secret "image-registry-tls" not found
Nov 03 19:31:14.441841 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:14.441407    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1753fd31-8933-4567-8146-07f3fd2d10a5-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-7hcf8\" (UID: \"1753fd31-8933-4567-8146-07f3fd2d10a5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7hcf8"
Nov 03 19:31:14.441841 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:14.441438    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-registry-tls podName:d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca nodeName:}" failed. No retries permitted until 2026-11-03 19:31:15.441422892 +0000 UTC m=+32.134867211 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-registry-tls") pod "image-registry-c5c44f4d-dv455" (UID: "d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca") : secret "image-registry-tls" not found
Nov 03 19:31:14.441841 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:14.441475    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e2b752d9-88c4-4dee-8b0c-7e9ac252adb3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kfdsx\" (UID: \"e2b752d9-88c4-4dee-8b0c-7e9ac252adb3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kfdsx"
Nov 03 19:31:14.441841 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:14.441489    2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 03 19:31:14.441841 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:14.441513    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/eb488ece-2ac3-499f-84c1-73534fcbb20b-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zpgl6\" (UID: \"eb488ece-2ac3-499f-84c1-73534fcbb20b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zpgl6"
Nov 03 19:31:14.441841 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:14.441523    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1753fd31-8933-4567-8146-07f3fd2d10a5-samples-operator-tls podName:1753fd31-8933-4567-8146-07f3fd2d10a5 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:15.441512351 +0000 UTC m=+32.134956699 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1753fd31-8933-4567-8146-07f3fd2d10a5-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-7hcf8" (UID: "1753fd31-8933-4567-8146-07f3fd2d10a5") : secret "samples-operator-tls" not found
Nov 03 19:31:14.441841 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:14.441563    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a023911d-3669-43e1-ac77-8a986f3715d6-metrics-certs\") pod \"router-default-569d4c45f-7pszr\" (UID: \"a023911d-3669-43e1-ac77-8a986f3715d6\") " pod="openshift-ingress/router-default-569d4c45f-7pszr"
Nov 03 19:31:14.441841 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:14.441578    2575 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 19:31:14.442410 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:14.441613    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb488ece-2ac3-499f-84c1-73534fcbb20b-networking-console-plugin-cert podName:eb488ece-2ac3-499f-84c1-73534fcbb20b nodeName:}" failed. No retries permitted until 2026-11-03 19:31:15.441602017 +0000 UTC m=+32.135046338 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/eb488ece-2ac3-499f-84c1-73534fcbb20b-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-zpgl6" (UID: "eb488ece-2ac3-499f-84c1-73534fcbb20b") : secret "networking-console-plugin-cert" not found
Nov 03 19:31:14.442410 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:14.441646    2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 03 19:31:14.442410 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:14.441732    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 19:31:14.442410 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:14.441745    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d74787d9c-nn7sk: secret "image-registry-tls" not found
Nov 03 19:31:14.442410 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:14.441810    2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 19:31:14.442410 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:14.441850    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 19:31:14.442410 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:14.441908    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a023911d-3669-43e1-ac77-8a986f3715d6-metrics-certs podName:a023911d-3669-43e1-ac77-8a986f3715d6 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:15.441895108 +0000 UTC m=+32.135339455 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a023911d-3669-43e1-ac77-8a986f3715d6-metrics-certs") pod "router-default-569d4c45f-7pszr" (UID: "a023911d-3669-43e1-ac77-8a986f3715d6") : secret "router-metrics-certs-default" not found
Nov 03 19:31:14.442410 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:14.441925    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/96c016f7-9098-4a39-9e72-e54da374d8fc-registry-tls podName:96c016f7-9098-4a39-9e72-e54da374d8fc nodeName:}" failed. No retries permitted until 2026-11-03 19:31:15.441916472 +0000 UTC m=+32.135360843 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/96c016f7-9098-4a39-9e72-e54da374d8fc-registry-tls") pod "image-registry-d74787d9c-nn7sk" (UID: "96c016f7-9098-4a39-9e72-e54da374d8fc") : secret "image-registry-tls" not found
Nov 03 19:31:14.442410 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:14.441941    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2b752d9-88c4-4dee-8b0c-7e9ac252adb3-cluster-monitoring-operator-tls podName:e2b752d9-88c4-4dee-8b0c-7e9ac252adb3 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:15.441932465 +0000 UTC m=+32.135376784 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e2b752d9-88c4-4dee-8b0c-7e9ac252adb3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-kfdsx" (UID: "e2b752d9-88c4-4dee-8b0c-7e9ac252adb3") : secret "cluster-monitoring-operator-tls" not found
Nov 03 19:31:14.442410 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:14.441957    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38a6add1-b051-450b-bb96-8ace8a0ccff3-metrics-tls podName:38a6add1-b051-450b-bb96-8ace8a0ccff3 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:15.441948495 +0000 UTC m=+32.135392813 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/38a6add1-b051-450b-bb96-8ace8a0ccff3-metrics-tls") pod "dns-default-wj8mp" (UID: "38a6add1-b051-450b-bb96-8ace8a0ccff3") : secret "dns-default-metrics-tls" not found
Nov 03 19:31:14.465216 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:14.465128    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-j7m2z"]
Nov 03 19:31:14.468787 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:31:14.468759    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod543b8778_baa4_4e09_b697_68e550e42942.slice/crio-cee9ee0075ad7fe8f9687cb833f49ea3f9a48bdeb6e0212e761a8acf0b06d211 WatchSource:0}: Error finding container cee9ee0075ad7fe8f9687cb833f49ea3f9a48bdeb6e0212e761a8acf0b06d211: Status 404 returned error can't find the container with id cee9ee0075ad7fe8f9687cb833f49ea3f9a48bdeb6e0212e761a8acf0b06d211
Nov 03 19:31:14.478770 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:14.478745    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-gzf8k"]
Nov 03 19:31:14.482454 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:14.482376    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-mh2tn"]
Nov 03 19:31:14.482549 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:31:14.482523    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f215b42_0f09_46a0_bc98_392c0fa909c9.slice/crio-760afcba9b261afe12f20cdbd1135d1967906cf78e08b9a5f83d27736a8afd7f WatchSource:0}: Error finding container 760afcba9b261afe12f20cdbd1135d1967906cf78e08b9a5f83d27736a8afd7f: Status 404 returned error can't find the container with id 760afcba9b261afe12f20cdbd1135d1967906cf78e08b9a5f83d27736a8afd7f
Nov 03 19:31:14.485849 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:31:14.485804    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fb8c33c_fe15_4daa_a01b_4cb2fc7eae7b.slice/crio-4c245f17b0c21d5f74bf92645ee2ed5737f7f55f6403eb89bac16ac2f44dd547 WatchSource:0}: Error finding container 4c245f17b0c21d5f74bf92645ee2ed5737f7f55f6403eb89bac16ac2f44dd547: Status 404 returned error can't find the container with id 4c245f17b0c21d5f74bf92645ee2ed5737f7f55f6403eb89bac16ac2f44dd547
Nov 03 19:31:14.749038 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:14.748998    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrxdt"
Nov 03 19:31:14.749397 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:14.749058    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njg27"
Nov 03 19:31:14.749575 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:14.749069    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rdr5d"
Nov 03 19:31:14.754384 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:14.754350    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 19:31:14.754384 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:14.754362    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-77jlf\""
Nov 03 19:31:14.754584 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:14.754366    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-khl78\""
Nov 03 19:31:14.754584 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:14.754449    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 19:31:14.754584 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:14.754495    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tdq2h\""
Nov 03 19:31:15.013859 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:15.013769    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cbhrt" event={"ID":"1611b730-6163-4776-bd2d-3d10c2cc9a88","Type":"ContainerStarted","Data":"03c21a45f12598b3509569e9e31fe33175fe0b2949aa92bf67867698de18dbce"}
Nov 03 19:31:15.016477 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:15.016387    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-xgbkd" event={"ID":"df9f9c4a-d054-42cf-b17c-0d23acd0459b","Type":"ContainerStarted","Data":"7cecbf58aac496a6eb877e75c848b3dd3578d1f56e55fdc212e85891d6cb09c9"}
Nov 03 19:31:15.016477 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:15.016446    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-xgbkd" event={"ID":"df9f9c4a-d054-42cf-b17c-0d23acd0459b","Type":"ContainerStarted","Data":"9fb06094152142396f464199b328c14b14771278ba0d9d040daab45a8693dcd2"}
Nov 03 19:31:15.019576 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:15.019255    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mh2tn" event={"ID":"9fb8c33c-fe15-4daa-a01b-4cb2fc7eae7b","Type":"ContainerStarted","Data":"4c245f17b0c21d5f74bf92645ee2ed5737f7f55f6403eb89bac16ac2f44dd547"}
Nov 03 19:31:15.025307 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:15.024655    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-42xn2" event={"ID":"8fa315db-e48d-4ac5-960a-23750974d431","Type":"ContainerStarted","Data":"a97c5cc92b130242d06fe0da8b87c4e5043b3c32bb0b8f70fecc2e92a3b93d37"}
Nov 03 19:31:15.025307 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:15.024732    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-42xn2" event={"ID":"8fa315db-e48d-4ac5-960a-23750974d431","Type":"ContainerStarted","Data":"c058fb511ce4a4026f63bb8f9a0cbc4eccf8ee9fea0ed0ed8993337b4a9e8166"}
Nov 03 19:31:15.030844 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:15.030648    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-gzf8k" event={"ID":"6f215b42-0f09-46a0-bc98-392c0fa909c9","Type":"ContainerStarted","Data":"760afcba9b261afe12f20cdbd1135d1967906cf78e08b9a5f83d27736a8afd7f"}
Nov 03 19:31:15.032813 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:15.032563    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-8kh6f" event={"ID":"3880f3d3-aad1-43c0-b704-01374aad4f78","Type":"ContainerStarted","Data":"e49beb36db5058ff6aafac0435a308b909777900afdd978ebec852e8d0fe35e2"}
Nov 03 19:31:15.034555 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:15.034465    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-tx285" event={"ID":"f89ef12d-9c60-4538-af9c-f97453bbc855","Type":"ContainerStarted","Data":"02550913e8d27c7c8560a212e5bd9d530b50751200373c6fac7af5b19bc5eba6"}
Nov 03 19:31:15.038195 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:15.037625    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-xgbkd" podStartSLOduration=429.037608023 podStartE2EDuration="7m9.037608023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:24:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:31:15.036424284 +0000 UTC m=+31.729868626" watchObservedRunningTime="2026-11-03 19:31:15.037608023 +0000 UTC m=+31.731052361"
Nov 03 19:31:15.039063 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:15.038966    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hxlrb" event={"ID":"e10dc749-bc71-456e-a315-6581add01a7b","Type":"ContainerStarted","Data":"417924c6ce13224daa5404d770644a58f2de02ee1c63dc259560510c12e4928c"}
Nov 03 19:31:15.041546 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:15.040930    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-wtgvw" event={"ID":"f97b063e-8403-46bd-9716-72411589a494","Type":"ContainerStarted","Data":"9c9e66af6e6baedbb9028a409366f1fce1b6d21b1dbc3c2fcdfe5c414bd8a267"}
Nov 03 19:31:15.041546 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:15.040976    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-wtgvw" event={"ID":"f97b063e-8403-46bd-9716-72411589a494","Type":"ContainerStarted","Data":"5a72890122c2ad33df5c500a2d864911fa62f753e832a2223fec4e3a9c675e44"}
Nov 03 19:31:15.044052 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:15.043980    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-j7m2z" event={"ID":"543b8778-baa4-4e09-b697-68e550e42942","Type":"ContainerStarted","Data":"cee9ee0075ad7fe8f9687cb833f49ea3f9a48bdeb6e0212e761a8acf0b06d211"}
Nov 03 19:31:15.056520 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:15.055874    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-42xn2" podStartSLOduration=429.055852259 podStartE2EDuration="7m9.055852259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:24:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:31:15.054385315 +0000 UTC m=+31.747829652" watchObservedRunningTime="2026-11-03 19:31:15.055852259 +0000 UTC m=+31.749296599"
Nov 03 19:31:15.454490 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:15.452896    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a023911d-3669-43e1-ac77-8a986f3715d6-metrics-certs\") pod \"router-default-569d4c45f-7pszr\" (UID: \"a023911d-3669-43e1-ac77-8a986f3715d6\") " pod="openshift-ingress/router-default-569d4c45f-7pszr"
Nov 03 19:31:15.454490 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:15.452987    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a023911d-3669-43e1-ac77-8a986f3715d6-service-ca-bundle\") pod \"router-default-569d4c45f-7pszr\" (UID: \"a023911d-3669-43e1-ac77-8a986f3715d6\") " pod="openshift-ingress/router-default-569d4c45f-7pszr"
Nov 03 19:31:15.454490 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:15.453015    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/96c016f7-9098-4a39-9e72-e54da374d8fc-registry-tls\") pod \"image-registry-d74787d9c-nn7sk\" (UID: \"96c016f7-9098-4a39-9e72-e54da374d8fc\") " pod="openshift-image-registry/image-registry-d74787d9c-nn7sk"
Nov 03 19:31:15.454490 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:15.453070    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-registry-tls\") pod \"image-registry-c5c44f4d-dv455\" (UID: \"d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca\") " pod="openshift-image-registry/image-registry-c5c44f4d-dv455"
Nov 03 19:31:15.454490 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:15.453091    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/38a6add1-b051-450b-bb96-8ace8a0ccff3-metrics-tls\") pod \"dns-default-wj8mp\" (UID: \"38a6add1-b051-450b-bb96-8ace8a0ccff3\") " pod="openshift-dns/dns-default-wj8mp"
Nov 03 19:31:15.454490 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:15.453130    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1753fd31-8933-4567-8146-07f3fd2d10a5-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-7hcf8\" (UID: \"1753fd31-8933-4567-8146-07f3fd2d10a5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7hcf8"
Nov 03 19:31:15.454490 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:15.453172    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e2b752d9-88c4-4dee-8b0c-7e9ac252adb3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kfdsx\" (UID: \"e2b752d9-88c4-4dee-8b0c-7e9ac252adb3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kfdsx"
Nov 03 19:31:15.454490 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:15.453206    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/eb488ece-2ac3-499f-84c1-73534fcbb20b-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zpgl6\" (UID: \"eb488ece-2ac3-499f-84c1-73534fcbb20b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zpgl6"
Nov 03 19:31:15.454490 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:15.453344    2575 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 19:31:15.454490 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:15.453411    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb488ece-2ac3-499f-84c1-73534fcbb20b-networking-console-plugin-cert podName:eb488ece-2ac3-499f-84c1-73534fcbb20b nodeName:}" failed. No retries permitted until 2026-11-03 19:31:17.453391182 +0000 UTC m=+34.146835503 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/eb488ece-2ac3-499f-84c1-73534fcbb20b-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-zpgl6" (UID: "eb488ece-2ac3-499f-84c1-73534fcbb20b") : secret "networking-console-plugin-cert" not found
Nov 03 19:31:15.454490 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:15.453842    2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 03 19:31:15.454490 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:15.453892    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a023911d-3669-43e1-ac77-8a986f3715d6-metrics-certs podName:a023911d-3669-43e1-ac77-8a986f3715d6 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:17.453878155 +0000 UTC m=+34.147322484 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a023911d-3669-43e1-ac77-8a986f3715d6-metrics-certs") pod "router-default-569d4c45f-7pszr" (UID: "a023911d-3669-43e1-ac77-8a986f3715d6") : secret "router-metrics-certs-default" not found
Nov 03 19:31:15.454490 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:15.453960    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a023911d-3669-43e1-ac77-8a986f3715d6-service-ca-bundle podName:a023911d-3669-43e1-ac77-8a986f3715d6 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:17.453949497 +0000 UTC m=+34.147393827 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a023911d-3669-43e1-ac77-8a986f3715d6-service-ca-bundle") pod "router-default-569d4c45f-7pszr" (UID: "a023911d-3669-43e1-ac77-8a986f3715d6") : configmap references non-existent config key: service-ca.crt
Nov 03 19:31:15.454490 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:15.454020    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 19:31:15.454490 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:15.454030    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d74787d9c-nn7sk: secret "image-registry-tls" not found
Nov 03 19:31:15.455298 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:15.454063    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/96c016f7-9098-4a39-9e72-e54da374d8fc-registry-tls podName:96c016f7-9098-4a39-9e72-e54da374d8fc nodeName:}" failed. No retries permitted until 2026-11-03 19:31:17.45405224 +0000 UTC m=+34.147496571 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/96c016f7-9098-4a39-9e72-e54da374d8fc-registry-tls") pod "image-registry-d74787d9c-nn7sk" (UID: "96c016f7-9098-4a39-9e72-e54da374d8fc") : secret "image-registry-tls" not found
Nov 03 19:31:15.455298 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:15.454118    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 19:31:15.455298 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:15.454127    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c5c44f4d-dv455: secret "image-registry-tls" not found
Nov 03 19:31:15.455298 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:15.454153    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-registry-tls podName:d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca nodeName:}" failed. No retries permitted until 2026-11-03 19:31:17.45414424 +0000 UTC m=+34.147588570 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-registry-tls") pod "image-registry-c5c44f4d-dv455" (UID: "d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca") : secret "image-registry-tls" not found
Nov 03 19:31:15.455298 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:15.454206    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 19:31:15.455298 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:15.454231    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38a6add1-b051-450b-bb96-8ace8a0ccff3-metrics-tls podName:38a6add1-b051-450b-bb96-8ace8a0ccff3 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:17.454223205 +0000 UTC m=+34.147667539 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/38a6add1-b051-450b-bb96-8ace8a0ccff3-metrics-tls") pod "dns-default-wj8mp" (UID: "38a6add1-b051-450b-bb96-8ace8a0ccff3") : secret "dns-default-metrics-tls" not found
Nov 03 19:31:15.455298 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:15.454280    2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 03 19:31:15.455298 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:15.454310    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1753fd31-8933-4567-8146-07f3fd2d10a5-samples-operator-tls podName:1753fd31-8933-4567-8146-07f3fd2d10a5 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:17.454300478 +0000 UTC m=+34.147744808 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1753fd31-8933-4567-8146-07f3fd2d10a5-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-7hcf8" (UID: "1753fd31-8933-4567-8146-07f3fd2d10a5") : secret "samples-operator-tls" not found
Nov 03 19:31:15.455298 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:15.454371    2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 19:31:15.455298 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:15.454398    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2b752d9-88c4-4dee-8b0c-7e9ac252adb3-cluster-monitoring-operator-tls podName:e2b752d9-88c4-4dee-8b0c-7e9ac252adb3 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:17.454390152 +0000 UTC m=+34.147834469 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e2b752d9-88c4-4dee-8b0c-7e9ac252adb3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-kfdsx" (UID: "e2b752d9-88c4-4dee-8b0c-7e9ac252adb3") : secret "cluster-monitoring-operator-tls" not found
Nov 03 19:31:17.055042 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:17.054998    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-tx285" event={"ID":"f89ef12d-9c60-4538-af9c-f97453bbc855","Type":"ContainerStarted","Data":"75cf790451f02e9283809ffad8fd7da8b850735bcfc8b463adb9cf685078168b"}
Nov 03 19:31:17.073815 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:17.073750    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-wtgvw" podStartSLOduration=431.07373203 podStartE2EDuration="7m11.07373203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:24:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:31:15.07401492 +0000 UTC m=+31.767459258" watchObservedRunningTime="2026-11-03 19:31:17.07373203 +0000 UTC m=+33.767176369"
Nov 03 19:31:17.480808 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:17.480765    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-registry-tls\") pod \"image-registry-c5c44f4d-dv455\" (UID: \"d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca\") " pod="openshift-image-registry/image-registry-c5c44f4d-dv455"
Nov 03 19:31:17.480941 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:17.480826    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/38a6add1-b051-450b-bb96-8ace8a0ccff3-metrics-tls\") pod \"dns-default-wj8mp\" (UID: \"38a6add1-b051-450b-bb96-8ace8a0ccff3\") " pod="openshift-dns/dns-default-wj8mp"
Nov 03 19:31:17.480941 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:17.480879    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1753fd31-8933-4567-8146-07f3fd2d10a5-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-7hcf8\" (UID: \"1753fd31-8933-4567-8146-07f3fd2d10a5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7hcf8"
Nov 03 19:31:17.480941 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:17.480915    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e2b752d9-88c4-4dee-8b0c-7e9ac252adb3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kfdsx\" (UID: \"e2b752d9-88c4-4dee-8b0c-7e9ac252adb3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kfdsx"
Nov 03 19:31:17.481038 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:17.480949    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/eb488ece-2ac3-499f-84c1-73534fcbb20b-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zpgl6\" (UID: \"eb488ece-2ac3-499f-84c1-73534fcbb20b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zpgl6"
Nov 03 19:31:17.481038 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:17.480989    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a023911d-3669-43e1-ac77-8a986f3715d6-metrics-certs\") pod \"router-default-569d4c45f-7pszr\" (UID: \"a023911d-3669-43e1-ac77-8a986f3715d6\") " pod="openshift-ingress/router-default-569d4c45f-7pszr"
Nov 03 19:31:17.481100 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:17.481034    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/790607e1-d57c-4622-9a72-1af902ab1d85-metrics-certs\") pod \"network-metrics-daemon-rdr5d\" (UID: \"790607e1-d57c-4622-9a72-1af902ab1d85\") " pod="openshift-multus/network-metrics-daemon-rdr5d"
Nov 03 19:31:17.481100 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:17.481064    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a023911d-3669-43e1-ac77-8a986f3715d6-service-ca-bundle\") pod \"router-default-569d4c45f-7pszr\" (UID: \"a023911d-3669-43e1-ac77-8a986f3715d6\") " pod="openshift-ingress/router-default-569d4c45f-7pszr"
Nov 03 19:31:17.481165 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:17.481102    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/96c016f7-9098-4a39-9e72-e54da374d8fc-registry-tls\") pod \"image-registry-d74787d9c-nn7sk\" (UID: \"96c016f7-9098-4a39-9e72-e54da374d8fc\") " pod="openshift-image-registry/image-registry-d74787d9c-nn7sk"
Nov 03 19:31:17.482317 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:17.481252    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 19:31:17.482317 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:17.481267    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d74787d9c-nn7sk: secret "image-registry-tls" not found
Nov 03 19:31:17.482317 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:17.481314    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/96c016f7-9098-4a39-9e72-e54da374d8fc-registry-tls podName:96c016f7-9098-4a39-9e72-e54da374d8fc nodeName:}" failed. No retries permitted until 2026-11-03 19:31:21.48130186 +0000 UTC m=+38.174746176 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/96c016f7-9098-4a39-9e72-e54da374d8fc-registry-tls") pod "image-registry-d74787d9c-nn7sk" (UID: "96c016f7-9098-4a39-9e72-e54da374d8fc") : secret "image-registry-tls" not found
Nov 03 19:31:17.482317 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:17.481637    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 19:31:17.482317 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:17.481650    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c5c44f4d-dv455: secret "image-registry-tls" not found
Nov 03 19:31:17.482317 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:17.481704    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-registry-tls podName:d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca nodeName:}" failed. No retries permitted until 2026-11-03 19:31:21.481691129 +0000 UTC m=+38.175135445 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-registry-tls") pod "image-registry-c5c44f4d-dv455" (UID: "d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca") : secret "image-registry-tls" not found
Nov 03 19:31:17.482317 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:17.481766    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 19:31:17.482317 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:17.481802    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38a6add1-b051-450b-bb96-8ace8a0ccff3-metrics-tls podName:38a6add1-b051-450b-bb96-8ace8a0ccff3 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:21.481791368 +0000 UTC m=+38.175235689 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/38a6add1-b051-450b-bb96-8ace8a0ccff3-metrics-tls") pod "dns-default-wj8mp" (UID: "38a6add1-b051-450b-bb96-8ace8a0ccff3") : secret "dns-default-metrics-tls" not found
Nov 03 19:31:17.482317 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:17.481855    2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 03 19:31:17.482317 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:17.481886    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1753fd31-8933-4567-8146-07f3fd2d10a5-samples-operator-tls podName:1753fd31-8933-4567-8146-07f3fd2d10a5 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:21.48187675 +0000 UTC m=+38.175321065 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1753fd31-8933-4567-8146-07f3fd2d10a5-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-7hcf8" (UID: "1753fd31-8933-4567-8146-07f3fd2d10a5") : secret "samples-operator-tls" not found
Nov 03 19:31:17.482317 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:17.481950    2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 19:31:17.482317 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:17.481980    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2b752d9-88c4-4dee-8b0c-7e9ac252adb3-cluster-monitoring-operator-tls podName:e2b752d9-88c4-4dee-8b0c-7e9ac252adb3 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:21.481970511 +0000 UTC m=+38.175414834 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e2b752d9-88c4-4dee-8b0c-7e9ac252adb3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-kfdsx" (UID: "e2b752d9-88c4-4dee-8b0c-7e9ac252adb3") : secret "cluster-monitoring-operator-tls" not found
Nov 03 19:31:17.482317 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:17.482029    2575 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 19:31:17.482317 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:17.482059    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb488ece-2ac3-499f-84c1-73534fcbb20b-networking-console-plugin-cert podName:eb488ece-2ac3-499f-84c1-73534fcbb20b nodeName:}" failed. No retries permitted until 2026-11-03 19:31:21.48204909 +0000 UTC m=+38.175493410 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/eb488ece-2ac3-499f-84c1-73534fcbb20b-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-zpgl6" (UID: "eb488ece-2ac3-499f-84c1-73534fcbb20b") : secret "networking-console-plugin-cert" not found
Nov 03 19:31:17.482317 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:17.482109    2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 03 19:31:17.482317 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:17.482138    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a023911d-3669-43e1-ac77-8a986f3715d6-metrics-certs podName:a023911d-3669-43e1-ac77-8a986f3715d6 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:21.482128354 +0000 UTC m=+38.175572671 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a023911d-3669-43e1-ac77-8a986f3715d6-metrics-certs") pod "router-default-569d4c45f-7pszr" (UID: "a023911d-3669-43e1-ac77-8a986f3715d6") : secret "router-metrics-certs-default" not found
Nov 03 19:31:17.482317 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:17.482188    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 03 19:31:17.483221 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:17.482215    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/790607e1-d57c-4622-9a72-1af902ab1d85-metrics-certs podName:790607e1-d57c-4622-9a72-1af902ab1d85 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:49.482205654 +0000 UTC m=+66.175649973 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/790607e1-d57c-4622-9a72-1af902ab1d85-metrics-certs") pod "network-metrics-daemon-rdr5d" (UID: "790607e1-d57c-4622-9a72-1af902ab1d85") : secret "metrics-daemon-secret" not found
Nov 03 19:31:17.483221 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:17.482274    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a023911d-3669-43e1-ac77-8a986f3715d6-service-ca-bundle podName:a023911d-3669-43e1-ac77-8a986f3715d6 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:21.482264329 +0000 UTC m=+38.175708649 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a023911d-3669-43e1-ac77-8a986f3715d6-service-ca-bundle") pod "router-default-569d4c45f-7pszr" (UID: "a023911d-3669-43e1-ac77-8a986f3715d6") : configmap references non-existent config key: service-ca.crt
Nov 03 19:31:17.582181 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:17.582145    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eee821c4-6fb0-4f84-a58d-23dc2ff77720-original-pull-secret\") pod \"global-pull-secret-syncer-hrxdt\" (UID: \"eee821c4-6fb0-4f84-a58d-23dc2ff77720\") " pod="kube-system/global-pull-secret-syncer-hrxdt"
Nov 03 19:31:17.582368 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:17.582244    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f75xn\" (UniqueName: \"kubernetes.io/projected/b46f51b2-436b-4967-8879-a16b58930591-kube-api-access-f75xn\") pod \"network-check-target-njg27\" (UID: \"b46f51b2-436b-4967-8879-a16b58930591\") " pod="openshift-network-diagnostics/network-check-target-njg27"
Nov 03 19:31:17.586783 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:17.586699    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f75xn\" (UniqueName: \"kubernetes.io/projected/b46f51b2-436b-4967-8879-a16b58930591-kube-api-access-f75xn\") pod \"network-check-target-njg27\" (UID: \"b46f51b2-436b-4967-8879-a16b58930591\") " pod="openshift-network-diagnostics/network-check-target-njg27"
Nov 03 19:31:17.588435 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:17.588402    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eee821c4-6fb0-4f84-a58d-23dc2ff77720-original-pull-secret\") pod \"global-pull-secret-syncer-hrxdt\" (UID: \"eee821c4-6fb0-4f84-a58d-23dc2ff77720\") " pod="kube-system/global-pull-secret-syncer-hrxdt"
Nov 03 19:31:17.761327 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:17.761104    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrxdt"
Nov 03 19:31:17.767212 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:17.767181    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njg27"
Nov 03 19:31:18.061194 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:18.061095    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gzf8k_6f215b42-0f09-46a0-bc98-392c0fa909c9/console-operator/0.log"
Nov 03 19:31:18.061194 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:18.061152    2575 generic.go:356] "Generic (PLEG): container finished" podID="6f215b42-0f09-46a0-bc98-392c0fa909c9" containerID="2fca0c29710b8817193d5e756fc99a0340a050d6eeab81f5670934e5c18dfe7e" exitCode=255
Nov 03 19:31:18.062128 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:18.061988    2575 scope.go:122] "RemoveContainer" containerID="2fca0c29710b8817193d5e756fc99a0340a050d6eeab81f5670934e5c18dfe7e"
Nov 03 19:31:18.062331 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:18.062309    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-gzf8k" event={"ID":"6f215b42-0f09-46a0-bc98-392c0fa909c9","Type":"ContainerDied","Data":"2fca0c29710b8817193d5e756fc99a0340a050d6eeab81f5670934e5c18dfe7e"}
Nov 03 19:31:18.068778 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:18.065705    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-tx285" podStartSLOduration=379.420129431 podStartE2EDuration="6m21.065691693s" totalImagesPullingTime="1.645562262s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:24:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:31:17.074275091 +0000 UTC m=+33.767719429" watchObservedRunningTime="2026-11-03 19:31:18.065691693 +0000 UTC m=+34.759136027"
Nov 03 19:31:18.068778 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:18.066641    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hrxdt"]
Nov 03 19:31:18.076600 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:18.076575    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-njg27"]
Nov 03 19:31:18.974626 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:18.972312    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-5l649"]
Nov 03 19:31:18.993877 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:18.993840    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-5l649"]
Nov 03 19:31:18.994034 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:18.993987    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5l649"
Nov 03 19:31:19.069446 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:19.069416    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gzf8k_6f215b42-0f09-46a0-bc98-392c0fa909c9/console-operator/1.log"
Nov 03 19:31:19.071286 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:19.070047    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gzf8k_6f215b42-0f09-46a0-bc98-392c0fa909c9/console-operator/0.log"
Nov 03 19:31:19.071286 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:19.070091    2575 generic.go:356] "Generic (PLEG): container finished" podID="6f215b42-0f09-46a0-bc98-392c0fa909c9" containerID="3895cd529be0be6af57557f5b4638c0b5aa21cd6a6d699b224ff7bffcb1bd8e3" exitCode=255
Nov 03 19:31:19.071286 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:19.070175    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-gzf8k" event={"ID":"6f215b42-0f09-46a0-bc98-392c0fa909c9","Type":"ContainerDied","Data":"3895cd529be0be6af57557f5b4638c0b5aa21cd6a6d699b224ff7bffcb1bd8e3"}
Nov 03 19:31:19.071286 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:19.070227    2575 scope.go:122] "RemoveContainer" containerID="2fca0c29710b8817193d5e756fc99a0340a050d6eeab81f5670934e5c18dfe7e"
Nov 03 19:31:19.071286 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:19.070583    2575 scope.go:122] "RemoveContainer" containerID="3895cd529be0be6af57557f5b4638c0b5aa21cd6a6d699b224ff7bffcb1bd8e3"
Nov 03 19:31:19.071916 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:19.071858    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-gzf8k_openshift-console-operator(6f215b42-0f09-46a0-bc98-392c0fa909c9)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-gzf8k" podUID="6f215b42-0f09-46a0-bc98-392c0fa909c9"
Nov 03 19:31:19.072752 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:19.072687    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hxlrb" event={"ID":"e10dc749-bc71-456e-a315-6581add01a7b","Type":"ContainerStarted","Data":"1c87bff726a20edd63a0001b79174fd2bfc95b7add8a7f63b480949c9844f331"}
Nov 03 19:31:19.073918 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:19.073897    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hrxdt" event={"ID":"eee821c4-6fb0-4f84-a58d-23dc2ff77720","Type":"ContainerStarted","Data":"614fce5f50618860328a6514e4a8ed906b17132e6d53aadfb45f8e50b6ebb3a7"}
Nov 03 19:31:19.075878 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:19.075849    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-njg27" event={"ID":"b46f51b2-436b-4967-8879-a16b58930591","Type":"ContainerStarted","Data":"6c8b4e67220dd8eff5e0a5c70f9e73180d809bb968c1192c2bdefd5ecc7f6800"}
Nov 03 19:31:19.086335 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:19.086304    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 03 19:31:19.086542 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:19.086452    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 03 19:31:19.086542 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:19.086527    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-2lfv2\""
Nov 03 19:31:19.102840 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:19.102801    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lgzq\" (UniqueName: \"kubernetes.io/projected/220ed31c-1577-4c4b-ab5b-e812fd63d193-kube-api-access-5lgzq\") pod \"migrator-66df5d496b-5l649\" (UID: \"220ed31c-1577-4c4b-ab5b-e812fd63d193\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5l649"
Nov 03 19:31:19.107529 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:19.107462    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hxlrb" podStartSLOduration=380.305900828 podStartE2EDuration="6m24.10744374s" totalImagesPullingTime="3.801542912s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:24:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:31:19.105600318 +0000 UTC m=+35.799044655" watchObservedRunningTime="2026-11-03 19:31:19.10744374 +0000 UTC m=+35.800888080"
Nov 03 19:31:19.203604 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:19.203563    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lgzq\" (UniqueName: \"kubernetes.io/projected/220ed31c-1577-4c4b-ab5b-e812fd63d193-kube-api-access-5lgzq\") pod \"migrator-66df5d496b-5l649\" (UID: \"220ed31c-1577-4c4b-ab5b-e812fd63d193\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5l649"
Nov 03 19:31:19.214150 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:19.214115    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lgzq\" (UniqueName: \"kubernetes.io/projected/220ed31c-1577-4c4b-ab5b-e812fd63d193-kube-api-access-5lgzq\") pod \"migrator-66df5d496b-5l649\" (UID: \"220ed31c-1577-4c4b-ab5b-e812fd63d193\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5l649"
Nov 03 19:31:19.322797 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:19.322758    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5l649"
Nov 03 19:31:19.766053 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:19.765982    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-5l649"]
Nov 03 19:31:19.769598 ip-10-0-134-222 kubenswrapper[2575]: W1103 19:31:19.769558    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod220ed31c_1577_4c4b_ab5b_e812fd63d193.slice/crio-e9319b6d883d1db6f7e71643f2a97b1f810903966a53f2ee00efe32b8cc288b2 WatchSource:0}: Error finding container e9319b6d883d1db6f7e71643f2a97b1f810903966a53f2ee00efe32b8cc288b2: Status 404 returned error can't find the container with id e9319b6d883d1db6f7e71643f2a97b1f810903966a53f2ee00efe32b8cc288b2
Nov 03 19:31:20.084293 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:20.084011    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-8kh6f" event={"ID":"3880f3d3-aad1-43c0-b704-01374aad4f78","Type":"ContainerStarted","Data":"b7ce8ebe35d2285a3f7ef842541ab9676c44e1038b05db62a869ad1d327de641"}
Nov 03 19:31:20.085798 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:20.085763    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5l649" event={"ID":"220ed31c-1577-4c4b-ab5b-e812fd63d193","Type":"ContainerStarted","Data":"e9319b6d883d1db6f7e71643f2a97b1f810903966a53f2ee00efe32b8cc288b2"}
Nov 03 19:31:20.089136 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:20.089094    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mh2tn" event={"ID":"9fb8c33c-fe15-4daa-a01b-4cb2fc7eae7b","Type":"ContainerStarted","Data":"d3620807293dfc2435cd5f84c226d2eb54765fd456022147d2ce289bc117fc51"}
Nov 03 19:31:20.092221 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:20.091661    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gzf8k_6f215b42-0f09-46a0-bc98-392c0fa909c9/console-operator/1.log"
Nov 03 19:31:20.092221 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:20.092178    2575 scope.go:122] "RemoveContainer" containerID="3895cd529be0be6af57557f5b4638c0b5aa21cd6a6d699b224ff7bffcb1bd8e3"
Nov 03 19:31:20.092397 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:20.092364    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-gzf8k_openshift-console-operator(6f215b42-0f09-46a0-bc98-392c0fa909c9)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-gzf8k" podUID="6f215b42-0f09-46a0-bc98-392c0fa909c9"
Nov 03 19:31:20.104500 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:20.104433    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-8kh6f" podStartSLOduration=378.045736324 podStartE2EDuration="6m23.104418593s" totalImagesPullingTime="5.058682269s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:24:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:31:20.103161462 +0000 UTC m=+36.796605800" watchObservedRunningTime="2026-11-03 19:31:20.104418593 +0000 UTC m=+36.797862931"
Nov 03 19:31:20.139272 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:20.139220    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mh2tn" podStartSLOduration=382.141880588 podStartE2EDuration="6m27.139206383s" totalImagesPullingTime="4.997325795s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:24:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:31:20.138987418 +0000 UTC m=+36.832431760" watchObservedRunningTime="2026-11-03 19:31:20.139206383 +0000 UTC m=+36.832650718"
Nov 03 19:31:20.768704 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:20.768605    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-cvll4"]
Nov 03 19:31:20.786136 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:20.786070    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-cvll4"]
Nov 03 19:31:20.786304 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:20.786196    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-cvll4"
Nov 03 19:31:20.791176 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:20.791111    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 19:31:20.792578 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:20.792500    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-nblsj\""
Nov 03 19:31:20.792578 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:20.792540    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 19:31:20.827696 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:20.827174    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/10137ba2-4beb-433e-b6df-199738351c61-data-volume\") pod \"insights-runtime-extractor-cvll4\" (UID: \"10137ba2-4beb-433e-b6df-199738351c61\") " pod="openshift-insights/insights-runtime-extractor-cvll4"
Nov 03 19:31:20.827696 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:20.827207    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/10137ba2-4beb-433e-b6df-199738351c61-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cvll4\" (UID: \"10137ba2-4beb-433e-b6df-199738351c61\") " pod="openshift-insights/insights-runtime-extractor-cvll4"
Nov 03 19:31:20.827940 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:20.827313    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s4qj\" (UniqueName: \"kubernetes.io/projected/10137ba2-4beb-433e-b6df-199738351c61-kube-api-access-2s4qj\") pod \"insights-runtime-extractor-cvll4\" (UID: \"10137ba2-4beb-433e-b6df-199738351c61\") " pod="openshift-insights/insights-runtime-extractor-cvll4"
Nov 03 19:31:20.827940 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:20.827748    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/10137ba2-4beb-433e-b6df-199738351c61-crio-socket\") pod \"insights-runtime-extractor-cvll4\" (UID: \"10137ba2-4beb-433e-b6df-199738351c61\") " pod="openshift-insights/insights-runtime-extractor-cvll4"
Nov 03 19:31:20.827940 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:20.827810    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/10137ba2-4beb-433e-b6df-199738351c61-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cvll4\" (UID: \"10137ba2-4beb-433e-b6df-199738351c61\") " pod="openshift-insights/insights-runtime-extractor-cvll4"
Nov 03 19:31:20.929004 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:20.928972    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s4qj\" (UniqueName: \"kubernetes.io/projected/10137ba2-4beb-433e-b6df-199738351c61-kube-api-access-2s4qj\") pod \"insights-runtime-extractor-cvll4\" (UID: \"10137ba2-4beb-433e-b6df-199738351c61\") " pod="openshift-insights/insights-runtime-extractor-cvll4"
Nov 03 19:31:20.929004 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:20.929011    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/10137ba2-4beb-433e-b6df-199738351c61-crio-socket\") pod \"insights-runtime-extractor-cvll4\" (UID: \"10137ba2-4beb-433e-b6df-199738351c61\") " pod="openshift-insights/insights-runtime-extractor-cvll4"
Nov 03 19:31:20.929230 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:20.929058    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/10137ba2-4beb-433e-b6df-199738351c61-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cvll4\" (UID: \"10137ba2-4beb-433e-b6df-199738351c61\") " pod="openshift-insights/insights-runtime-extractor-cvll4"
Nov 03 19:31:20.929230 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:20.929140    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/10137ba2-4beb-433e-b6df-199738351c61-data-volume\") pod \"insights-runtime-extractor-cvll4\" (UID: \"10137ba2-4beb-433e-b6df-199738351c61\") " pod="openshift-insights/insights-runtime-extractor-cvll4"
Nov 03 19:31:20.929230 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:20.929160    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/10137ba2-4beb-433e-b6df-199738351c61-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cvll4\" (UID: \"10137ba2-4beb-433e-b6df-199738351c61\") " pod="openshift-insights/insights-runtime-extractor-cvll4"
Nov 03 19:31:20.929389 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:20.929266    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 19:31:20.929389 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:20.929317    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10137ba2-4beb-433e-b6df-199738351c61-insights-runtime-extractor-tls podName:10137ba2-4beb-433e-b6df-199738351c61 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:21.429305522 +0000 UTC m=+38.122749838 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/10137ba2-4beb-433e-b6df-199738351c61-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cvll4" (UID: "10137ba2-4beb-433e-b6df-199738351c61") : secret "insights-runtime-extractor-tls" not found
Nov 03 19:31:20.929710 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:20.929564    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/10137ba2-4beb-433e-b6df-199738351c61-crio-socket\") pod \"insights-runtime-extractor-cvll4\" (UID: \"10137ba2-4beb-433e-b6df-199738351c61\") " pod="openshift-insights/insights-runtime-extractor-cvll4"
Nov 03 19:31:20.929710 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:20.929574    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 19:31:20.929710 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:20.929642    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/10137ba2-4beb-433e-b6df-199738351c61-kube-rbac-proxy-cm podName:10137ba2-4beb-433e-b6df-199738351c61 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:21.429624059 +0000 UTC m=+38.123068405 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/10137ba2-4beb-433e-b6df-199738351c61-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cvll4" (UID: "10137ba2-4beb-433e-b6df-199738351c61") : configmap "kube-rbac-proxy" not found
Nov 03 19:31:20.930037 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:20.930013    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/10137ba2-4beb-433e-b6df-199738351c61-data-volume\") pod \"insights-runtime-extractor-cvll4\" (UID: \"10137ba2-4beb-433e-b6df-199738351c61\") " pod="openshift-insights/insights-runtime-extractor-cvll4"
Nov 03 19:31:20.940156 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:20.940120    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s4qj\" (UniqueName: \"kubernetes.io/projected/10137ba2-4beb-433e-b6df-199738351c61-kube-api-access-2s4qj\") pod \"insights-runtime-extractor-cvll4\" (UID: \"10137ba2-4beb-433e-b6df-199738351c61\") " pod="openshift-insights/insights-runtime-extractor-cvll4"
Nov 03 19:31:21.100012 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:21.099236    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-j7m2z" event={"ID":"543b8778-baa4-4e09-b697-68e550e42942","Type":"ContainerStarted","Data":"e3909838200ef62b70277c3459a0c60478ad8228602e6cc356998aac9252d41b"}
Nov 03 19:31:21.103784 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:21.103733    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cbhrt" event={"ID":"1611b730-6163-4776-bd2d-3d10c2cc9a88","Type":"ContainerStarted","Data":"653f8085fe7cd6e9c008c328cd7efce712fcc30d8bab7441f51b1df091921489"}
Nov 03 19:31:21.120250 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:21.120186    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-j7m2z" podStartSLOduration=2.032204236 podStartE2EDuration="8.120168393s" totalImagesPullingTime="6.087964157s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:31:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:31:21.118222323 +0000 UTC m=+37.811666661" watchObservedRunningTime="2026-11-03 19:31:21.120168393 +0000 UTC m=+37.813612732"
Nov 03 19:31:21.140019 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:21.139966    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cbhrt" podStartSLOduration=380.002073317 podStartE2EDuration="6m26.139950743s" totalImagesPullingTime="6.137877426s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:24:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:31:21.138411906 +0000 UTC m=+37.831856244" watchObservedRunningTime="2026-11-03 19:31:21.139950743 +0000 UTC m=+37.833395128"
Nov 03 19:31:21.433048 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:21.433020    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/10137ba2-4beb-433e-b6df-199738351c61-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cvll4\" (UID: \"10137ba2-4beb-433e-b6df-199738351c61\") " pod="openshift-insights/insights-runtime-extractor-cvll4"
Nov 03 19:31:21.433165 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:21.433140    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 19:31:21.433215 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:21.433142    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/10137ba2-4beb-433e-b6df-199738351c61-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cvll4\" (UID: \"10137ba2-4beb-433e-b6df-199738351c61\") " pod="openshift-insights/insights-runtime-extractor-cvll4"
Nov 03 19:31:21.433260 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:21.433214    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/10137ba2-4beb-433e-b6df-199738351c61-kube-rbac-proxy-cm podName:10137ba2-4beb-433e-b6df-199738351c61 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:22.43319667 +0000 UTC m=+39.126641003 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/10137ba2-4beb-433e-b6df-199738351c61-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cvll4" (UID: "10137ba2-4beb-433e-b6df-199738351c61") : configmap "kube-rbac-proxy" not found
Nov 03 19:31:21.433260 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:21.433219    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 19:31:21.433349 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:21.433263    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10137ba2-4beb-433e-b6df-199738351c61-insights-runtime-extractor-tls podName:10137ba2-4beb-433e-b6df-199738351c61 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:22.433253051 +0000 UTC m=+39.126697387 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/10137ba2-4beb-433e-b6df-199738351c61-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cvll4" (UID: "10137ba2-4beb-433e-b6df-199738351c61") : secret "insights-runtime-extractor-tls" not found
Nov 03 19:31:21.534390 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:21.534349    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e2b752d9-88c4-4dee-8b0c-7e9ac252adb3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kfdsx\" (UID: \"e2b752d9-88c4-4dee-8b0c-7e9ac252adb3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kfdsx"
Nov 03 19:31:21.534643 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:21.534399    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/eb488ece-2ac3-499f-84c1-73534fcbb20b-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zpgl6\" (UID: \"eb488ece-2ac3-499f-84c1-73534fcbb20b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zpgl6"
Nov 03 19:31:21.534643 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:21.534498    2575 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 19:31:21.534643 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:21.534512    2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 19:31:21.534643 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:21.534551    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb488ece-2ac3-499f-84c1-73534fcbb20b-networking-console-plugin-cert podName:eb488ece-2ac3-499f-84c1-73534fcbb20b nodeName:}" failed. No retries permitted until 2026-11-03 19:31:29.53453891 +0000 UTC m=+46.227983227 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/eb488ece-2ac3-499f-84c1-73534fcbb20b-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-zpgl6" (UID: "eb488ece-2ac3-499f-84c1-73534fcbb20b") : secret "networking-console-plugin-cert" not found
Nov 03 19:31:21.534643 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:21.534578    2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 03 19:31:21.534643 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:21.534520    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a023911d-3669-43e1-ac77-8a986f3715d6-metrics-certs\") pod \"router-default-569d4c45f-7pszr\" (UID: \"a023911d-3669-43e1-ac77-8a986f3715d6\") " pod="openshift-ingress/router-default-569d4c45f-7pszr"
Nov 03 19:31:21.534643 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:21.534581    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2b752d9-88c4-4dee-8b0c-7e9ac252adb3-cluster-monitoring-operator-tls podName:e2b752d9-88c4-4dee-8b0c-7e9ac252adb3 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:29.5345615 +0000 UTC m=+46.228007693 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e2b752d9-88c4-4dee-8b0c-7e9ac252adb3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-kfdsx" (UID: "e2b752d9-88c4-4dee-8b0c-7e9ac252adb3") : secret "cluster-monitoring-operator-tls" not found
Nov 03 19:31:21.534643 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:21.534645    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a023911d-3669-43e1-ac77-8a986f3715d6-metrics-certs podName:a023911d-3669-43e1-ac77-8a986f3715d6 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:29.534634697 +0000 UTC m=+46.228079013 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a023911d-3669-43e1-ac77-8a986f3715d6-metrics-certs") pod "router-default-569d4c45f-7pszr" (UID: "a023911d-3669-43e1-ac77-8a986f3715d6") : secret "router-metrics-certs-default" not found
Nov 03 19:31:21.535042 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:21.534730    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a023911d-3669-43e1-ac77-8a986f3715d6-service-ca-bundle\") pod \"router-default-569d4c45f-7pszr\" (UID: \"a023911d-3669-43e1-ac77-8a986f3715d6\") " pod="openshift-ingress/router-default-569d4c45f-7pszr"
Nov 03 19:31:21.535042 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:21.534757    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/96c016f7-9098-4a39-9e72-e54da374d8fc-registry-tls\") pod \"image-registry-d74787d9c-nn7sk\" (UID: \"96c016f7-9098-4a39-9e72-e54da374d8fc\") " pod="openshift-image-registry/image-registry-d74787d9c-nn7sk"
Nov 03 19:31:21.535042 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:21.534786    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-registry-tls\") pod \"image-registry-c5c44f4d-dv455\" (UID: \"d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca\") " pod="openshift-image-registry/image-registry-c5c44f4d-dv455"
Nov 03 19:31:21.535042 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:21.534806    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/38a6add1-b051-450b-bb96-8ace8a0ccff3-metrics-tls\") pod \"dns-default-wj8mp\" (UID: \"38a6add1-b051-450b-bb96-8ace8a0ccff3\") " pod="openshift-dns/dns-default-wj8mp"
Nov 03 19:31:21.535042 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:21.534881    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 19:31:21.535042 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:21.534887    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a023911d-3669-43e1-ac77-8a986f3715d6-service-ca-bundle podName:a023911d-3669-43e1-ac77-8a986f3715d6 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:29.534871761 +0000 UTC m=+46.228316108 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a023911d-3669-43e1-ac77-8a986f3715d6-service-ca-bundle") pod "router-default-569d4c45f-7pszr" (UID: "a023911d-3669-43e1-ac77-8a986f3715d6") : configmap references non-existent config key: service-ca.crt
Nov 03 19:31:21.535042 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:21.534896    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 19:31:21.535042 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:21.534913    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c5c44f4d-dv455: secret "image-registry-tls" not found
Nov 03 19:31:21.535042 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:21.534918    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1753fd31-8933-4567-8146-07f3fd2d10a5-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-7hcf8\" (UID: \"1753fd31-8933-4567-8146-07f3fd2d10a5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7hcf8"
Nov 03 19:31:21.535042 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:21.534943    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-registry-tls podName:d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca nodeName:}" failed. No retries permitted until 2026-11-03 19:31:29.534932635 +0000 UTC m=+46.228376992 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-registry-tls") pod "image-registry-c5c44f4d-dv455" (UID: "d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca") : secret "image-registry-tls" not found
Nov 03 19:31:21.535042 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:21.534898    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d74787d9c-nn7sk: secret "image-registry-tls" not found
Nov 03 19:31:21.535042 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:21.534885    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 19:31:21.535042 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:21.534979    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/96c016f7-9098-4a39-9e72-e54da374d8fc-registry-tls podName:96c016f7-9098-4a39-9e72-e54da374d8fc nodeName:}" failed. No retries permitted until 2026-11-03 19:31:29.534971272 +0000 UTC m=+46.228415588 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/96c016f7-9098-4a39-9e72-e54da374d8fc-registry-tls") pod "image-registry-d74787d9c-nn7sk" (UID: "96c016f7-9098-4a39-9e72-e54da374d8fc") : secret "image-registry-tls" not found
Nov 03 19:31:21.535042 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:21.534997    2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 03 19:31:21.535042 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:21.535029    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38a6add1-b051-450b-bb96-8ace8a0ccff3-metrics-tls podName:38a6add1-b051-450b-bb96-8ace8a0ccff3 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:29.535011535 +0000 UTC m=+46.228455865 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/38a6add1-b051-450b-bb96-8ace8a0ccff3-metrics-tls") pod "dns-default-wj8mp" (UID: "38a6add1-b051-450b-bb96-8ace8a0ccff3") : secret "dns-default-metrics-tls" not found
Nov 03 19:31:21.535042 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:21.535043    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1753fd31-8933-4567-8146-07f3fd2d10a5-samples-operator-tls podName:1753fd31-8933-4567-8146-07f3fd2d10a5 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:29.535037365 +0000 UTC m=+46.228481681 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1753fd31-8933-4567-8146-07f3fd2d10a5-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-7hcf8" (UID: "1753fd31-8933-4567-8146-07f3fd2d10a5") : secret "samples-operator-tls" not found
Nov 03 19:31:22.107878 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:22.107837    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-njg27" event={"ID":"b46f51b2-436b-4967-8879-a16b58930591","Type":"ContainerStarted","Data":"d21c68dd8dec5fc1c36a8ee89dab92673bc945a68883867ca595e2a64bef1a83"}
Nov 03 19:31:22.108349 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:22.107900    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-njg27"
Nov 03 19:31:22.109240 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:22.109214    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5l649" event={"ID":"220ed31c-1577-4c4b-ab5b-e812fd63d193","Type":"ContainerStarted","Data":"b8b1e7dae9c457f8b404c2cff785bde1350eefc39d8ed99a145746a9f1609967"}
Nov 03 19:31:22.125421 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:22.125165    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-njg27" podStartSLOduration=35.835009611 podStartE2EDuration="39.125150764s" totalImagesPullingTime="3.290141153s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:30:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:31:22.125150422 +0000 UTC m=+38.818594759" watchObservedRunningTime="2026-11-03 19:31:22.125150764 +0000 UTC m=+38.818595102"
Nov 03 19:31:22.143442 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:22.143394    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5l649" podStartSLOduration=2.3131091870000002 podStartE2EDuration="4.143375588s" totalImagesPullingTime="1.830266401s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:31:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:31:22.142616576 +0000 UTC m=+38.836060915" watchObservedRunningTime="2026-11-03 19:31:22.143375588 +0000 UTC m=+38.836819929"
Nov 03 19:31:22.442756 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:22.442658    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/10137ba2-4beb-433e-b6df-199738351c61-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cvll4\" (UID: \"10137ba2-4beb-433e-b6df-199738351c61\") " pod="openshift-insights/insights-runtime-extractor-cvll4"
Nov 03 19:31:22.442909 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:22.442802    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/10137ba2-4beb-433e-b6df-199738351c61-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cvll4\" (UID: \"10137ba2-4beb-433e-b6df-199738351c61\") " pod="openshift-insights/insights-runtime-extractor-cvll4"
Nov 03 19:31:22.442909 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:22.442811    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 19:31:22.442987 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:22.442918    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10137ba2-4beb-433e-b6df-199738351c61-insights-runtime-extractor-tls podName:10137ba2-4beb-433e-b6df-199738351c61 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:24.442898842 +0000 UTC m=+41.136343162 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/10137ba2-4beb-433e-b6df-199738351c61-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cvll4" (UID: "10137ba2-4beb-433e-b6df-199738351c61") : secret "insights-runtime-extractor-tls" not found
Nov 03 19:31:22.442987 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:22.442938    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 19:31:22.442987 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:22.442985    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/10137ba2-4beb-433e-b6df-199738351c61-kube-rbac-proxy-cm podName:10137ba2-4beb-433e-b6df-199738351c61 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:24.44297327 +0000 UTC m=+41.136417597 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/10137ba2-4beb-433e-b6df-199738351c61-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cvll4" (UID: "10137ba2-4beb-433e-b6df-199738351c61") : configmap "kube-rbac-proxy" not found
Nov 03 19:31:24.098562 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:24.098528    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-gzf8k"
Nov 03 19:31:24.098967 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:24.098619    2575 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-gzf8k"
Nov 03 19:31:24.099011 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:24.098984    2575 scope.go:122] "RemoveContainer" containerID="3895cd529be0be6af57557f5b4638c0b5aa21cd6a6d699b224ff7bffcb1bd8e3"
Nov 03 19:31:24.099199 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:24.099177    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-gzf8k_openshift-console-operator(6f215b42-0f09-46a0-bc98-392c0fa909c9)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-gzf8k" podUID="6f215b42-0f09-46a0-bc98-392c0fa909c9"
Nov 03 19:31:24.460216 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:24.460124    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/10137ba2-4beb-433e-b6df-199738351c61-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cvll4\" (UID: \"10137ba2-4beb-433e-b6df-199738351c61\") " pod="openshift-insights/insights-runtime-extractor-cvll4"
Nov 03 19:31:24.460393 ip-10-0-134-222 kubenswrapper[2575]: I1103 19:31:24.460243    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/10137ba2-4beb-433e-b6df-199738351c61-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cvll4\" (UID: \"10137ba2-4beb-433e-b6df-199738351c61\") " pod="openshift-insights/insights-runtime-extractor-cvll4"
Nov 03 19:31:24.460393 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:24.460283    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 19:31:24.460393 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:24.460380    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/10137ba2-4beb-433e-b6df-199738351c61-kube-rbac-proxy-cm podName:10137ba2-4beb-433e-b6df-199738351c61 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:28.460358388 +0000 UTC m=+45.153802727 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/10137ba2-4beb-433e-b6df-199738351c61-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cvll4" (UID: "10137ba2-4beb-433e-b6df-199738351c61") : configmap "kube-rbac-proxy" not found
Nov 03 19:31:24.460555 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:24.460397    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 19:31:24.460555 ip-10-0-134-222 kubenswrapper[2575]: E1103 19:31:24.460465    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10137ba2-4beb-433e-b6df-199738351c61-insights-runtime-extractor-tls podName:10137ba2-4beb-433e-b6df-199738351c61 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:28.460453407 +0000 UTC m=+45.153897734 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/10137ba2-4beb-433e-b6df-199738351c61-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cvll4" (UID: "10137ba2-4beb-433e-b6df-199738351c61") : secret "insights-runtime-extractor-tls" not found
Nov 03 19:31:26.685712 ip-10-0-134-222 systemd[1]: Stopping Kubernetes Kubelet...
Nov 03 19:31:26.699030 ip-10-0-134-222 systemd[1]: kubelet.service: Deactivated successfully.
Nov 03 19:31:26.699198 ip-10-0-134-222 systemd[1]: Stopped Kubernetes Kubelet.
Nov 03 19:31:26.699247 ip-10-0-134-222 systemd[1]: kubelet.service: Consumed 3.202s CPU time.
Nov 03 19:31:26.710279 ip-10-0-134-222 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 19:31:26.794509 ip-10-0-134-222 kubenswrapper[4921]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 19:31:26.794509 ip-10-0-134-222 kubenswrapper[4921]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 19:31:26.794509 ip-10-0-134-222 kubenswrapper[4921]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 19:31:26.794509 ip-10-0-134-222 kubenswrapper[4921]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 19:31:26.796685 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796575    4921 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:31:26.796685 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796683    4921 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:31:26.796685 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796690    4921 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:31:26.796837 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796695    4921 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:31:26.796837 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796699    4921 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:31:26.796837 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796703    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:31:26.796837 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796706    4921 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:31:26.796837 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796709    4921 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:31:26.796837 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796712    4921 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:31:26.796837 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796714    4921 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:31:26.796837 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796717    4921 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:31:26.796837 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796720    4921 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:31:26.796837 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796722    4921 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:31:26.796837 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796733    4921 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:31:26.796837 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796737    4921 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:31:26.796837 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796740    4921 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:31:26.796837 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796743    4921 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:31:26.796837 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796745    4921 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:31:26.796837 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796748    4921 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:31:26.796837 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796751    4921 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:31:26.796837 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796753    4921 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:31:26.796837 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796756    4921 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:31:26.796837 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796759    4921 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:31:26.797353 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796761    4921 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:31:26.797353 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796764    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:31:26.797353 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796767    4921 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:31:26.797353 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796769    4921 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:31:26.797353 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796772    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:31:26.797353 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796774    4921 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:31:26.797353 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796777    4921 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:31:26.797353 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796780    4921 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:31:26.797353 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796783    4921 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:31:26.797353 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796786    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:31:26.797353 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796789    4921 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:31:26.797353 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796792    4921 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:31:26.797353 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796795    4921 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:31:26.797353 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796797    4921 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:31:26.797353 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796800    4921 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:31:26.797353 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796803    4921 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:31:26.797353 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796806    4921 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:31:26.797353 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796808    4921 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:31:26.797353 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796811    4921 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:31:26.797852 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796813    4921 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:31:26.797852 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796816    4921 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:31:26.797852 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796819    4921 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:31:26.797852 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796821    4921 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:31:26.797852 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796826    4921 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:31:26.797852 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796831    4921 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:31:26.797852 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796834    4921 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:31:26.797852 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796838    4921 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:31:26.797852 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796842    4921 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:31:26.797852 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796845    4921 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:31:26.797852 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796847    4921 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:31:26.797852 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796851    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:31:26.797852 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796854    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:31:26.797852 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796857    4921 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:31:26.797852 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796859    4921 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:31:26.797852 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796863    4921 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:31:26.797852 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796866    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:31:26.797852 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796868    4921 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:31:26.797852 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796871    4921 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:31:26.797852 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796874    4921 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:31:26.798349 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796877    4921 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:31:26.798349 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796880    4921 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:31:26.798349 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796883    4921 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:31:26.798349 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796886    4921 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:31:26.798349 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796889    4921 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:31:26.798349 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796891    4921 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:31:26.798349 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796894    4921 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:31:26.798349 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796896    4921 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:31:26.798349 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796899    4921 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:31:26.798349 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796902    4921 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:31:26.798349 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796905    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:31:26.798349 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796907    4921 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:31:26.798349 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796910    4921 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:31:26.798349 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796915    4921 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:31:26.798349 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796918    4921 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:31:26.798349 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796920    4921 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:31:26.798349 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796923    4921 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:31:26.798349 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796926    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:31:26.798349 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796929    4921 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:31:26.798349 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796931    4921 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:31:26.798878 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796934    4921 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:31:26.798878 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796936    4921 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:31:26.798878 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796939    4921 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:31:26.798878 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796941    4921 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:31:26.798878 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796944    4921 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:31:26.798878 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796947    4921 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:31:26.798878 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796949    4921 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:31:26.798878 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796952    4921 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:31:26.798878 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796954    4921 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:31:26.798878 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796957    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:31:26.798878 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796960    4921 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:31:26.798878 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796962    4921 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:31:26.798878 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796965    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:31:26.798878 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796968    4921 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:31:26.798878 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796971    4921 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:31:26.798878 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796973    4921 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:31:26.798878 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796976    4921 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:31:26.798878 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796980    4921 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:31:26.798878 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796984    4921 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:31:26.799352 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796988    4921 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:31:26.799352 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796991    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:31:26.799352 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796994    4921 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:31:26.799352 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796996    4921 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:31:26.799352 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.796999    4921 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:31:26.799352 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.797784    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:31:26.799352 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.797793    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:31:26.799352 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.797796    4921 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:31:26.799352 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.797799    4921 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:31:26.799352 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.797801    4921 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:31:26.799352 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.797805    4921 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:31:26.799352 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.797809    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:31:26.799352 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.797812    4921 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:31:26.799352 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.797815    4921 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:31:26.799352 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.797818    4921 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:31:26.799352 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.797821    4921 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:31:26.799352 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.797823    4921 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:31:26.799352 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.797826    4921 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:31:26.799352 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.797828    4921 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:31:26.799352 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.797831    4921 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:31:26.799913 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.797834    4921 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:31:26.799913 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.797836    4921 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:31:26.799913 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.797839    4921 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:31:26.799913 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.797842    4921 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:31:26.799913 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.797844    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:31:26.799913 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.797847    4921 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:31:26.799913 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.797849    4921 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:31:26.799913 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.797854    4921 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:31:26.799913 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.797859    4921 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:31:26.799913 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.797862    4921 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:31:26.799913 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.797866    4921 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:31:26.799913 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.797869    4921 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:31:26.799913 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.797871    4921 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:31:26.799913 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.797874    4921 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:31:26.799913 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.797877    4921 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:31:26.799913 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.797879    4921 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:31:26.799913 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.797882    4921 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:31:26.799913 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.797885    4921 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:31:26.799913 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.797888    4921 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:31:26.800408 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.797891    4921 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:31:26.800408 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.797893    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:31:26.800408 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.797896    4921 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:31:26.800408 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.797898    4921 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:31:26.800408 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.797905    4921 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:31:26.800408 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.797911    4921 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:31:26.800408 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.797914    4921 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:31:26.800408 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.797916    4921 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:31:26.800408 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.797919    4921 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:31:26.800408 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.797921    4921 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:31:26.800408 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.797924    4921 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:31:26.800408 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.797926    4921 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:31:26.800408 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.797929    4921 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:31:26.800408 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.797931    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:31:26.800408 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.797934    4921 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:31:26.800408 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.797936    4921 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:31:26.800408 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.797939    4921 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:31:26.800408 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.797941    4921 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:31:26.800408 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.797944    4921 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:31:26.800911 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.797947    4921 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:31:26.800911 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.797949    4921 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:31:26.800911 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.797952    4921 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:31:26.800911 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.797956    4921 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:31:26.800911 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.797958    4921 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:31:26.800911 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.797961    4921 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:31:26.800911 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.797964    4921 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:31:26.800911 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.797967    4921 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:31:26.800911 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.797969    4921 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:31:26.800911 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.797972    4921 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:31:26.800911 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.797975    4921 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:31:26.800911 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.797977    4921 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:31:26.800911 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.797980    4921 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:31:26.800911 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.797983    4921 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:31:26.800911 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.797985    4921 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:31:26.800911 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.797988    4921 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:31:26.800911 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.797990    4921 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:31:26.800911 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.797993    4921 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:31:26.800911 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.797996    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:31:26.800911 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798004    4921 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:31:26.801407 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798007    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:31:26.801407 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798009    4921 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:31:26.801407 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798012    4921 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:31:26.801407 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798014    4921 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:31:26.801407 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798017    4921 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:31:26.801407 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798019    4921 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:31:26.801407 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798022    4921 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:31:26.801407 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798024    4921 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:31:26.801407 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798032    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:31:26.801407 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798035    4921 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:31:26.801407 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798038    4921 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:31:26.801407 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798041    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:31:26.801407 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798043    4921 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:31:26.801407 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798046    4921 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:31:26.801407 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798049    4921 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:31:26.801407 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798051    4921 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:31:26.801407 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798054    4921 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:31:26.801407 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798057    4921 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:31:26.801407 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798059    4921 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:31:26.801407 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798062    4921 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:31:26.801931 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798065    4921 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:31:26.801931 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798068    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:31:26.801931 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798071    4921 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:31:26.801931 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798073    4921 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:31:26.801931 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798076    4921 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:31:26.801931 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798079    4921 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:31:26.801931 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798081    4921 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:31:26.801931 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798084    4921 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:31:26.801931 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798086    4921 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:31:26.801931 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798089    4921 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:31:26.801931 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798091    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:31:26.801931 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798094    4921 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:31:26.801931 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798102    4921 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:31:26.801931 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798615    4921 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:31:26.801931 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798621    4921 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:31:26.801931 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798624    4921 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:31:26.801931 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798627    4921 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:31:26.801931 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798630    4921 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:31:26.801931 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798632    4921 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:31:26.802508 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798635    4921 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:31:26.802508 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798638    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:31:26.802508 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798640    4921 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:31:26.802508 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798643    4921 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:31:26.802508 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798646    4921 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:31:26.802508 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798649    4921 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:31:26.802508 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798651    4921 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:31:26.802508 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798654    4921 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:31:26.802508 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798656    4921 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:31:26.802508 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798661    4921 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:31:26.802508 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798679    4921 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:31:26.802508 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798685    4921 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:31:26.802508 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798689    4921 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:31:26.802508 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798693    4921 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:31:26.802508 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798701    4921 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:31:26.802508 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798704    4921 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:31:26.802508 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798707    4921 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:31:26.802508 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798709    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:31:26.802508 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798712    4921 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:31:26.803018 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798714    4921 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:31:26.803018 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798717    4921 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:31:26.803018 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798720    4921 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:31:26.803018 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798722    4921 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:31:26.803018 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798725    4921 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:31:26.803018 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798727    4921 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:31:26.803018 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798733    4921 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:31:26.803018 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798736    4921 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:31:26.803018 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798739    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:31:26.803018 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798742    4921 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:31:26.803018 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798744    4921 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:31:26.803018 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798747    4921 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:31:26.803018 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798749    4921 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:31:26.803018 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798753    4921 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:31:26.803018 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798756    4921 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:31:26.803018 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798759    4921 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:31:26.803018 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798762    4921 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:31:26.803018 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798764    4921 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:31:26.803018 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798769    4921 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:31:26.803491 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798771    4921 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:31:26.803491 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798774    4921 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:31:26.803491 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798776    4921 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:31:26.803491 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798779    4921 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:31:26.803491 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798782    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:31:26.803491 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798784    4921 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:31:26.803491 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798787    4921 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:31:26.803491 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798790    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:31:26.803491 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798794    4921 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:31:26.803491 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798796    4921 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:31:26.803491 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798799    4921 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:31:26.803491 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798801    4921 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:31:26.803491 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798804    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:31:26.803491 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798806    4921 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:31:26.803491 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798809    4921 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:31:26.803491 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798811    4921 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:31:26.803491 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798814    4921 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:31:26.803491 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798817    4921 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:31:26.803491 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798819    4921 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:31:26.803974 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798822    4921 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:31:26.803974 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798825    4921 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:31:26.803974 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798827    4921 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:31:26.803974 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798830    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:31:26.803974 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798833    4921 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:31:26.803974 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798836    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:31:26.803974 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798838    4921 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:31:26.803974 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798841    4921 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:31:26.803974 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798843    4921 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:31:26.803974 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798846    4921 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:31:26.803974 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798848    4921 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:31:26.803974 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798851    4921 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:31:26.803974 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798855    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:31:26.803974 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798857    4921 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:31:26.803974 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798860    4921 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:31:26.803974 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798863    4921 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:31:26.803974 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798865    4921 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:31:26.803974 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798868    4921 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:31:26.803974 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798871    4921 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:31:26.803974 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798873    4921 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:31:26.804734 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798876    4921 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:31:26.804734 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798880    4921 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:31:26.804734 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798883    4921 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:31:26.804734 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798885    4921 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:31:26.804734 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798888    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:31:26.804734 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798891    4921 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:31:26.804734 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798894    4921 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:31:26.804734 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798896    4921 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:31:26.804734 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798899    4921 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:31:26.804734 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798901    4921 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:31:26.804734 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798904    4921 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:31:26.804734 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798907    4921 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:31:26.804734 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798909    4921 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:31:26.804734 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798911    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:31:26.804734 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798915    4921 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:31:26.804734 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798917    4921 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:31:26.804734 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798920    4921 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:31:26.804734 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798923    4921 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:31:26.804734 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798925    4921 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:31:26.804734 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798928    4921 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:31:26.805419 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798930    4921 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:31:26.805419 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798933    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:31:26.805419 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.798935    4921 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:31:26.805419 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799025    4921 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 19:31:26.805419 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799035    4921 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 19:31:26.805419 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799043    4921 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 19:31:26.805419 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799050    4921 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 19:31:26.805419 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799057    4921 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 19:31:26.805419 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799062    4921 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 19:31:26.805419 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799067    4921 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 19:31:26.805419 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799072    4921 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 19:31:26.805419 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799075    4921 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 19:31:26.805419 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799078    4921 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 19:31:26.805419 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799083    4921 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 19:31:26.805419 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799087    4921 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 19:31:26.805419 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799090    4921 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 19:31:26.805419 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799094    4921 flags.go:64] FLAG: --cgroup-root=""
Nov 03 19:31:26.805419 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799097    4921 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 19:31:26.805419 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799100    4921 flags.go:64] FLAG: --client-ca-file=""
Nov 03 19:31:26.805419 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799103    4921 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 19:31:26.805419 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799106    4921 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 19:31:26.805419 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799111    4921 flags.go:64] FLAG: --cluster-domain=""
Nov 03 19:31:26.805419 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799114    4921 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 19:31:26.805419 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799118    4921 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 19:31:26.806063 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799121    4921 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 19:31:26.806063 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799124    4921 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 19:31:26.806063 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799133    4921 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 19:31:26.806063 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799137    4921 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 19:31:26.806063 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799140    4921 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 19:31:26.806063 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799144    4921 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 19:31:26.806063 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799147    4921 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 19:31:26.806063 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799150    4921 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 19:31:26.806063 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799154    4921 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 19:31:26.806063 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799157    4921 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 19:31:26.806063 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799160    4921 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 19:31:26.806063 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799164    4921 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 19:31:26.806063 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799169    4921 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 19:31:26.806063 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799173    4921 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 19:31:26.806063 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799176    4921 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 19:31:26.806063 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799179    4921 flags.go:64] FLAG: --enable-server="true"
Nov 03 19:31:26.806063 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799182    4921 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 19:31:26.806063 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799186    4921 flags.go:64] FLAG: --event-burst="100"
Nov 03 19:31:26.806063 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799190    4921 flags.go:64] FLAG: --event-qps="50"
Nov 03 19:31:26.806063 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799193    4921 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 19:31:26.806063 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799196    4921 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 19:31:26.806063 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799201    4921 flags.go:64] FLAG: --eviction-hard=""
Nov 03 19:31:26.806063 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799206    4921 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 19:31:26.806063 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799209    4921 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 19:31:26.806729 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799212    4921 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 19:31:26.806729 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799215    4921 flags.go:64] FLAG: --eviction-soft=""
Nov 03 19:31:26.806729 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799218    4921 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 19:31:26.806729 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799221    4921 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 19:31:26.806729 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799225    4921 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 19:31:26.806729 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799228    4921 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 19:31:26.806729 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799231    4921 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 19:31:26.806729 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799234    4921 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 19:31:26.806729 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799237    4921 flags.go:64] FLAG: --feature-gates=""
Nov 03 19:31:26.806729 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799241    4921 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 19:31:26.806729 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799244    4921 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 19:31:26.806729 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799248    4921 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 19:31:26.806729 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799251    4921 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 19:31:26.806729 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799254    4921 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 19:31:26.806729 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799258    4921 flags.go:64] FLAG: --help="false"
Nov 03 19:31:26.806729 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799261    4921 flags.go:64] FLAG: --hostname-override="ip-10-0-134-222.us-east-2.compute.internal"
Nov 03 19:31:26.806729 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799264    4921 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 19:31:26.806729 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799267    4921 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 19:31:26.806729 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799270    4921 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 19:31:26.806729 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799273    4921 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 19:31:26.806729 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799279    4921 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 19:31:26.806729 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799282    4921 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 19:31:26.806729 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799285    4921 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 19:31:26.807335 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799288    4921 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 19:31:26.807335 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799291    4921 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 19:31:26.807335 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799294    4921 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 19:31:26.807335 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799297    4921 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 19:31:26.807335 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799301    4921 flags.go:64] FLAG: --kube-reserved=""
Nov 03 19:31:26.807335 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799304    4921 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 19:31:26.807335 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799308    4921 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 19:31:26.807335 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799311    4921 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 19:31:26.807335 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799314    4921 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 19:31:26.807335 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799317    4921 flags.go:64] FLAG: --lock-file=""
Nov 03 19:31:26.807335 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799320    4921 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 19:31:26.807335 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799323    4921 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 19:31:26.807335 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799326    4921 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 19:31:26.807335 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799332    4921 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 19:31:26.807335 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799336    4921 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 19:31:26.807335 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799339    4921 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 19:31:26.807335 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799342    4921 flags.go:64] FLAG: --logging-format="text"
Nov 03 19:31:26.807335 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799345    4921 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 19:31:26.807335 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799349    4921 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 19:31:26.807335 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799352    4921 flags.go:64] FLAG: --manifest-url=""
Nov 03 19:31:26.807335 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799355    4921 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 19:31:26.807335 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799360    4921 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 19:31:26.807335 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799364    4921 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 19:31:26.807335 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799368    4921 flags.go:64] FLAG: --max-pods="110"
Nov 03 19:31:26.807335 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799371    4921 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 19:31:26.807964 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799374    4921 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 19:31:26.807964 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799377    4921 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 19:31:26.807964 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799380    4921 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 19:31:26.807964 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799383    4921 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 19:31:26.807964 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799387    4921 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 19:31:26.807964 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799391    4921 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 19:31:26.807964 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799397    4921 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 19:31:26.807964 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799401    4921 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 19:31:26.807964 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799404    4921 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 19:31:26.807964 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799407    4921 flags.go:64] FLAG: --pod-cidr=""
Nov 03 19:31:26.807964 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799410    4921 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 19:31:26.807964 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799413    4921 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 19:31:26.807964 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799416    4921 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 19:31:26.807964 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799420    4921 flags.go:64] FLAG: --port="10250"
Nov 03 19:31:26.807964 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799423    4921 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 19:31:26.807964 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799426    4921 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0e99d08dc592670ed"
Nov 03 19:31:26.807964 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799429    4921 flags.go:64] FLAG: --qos-reserved=""
Nov 03 19:31:26.807964 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799432    4921 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 19:31:26.807964 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799435    4921 flags.go:64] FLAG: --register-node="true"
Nov 03 19:31:26.807964 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799438    4921 flags.go:64] FLAG: --register-with-taints=""
Nov 03 19:31:26.807964 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799442    4921 flags.go:64] FLAG: --registry-burst="10"
Nov 03 19:31:26.807964 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799445    4921 flags.go:64] FLAG: --registry-qps="5"
Nov 03 19:31:26.807964 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799448    4921 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 19:31:26.807964 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799451    4921 flags.go:64] FLAG: --reserved-memory=""
Nov 03 19:31:26.807964 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799455    4921 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 19:31:26.807964 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799458    4921 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 19:31:26.808599 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799461    4921 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 19:31:26.808599 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799464    4921 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 19:31:26.808599 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799467    4921 flags.go:64] FLAG: --runonce="false"
Nov 03 19:31:26.808599 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799470    4921 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 19:31:26.808599 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799474    4921 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 19:31:26.808599 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799477    4921 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 19:31:26.808599 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799481    4921 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 19:31:26.808599 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799484    4921 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 19:31:26.808599 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799487    4921 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 19:31:26.808599 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799490    4921 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 19:31:26.808599 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799495    4921 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 19:31:26.808599 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799498    4921 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 19:31:26.808599 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799501    4921 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 19:31:26.808599 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799504    4921 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 19:31:26.808599 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799507    4921 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 19:31:26.808599 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799510    4921 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 19:31:26.808599 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799513    4921 flags.go:64] FLAG: --system-cgroups=""
Nov 03 19:31:26.808599 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799517    4921 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 19:31:26.808599 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799522    4921 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 19:31:26.808599 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799525    4921 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 19:31:26.808599 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799528    4921 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 19:31:26.808599 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799533    4921 flags.go:64] FLAG: --tls-min-version=""
Nov 03 19:31:26.808599 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799536    4921 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 19:31:26.808599 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799539    4921 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 19:31:26.808599 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799542    4921 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 19:31:26.809234 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799545    4921 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 19:31:26.809234 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799548    4921 flags.go:64] FLAG: --v="2"
Nov 03 19:31:26.809234 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799552    4921 flags.go:64] FLAG: --version="false"
Nov 03 19:31:26.809234 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799557    4921 flags.go:64] FLAG: --vmodule=""
Nov 03 19:31:26.809234 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799561    4921 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 19:31:26.809234 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.799564    4921 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 19:31:26.809234 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799706    4921 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:31:26.809234 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799710    4921 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:31:26.809234 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799715    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:31:26.809234 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799718    4921 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:31:26.809234 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799721    4921 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:31:26.809234 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799724    4921 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:31:26.809234 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799727    4921 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:31:26.809234 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799730    4921 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:31:26.809234 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799732    4921 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:31:26.809234 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799735    4921 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:31:26.809234 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799738    4921 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:31:26.809234 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799742    4921 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:31:26.809234 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799744    4921 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:31:26.809234 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799747    4921 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:31:26.809234 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799750    4921 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:31:26.809774 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799752    4921 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:31:26.809774 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799756    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:31:26.809774 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799758    4921 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:31:26.809774 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799761    4921 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:31:26.809774 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799764    4921 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:31:26.809774 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799767    4921 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:31:26.809774 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799769    4921 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:31:26.809774 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799772    4921 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:31:26.809774 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799774    4921 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:31:26.809774 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799777    4921 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:31:26.809774 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799780    4921 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:31:26.809774 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799782    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:31:26.809774 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799785    4921 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:31:26.809774 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799788    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:31:26.809774 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799791    4921 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:31:26.809774 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799794    4921 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:31:26.809774 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799796    4921 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:31:26.809774 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799799    4921 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:31:26.809774 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799801    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:31:26.809774 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799804    4921 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:31:26.810297 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799806    4921 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:31:26.810297 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799809    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:31:26.810297 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799811    4921 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:31:26.810297 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799814    4921 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:31:26.810297 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799818    4921 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:31:26.810297 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799820    4921 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:31:26.810297 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799823    4921 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:31:26.810297 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799825    4921 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:31:26.810297 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799829    4921 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:31:26.810297 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799832    4921 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:31:26.810297 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799835    4921 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:31:26.810297 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799838    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:31:26.810297 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799854    4921 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:31:26.810297 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799859    4921 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:31:26.810297 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799862    4921 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:31:26.810297 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799865    4921 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:31:26.810297 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799868    4921 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:31:26.810297 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799871    4921 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:31:26.810297 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799873    4921 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:31:26.810297 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799876    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:31:26.810931 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799879    4921 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:31:26.810931 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799882    4921 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:31:26.810931 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799885    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:31:26.810931 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799887    4921 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:31:26.810931 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799892    4921 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:31:26.810931 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799895    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:31:26.810931 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799897    4921 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:31:26.810931 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799900    4921 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:31:26.810931 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799903    4921 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:31:26.810931 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799906    4921 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:31:26.810931 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799908    4921 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:31:26.810931 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799911    4921 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:31:26.810931 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799915    4921 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:31:26.810931 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799920    4921 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:31:26.810931 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799923    4921 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:31:26.810931 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799926    4921 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:31:26.810931 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799930    4921 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:31:26.810931 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799933    4921 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:31:26.810931 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799936    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:31:26.811474 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799938    4921 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:31:26.811474 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799942    4921 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:31:26.811474 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799945    4921 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:31:26.811474 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799947    4921 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:31:26.811474 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799951    4921 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:31:26.811474 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799953    4921 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:31:26.811474 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799956    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:31:26.811474 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799959    4921 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:31:26.811474 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799961    4921 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:31:26.811474 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799964    4921 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:31:26.811474 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799967    4921 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:31:26.811474 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799970    4921 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:31:26.811474 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799972    4921 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:31:26.811474 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799975    4921 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:31:26.811474 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799978    4921 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:31:26.811474 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799980    4921 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:31:26.811474 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799983    4921 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:31:26.811474 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799987    4921 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:31:26.811474 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799990    4921 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:31:26.811474 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799993    4921 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:31:26.812013 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799995    4921 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:31:26.812013 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.799998    4921 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:31:26.812013 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800000    4921 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:31:26.812013 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800003    4921 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:31:26.812013 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800006    4921 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:31:26.812013 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800009    4921 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:31:26.812013 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800013    4921 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:31:26.812013 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800016    4921 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:31:26.812013 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800019    4921 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:31:26.812013 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800023    4921 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:31:26.812013 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800025    4921 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:31:26.812013 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800028    4921 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:31:26.812013 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.800064    4921 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 19:31:26.812013 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800101    4921 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:31:26.812013 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800108    4921 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:31:26.812384 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800111    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:31:26.812384 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800114    4921 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:31:26.812384 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800117    4921 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:31:26.812384 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800119    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:31:26.812384 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800122    4921 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:31:26.812384 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800125    4921 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:31:26.812384 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800128    4921 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:31:26.812384 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800130    4921 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:31:26.812384 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800133    4921 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:31:26.812384 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800136    4921 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:31:26.812384 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800140    4921 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:31:26.812384 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800143    4921 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:31:26.812384 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800146    4921 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:31:26.812384 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800149    4921 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:31:26.812384 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800154    4921 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:31:26.812384 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800157    4921 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:31:26.812384 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800159    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:31:26.812384 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800162    4921 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:31:26.812384 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800165    4921 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:31:26.812987 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800167    4921 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:31:26.812987 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800170    4921 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:31:26.812987 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800173    4921 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:31:26.812987 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800176    4921 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:31:26.812987 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800178    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:31:26.812987 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800181    4921 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:31:26.812987 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800184    4921 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:31:26.812987 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800186    4921 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:31:26.812987 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800189    4921 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:31:26.812987 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800191    4921 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:31:26.812987 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800194    4921 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:31:26.812987 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800197    4921 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:31:26.812987 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800202    4921 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:31:26.812987 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800205    4921 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:31:26.812987 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800208    4921 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:31:26.812987 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800210    4921 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:31:26.812987 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800213    4921 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:31:26.812987 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800216    4921 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:31:26.812987 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800218    4921 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:31:26.813508 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800221    4921 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:31:26.813508 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800223    4921 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:31:26.813508 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800226    4921 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:31:26.813508 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800229    4921 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:31:26.813508 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800231    4921 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:31:26.813508 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800235    4921 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:31:26.813508 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800239    4921 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:31:26.813508 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800242    4921 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:31:26.813508 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800247    4921 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:31:26.813508 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800249    4921 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:31:26.813508 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800252    4921 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:31:26.813508 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800255    4921 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:31:26.813508 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800257    4921 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:31:26.813508 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800261    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:31:26.813508 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800263    4921 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:31:26.813508 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800266    4921 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:31:26.813508 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800268    4921 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:31:26.813508 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800271    4921 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:31:26.813508 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800273    4921 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:31:26.813508 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800276    4921 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:31:26.814109 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800279    4921 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:31:26.814109 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800282    4921 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:31:26.814109 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800284    4921 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:31:26.814109 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800287    4921 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:31:26.814109 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800290    4921 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:31:26.814109 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800294    4921 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:31:26.814109 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800297    4921 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:31:26.814109 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800300    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:31:26.814109 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800302    4921 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:31:26.814109 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800305    4921 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:31:26.814109 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800308    4921 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:31:26.814109 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800311    4921 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:31:26.814109 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800313    4921 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:31:26.814109 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800316    4921 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:31:26.814109 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800319    4921 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:31:26.814109 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800321    4921 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:31:26.814109 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800324    4921 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:31:26.814109 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800327    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:31:26.814109 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800329    4921 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:31:26.814562 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800332    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:31:26.814562 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800336    4921 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:31:26.814562 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800339    4921 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:31:26.814562 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800342    4921 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:31:26.814562 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800345    4921 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:31:26.814562 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800348    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:31:26.814562 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800350    4921 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:31:26.814562 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800353    4921 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:31:26.814562 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800355    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:31:26.814562 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800358    4921 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:31:26.814562 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800360    4921 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:31:26.814562 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800363    4921 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:31:26.814562 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800366    4921 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:31:26.814562 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800368    4921 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:31:26.814562 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800371    4921 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:31:26.814562 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800373    4921 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:31:26.814562 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800376    4921 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:31:26.814562 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800378    4921 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:31:26.814562 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800383    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:31:26.814562 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800386    4921 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:31:26.815062 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800389    4921 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:31:26.815062 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800392    4921 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:31:26.815062 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800394    4921 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:31:26.815062 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800397    4921 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:31:26.815062 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800400    4921 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:31:26.815062 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800402    4921 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:31:26.815062 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.800405    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:31:26.815062 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.804485    4921 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 19:31:26.815062 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.804507    4921 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 19:31:26.815062 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.804532    4921 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 19:31:26.815062 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.804539    4921 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 19:31:26.815062 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.804737    4921 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:31:26.815062 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.804746    4921 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:31:26.815062 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.804754    4921 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:31:26.815062 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.804759    4921 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:31:26.815062 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.804764    4921 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:31:26.815062 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.804775    4921 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:31:26.815062 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.804780    4921 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:31:26.815062 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.804784    4921 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:31:26.815062 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.804789    4921 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:31:26.815551 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.804794    4921 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:31:26.815551 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.804798    4921 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:31:26.815551 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.804803    4921 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:31:26.815551 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.804807    4921 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:31:26.815551 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.804812    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:31:26.815551 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.804817    4921 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:31:26.815551 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.804821    4921 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:31:26.815551 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.804825    4921 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:31:26.815551 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.804829    4921 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:31:26.815551 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.804839    4921 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:31:26.815551 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.804843    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:31:26.815551 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.804848    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:31:26.815551 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.804853    4921 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:31:26.815551 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.804858    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:31:26.815551 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.804863    4921 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:31:26.815551 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.804868    4921 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:31:26.815551 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.804873    4921 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:31:26.815551 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.804878    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:31:26.815551 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.804883    4921 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:31:26.815551 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.804887    4921 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:31:26.816080 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.804891    4921 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:31:26.816080 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.804901    4921 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:31:26.816080 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.804905    4921 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:31:26.816080 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.804909    4921 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:31:26.816080 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.804913    4921 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:31:26.816080 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.804918    4921 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:31:26.816080 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.804922    4921 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:31:26.816080 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.804927    4921 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:31:26.816080 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.804931    4921 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:31:26.816080 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.804936    4921 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:31:26.816080 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.804940    4921 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:31:26.816080 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.804946    4921 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:31:26.816080 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.804951    4921 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:31:26.816080 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.804955    4921 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:31:26.816080 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.804964    4921 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:31:26.816080 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.804969    4921 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:31:26.816080 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.804973    4921 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:31:26.816080 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.804977    4921 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:31:26.816080 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.804982    4921 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:31:26.816080 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.804986    4921 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:31:26.816560 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.804991    4921 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:31:26.816560 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.804996    4921 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:31:26.816560 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805000    4921 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:31:26.816560 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805004    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:31:26.816560 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805008    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:31:26.816560 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805013    4921 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:31:26.816560 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805023    4921 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:31:26.816560 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805027    4921 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:31:26.816560 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805032    4921 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:31:26.816560 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805036    4921 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:31:26.816560 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805040    4921 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:31:26.816560 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805045    4921 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:31:26.816560 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805049    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:31:26.816560 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805053    4921 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:31:26.816560 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805057    4921 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:31:26.816560 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805062    4921 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:31:26.816560 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805066    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:31:26.816560 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805071    4921 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:31:26.816560 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805075    4921 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:31:26.816560 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805083    4921 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:31:26.817071 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805088    4921 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:31:26.817071 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805092    4921 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:31:26.817071 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805096    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:31:26.817071 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805101    4921 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:31:26.817071 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805111    4921 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:31:26.817071 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805116    4921 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:31:26.817071 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805120    4921 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:31:26.817071 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805124    4921 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:31:26.817071 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805131    4921 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:31:26.817071 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805138    4921 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:31:26.817071 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805143    4921 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:31:26.817071 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805148    4921 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:31:26.817071 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805158    4921 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:31:26.817071 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805163    4921 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:31:26.817071 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805167    4921 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:31:26.817071 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805171    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:31:26.817071 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805176    4921 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:31:26.817071 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805180    4921 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:31:26.817071 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805185    4921 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:31:26.817549 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805190    4921 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:31:26.817549 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805195    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:31:26.817549 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805199    4921 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:31:26.817549 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805204    4921 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:31:26.817549 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805208    4921 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:31:26.817549 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805257    4921 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:31:26.817549 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805261    4921 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:31:26.817549 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805265    4921 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:31:26.817549 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805267    4921 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:31:26.817549 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805270    4921 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:31:26.817549 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805273    4921 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:31:26.817549 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805276    4921 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:31:26.817549 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805295    4921 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:31:26.817549 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805298    4921 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:31:26.817549 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805313    4921 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:31:26.817549 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805316    4921 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:31:26.817549 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805431    4921 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:31:26.817549 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805436    4921 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:31:26.818166 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.805479    4921 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 19:31:26.818166 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805664    4921 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:31:26.818166 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805687    4921 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:31:26.818166 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805691    4921 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:31:26.818166 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805695    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:31:26.818166 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805698    4921 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:31:26.818166 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805700    4921 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:31:26.818166 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805704    4921 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:31:26.818166 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805706    4921 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:31:26.818166 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805710    4921 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:31:26.818166 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805712    4921 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:31:26.818166 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805715    4921 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:31:26.818166 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805718    4921 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:31:26.818166 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805721    4921 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:31:26.818166 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805723    4921 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:31:26.818166 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805727    4921 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:31:26.818554 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805730    4921 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:31:26.818554 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805732    4921 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:31:26.818554 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805735    4921 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:31:26.818554 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805737    4921 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:31:26.818554 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805740    4921 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:31:26.818554 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805743    4921 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:31:26.818554 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805745    4921 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:31:26.818554 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805748    4921 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:31:26.818554 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805751    4921 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:31:26.818554 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805753    4921 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:31:26.818554 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805756    4921 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:31:26.818554 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805759    4921 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:31:26.818554 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805762    4921 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:31:26.818554 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805764    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:31:26.818554 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805767    4921 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:31:26.818554 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805769    4921 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:31:26.818554 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805772    4921 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:31:26.818554 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805775    4921 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:31:26.818554 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805778    4921 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:31:26.818554 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805780    4921 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:31:26.819081 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805784    4921 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:31:26.819081 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805789    4921 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:31:26.819081 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805793    4921 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:31:26.819081 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805797    4921 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:31:26.819081 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805800    4921 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:31:26.819081 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805803    4921 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:31:26.819081 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805806    4921 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:31:26.819081 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805809    4921 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:31:26.819081 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805811    4921 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:31:26.819081 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805814    4921 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:31:26.819081 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805817    4921 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:31:26.819081 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805819    4921 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:31:26.819081 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805822    4921 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:31:26.819081 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805825    4921 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:31:26.819081 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805827    4921 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:31:26.819081 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805830    4921 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:31:26.819081 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805833    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:31:26.819081 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805835    4921 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:31:26.819081 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805838    4921 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:31:26.819081 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805840    4921 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:31:26.819561 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805843    4921 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:31:26.819561 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805846    4921 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:31:26.819561 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805848    4921 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:31:26.819561 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805851    4921 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:31:26.819561 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805854    4921 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:31:26.819561 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805857    4921 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:31:26.819561 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805859    4921 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:31:26.819561 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805862    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:31:26.819561 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805865    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:31:26.819561 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805868    4921 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:31:26.819561 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805870    4921 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:31:26.819561 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805873    4921 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:31:26.819561 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805875    4921 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:31:26.819561 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805878    4921 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:31:26.819561 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805881    4921 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:31:26.819561 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805883    4921 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:31:26.819561 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805886    4921 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:31:26.819561 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805889    4921 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:31:26.819561 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805892    4921 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:31:26.820064 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805897    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:31:26.820064 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805899    4921 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:31:26.820064 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805902    4921 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:31:26.820064 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805904    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:31:26.820064 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805907    4921 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:31:26.820064 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805910    4921 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:31:26.820064 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805912    4921 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:31:26.820064 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805915    4921 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:31:26.820064 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805918    4921 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:31:26.820064 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805921    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:31:26.820064 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805923    4921 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:31:26.820064 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805926    4921 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:31:26.820064 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805929    4921 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:31:26.820064 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805932    4921 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:31:26.820064 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805934    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:31:26.820064 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805937    4921 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:31:26.820064 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805940    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:31:26.820064 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805942    4921 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:31:26.820064 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805945    4921 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:31:26.820064 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805947    4921 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:31:26.820629 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805950    4921 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:31:26.820629 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805953    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:31:26.820629 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805955    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:31:26.820629 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805958    4921 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:31:26.820629 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805960    4921 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:31:26.820629 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805964    4921 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:31:26.820629 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805968    4921 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:31:26.820629 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805971    4921 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:31:26.820629 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805974    4921 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:31:26.820629 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805976    4921 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:31:26.820629 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805980    4921 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:31:26.820629 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.805983    4921 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:31:26.820629 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.806008    4921 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 19:31:26.820629 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806058    4921 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:31:26.820629 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806063    4921 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:31:26.820629 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806066    4921 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:31:26.821315 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806069    4921 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:31:26.821315 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806072    4921 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:31:26.821315 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806076    4921 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:31:26.821315 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806079    4921 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:31:26.821315 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806082    4921 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:31:26.821315 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806084    4921 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:31:26.821315 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806087    4921 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:31:26.821315 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806090    4921 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:31:26.821315 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806092    4921 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:31:26.821315 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806095    4921 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:31:26.821315 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806097    4921 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:31:26.821315 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806100    4921 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:31:26.821315 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806103    4921 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:31:26.821315 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806105    4921 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:31:26.821315 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806108    4921 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:31:26.821315 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806111    4921 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:31:26.821315 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806113    4921 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:31:26.821315 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806116    4921 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:31:26.821315 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806118    4921 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:31:26.822038 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806121    4921 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:31:26.822038 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806124    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:31:26.822038 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806126    4921 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:31:26.822038 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806129    4921 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:31:26.822038 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806132    4921 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:31:26.822038 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806134    4921 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:31:26.822038 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806137    4921 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:31:26.822038 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806140    4921 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:31:26.822038 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806145    4921 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:31:26.822038 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806152    4921 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:31:26.822038 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806156    4921 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:31:26.822038 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806160    4921 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:31:26.822038 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806163    4921 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:31:26.822038 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806165    4921 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:31:26.822038 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806168    4921 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:31:26.822038 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806171    4921 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:31:26.822038 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806173    4921 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:31:26.822038 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806176    4921 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:31:26.822038 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806179    4921 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:31:26.822038 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806182    4921 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:31:26.822528 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806184    4921 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:31:26.822528 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806187    4921 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:31:26.822528 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806190    4921 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:31:26.822528 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806192    4921 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:31:26.822528 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806195    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:31:26.822528 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806198    4921 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:31:26.822528 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806200    4921 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:31:26.822528 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806203    4921 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:31:26.822528 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806206    4921 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:31:26.822528 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806208    4921 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:31:26.822528 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806211    4921 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:31:26.822528 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806213    4921 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:31:26.822528 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806216    4921 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:31:26.822528 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806219    4921 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:31:26.822528 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806221    4921 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:31:26.822528 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806224    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:31:26.822528 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806227    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:31:26.822528 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806229    4921 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:31:26.822528 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806232    4921 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:31:26.823102 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806234    4921 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:31:26.823102 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806237    4921 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:31:26.823102 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806242    4921 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:31:26.823102 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806245    4921 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:31:26.823102 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806248    4921 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:31:26.823102 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806251    4921 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:31:26.823102 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806253    4921 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:31:26.823102 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806256    4921 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:31:26.823102 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806259    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:31:26.823102 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806262    4921 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:31:26.823102 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806264    4921 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:31:26.823102 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806267    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:31:26.823102 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806270    4921 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:31:26.823102 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806272    4921 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:31:26.823102 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806275    4921 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:31:26.823102 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806277    4921 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:31:26.823102 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806280    4921 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:31:26.823102 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806283    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:31:26.823102 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806286    4921 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:31:26.823560 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806289    4921 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:31:26.823560 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806292    4921 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:31:26.823560 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806294    4921 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:31:26.823560 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806297    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:31:26.823560 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806300    4921 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:31:26.823560 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806303    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:31:26.823560 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806305    4921 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:31:26.823560 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806308    4921 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:31:26.823560 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806310    4921 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:31:26.823560 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806313    4921 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:31:26.823560 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806315    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:31:26.823560 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806318    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:31:26.823560 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806321    4921 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:31:26.823560 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806323    4921 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:31:26.823560 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806327    4921 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:31:26.823560 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806332    4921 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:31:26.823560 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806336    4921 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:31:26.823560 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806339    4921 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:31:26.823560 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806342    4921 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:31:26.824036 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806345    4921 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:31:26.824036 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806348    4921 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:31:26.824036 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806351    4921 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:31:26.824036 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806353    4921 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:31:26.824036 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806356    4921 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:31:26.824036 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806359    4921 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:31:26.824036 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:26.806362    4921 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:31:26.824036 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.806421    4921 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 19:31:26.824036 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.807388    4921 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 03 19:31:26.824036 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.807440    4921 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 03 19:31:26.824036 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.807613    4921 server.go:1014] "Starting client certificate rotation"
Nov 03 19:31:26.824036 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.807766    4921 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 19:31:26.824036 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.807828    4921 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 19:25:43 +0000 UTC" deadline="2028-06-22 11:04:30.115708007 +0000 UTC"
Nov 03 19:31:26.824036 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.807866    4921 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14319h33m3.307845239s"
Nov 03 19:31:26.824036 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.808167    4921 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 19:31:26.824036 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.809320    4921 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 19:31:26.824465 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.810820    4921 log.go:25] "Validated CRI v1 runtime API"
Nov 03 19:31:26.824465 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.812443    4921 log.go:25] "Validated CRI v1 image API"
Nov 03 19:31:26.824465 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.813220    4921 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 19:31:26.824465 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.815120    4921 fs.go:135] Filesystem UUIDs: map[2ac96c12-af93-455d-b27a-5d1a5d26e1de:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 a137b2b5-e435-49c9-b1c4-e835406e68cf:/dev/nvme0n1p3]
Nov 03 19:31:26.824609 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.815141    4921 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/02550913e8d27c7c8560a212e5bd9d530b50751200373c6fac7af5b19bc5eba6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/02550913e8d27c7c8560a212e5bd9d530b50751200373c6fac7af5b19bc5eba6/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/03c21a45f12598b3509569e9e31fe33175fe0b2949aa92bf67867698de18dbce/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/03c21a45f12598b3509569e9e31fe33175fe0b2949aa92bf67867698de18dbce/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0ed2a89a99f0a7292450791623764b1318e836e96f0da6d2b1da8e710c76aff0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0ed2a89a99f0a7292450791623764b1318e836e96f0da6d2b1da8e710c76aff0/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/24683f3bceaa61bf9e0e4d275b35e9a826b29dfed6d50b5f0ca3e5f14ddb40fa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/24683f3bceaa61bf9e0e4d275b35e9a826b29dfed6d50b5f0ca3e5f14ddb40fa/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/342234a86ae594f8fc680a4b71c5d32cc74809d11193bde259907c98dbdfbe63/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/342234a86ae594f8fc680a4b71c5d32cc74809d11193bde259907c98dbdfbe63/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/417924c6ce13224daa5404d770644a58f2de02ee1c63dc259560510c12e4928c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/417924c6ce13224daa5404d770644a58f2de02ee1c63dc259560510c12e4928c/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4c245f17b0c21d5f74bf92645ee2ed5737f7f55f6403eb89bac16ac2f44dd547/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4c245f17b0c21d5f74bf92645ee2ed5737f7f55f6403eb89bac16ac2f44dd547/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5a72890122c2ad33df5c500a2d864911fa62f753e832a2223fec4e3a9c675e44/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5a72890122c2ad33df5c500a2d864911fa62f753e832a2223fec4e3a9c675e44/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/614fce5f50618860328a6514e4a8ed906b17132e6d53aadfb45f8e50b6ebb3a7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/614fce5f50618860328a6514e4a8ed906b17132e6d53aadfb45f8e50b6ebb3a7/userdata/shm major:0 minor:302 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6c8b4e67220dd8eff5e0a5c70f9e73180d809bb968c1192c2bdefd5ecc7f6800/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6c8b4e67220dd8eff5e0a5c70f9e73180d809bb968c1192c2bdefd5ecc7f6800/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/760afcba9b261afe12f20cdbd1135d1967906cf78e08b9a5f83d27736a8afd7f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/760afcba9b261afe12f20cdbd1135d1967906cf78e08b9a5f83d27736a8afd7f/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8956f25a94b043ba181c79bc664f46b9ec9492e4feead98a59cff77e662a4726/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8956f25a94b043ba181c79bc664f46b9ec9492e4feead98a59cff77e662a4726/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/97697af028d0d1433e902ba6ee03eddadbd380fa856a9761702510919ef2426d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/97697af028d0d1433e902ba6ee03eddadbd380fa856a9761702510919ef2426d/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9fb06094152142396f464199b328c14b14771278ba0d9d040daab45a8693dcd2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9fb06094152142396f464199b328c14b14771278ba0d9d040daab45a8693dcd2/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c058fb511ce4a4026f63bb8f9a0cbc4eccf8ee9fea0ed0ed8993337b4a9e8166/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c058fb511ce4a4026f63bb8f9a0cbc4eccf8ee9fea0ed0ed8993337b4a9e8166/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cee9ee0075ad7fe8f9687cb833f49ea3f9a48bdeb6e0212e761a8acf0b06d211/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cee9ee0075ad7fe8f9687cb833f49ea3f9a48bdeb6e0212e761a8acf0b06d211/userdata/shm major:0 minor:245 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/df61d6453bb2b7ff05aa8df94e63ef5f6d6616ebd51be5be599aa73f996b191d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/df61d6453bb2b7ff05aa8df94e63ef5f6d6616ebd51be5be599aa73f996b191d/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e49beb36db5058ff6aafac0435a308b909777900afdd978ebec852e8d0fe35e2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e49beb36db5058ff6aafac0435a308b909777900afdd978ebec852e8d0fe35e2/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e666c09aad34a256678922fa6e602d5132bb9d5e8d6c6b094083ed27b1f676ce/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e666c09aad34a256678922fa6e602d5132bb9d5e8d6c6b094083ed27b1f676ce/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e9319b6d883d1db6f7e71643f2a97b1f810903966a53f2ee00efe32b8cc288b2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e9319b6d883d1db6f7e71643f2a97b1f810903966a53f2ee00efe32b8cc288b2/userdata/shm major:0 minor:300 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e997c0a1e8ead87756bf6ba3e07e296f5523e84fa98d10255af27065d299ac79/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e997c0a1e8ead87756bf6ba3e07e296f5523e84fa98d10255af27065d299ac79/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f8ab706e01419505523c810aac56fba7d29a8c4703c9b0f23565593f44ef80a7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f8ab706e01419505523c810aac56fba7d29a8c4703c9b0f23565593f44ef80a7/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fe3b22311179a16c8c3eeab442b74b9b6046baa06d8bafb32318503a0d2e7908/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fe3b22311179a16c8c3eeab442b74b9b6046baa06d8bafb32318503a0d2e7908/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25/volumes/kubernetes.io~empty-dir/tmp major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25/volumes/kubernetes.io~projected/kube-api-access-w5pj6:{mountpoint:/var/lib/kubelet/pods/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25/volumes/kubernetes.io~projected/kube-api-access-w5pj6 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/10137ba2-4beb-433e-b6df-199738351c61/volumes/kubernetes.io~projected/kube-api-access-2s4qj:{mountpoint:/var/lib/kubelet/pods/10137ba2-4beb-433e-b6df-199738351c61/volumes/kubernetes.io~projected/kube-api-access-2s4qj major:0 minor:329 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1611b730-6163-4776-bd2d-3d10c2cc9a88/volumes/kubernetes.io~projected/kube-api-access-2287d:{mountpoint:/var/lib/kubelet/pods/1611b730-6163-4776-bd2d-3d10c2cc9a88/volumes/kubernetes.io~projected/kube-api-access-2287d major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1753fd31-8933-4567-8146-07f3fd2d10a5/volumes/kubernetes.io~projected/kube-api-access-6n7xj:{mountpoint:/var/lib/kubelet/pods/1753fd31-8933-4567-8146-07f3fd2d10a5/volumes/kubernetes.io~projected/kube-api-access-6n7xj major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1c00e787-a35d-4ac4-95a4-2546e29602aa/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/1c00e787-a35d-4ac4-95a4-2546e29602aa/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/220ed31c-1577-4c4b-ab5b-e812fd63d193/volumes/kubernetes.io~projected/kube-api-access-5lgzq:{mountpoint:/var/lib/kubelet/pods/220ed31c-1577-4c4b-ab5b-e812fd63d193/volumes/kubernetes.io~projected/kube-api-access-5lgzq major:0 minor:299 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3880f3d3-aad1-43c0-b704-01374aad4f78/volumes/kubernetes.io~projected/kube-api-access-8mh9g:{mountpoint:/var/lib/kubelet/pods/3880f3d3-aad1-43c0-b704-01374aad4f78/volumes/kubernetes.io~projected/kube-api-access-8mh9g major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3880f3d3-aad1-43c0-b704-01374aad4f78/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/3880f3d3-aad1-43c0-b704-01374aad4f78/volumes/kubernetes.io~secret/serving-cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/38a6add1-b051-450b-bb96-8ace8a0ccff3/volumes/kubernetes.io~projected/kube-api-access-ngpxm:{mountpoint:/var/lib/kubelet/pods/38a6add1-b051-450b-bb96-8ace8a0ccff3/volumes/kubernetes.io~projected/kube-api-access-ngpxm major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/543b8778-baa4-4e09-b697-68e550e42942/volumes/kubernetes.io~projected/kube-api-access-gj2lf:{mountpoint:/var/lib/kubelet/pods/543b8778-baa4-4e09-b697-68e550e42942/volumes/kubernetes.io~projected/kube-api-access-gj2lf major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/543b8778-baa4-4e09-b697-68e550e42942/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/543b8778-baa4-4e09-b697-68e550e42942/volumes/kubernetes.io~secret/cert major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/545f3c9a-2ac6-44a3-be0b-cf71b1592d42/volumes/kubernetes.io~projected/kube-api-access-55hqg:{mountpoint:/var/lib/kubelet/pods/545f3c9a-2ac6-44a3-be0b-cf71b1592d42/volumes/kubernetes.io~projected/kube-api-access-55hqg major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/56e21184-e568-46bc-8e90-204684d6c366/volumes/kubernetes.io~projected/kube-api-access-6wk2r:{mountpoint:/var/lib/kubelet/pods/56e21184-e568-46bc-8e90-204684d6c366/volumes/kubernetes.io~projected/kube-api-access-6wk2r major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/653db542-8702-4eaa-abc1-a3bb866163c6/volumes/kubernetes.io~projected/kube-api-access-mpwxv:{mountpoint:/var/lib/kubelet/pods/653db542-8702-4eaa-abc1-a3bb866163c6/volumes/kubernetes.io~projected/kube-api-access-mpwxv major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6f215b42-0f09-46a0-bc98-392c0fa909c9/volumes/kubernetes.io~projected/kube-api-access-qjtq4:{mountpoint:/var/lib/kubelet/pods/6f215b42-0f09-46a0-bc98-392c0fa909c9/volumes/kubernetes.io~projected/kube-api-access-qjtq4 major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6f215b42-0f09-46a0-bc98-392c0fa909c9/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/6f215b42-0f09-46a0-bc98-392c0fa909c9/volumes/kubernetes.io~secret/serving-cert major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/790607e1-d57c-4622-9a72-1af902ab1d85/volumes/kubernetes.io~projected/kube-api-access-hgtvt:{mountpoint:/var/lib/kubelet/pods/790607e1-d57c-4622-9a72-1af902ab1d85/volumes/kubernetes.io~projected/kube-api-access-hgtvt major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8b5e1b09-699d-4db8-9eb0-0b71bdda858f/volumes/kubernetes.io~projected/kube-api-access-2g8fh:{mountpoint:/var/lib/kubelet/pods/8b5e1b09-699d-4db8-9eb0-0b71bdda858f/volumes/kubernetes.io~projected/kube-api-access-2g8fh major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce/volumes/kubernetes.io~projected/kube-api-access-mtt8c:{mountpoint:/var/lib/kubelet/pods/8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce/volumes/kubernetes.io~projected/kube-api-access-mtt8c major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8fa315db-e48d-4ac5-960a-23750974d431/volumes/kubernetes.io~projected/kube-api-access-jtqhk:{mountpoint:/var/lib/kubelet/pods/8fa315db-e48d-4ac5-960a-23750974d431/volumes/kubernetes.io~projected/kube-api-access-jtqhk major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/96c016f7-9098-4a39-9e72-e54da374d8fc/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/96c016f7-9098-4a39-9e72-e54da374d8fc/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/96c016f7-9098-4a39-9e72-e54da374d8fc/volumes/kubernetes.io~projected/kube-api-access-4w42k:{mountpoint:/var/lib/kubelet/pods/96c016f7-9098-4a39-9e72-e54da374d8fc/volumes/kubernetes.io~projected/kube-api-access-4w42k major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/96c016f7-9098-4a39-9e72-e54da374d8fc/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/96c016f7-9098-4a39-9e72-e54da374d8fc/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/96c016f7-9098-4a39-9e72-e54da374d8fc/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/96c016f7-9098-4a39-9e72-e54da374d8fc/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9fb8c33c-fe15-4daa-a01b-4cb2fc7eae7b/volumes/kubernetes.io~projected/kube-api-access-q68w5:{mountpoint:/var/lib/kubelet/pods/9fb8c33c-fe15-4daa-a01b-4cb2fc7eae7b/volumes/kubernetes.io~projected/kube-api-access-q68w5 major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9fb8c33c-fe15-4daa-a01b-4cb2fc7eae7b/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/9fb8c33c-fe15-4daa-a01b-4cb2fc7eae7b/volumes/kubernetes.io~secret/serving-cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a023911d-3669-43e1-ac77-8a986f3715d6/volumes/kubernetes.io~projected/kube-api-access-ftr5x:{mountpoint:/var/lib/kubelet/pods/a023911d-3669-43e1-ac77-8a986f3715d6/volumes/kubernetes.io~projected/kube-api-access-ftr5x major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a023911d-3669-43e1-ac77-8a986f3715d6/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/a023911d-3669-43e1-ac77-8a986f3715d6/volumes/kubernetes.io~secret/default-certificate major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a023911d-3669-43e1-ac77-8a986f3715d6/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/a023911d-3669-43e1-ac77-8a986f3715d6/volumes/kubernetes.io~secret/stats-auth major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b46f51b2-436b-4967-8879-a16b58930591/volumes/kubernetes.io~projected/kube-api-access-f75xn:{mountpoint:/var/lib/kubelet/pods/b46f51b2-436b-4967-8879-a16b58930591/volumes/kubernetes.io~projected/kube-api-access-f75xn major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b797bdb0-b354-4dfc-a50a-3c60ecacbf71/volumes/kubernetes.io~projected/kube-api-access-xr684:{mountpoint:/var/lib/kubelet/pods/b797bdb0-b354-4dfc-a50a-3c60ecacbf71/volumes/kubernetes.io~projected/kube-api-access-xr684 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d0ab7c64-a85f-4713-a270-f0cf711fcee5/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/d0ab7c64-a85f-4713-a270-f0cf711fcee5/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d0ab7c64-a85f-4713-a270-f0cf711fcee5/volumes/kubernetes.io~projected/kube-api-access-jc8v6:{mountpoint:/var/lib/kubelet/pods/d0ab7c64-a85f-4713-a270-f0cf711fcee5/volumes/kubernetes.io~projected/kube-api-access-jc8v6 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d0ab7c64-a85f-4713-a270-f0cf711fcee5/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/d0ab7c64-a85f-4713-a270-f0cf711fcee5/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca/volumes/kubernetes.io~projected/kube-api-access-8gm62:{mountpoint:/var/lib/kubelet/pods/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca/volumes/kubernetes.io~projected/kube-api-access-8gm62 major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/df9f9c4a-d054-42cf-b17c-0d23acd0459b/volumes/kubernetes.io~projected/kube-api-access-ktkfm:{mountpoint:/var/lib/kubelet/pods/df9f9c4a-d054-42cf-b17c-0d23acd0459b/volumes/kubernetes.io~projected/kube-api-access-ktkfm major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e10dc749-bc71-456e-a315-6581add01a7b/volumes/kubernetes.io~projected/kube-api-access-nfg8j:{mountpoint:/var/lib/kubelet/pods/e10dc749-bc71-456e-a315-6581add01a7b/volumes/kubernetes.io~projected/kube-api-access-nfg8j major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e10dc749-bc71-456e-a315-6581add01a7b/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/e10dc749-bc71-456e-a315-6581add01a7b/volumes/kubernetes.io~secret/serving-cert major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e2b752d9-88c4-4dee-8b0c-7e9ac252adb3/volumes/kubernetes.io~projected/kube-api-access-xz8xk:{mountpoint:/var/lib/kubelet/pods/e2b752d9-88c4-4dee-8b0c-7e9ac252adb3/volumes/kubernetes.io~projected/kube-api-access-xz8xk major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eee821c4-6fb0-4f84-a58d-23dc2ff77720/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/eee821c4-6fb0-4f84-a58d-23dc2ff77720/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f89ef12d-9c60-4538-af9c-f97453bbc855/volumes/kubernetes.io~projected/kube-api-access-xdw57:{mountpoint:/var/lib/kubelet/pods/f89ef12d-9c60-4538-af9c-f97453bbc855/volumes/kubernetes.io~projected/kube-api-access-xdw57 major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f97b063e-8403-46bd-9716-72411589a494/volumes/kubernetes.io~projected/kube-api-access-p288b:{mountpoint:/var/lib/kubelet/pods/f97b063e-8403-46bd-9716-72411589a494/volumes/kubernetes.io~projected/kube-api-access-p288b major:0 minor:223 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/c8d0acd47331a7e547b5d3794c460dfc66c832b256d88dbc8ce29a7dc5437ac4/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/5a04e110f89571b0eaf4d643278b40c4c24067cffc465fe8c73b8167c1b49cdc/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/dd1a7eb722ab67fbd93b600daedb0caf076fcaaeb8343cd027d5b1a5e6459a24/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/19d1eb283a42bf67b4c11aae8b48f0b18f4e9381e4682929362b123ce3804f33/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/2f203b81b44083ec154436c7dab13fc113a92a1f995ab092e0341b7099fcff55/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/cb878f3278e851c6a15853e5830fb3120a9ee3b9ab887e6be901bd5daff062a6/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/b69d8d1b73881e4405aa209bead42024edf3867bc833451b58f93cac2d10e342/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/45e29123abcd843a08881c68689fe1441501e4aef739deac18d635a3a9d6f79d/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/49d53c96c40a91b589f7d229dc13b569cfd49ede2aea5f1fa104127fa24ff52b/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/2dd7afe3f5c33c26c334c50e2f05677a776ce8fc91b3f69d8dd6d068bae62fff/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/7aa83e253285cc3854cbc4df60577249f1dc9b13a09f7e9091c3e4f97dea53e2/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/80420ed67c78c9747b080fa32d12b2e3957466a41e641e127c4b80cf44868165/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/14861f5d03cf9c7523011c8abe4575683b5f2a865767676857e791d9ea8fbd5b/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-159:{mountpoint:/var/lib/containers/storage/overlay/6e00dc62da543aa8bc6465c387b7788eafde6b19be0d6740d2f35d2583375ffa/merged major:0 minor:159 fsType:overlay blockSize:0} overlay_0-164:{mountpoint:/var/lib/containers/storage/overlay/187c0c98b9a059de2c687ee8f888d9dce21f61e924428f683a54289ecec64a4b/merged major:0 minor:164 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/d0e4f67e879b1722cf3ee5d51482bd7b61d374a04e1ddfd0559a5b1978400c87/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-174:{mountpoint:/var/lib/containers/storage/overlay/67c5c9db0e3bc61f0438df316eaa7843f2c57bbac8eeed11c4b6e9e196ea2694/merged major:0 minor:174 fsType:overlay blockSize:0} overlay_0-179:{mountpoint:/var/lib/containers/storage/overlay/a19e15fdbd66cdf261a4a3f7729a0c8fc5e7dd6a69f84334278be28b73745088/merged major:0 minor:179 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/56cb92df91c8d1c5e6ebc44e7e7a7672aa50a393f308782c074e200da6d1ceca/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/d5db1c1f95bfbb23477b8a73a0c0e5b4280eda897283c4a26e90f7c423920582/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/0e5baebaaa2fc7eb1d466d35e5b6291b685d77586fe7392003ef35cc3f3c25c7/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/7e980cbdfb54ba4ec815628da896228c282c268dae43a10493c3460896d5b944/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/18fdbd1827c128432e035ed030005ed43c40f249cba64448408a91fb8d77e756/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/7eed4afc5948671b821b15583e16e5524265a29ca37ecac5170aae3792562392/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/1762cbf45d4176a97d1c778d2a31e8f95d29f95ed5e4cbd79802fdfb11e98ea5/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/e6d971024dd6c622fd935fb577ee0c0947c8bd8fd1c499de3b41b8684c2744ff/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/f3702f1f4a30c1fd3c9ef210177569e2e2d2b175bdab80cfac1796b73844c4cc/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/7518252c373875fde2109439b17eedc1478db834d656a534c345b1c28cd10b72/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/e4982d562ebbc8cf29fadd81fa3e3a7480ffc9283cf65094931e8053fb34ee94/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/99a176a9280071ff5303b0d4255ab29632a9a9cfdb6364fe9c36455a58fd41ac/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-269:{mountpoint:/var/lib/containers/storage/overlay/53c11e6234e0c927643c1366c3a2aff8f42f4d79fc076d32295d46e4c0c895d5/merged major:0 minor:269 fsType:overlay blockSize:0} overlay_0-271:{mountpoint:/var/lib/containers/storage/overlay/0cce023611186cc133e466973116926b02db8edb9468fefbc1a6fa7ae11e8cc7/merged major:0 minor:271 fsType:overlay blockSize:0} overlay_0-273:{mountpoint:/var/lib/containers/storage/overlay/d5ae505931a940fa895c5bba74abbd09ccea45707bbb051dd10cba18911a1916/merged major:0 minor:273 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/0369eebc31266b0431ce6b699dcba79f21032caec6e468f0c54498222cbc9a65/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-285:{mountpoint:/var/lib/containers/storage/overlay/6b9bb64f34af3679ad7f022c82d05a8b6f1e300aa17d2251c25a731d50803d93/merged major:0 minor:285 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/6e8935bef85a27f7cf8c53f8ffa62fd5e2f273475da3ccc8b17dbe340a9f8ae5/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-297:{mountpoint:/var/lib/containers/storage/overlay/04c72b699ceb7bf52b521f49fbf436e1f5c25e3bef0530050d7b54b8d39868ba/merged major:0 minor:297 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/012a74ad03cf192ddc7fede24f2348f5e96a84d72b1e7fd2cabbd6a58ce72008/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-307:{mountpoint:/var/lib/containers/storage/overlay/63710551b72ff9c73f2bbe581f2ca011c323ffe5d894c338a5720b731a06eca8/merged major:0 minor:307 fsType:overlay blockSize:0} overlay_0-309:{mountpoint:/var/lib/containers/storage/overlay/e1353f5a26786ce9635ecf1a3b516c2eb024bf69db1ad023a1219229b7d2d874/merged major:0 minor:309 fsType:overlay blockSize:0} overlay_0-311:{mountpoint:/var/lib/containers/storage/overlay/6e9b43a8f58934fe141aabb3ee38751dc766ad3c00003f13644f71d27dcf3175/merged major:0 minor:311 fsType:overlay blockSize:0} overlay_0-325:{mountpoint:/var/lib/containers/storage/overlay/45b7c841ed3c4e900596a09dc48f9335b8f03b3144ff5085d1c74d1c423bd96b/merged major:0 minor:325 fsType:overlay blockSize:0} overlay_0-327:{mountpoint:/var/lib/containers/storage/overlay/500f88ae6115d8f86dfd3031fe726b062a93b6b281b12fc2206b45d59df47f0b/merged major:0 minor:327 fsType:overlay blockSize:0} overlay_0-330:{mountpoint:/var/lib/containers/storage/overlay/dffcd955b5c4dc37f188e8dbe4a67ace108858d6dd25d39c08f20cc7a2d5ac23/merged major:0 minor:330 fsType:overlay blockSize:0} overlay_0-340:{mountpoint:/var/lib/containers/storage/overlay/8f741b1a980ac8acc8502932e57fb5e65226a3a8566deffa8b92fb9f3ec9f50c/merged major:0 minor:340 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/00159d703f85c9e5d2964ba7fb0f6f11d491354299c744abfea1793e53d6d84b/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/0ff8f860763b94c3d99f03e2de827f656d7aba44e398ac43f9a16b3eaa638bb9/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/03532adb958156d6efeb87fe0e8bd0e2aa5879ea8ac3f2aa71a0241e3fd13c52/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/90fecdd44ad4f3182cb8679329f0ad242c353a9a28bf75972d6e074eb609be28/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/690a9b226cc71e0e88ef5682c7bac05a2b248440235d0c6f97c537a48e41a119/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/c44ec668d4dea6e7201da20cd2fc4e3034b75d25bce17151190377a27f765789/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/59211196dbf9a4d1a8953f87a2ac0697a306c0c5981de8bdf81757522ecb8740/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/b518d760a2f8c0007272b5a7355aff973de16d5bd10f61d9ebe43b6c3b6620c1/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/427c4c9cc6f07de2e3ef2c0715c8096e99ae4d3dbe89f63b8e3576b3509a06c6/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/c1107b4b7ff98eae16f9af845f0d9bc5c2150e36518e0d9c8f63e7edf16db099/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/ff2d9e707386265b3ecdaa6e3bb4655a8394e340b96714467913cbc660d6c8a2/merged major:0 minor:96 fsType:overlay blockSize:0}]
Nov 03 19:31:26.824843 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.823815    4921 manager.go:217] Machine: {Timestamp:2026-11-03 19:31:26.823335379 +0000 UTC m=+0.067300835 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100694 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24b01d75922ceb6cac702efe3d6796 SystemUUID:ec24b01d-7592-2ceb-6cac-702efe3d6796 BootID:6155d873-3697-438a-a3c0-03814e53fc30 Filesystems:[{Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d0ab7c64-a85f-4713-a270-f0cf711fcee5/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8b5e1b09-699d-4db8-9eb0-0b71bdda858f/volumes/kubernetes.io~projected/kube-api-access-2g8fh DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e666c09aad34a256678922fa6e602d5132bb9d5e8d6c6b094083ed27b1f676ce/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:204 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1611b730-6163-4776-bd2d-3d10c2cc9a88/volumes/kubernetes.io~projected/kube-api-access-2287d DeviceMajor:0 DeviceMinor:225 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f89ef12d-9c60-4538-af9c-f97453bbc855/volumes/kubernetes.io~projected/kube-api-access-xdw57 DeviceMajor:0 DeviceMinor:227 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e997c0a1e8ead87756bf6ba3e07e296f5523e84fa98d10255af27065d299ac79/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/342234a86ae594f8fc680a4b71c5d32cc74809d11193bde259907c98dbdfbe63/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/56e21184-e568-46bc-8e90-204684d6c366/volumes/kubernetes.io~projected/kube-api-access-6wk2r DeviceMajor:0 DeviceMinor:61 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b797bdb0-b354-4dfc-a50a-3c60ecacbf71/volumes/kubernetes.io~projected/kube-api-access-xr684 DeviceMajor:0 DeviceMinor:65 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/df61d6453bb2b7ff05aa8df94e63ef5f6d6616ebd51be5be599aa73f996b191d/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-159 DeviceMajor:0 DeviceMinor:159 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cee9ee0075ad7fe8f9687cb833f49ea3f9a48bdeb6e0212e761a8acf0b06d211/userdata/shm DeviceMajor:0 DeviceMinor:245 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-311 DeviceMajor:0 DeviceMinor:311 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/96c016f7-9098-4a39-9e72-e54da374d8fc/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:206 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1753fd31-8933-4567-8146-07f3fd2d10a5/volumes/kubernetes.io~projected/kube-api-access-6n7xj DeviceMajor:0 DeviceMinor:209 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fe3b22311179a16c8c3eeab442b74b9b6046baa06d8bafb32318503a0d2e7908/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f8ab706e01419505523c810aac56fba7d29a8c4703c9b0f23565593f44ef80a7/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/543b8778-baa4-4e09-b697-68e550e42942/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:207 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c058fb511ce4a4026f63bb8f9a0cbc4eccf8ee9fea0ed0ed8993337b4a9e8166/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/24683f3bceaa61bf9e0e4d275b35e9a826b29dfed6d50b5f0ca3e5f14ddb40fa/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/653db542-8702-4eaa-abc1-a3bb866163c6/volumes/kubernetes.io~projected/kube-api-access-mpwxv DeviceMajor:0 DeviceMinor:60 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce/volumes/kubernetes.io~projected/kube-api-access-mtt8c DeviceMajor:0 DeviceMinor:66 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/df9f9c4a-d054-42cf-b17c-0d23acd0459b/volumes/kubernetes.io~projected/kube-api-access-ktkfm DeviceMajor:0 DeviceMinor:215 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/02550913e8d27c7c8560a212e5bd9d530b50751200373c6fac7af5b19bc5eba6/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6c8b4e67220dd8eff5e0a5c70f9e73180d809bb968c1192c2bdefd5ecc7f6800/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca/volumes/kubernetes.io~projected/kube-api-access-8gm62 DeviceMajor:0 DeviceMinor:217 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6f215b42-0f09-46a0-bc98-392c0fa909c9/volumes/kubernetes.io~projected/kube-api-access-qjtq4 DeviceMajor:0 DeviceMinor:226 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-309 DeviceMajor:0 DeviceMinor:309 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9fb8c33c-fe15-4daa-a01b-4cb2fc7eae7b/volumes/kubernetes.io~projected/kube-api-access-q68w5 DeviceMajor:0 DeviceMinor:222 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a023911d-3669-43e1-ac77-8a986f3715d6/volumes/kubernetes.io~projected/kube-api-access-ftr5x DeviceMajor:0 DeviceMinor:224 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/790607e1-d57c-4622-9a72-1af902ab1d85/volumes/kubernetes.io~projected/kube-api-access-hgtvt DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:62 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0ed2a89a99f0a7292450791623764b1318e836e96f0da6d2b1da8e710c76aff0/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/96c016f7-9098-4a39-9e72-e54da374d8fc/volumes/kubernetes.io~projected/kube-api-access-4w42k DeviceMajor:0 DeviceMinor:212 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/220ed31c-1577-4c4b-ab5b-e812fd63d193/volumes/kubernetes.io~projected/kube-api-access-5lgzq DeviceMajor:0 DeviceMinor:299 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-330 DeviceMajor:0 DeviceMinor:330 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/97697af028d0d1433e902ba6ee03eddadbd380fa856a9761702510919ef2426d/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/543b8778-baa4-4e09-b697-68e550e42942/volumes/kubernetes.io~projected/kube-api-access-gj2lf DeviceMajor:0 DeviceMinor:214 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-297 DeviceMajor:0 DeviceMinor:297 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-164 DeviceMajor:0 DeviceMinor:164 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a023911d-3669-43e1-ac77-8a986f3715d6/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/96c016f7-9098-4a39-9e72-e54da374d8fc/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:208 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/417924c6ce13224daa5404d770644a58f2de02ee1c63dc259560510c12e4928c/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e10dc749-bc71-456e-a315-6581add01a7b/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:210 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-325 DeviceMajor:0 DeviceMinor:325 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/10137ba2-4beb-433e-b6df-199738351c61/volumes/kubernetes.io~projected/kube-api-access-2s4qj DeviceMajor:0 DeviceMinor:329 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e10dc749-bc71-456e-a315-6581add01a7b/volumes/kubernetes.io~projected/kube-api-access-nfg8j DeviceMajor:0 DeviceMinor:216 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-273 DeviceMajor:0 DeviceMinor:273 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3880f3d3-aad1-43c0-b704-01374aad4f78/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e2b752d9-88c4-4dee-8b0c-7e9ac252adb3/volumes/kubernetes.io~projected/kube-api-access-xz8xk DeviceMajor:0 DeviceMinor:221 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9fb06094152142396f464199b328c14b14771278ba0d9d040daab45a8693dcd2/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d0ab7c64-a85f-4713-a270-f0cf711fcee5/volumes/kubernetes.io~projected/kube-api-access-jc8v6 DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:63 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/545f3c9a-2ac6-44a3-be0b-cf71b1592d42/volumes/kubernetes.io~projected/kube-api-access-55hqg DeviceMajor:0 DeviceMinor:64 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e49beb36db5058ff6aafac0435a308b909777900afdd978ebec852e8d0fe35e2/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-174 DeviceMajor:0 DeviceMinor:174 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1c00e787-a35d-4ac4-95a4-2546e29602aa/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a023911d-3669-43e1-ac77-8a986f3715d6/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:205 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/760afcba9b261afe12f20cdbd1135d1967906cf78e08b9a5f83d27736a8afd7f/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/614fce5f50618860328a6514e4a8ed906b17132e6d53aadfb45f8e50b6ebb3a7/userdata/shm DeviceMajor:0 DeviceMinor:302 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-179 DeviceMajor:0 DeviceMinor:179 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d0ab7c64-a85f-4713-a270-f0cf711fcee5/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4c245f17b0c21d5f74bf92645ee2ed5737f7f55f6403eb89bac16ac2f44dd547/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/96c016f7-9098-4a39-9e72-e54da374d8fc/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/03c21a45f12598b3509569e9e31fe33175fe0b2949aa92bf67867698de18dbce/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-285 DeviceMajor:0 DeviceMinor:285 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e9319b6d883d1db6f7e71643f2a97b1f810903966a53f2ee00efe32b8cc288b2/userdata/shm DeviceMajor:0 DeviceMinor:300 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3880f3d3-aad1-43c0-b704-01374aad4f78/volumes/kubernetes.io~projected/kube-api-access-8mh9g DeviceMajor:0 DeviceMinor:213 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8fa315db-e48d-4ac5-960a-23750974d431/volumes/kubernetes.io~projected/kube-api-access-jtqhk DeviceMajor:0 DeviceMinor:219 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5a72890122c2ad33df5c500a2d864911fa62f753e832a2223fec4e3a9c675e44/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-269 DeviceMajor:0 DeviceMinor:269 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6f215b42-0f09-46a0-bc98-392c0fa909c9/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:211 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f97b063e-8403-46bd-9716-72411589a494/volumes/kubernetes.io~projected/kube-api-access-p288b DeviceMajor:0 DeviceMinor:223 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-271 DeviceMajor:0 DeviceMinor:271 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25/volumes/kubernetes.io~projected/kube-api-access-w5pj6 DeviceMajor:0 DeviceMinor:67 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9fb8c33c-fe15-4daa-a01b-4cb2fc7eae7b/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:203 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:220 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/eee821c4-6fb0-4f84-a58d-23dc2ff77720/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:293 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-307 DeviceMajor:0 DeviceMinor:307 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-327 DeviceMajor:0 DeviceMinor:327 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8956f25a94b043ba181c79bc664f46b9ec9492e4feead98a59cff77e662a4726/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b46f51b2-436b-4967-8879-a16b58930591/volumes/kubernetes.io~projected/kube-api-access-f75xn DeviceMajor:0 DeviceMinor:288 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-340 DeviceMajor:0 DeviceMinor:340 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/38a6add1-b051-450b-bb96-8ace8a0ccff3/volumes/kubernetes.io~projected/kube-api-access-ngpxm DeviceMajor:0 DeviceMinor:218 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:02550913e8d27c7 MacAddress:d2:aa:9a:16:a2:ab Speed:10000 Mtu:8901} {Name:03c21a45f12598b MacAddress:82:b5:6f:db:dd:4a Speed:10000 Mtu:8901} {Name:417924c6ce13224 MacAddress:5e:6a:75:5a:03:f8 Speed:10000 Mtu:8901} {Name:4c245f17b0c21d5 MacAddress:c2:28:ee:dc:92:38 Speed:10000 Mtu:8901} {Name:5a72890122c2ad3 MacAddress:f2:0e:42:9e:2d:4a Speed:10000 Mtu:8901} {Name:614fce5f5061886 MacAddress:42:7f:77:29:71:83 Speed:10000 Mtu:8901} {Name:6c8b4e67220dd8e MacAddress:5e:46:ff:26:4a:b5 Speed:10000 Mtu:8901} {Name:760afcba9b261af MacAddress:be:c5:9c:85:31:9c Speed:10000 Mtu:8901} {Name:9fb060941521423 MacAddress:8a:d6:86:01:b5:fe Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:ff:bc:59:77 Speed:0 Mtu:9001} {Name:br-int MacAddress:e6:79:5b:0e:f5:a3 Speed:0 Mtu:8901} {Name:c058fb511ce4a40 MacAddress:ee:bb:5e:99:42:c6 Speed:10000 Mtu:8901} {Name:cee9ee0075ad7fe MacAddress:de:87:a4:a6:a9:c6 Speed:10000 Mtu:8901} {Name:e49beb36db5058f MacAddress:c2:06:62:21:78:b1 Speed:10000 Mtu:8901} {Name:e9319b6d883d1db MacAddress:de:55:e0:9c:fd:c3 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:ff:bc:59:77 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:0e:2f:05:a4:f6:3d Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:1a:b6:0e:89:bb:b2 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 19:31:26.824843 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.824174    4921 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 19:31:26.824843 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.824223    4921 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 19:31:26.824843 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.824514    4921 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 19:31:26.824843 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.824543    4921 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-222.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 19:31:26.824843 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.824690    4921 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 19:31:26.824843 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.824698    4921 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 19:31:26.824843 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.824710    4921 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 19:31:26.824843 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.824729    4921 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 19:31:26.824843 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.824749    4921 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 19:31:26.825290 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.824906    4921 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 19:31:26.825290 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.825010    4921 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 19:31:26.825290 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.825043    4921 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 19:31:26.825290 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.825056    4921 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 19:31:26.825290 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.825066    4921 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 19:31:26.825290 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.825075    4921 kubelet.go:405] "Adding apiserver pod source"
Nov 03 19:31:26.825290 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.825083    4921 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 19:31:26.825873 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.825860    4921 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 19:31:26.825909 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.825874    4921 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 19:31:26.826430 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.826407    4921 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 19:31:26.829972 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.829955    4921 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 19:31:26.829972 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.829971    4921 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 19:31:26.830093 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.830075    4921 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 03 19:31:26.830305 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.830293    4921 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 19:31:26.830336 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.830323    4921 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 19:31:26.830433 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.830424    4921 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 19:31:26.830466 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.830438    4921 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 19:31:26.830466 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.830445    4921 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 19:31:26.830466 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.830452    4921 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 19:31:26.830466 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.830457    4921 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 19:31:26.830466 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.830463    4921 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 19:31:26.830595 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.830472    4921 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 19:31:26.830595 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.830481    4921 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 19:31:26.830595 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.830488    4921 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 19:31:26.830595 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.830499    4921 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 19:31:26.830595 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.830513    4921 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 19:31:26.830595 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.830524    4921 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 19:31:26.830595 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.830551    4921 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 19:31:26.830595 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.830561    4921 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 19:31:26.834882 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.834866    4921 server.go:1263] "Started kubelet"
Nov 03 19:31:26.835046 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.835002    4921 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 19:31:26.835150 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.835135    4921 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 19:31:26.835390 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.835343    4921 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 19:31:26.835511 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.835478    4921 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 19:31:26.835463 ip-10-0-134-222 systemd[1]: Started Kubernetes Kubelet.
Nov 03 19:31:26.840206 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.837953    4921 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 19:31:26.841048 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.841030    4921 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 19:31:26.841250 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:31:26.841223    4921 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-222.us-east-2.compute.internal\" not found"
Nov 03 19:31:26.841328 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.841257    4921 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 19:31:26.841328 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.841275    4921 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 19:31:26.841436 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.841408    4921 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 19:31:26.842527 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.842373    4921 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 19:31:26.844045 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.844020    4921 factory.go:55] Registering systemd factory
Nov 03 19:31:26.844045 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.844047    4921 factory.go:223] Registration of the systemd container factory successfully
Nov 03 19:31:26.844291 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.844274    4921 factory.go:153] Registering CRI-O factory
Nov 03 19:31:26.844355 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.844296    4921 factory.go:223] Registration of the crio container factory successfully
Nov 03 19:31:26.844402 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.844367    4921 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 19:31:26.844402 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.844391    4921 factory.go:103] Registering Raw factory
Nov 03 19:31:26.844501 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.844410    4921 manager.go:1199] Started watching for new ooms in manager
Nov 03 19:31:26.844822 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.844801    4921 manager.go:319] Starting recovery of all containers
Nov 03 19:31:26.844914 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.844801    4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9fb8c33c-fe15-4daa-a01b-4cb2fc7eae7b" volumeName="kubernetes.io/configmap/9fb8c33c-fe15-4daa-a01b-4cb2fc7eae7b-config" seLinuxMountContext=""
Nov 03 19:31:26.844914 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.844839    4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9fb8c33c-fe15-4daa-a01b-4cb2fc7eae7b" volumeName="kubernetes.io/projected/9fb8c33c-fe15-4daa-a01b-4cb2fc7eae7b-kube-api-access-q68w5" seLinuxMountContext=""
Nov 03 19:31:26.844914 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.844853    4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca" volumeName="kubernetes.io/secret/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-image-registry-private-configuration" seLinuxMountContext=""
Nov 03 19:31:26.844914 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.844866    4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3880f3d3-aad1-43c0-b704-01374aad4f78" volumeName="kubernetes.io/configmap/3880f3d3-aad1-43c0-b704-01374aad4f78-service-ca-bundle" seLinuxMountContext=""
Nov 03 19:31:26.844914 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.844879    4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="96c016f7-9098-4a39-9e72-e54da374d8fc" volumeName="kubernetes.io/projected/96c016f7-9098-4a39-9e72-e54da374d8fc-bound-sa-token" seLinuxMountContext=""
Nov 03 19:31:26.844914 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.844890    4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f89ef12d-9c60-4538-af9c-f97453bbc855" volumeName="kubernetes.io/projected/f89ef12d-9c60-4538-af9c-f97453bbc855-kube-api-access-xdw57" seLinuxMountContext=""
Nov 03 19:31:26.844914 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.844902    4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3880f3d3-aad1-43c0-b704-01374aad4f78" volumeName="kubernetes.io/empty-dir/3880f3d3-aad1-43c0-b704-01374aad4f78-snapshots" seLinuxMountContext=""
Nov 03 19:31:26.844914 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.844914    4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="543b8778-baa4-4e09-b697-68e550e42942" volumeName="kubernetes.io/secret/543b8778-baa4-4e09-b697-68e550e42942-cert" seLinuxMountContext=""
Nov 03 19:31:26.845140 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.844928    4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="653db542-8702-4eaa-abc1-a3bb866163c6" volumeName="kubernetes.io/configmap/653db542-8702-4eaa-abc1-a3bb866163c6-cni-binary-copy" seLinuxMountContext=""
Nov 03 19:31:26.845140 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.844940    4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="653db542-8702-4eaa-abc1-a3bb866163c6" volumeName="kubernetes.io/projected/653db542-8702-4eaa-abc1-a3bb866163c6-kube-api-access-mpwxv" seLinuxMountContext=""
Nov 03 19:31:26.845140 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.844952    4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b5e1b09-699d-4db8-9eb0-0b71bdda858f" volumeName="kubernetes.io/configmap/8b5e1b09-699d-4db8-9eb0-0b71bdda858f-iptables-alerter-script" seLinuxMountContext=""
Nov 03 19:31:26.845140 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.844966    4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="96c016f7-9098-4a39-9e72-e54da374d8fc" volumeName="kubernetes.io/empty-dir/96c016f7-9098-4a39-9e72-e54da374d8fc-ca-trust-extracted" seLinuxMountContext=""
Nov 03 19:31:26.845140 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.844977    4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b797bdb0-b354-4dfc-a50a-3c60ecacbf71" volumeName="kubernetes.io/configmap/b797bdb0-b354-4dfc-a50a-3c60ecacbf71-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 03 19:31:26.845140 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.844990    4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0ab7c64-a85f-4713-a270-f0cf711fcee5" volumeName="kubernetes.io/secret/d0ab7c64-a85f-4713-a270-f0cf711fcee5-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 03 19:31:26.845140 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.845005    4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10137ba2-4beb-433e-b6df-199738351c61" volumeName="kubernetes.io/projected/10137ba2-4beb-433e-b6df-199738351c61-kube-api-access-2s4qj" seLinuxMountContext=""
Nov 03 19:31:26.845140 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.845022    4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1611b730-6163-4776-bd2d-3d10c2cc9a88" volumeName="kubernetes.io/projected/1611b730-6163-4776-bd2d-3d10c2cc9a88-kube-api-access-2287d" seLinuxMountContext=""
Nov 03 19:31:26.845140 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.845037    4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="96c016f7-9098-4a39-9e72-e54da374d8fc" volumeName="kubernetes.io/configmap/96c016f7-9098-4a39-9e72-e54da374d8fc-registry-certificates" seLinuxMountContext=""
Nov 03 19:31:26.845140 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.845049    4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="96c016f7-9098-4a39-9e72-e54da374d8fc" volumeName="kubernetes.io/projected/96c016f7-9098-4a39-9e72-e54da374d8fc-kube-api-access-4w42k" seLinuxMountContext=""
Nov 03 19:31:26.845140 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.845060    4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a023911d-3669-43e1-ac77-8a986f3715d6" volumeName="kubernetes.io/projected/a023911d-3669-43e1-ac77-8a986f3715d6-kube-api-access-ftr5x" seLinuxMountContext=""
Nov 03 19:31:26.845140 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.845072    4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca" volumeName="kubernetes.io/configmap/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-trusted-ca" seLinuxMountContext=""
Nov 03 19:31:26.845140 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.845084    4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca" volumeName="kubernetes.io/projected/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-kube-api-access-8gm62" seLinuxMountContext=""
Nov 03 19:31:26.845140 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.845095    4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca" volumeName="kubernetes.io/secret/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-installation-pull-secrets" seLinuxMountContext=""
Nov 03 19:31:26.845140 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.845125    4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1753fd31-8933-4567-8146-07f3fd2d10a5" volumeName="kubernetes.io/configmap/1753fd31-8933-4567-8146-07f3fd2d10a5-samples-operator-config" seLinuxMountContext=""
Nov 03 19:31:26.845538 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.845139    4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="38a6add1-b051-450b-bb96-8ace8a0ccff3" volumeName="kubernetes.io/projected/38a6add1-b051-450b-bb96-8ace8a0ccff3-kube-api-access-ngpxm" seLinuxMountContext=""
Nov 03 19:31:26.845538 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.845158    4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="545f3c9a-2ac6-44a3-be0b-cf71b1592d42" volumeName="kubernetes.io/projected/545f3c9a-2ac6-44a3-be0b-cf71b1592d42-kube-api-access-55hqg" seLinuxMountContext=""
Nov 03 19:31:26.845538 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.845170    4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56e21184-e568-46bc-8e90-204684d6c366" volumeName="kubernetes.io/projected/56e21184-e568-46bc-8e90-204684d6c366-kube-api-access-6wk2r" seLinuxMountContext=""
Nov 03 19:31:26.845538 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.845181    4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="653db542-8702-4eaa-abc1-a3bb866163c6" volumeName="kubernetes.io/configmap/653db542-8702-4eaa-abc1-a3bb866163c6-multus-daemon-config" seLinuxMountContext=""
Nov 03 19:31:26.845538 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.845194    4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="96c016f7-9098-4a39-9e72-e54da374d8fc" volumeName="kubernetes.io/secret/96c016f7-9098-4a39-9e72-e54da374d8fc-image-registry-private-configuration" seLinuxMountContext=""
Nov 03 19:31:26.845538 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.845227    4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a023911d-3669-43e1-ac77-8a986f3715d6" volumeName="kubernetes.io/secret/a023911d-3669-43e1-ac77-8a986f3715d6-default-certificate" seLinuxMountContext=""
Nov 03 19:31:26.845538 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.845239    4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df9f9c4a-d054-42cf-b17c-0d23acd0459b" volumeName="kubernetes.io/projected/df9f9c4a-d054-42cf-b17c-0d23acd0459b-kube-api-access-ktkfm" seLinuxMountContext=""
Nov 03 19:31:26.845538 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.845252    4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25" volumeName="kubernetes.io/projected/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25-kube-api-access-w5pj6" seLinuxMountContext=""
Nov 03 19:31:26.845538 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.845265    4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c00e787-a35d-4ac4-95a4-2546e29602aa" volumeName="kubernetes.io/configmap/1c00e787-a35d-4ac4-95a4-2546e29602aa-konnectivity-ca" seLinuxMountContext=""
Nov 03 19:31:26.845538 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.845278    4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c00e787-a35d-4ac4-95a4-2546e29602aa" volumeName="kubernetes.io/secret/1c00e787-a35d-4ac4-95a4-2546e29602aa-agent-certs" seLinuxMountContext=""
Nov 03 19:31:26.845538 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.845290    4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="96c016f7-9098-4a39-9e72-e54da374d8fc" volumeName="kubernetes.io/secret/96c016f7-9098-4a39-9e72-e54da374d8fc-installation-pull-secrets" seLinuxMountContext=""
Nov 03 19:31:26.845538 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.845304    4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca" volumeName="kubernetes.io/projected/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-bound-sa-token" seLinuxMountContext=""
Nov 03 19:31:26.845538 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.845315    4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e10dc749-bc71-456e-a315-6581add01a7b" volumeName="kubernetes.io/projected/e10dc749-bc71-456e-a315-6581add01a7b-kube-api-access-nfg8j" seLinuxMountContext=""
Nov 03 19:31:26.845538 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.845327    4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e10dc749-bc71-456e-a315-6581add01a7b" volumeName="kubernetes.io/secret/e10dc749-bc71-456e-a315-6581add01a7b-serving-cert" seLinuxMountContext=""
Nov 03 19:31:26.845538 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.845351    4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f215b42-0f09-46a0-bc98-392c0fa909c9" volumeName="kubernetes.io/secret/6f215b42-0f09-46a0-bc98-392c0fa909c9-serving-cert" seLinuxMountContext=""
Nov 03 19:31:26.845538 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.845363    4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0ab7c64-a85f-4713-a270-f0cf711fcee5" volumeName="kubernetes.io/configmap/d0ab7c64-a85f-4713-a270-f0cf711fcee5-env-overrides" seLinuxMountContext=""
Nov 03 19:31:26.845538 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.845375    4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca" volumeName="kubernetes.io/empty-dir/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-ca-trust-extracted" seLinuxMountContext=""
Nov 03 19:31:26.845538 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.845402    4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25" volumeName="kubernetes.io/empty-dir/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25-etc-tuned" seLinuxMountContext=""
Nov 03 19:31:26.845538 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.845416    4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3880f3d3-aad1-43c0-b704-01374aad4f78" volumeName="kubernetes.io/empty-dir/3880f3d3-aad1-43c0-b704-01374aad4f78-tmp" seLinuxMountContext=""
Nov 03 19:31:26.845538 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.845429    4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3880f3d3-aad1-43c0-b704-01374aad4f78" volumeName="kubernetes.io/projected/3880f3d3-aad1-43c0-b704-01374aad4f78-kube-api-access-8mh9g" seLinuxMountContext=""
Nov 03 19:31:26.845538 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.845446    4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f215b42-0f09-46a0-bc98-392c0fa909c9" volumeName="kubernetes.io/configmap/6f215b42-0f09-46a0-bc98-392c0fa909c9-config" seLinuxMountContext=""
Nov 03 19:31:26.845538 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.845458    4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8fa315db-e48d-4ac5-960a-23750974d431" volumeName="kubernetes.io/projected/8fa315db-e48d-4ac5-960a-23750974d431-kube-api-access-jtqhk" seLinuxMountContext=""
Nov 03 19:31:26.845538 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.845477    4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0ab7c64-a85f-4713-a270-f0cf711fcee5" volumeName="kubernetes.io/configmap/d0ab7c64-a85f-4713-a270-f0cf711fcee5-ovnkube-config" seLinuxMountContext=""
Nov 03 19:31:26.845538 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.845501    4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e2b752d9-88c4-4dee-8b0c-7e9ac252adb3" volumeName="kubernetes.io/configmap/e2b752d9-88c4-4dee-8b0c-7e9ac252adb3-telemetry-config" seLinuxMountContext=""
Nov 03 19:31:26.845538 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.845520    4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f97b063e-8403-46bd-9716-72411589a494" volumeName="kubernetes.io/projected/f97b063e-8403-46bd-9716-72411589a494-kube-api-access-p288b" seLinuxMountContext=""
Nov 03 19:31:26.845538 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.845542    4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f215b42-0f09-46a0-bc98-392c0fa909c9" volumeName="kubernetes.io/configmap/6f215b42-0f09-46a0-bc98-392c0fa909c9-trusted-ca" seLinuxMountContext=""
Nov 03 19:31:26.846352 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.845561    4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="96c016f7-9098-4a39-9e72-e54da374d8fc" volumeName="kubernetes.io/configmap/96c016f7-9098-4a39-9e72-e54da374d8fc-trusted-ca" seLinuxMountContext=""
Nov 03 19:31:26.846352 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.845580    4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca" volumeName="kubernetes.io/configmap/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-registry-certificates" seLinuxMountContext=""
Nov 03 19:31:26.846352 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.845603    4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1753fd31-8933-4567-8146-07f3fd2d10a5" volumeName="kubernetes.io/projected/1753fd31-8933-4567-8146-07f3fd2d10a5-kube-api-access-6n7xj" seLinuxMountContext=""
Nov 03 19:31:26.846352 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.845629    4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="790607e1-d57c-4622-9a72-1af902ab1d85" volumeName="kubernetes.io/projected/790607e1-d57c-4622-9a72-1af902ab1d85-kube-api-access-hgtvt" seLinuxMountContext=""
Nov 03 19:31:26.846352 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.845655    4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b797bdb0-b354-4dfc-a50a-3c60ecacbf71" volumeName="kubernetes.io/configmap/b797bdb0-b354-4dfc-a50a-3c60ecacbf71-cni-binary-copy" seLinuxMountContext=""
Nov 03 19:31:26.846352 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.845693    4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0ab7c64-a85f-4713-a270-f0cf711fcee5" volumeName="kubernetes.io/projected/d0ab7c64-a85f-4713-a270-f0cf711fcee5-kube-api-access-jc8v6" seLinuxMountContext=""
Nov 03 19:31:26.846352 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.845710    4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a023911d-3669-43e1-ac77-8a986f3715d6" volumeName="kubernetes.io/secret/a023911d-3669-43e1-ac77-8a986f3715d6-stats-auth" seLinuxMountContext=""
Nov 03 19:31:26.846352 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.845740    4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b797bdb0-b354-4dfc-a50a-3c60ecacbf71" volumeName="kubernetes.io/configmap/b797bdb0-b354-4dfc-a50a-3c60ecacbf71-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 03 19:31:26.846352 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.845757    4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25" volumeName="kubernetes.io/empty-dir/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25-tmp" seLinuxMountContext=""
Nov 03 19:31:26.846352 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.845776    4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="543b8778-baa4-4e09-b697-68e550e42942" volumeName="kubernetes.io/projected/543b8778-baa4-4e09-b697-68e550e42942-kube-api-access-gj2lf" seLinuxMountContext=""
Nov 03 19:31:26.846352 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.845792    4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b46f51b2-436b-4967-8879-a16b58930591" volumeName="kubernetes.io/projected/b46f51b2-436b-4967-8879-a16b58930591-kube-api-access-f75xn" seLinuxMountContext=""
Nov 03 19:31:26.846352 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.845809    4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b797bdb0-b354-4dfc-a50a-3c60ecacbf71" volumeName="kubernetes.io/projected/b797bdb0-b354-4dfc-a50a-3c60ecacbf71-kube-api-access-xr684" seLinuxMountContext=""
Nov 03 19:31:26.846352 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.845827    4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e10dc749-bc71-456e-a315-6581add01a7b" volumeName="kubernetes.io/configmap/e10dc749-bc71-456e-a315-6581add01a7b-config" seLinuxMountContext=""
Nov 03 19:31:26.846352 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.845844    4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb488ece-2ac3-499f-84c1-73534fcbb20b" volumeName="kubernetes.io/configmap/eb488ece-2ac3-499f-84c1-73534fcbb20b-nginx-conf" seLinuxMountContext=""
Nov 03 19:31:26.846352 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.845859    4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3880f3d3-aad1-43c0-b704-01374aad4f78" volumeName="kubernetes.io/configmap/3880f3d3-aad1-43c0-b704-01374aad4f78-trusted-ca-bundle" seLinuxMountContext=""
Nov 03 19:31:26.846352 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.845934    4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="38a6add1-b051-450b-bb96-8ace8a0ccff3" volumeName="kubernetes.io/configmap/38a6add1-b051-450b-bb96-8ace8a0ccff3-config-volume" seLinuxMountContext=""
Nov 03 19:31:26.846352 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.845954    4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="38a6add1-b051-450b-bb96-8ace8a0ccff3" volumeName="kubernetes.io/empty-dir/38a6add1-b051-450b-bb96-8ace8a0ccff3-tmp-dir" seLinuxMountContext=""
Nov 03 19:31:26.846352 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.845972    4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56e21184-e568-46bc-8e90-204684d6c366" volumeName="kubernetes.io/empty-dir/56e21184-e568-46bc-8e90-204684d6c366-tmp-dir" seLinuxMountContext=""
Nov 03 19:31:26.846352 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.845990    4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f215b42-0f09-46a0-bc98-392c0fa909c9" volumeName="kubernetes.io/projected/6f215b42-0f09-46a0-bc98-392c0fa909c9-kube-api-access-qjtq4" seLinuxMountContext=""
Nov 03 19:31:26.846352 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.846008    4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="220ed31c-1577-4c4b-ab5b-e812fd63d193" volumeName="kubernetes.io/projected/220ed31c-1577-4c4b-ab5b-e812fd63d193-kube-api-access-5lgzq" seLinuxMountContext=""
Nov 03 19:31:26.846352 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.846028    4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0ab7c64-a85f-4713-a270-f0cf711fcee5" volumeName="kubernetes.io/configmap/d0ab7c64-a85f-4713-a270-f0cf711fcee5-ovnkube-script-lib" seLinuxMountContext=""
Nov 03 19:31:26.846352 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.846047    4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e2b752d9-88c4-4dee-8b0c-7e9ac252adb3" volumeName="kubernetes.io/projected/e2b752d9-88c4-4dee-8b0c-7e9ac252adb3-kube-api-access-xz8xk" seLinuxMountContext=""
Nov 03 19:31:26.846352 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.846065    4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eee821c4-6fb0-4f84-a58d-23dc2ff77720" volumeName="kubernetes.io/secret/eee821c4-6fb0-4f84-a58d-23dc2ff77720-original-pull-secret" seLinuxMountContext=""
Nov 03 19:31:26.846352 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.846084    4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10137ba2-4beb-433e-b6df-199738351c61" volumeName="kubernetes.io/empty-dir/10137ba2-4beb-433e-b6df-199738351c61-data-volume" seLinuxMountContext=""
Nov 03 19:31:26.846352 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.846102    4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3880f3d3-aad1-43c0-b704-01374aad4f78" volumeName="kubernetes.io/secret/3880f3d3-aad1-43c0-b704-01374aad4f78-serving-cert" seLinuxMountContext=""
Nov 03 19:31:26.846352 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.846119    4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce" volumeName="kubernetes.io/projected/8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce-kube-api-access-mtt8c" seLinuxMountContext=""
Nov 03 19:31:26.846352 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.846136    4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9fb8c33c-fe15-4daa-a01b-4cb2fc7eae7b" volumeName="kubernetes.io/secret/9fb8c33c-fe15-4daa-a01b-4cb2fc7eae7b-serving-cert" seLinuxMountContext=""
Nov 03 19:31:26.846352 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.846155    4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="545f3c9a-2ac6-44a3-be0b-cf71b1592d42" volumeName="kubernetes.io/configmap/545f3c9a-2ac6-44a3-be0b-cf71b1592d42-serviceca" seLinuxMountContext=""
Nov 03 19:31:26.846352 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.846173    4921 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b5e1b09-699d-4db8-9eb0-0b71bdda858f" volumeName="kubernetes.io/projected/8b5e1b09-699d-4db8-9eb0-0b71bdda858f-kube-api-access-2g8fh" seLinuxMountContext=""
Nov 03 19:31:26.846352 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.846184    4921 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 19:31:26.846352 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.846192    4921 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 19:31:26.847920 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.846702    4921 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 19:25:45 +0000 UTC" deadline="2028-07-11 13:37:42.505289822 +0000 UTC"
Nov 03 19:31:26.847920 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.846730    4921 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14778h6m15.658563291s"
Nov 03 19:31:26.849525 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.849509    4921 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 03 19:31:26.862780 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.862752    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gzf8k_6f215b42-0f09-46a0-bc98-392c0fa909c9/console-operator/1.log"
Nov 03 19:31:26.863105 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.863076    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gzf8k_6f215b42-0f09-46a0-bc98-392c0fa909c9/console-operator/1.log"
Nov 03 19:31:26.870041 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.870003    4921 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 19:31:26.880117 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.880082    4921 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 19:31:26.880117 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.880117    4921 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 19:31:26.880304 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.880142    4921 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 19:31:26.880304 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:31:26.880182    4921 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 03 19:31:26.882574 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.882535    4921 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 19:31:26.887191 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.887165    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gzf8k_6f215b42-0f09-46a0-bc98-392c0fa909c9/console-operator/1.log"
Nov 03 19:31:26.887318 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.887205    4921 generic.go:356] "Generic (PLEG): container finished" podID="6f215b42-0f09-46a0-bc98-392c0fa909c9" containerID="3895cd529be0be6af57557f5b4638c0b5aa21cd6a6d699b224ff7bffcb1bd8e3" exitCode=255
Nov 03 19:31:26.918281 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.918245    4921 generic.go:356] "Generic (PLEG): container finished" podID="c5e667f56d4cbe64bbf7db6ca2bb9e9c" containerID="e6bc859a65a6a89039d5735a35c1672bcd570c1617da4abc3ea0c18161f936a3" exitCode=0
Nov 03 19:31:26.920661 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.920634    4921 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 19:31:26.921086 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.921061    4921 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 19:31:26.922207 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.922182    4921 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 19:31:26.929886 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.929858    4921 generic.go:356] "Generic (PLEG): container finished" podID="b797bdb0-b354-4dfc-a50a-3c60ecacbf71" containerID="0344486317f9cc5b931a45412d2bb159d13c62228ed9187fe6e6cd5696fefc3a" exitCode=0
Nov 03 19:31:26.929886 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.929883    4921 generic.go:356] "Generic (PLEG): container finished" podID="b797bdb0-b354-4dfc-a50a-3c60ecacbf71" containerID="dbc810b52fc22b02df9247e246c8490e439e08074258cf7ec5cbab44613ba588" exitCode=0
Nov 03 19:31:26.929886 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.929889    4921 generic.go:356] "Generic (PLEG): container finished" podID="b797bdb0-b354-4dfc-a50a-3c60ecacbf71" containerID="1264fc123e789534b0b6945171592450d6d144610484964413f368b833942c14" exitCode=0
Nov 03 19:31:26.930122 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.929897    4921 generic.go:356] "Generic (PLEG): container finished" podID="b797bdb0-b354-4dfc-a50a-3c60ecacbf71" containerID="2624e372a7710aed510c1f076341cadeafcf07abf8774723ca8bd97ab5da6952" exitCode=0
Nov 03 19:31:26.930122 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.929906    4921 generic.go:356] "Generic (PLEG): container finished" podID="b797bdb0-b354-4dfc-a50a-3c60ecacbf71" containerID="6016ea8b4a92ebd0d32f6649bfa7d02e3426bf2f54d88a9b56d2d2de6ca937ce" exitCode=0
Nov 03 19:31:26.930122 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.929912    4921 generic.go:356] "Generic (PLEG): container finished" podID="b797bdb0-b354-4dfc-a50a-3c60ecacbf71" containerID="fe5e127fb70e46591cc8c1a265341411f5a2987c182946b6b668ef2f62cdf0df" exitCode=0
Nov 03 19:31:26.942345 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.942283    4921 manager.go:324] Recovery completed
Nov 03 19:31:26.964663 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.964633    4921 cpu_manager.go:226] "Starting" policy="none"
Nov 03 19:31:26.964663 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.964654    4921 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 19:31:26.964859 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.964693    4921 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 19:31:26.964889 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.964852    4921 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 03 19:31:26.964889 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.964868    4921 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 03 19:31:26.964946 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.964890    4921 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 03 19:31:26.964946 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.964898    4921 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 03 19:31:26.964946 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.964904    4921 policy_none.go:50] "Start"
Nov 03 19:31:26.964946 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.964916    4921 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 19:31:26.964946 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.964930    4921 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 19:31:26.965072 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.965057    4921 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 03 19:31:26.965072 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.965064    4921 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 03 19:31:26.965072 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.965069    4921 policy_none.go:44] "Start"
Nov 03 19:31:26.969409 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.969375    4921 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 19:31:26.971919 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.971902    4921 manager.go:341] "Starting Device Plugin manager"
Nov 03 19:31:26.972597 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:31:26.971933    4921 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 19:31:26.972638 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.972604    4921 server.go:87] "Starting device plugin registration server"
Nov 03 19:31:26.972962 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.972945    4921 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 19:31:26.972998 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.972977    4921 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 19:31:26.973095 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.973067    4921 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 19:31:26.973237 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.973195    4921 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 19:31:26.973324 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.973249    4921 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 19:31:26.973324 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.973257    4921 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 19:31:26.973416 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.973290    4921 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T19:31:26.97323515Z","UUID":"e459cf3d-590f-4372-8e00-85d49b394877","Handler":null,"Name":"","Endpoint":""}
Nov 03 19:31:26.974745 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.974725    4921 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 19:31:26.974745 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.974755    4921 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 19:31:26.974905 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:31:26.974889    4921 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 19:31:26.980281 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.980236    4921 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-222.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-222.us-east-2.compute.internal"]
Nov 03 19:31:26.981526 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.981503    4921 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-222.us-east-2.compute.internal"
Nov 03 19:31:26.982339 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.982289    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-222.us-east-2.compute.internal" event={"ID":"c5e667f56d4cbe64bbf7db6ca2bb9e9c","Type":"ContainerStarted","Data":"7f332c31d651aa688a23cd0d382ff66341dd6f859152d99940ed4ec353703da7"}
Nov 03 19:31:26.982423 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.982359    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-222.us-east-2.compute.internal" event={"ID":"c5e667f56d4cbe64bbf7db6ca2bb9e9c","Type":"ContainerDied","Data":"e6bc859a65a6a89039d5735a35c1672bcd570c1617da4abc3ea0c18161f936a3"}
Nov 03 19:31:26.982423 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.982381    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-222.us-east-2.compute.internal" event={"ID":"c5e667f56d4cbe64bbf7db6ca2bb9e9c","Type":"ContainerStarted","Data":"24683f3bceaa61bf9e0e4d275b35e9a826b29dfed6d50b5f0ca3e5f14ddb40fa"}
Nov 03 19:31:26.982540 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.982448    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-222.us-east-2.compute.internal" event={"ID":"66cbb2b4d2abe69f7cf665c408777b87","Type":"ContainerStarted","Data":"60a9b190e093f6a47399ce390e94b17004d2297d1f2a0e306d22eb797dab4b3b"}
Nov 03 19:31:26.982540 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.982473    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-222.us-east-2.compute.internal" event={"ID":"66cbb2b4d2abe69f7cf665c408777b87","Type":"ContainerStarted","Data":"fe3b22311179a16c8c3eeab442b74b9b6046baa06d8bafb32318503a0d2e7908"}
Nov 03 19:31:26.982540 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.982518    4921 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-222.us-east-2.compute.internal"
Nov 03 19:31:26.991947 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.991927    4921 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 19:31:26.992062 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:31:26.991972    4921 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-222.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-222.us-east-2.compute.internal"
Nov 03 19:31:26.992427 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:26.992408    4921 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 19:31:26.992475 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:31:26.992460    4921 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-222.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-222.us-east-2.compute.internal"
Nov 03 19:31:27.051038 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.051004    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c5e667f56d4cbe64bbf7db6ca2bb9e9c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-222.us-east-2.compute.internal\" (UID: \"c5e667f56d4cbe64bbf7db6ca2bb9e9c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-222.us-east-2.compute.internal"
Nov 03 19:31:27.051038 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.051037    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5e667f56d4cbe64bbf7db6ca2bb9e9c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-222.us-east-2.compute.internal\" (UID: \"c5e667f56d4cbe64bbf7db6ca2bb9e9c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-222.us-east-2.compute.internal"
Nov 03 19:31:27.051229 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.051064    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/66cbb2b4d2abe69f7cf665c408777b87-config\") pod \"kube-apiserver-proxy-ip-10-0-134-222.us-east-2.compute.internal\" (UID: \"66cbb2b4d2abe69f7cf665c408777b87\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-222.us-east-2.compute.internal"
Nov 03 19:31:27.073295 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.073261    4921 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 19:31:27.074574 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.074552    4921 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-222.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 19:31:27.074679 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.074585    4921 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-222.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 19:31:27.074679 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.074596    4921 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-222.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 19:31:27.074679 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.074639    4921 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-222.us-east-2.compute.internal"
Nov 03 19:31:27.086481 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.086459    4921 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-134-222.us-east-2.compute.internal"
Nov 03 19:31:27.086574 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.086531    4921 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-222.us-east-2.compute.internal"
Nov 03 19:31:27.151567 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.151523    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c5e667f56d4cbe64bbf7db6ca2bb9e9c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-222.us-east-2.compute.internal\" (UID: \"c5e667f56d4cbe64bbf7db6ca2bb9e9c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-222.us-east-2.compute.internal"
Nov 03 19:31:27.151567 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.151570    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5e667f56d4cbe64bbf7db6ca2bb9e9c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-222.us-east-2.compute.internal\" (UID: \"c5e667f56d4cbe64bbf7db6ca2bb9e9c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-222.us-east-2.compute.internal"
Nov 03 19:31:27.151799 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.151590    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/66cbb2b4d2abe69f7cf665c408777b87-config\") pod \"kube-apiserver-proxy-ip-10-0-134-222.us-east-2.compute.internal\" (UID: \"66cbb2b4d2abe69f7cf665c408777b87\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-222.us-east-2.compute.internal"
Nov 03 19:31:27.151799 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.151650    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c5e667f56d4cbe64bbf7db6ca2bb9e9c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-222.us-east-2.compute.internal\" (UID: \"c5e667f56d4cbe64bbf7db6ca2bb9e9c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-222.us-east-2.compute.internal"
Nov 03 19:31:27.151799 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.151711    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/66cbb2b4d2abe69f7cf665c408777b87-config\") pod \"kube-apiserver-proxy-ip-10-0-134-222.us-east-2.compute.internal\" (UID: \"66cbb2b4d2abe69f7cf665c408777b87\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-222.us-east-2.compute.internal"
Nov 03 19:31:27.151799 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.151738    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5e667f56d4cbe64bbf7db6ca2bb9e9c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-222.us-east-2.compute.internal\" (UID: \"c5e667f56d4cbe64bbf7db6ca2bb9e9c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-222.us-east-2.compute.internal"
Nov 03 19:31:27.825296 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.825258    4921 apiserver.go:52] "Watching apiserver"
Nov 03 19:31:27.832968 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.832932    4921 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 19:31:27.833581 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.833559    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-9ck9p","openshift-network-console/networking-console-plugin-86c478bb88-zpgl6","kube-system/kube-apiserver-proxy-ip-10-0-134-222.us-east-2.compute.internal","openshift-network-diagnostics/network-check-source-6f78d59f99-cbhrt","openshift-image-registry/image-registry-d74787d9c-nn7sk","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-222.us-east-2.compute.internal","openshift-monitoring/cluster-monitoring-operator-884576c57-kfdsx","kube-system/kas-connection-checker-685895c7c9-42xn2","openshift-ingress/router-default-569d4c45f-7pszr","openshift-cluster-node-tuning-operator/tuned-l98pn","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mcxcv","openshift-dns/node-resolver-p2p8d","openshift-insights/insights-operator-7c4986d449-8kh6f","openshift-multus/network-metrics-daemon-rdr5d","openshift-image-registry/node-ca-vv2s2","kube-system/global-pull-secret-syncer-hrxdt","kube-system/kas-connection-checker-685895c7c9-wtgvw","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7hcf8","openshift-image-registry/image-registry-c5c44f4d-dv455","kube-system/kas-connection-checker-685895c7c9-xgbkd","openshift-dns/dns-default-wj8mp","openshift-network-diagnostics/network-check-target-njg27","openshift-service-ca-operator/service-ca-operator-544d749c8c-mh2tn","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hxlrb","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-tx285","openshift-kube-storage-version-migrator/migrator-66df5d496b-5l649","openshift-insights/insights-runtime-extractor-cvll4","openshift-ingress-canary/ingress-canary-j7m2z","openshift-ovn-kubernetes/ovnkube-node-k5nbr","openshift-console-operator/console-operator-7cd56b8dbc-gzf8k","openshift-multus/multus-additional-cni-plugins-g67bt","openshift-network-operator/iptables-alerter-wkrtt","kube-system/konnectivity-agent-jwrp9"]
Nov 03 19:31:27.835827 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.835803    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-569d4c45f-7pszr"
Nov 03 19:31:27.836489 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.836473    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kfdsx"
Nov 03 19:31:27.837928 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.837912    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7hcf8"
Nov 03 19:31:27.840158 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.840124    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-zpgl6"
Nov 03 19:31:27.841298 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.841279    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d74787d9c-nn7sk"
Nov 03 19:31:27.842959 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.842938    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Nov 03 19:31:27.843086 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.843073    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Nov 03 19:31:27.843262 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.843206    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Nov 03 19:31:27.843409 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.843389    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Nov 03 19:31:27.843709 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.843690    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Nov 03 19:31:27.843779 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.843358    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 19:31:27.843829 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.843312    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Nov 03 19:31:27.843895 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.843877    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Nov 03 19:31:27.843949 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.843904    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Nov 03 19:31:27.843999 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.843956    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 03 19:31:27.843999 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.843969    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 03 19:31:27.844395 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.844373    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Nov 03 19:31:27.844498 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.844438    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 19:31:27.844498 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.844450    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Nov 03 19:31:27.844498 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.844442    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Nov 03 19:31:27.844498 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.844458    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 19:31:27.844635 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.844504    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 19:31:27.844635 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.844532    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Nov 03 19:31:27.844635 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.844561    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Nov 03 19:31:27.845301 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.845282    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Nov 03 19:31:27.845393 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.845341    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 19:31:27.845393 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.845353    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 19:31:27.845393 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.845359    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Nov 03 19:31:27.845686 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.845654    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 03 19:31:27.845737 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.845696    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Nov 03 19:31:27.845737 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.845723    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Nov 03 19:31:27.845894 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.845872    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Nov 03 19:31:27.845995 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.845902    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Nov 03 19:31:27.846105 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.846086    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rdr5d"
Nov 03 19:31:27.846490 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.846472    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Nov 03 19:31:27.846721 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.846704    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 19:31:27.846806 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.846750    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 19:31:27.846868 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.846828    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 03 19:31:27.846917 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.846877    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 19:31:27.847935 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.847913    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 19:31:27.848017 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.847976    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 19:31:27.848095 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.848077    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 03 19:31:27.848156 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.848110    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 19:31:27.848519 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.848504    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 19:31:27.848589 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.848530    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 19:31:27.850225 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.850207    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 19:31:27.850225 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.850208    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 19:31:27.850338 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.850249    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 19:31:27.850338 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.850253    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 19:31:27.850338 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.850289    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 19:31:27.850505 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.850360    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 19:31:27.850699 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.850681    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 19:31:27.850868 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.850854    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 19:31:27.851066 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.851050    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c5c44f4d-dv455"
Nov 03 19:31:27.851341 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.851325    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 19:31:27.851393 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.851345    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 19:31:27.851393 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.851379    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 19:31:27.851456 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.851427    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 19:31:27.851780 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.851761    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 19:31:27.852093 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.852081    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 19:31:27.852315 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.852297    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wj8mp"
Nov 03 19:31:27.852521 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.852505    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 19:31:27.852603 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.852526    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 19:31:27.855110 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.855086    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-cvll4"
Nov 03 19:31:27.856243 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.856225    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 19:31:27.856334 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.856307    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 19:31:27.856426 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.856412    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 19:31:27.856845 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.856825    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9fb8c33c-fe15-4daa-a01b-4cb2fc7eae7b-serving-cert\") pod \"service-ca-operator-544d749c8c-mh2tn\" (UID: \"9fb8c33c-fe15-4daa-a01b-4cb2fc7eae7b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mh2tn"
Nov 03 19:31:27.856885 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.856867    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e2b752d9-88c4-4dee-8b0c-7e9ac252adb3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kfdsx\" (UID: \"e2b752d9-88c4-4dee-8b0c-7e9ac252adb3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kfdsx"
Nov 03 19:31:27.856921 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.856899    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xz8xk\" (UniqueName: \"kubernetes.io/projected/e2b752d9-88c4-4dee-8b0c-7e9ac252adb3-kube-api-access-xz8xk\") pod \"cluster-monitoring-operator-884576c57-kfdsx\" (UID: \"e2b752d9-88c4-4dee-8b0c-7e9ac252adb3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kfdsx"
Nov 03 19:31:27.856953 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.856929    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e10dc749-bc71-456e-a315-6581add01a7b-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hxlrb\" (UID: \"e10dc749-bc71-456e-a315-6581add01a7b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hxlrb"
Nov 03 19:31:27.856996 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.856957    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/96c016f7-9098-4a39-9e72-e54da374d8fc-installation-pull-secrets\") pod \"image-registry-d74787d9c-nn7sk\" (UID: \"96c016f7-9098-4a39-9e72-e54da374d8fc\") " pod="openshift-image-registry/image-registry-d74787d9c-nn7sk"
Nov 03 19:31:27.856996 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.856982    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3880f3d3-aad1-43c0-b704-01374aad4f78-tmp\") pod \"insights-operator-7c4986d449-8kh6f\" (UID: \"3880f3d3-aad1-43c0-b704-01374aad4f78\") " pod="openshift-insights/insights-operator-7c4986d449-8kh6f"
Nov 03 19:31:27.857086 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.857068    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3880f3d3-aad1-43c0-b704-01374aad4f78-tmp\") pod \"insights-operator-7c4986d449-8kh6f\" (UID: \"3880f3d3-aad1-43c0-b704-01374aad4f78\") " pod="openshift-insights/insights-operator-7c4986d449-8kh6f"
Nov 03 19:31:27.857182 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.857161    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3880f3d3-aad1-43c0-b704-01374aad4f78-snapshots\") pod \"insights-operator-7c4986d449-8kh6f\" (UID: \"3880f3d3-aad1-43c0-b704-01374aad4f78\") " pod="openshift-insights/insights-operator-7c4986d449-8kh6f"
Nov 03 19:31:27.857267 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.857251    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3880f3d3-aad1-43c0-b704-01374aad4f78-snapshots\") pod \"insights-operator-7c4986d449-8kh6f\" (UID: \"3880f3d3-aad1-43c0-b704-01374aad4f78\") " pod="openshift-insights/insights-operator-7c4986d449-8kh6f"
Nov 03 19:31:27.857325 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.857272    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 19:31:27.857325 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.857274    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdw57\" (UniqueName: \"kubernetes.io/projected/f89ef12d-9c60-4538-af9c-f97453bbc855-kube-api-access-xdw57\") pod \"volume-data-source-validator-58c9f5f46d-tx285\" (UID: \"f89ef12d-9c60-4538-af9c-f97453bbc855\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-tx285"
Nov 03 19:31:27.857325 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.857300    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3880f3d3-aad1-43c0-b704-01374aad4f78-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-8kh6f\" (UID: \"3880f3d3-aad1-43c0-b704-01374aad4f78\") " pod="openshift-insights/insights-operator-7c4986d449-8kh6f"
Nov 03 19:31:27.857325 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.857321    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3880f3d3-aad1-43c0-b704-01374aad4f78-service-ca-bundle\") pod \"insights-operator-7c4986d449-8kh6f\" (UID: \"3880f3d3-aad1-43c0-b704-01374aad4f78\") " pod="openshift-insights/insights-operator-7c4986d449-8kh6f"
Nov 03 19:31:27.857513 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.857344    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3880f3d3-aad1-43c0-b704-01374aad4f78-serving-cert\") pod \"insights-operator-7c4986d449-8kh6f\" (UID: \"3880f3d3-aad1-43c0-b704-01374aad4f78\") " pod="openshift-insights/insights-operator-7c4986d449-8kh6f"
Nov 03 19:31:27.857513 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.857361    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/96c016f7-9098-4a39-9e72-e54da374d8fc-ca-trust-extracted\") pod \"image-registry-d74787d9c-nn7sk\" (UID: \"96c016f7-9098-4a39-9e72-e54da374d8fc\") " pod="openshift-image-registry/image-registry-d74787d9c-nn7sk"
Nov 03 19:31:27.857513 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.857377    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e2b752d9-88c4-4dee-8b0c-7e9ac252adb3-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-kfdsx\" (UID: \"e2b752d9-88c4-4dee-8b0c-7e9ac252adb3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kfdsx"
Nov 03 19:31:27.857513 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.857276    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e10dc749-bc71-456e-a315-6581add01a7b-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hxlrb\" (UID: \"e10dc749-bc71-456e-a315-6581add01a7b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hxlrb"
Nov 03 19:31:27.857513 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.857398    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e10dc749-bc71-456e-a315-6581add01a7b-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hxlrb\" (UID: \"e10dc749-bc71-456e-a315-6581add01a7b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hxlrb"
Nov 03 19:31:27.857513 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.857456    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfg8j\" (UniqueName: \"kubernetes.io/projected/e10dc749-bc71-456e-a315-6581add01a7b-kube-api-access-nfg8j\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hxlrb\" (UID: \"e10dc749-bc71-456e-a315-6581add01a7b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hxlrb"
Nov 03 19:31:27.857513 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.857486    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3880f3d3-aad1-43c0-b704-01374aad4f78-service-ca-bundle\") pod \"insights-operator-7c4986d449-8kh6f\" (UID: \"3880f3d3-aad1-43c0-b704-01374aad4f78\") " pod="openshift-insights/insights-operator-7c4986d449-8kh6f"
Nov 03 19:31:27.857513 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.857495    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f75xn\" (UniqueName: \"kubernetes.io/projected/b46f51b2-436b-4967-8879-a16b58930591-kube-api-access-f75xn\") pod \"network-check-target-njg27\" (UID: \"b46f51b2-436b-4967-8879-a16b58930591\") " pod="openshift-network-diagnostics/network-check-target-njg27"
Nov 03 19:31:27.857513 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.857506    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/96c016f7-9098-4a39-9e72-e54da374d8fc-ca-trust-extracted\") pod \"image-registry-d74787d9c-nn7sk\" (UID: \"96c016f7-9098-4a39-9e72-e54da374d8fc\") " pod="openshift-image-registry/image-registry-d74787d9c-nn7sk"
Nov 03 19:31:27.857861 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.857528    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56e21184-e568-46bc-8e90-204684d6c366-tmp-dir\") pod \"node-resolver-p2p8d\" (UID: \"56e21184-e568-46bc-8e90-204684d6c366\") " pod="openshift-dns/node-resolver-p2p8d"
Nov 03 19:31:27.857861 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.857565    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a023911d-3669-43e1-ac77-8a986f3715d6-service-ca-bundle\") pod \"router-default-569d4c45f-7pszr\" (UID: \"a023911d-3669-43e1-ac77-8a986f3715d6\") " pod="openshift-ingress/router-default-569d4c45f-7pszr"
Nov 03 19:31:27.857861 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.857649    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56e21184-e568-46bc-8e90-204684d6c366-tmp-dir\") pod \"node-resolver-p2p8d\" (UID: \"56e21184-e568-46bc-8e90-204684d6c366\") " pod="openshift-dns/node-resolver-p2p8d"
Nov 03 19:31:27.857861 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.857650    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3880f3d3-aad1-43c0-b704-01374aad4f78-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-8kh6f\" (UID: \"3880f3d3-aad1-43c0-b704-01374aad4f78\") " pod="openshift-insights/insights-operator-7c4986d449-8kh6f"
Nov 03 19:31:27.857861 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.857661    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtqhk\" (UniqueName: \"kubernetes.io/projected/8fa315db-e48d-4ac5-960a-23750974d431-kube-api-access-jtqhk\") pod \"kas-connection-checker-685895c7c9-42xn2\" (UID: \"8fa315db-e48d-4ac5-960a-23750974d431\") " pod="kube-system/kas-connection-checker-685895c7c9-42xn2"
Nov 03 19:31:27.857861 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.857725    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a023911d-3669-43e1-ac77-8a986f3715d6-metrics-certs\") pod \"router-default-569d4c45f-7pszr\" (UID: \"a023911d-3669-43e1-ac77-8a986f3715d6\") " pod="openshift-ingress/router-default-569d4c45f-7pszr"
Nov 03 19:31:27.857861 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.857722    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e2b752d9-88c4-4dee-8b0c-7e9ac252adb3-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-kfdsx\" (UID: \"e2b752d9-88c4-4dee-8b0c-7e9ac252adb3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kfdsx"
Nov 03 19:31:27.857861 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.857826    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 03 19:31:27.857861 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.857852    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftr5x\" (UniqueName: \"kubernetes.io/projected/a023911d-3669-43e1-ac77-8a986f3715d6-kube-api-access-ftr5x\") pod \"router-default-569d4c45f-7pszr\" (UID: \"a023911d-3669-43e1-ac77-8a986f3715d6\") " pod="openshift-ingress/router-default-569d4c45f-7pszr"
Nov 03 19:31:27.858239 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.857928    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6f215b42-0f09-46a0-bc98-392c0fa909c9-config\") pod \"console-operator-7cd56b8dbc-gzf8k\" (UID: \"6f215b42-0f09-46a0-bc98-392c0fa909c9\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gzf8k"
Nov 03 19:31:27.858239 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.857981    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mh9g\" (UniqueName: \"kubernetes.io/projected/3880f3d3-aad1-43c0-b704-01374aad4f78-kube-api-access-8mh9g\") pod \"insights-operator-7c4986d449-8kh6f\" (UID: \"3880f3d3-aad1-43c0-b704-01374aad4f78\") " pod="openshift-insights/insights-operator-7c4986d449-8kh6f"
Nov 03 19:31:27.858239 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.858024    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/eb488ece-2ac3-499f-84c1-73534fcbb20b-nginx-conf\") pod \"networking-console-plugin-86c478bb88-zpgl6\" (UID: \"eb488ece-2ac3-499f-84c1-73534fcbb20b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zpgl6"
Nov 03 19:31:27.858239 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.858061    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/eb488ece-2ac3-499f-84c1-73534fcbb20b-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zpgl6\" (UID: \"eb488ece-2ac3-499f-84c1-73534fcbb20b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zpgl6"
Nov 03 19:31:27.858239 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.858113    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/96c016f7-9098-4a39-9e72-e54da374d8fc-image-registry-private-configuration\") pod \"image-registry-d74787d9c-nn7sk\" (UID: \"96c016f7-9098-4a39-9e72-e54da374d8fc\") " pod="openshift-image-registry/image-registry-d74787d9c-nn7sk"
Nov 03 19:31:27.858239 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.858140    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 03 19:31:27.858239 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.858160    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/96c016f7-9098-4a39-9e72-e54da374d8fc-registry-certificates\") pod \"image-registry-d74787d9c-nn7sk\" (UID: \"96c016f7-9098-4a39-9e72-e54da374d8fc\") " pod="openshift-image-registry/image-registry-d74787d9c-nn7sk"
Nov 03 19:31:27.858239 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.858179    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6f215b42-0f09-46a0-bc98-392c0fa909c9-config\") pod \"console-operator-7cd56b8dbc-gzf8k\" (UID: \"6f215b42-0f09-46a0-bc98-392c0fa909c9\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gzf8k"
Nov 03 19:31:27.858239 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.858180    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w42k\" (UniqueName: \"kubernetes.io/projected/96c016f7-9098-4a39-9e72-e54da374d8fc-kube-api-access-4w42k\") pod \"image-registry-d74787d9c-nn7sk\" (UID: \"96c016f7-9098-4a39-9e72-e54da374d8fc\") " pod="openshift-image-registry/image-registry-d74787d9c-nn7sk"
Nov 03 19:31:27.858605 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.858234    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q68w5\" (UniqueName: \"kubernetes.io/projected/9fb8c33c-fe15-4daa-a01b-4cb2fc7eae7b-kube-api-access-q68w5\") pod \"service-ca-operator-544d749c8c-mh2tn\" (UID: \"9fb8c33c-fe15-4daa-a01b-4cb2fc7eae7b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mh2tn"
Nov 03 19:31:27.858605 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.858277    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2287d\" (UniqueName: \"kubernetes.io/projected/1611b730-6163-4776-bd2d-3d10c2cc9a88-kube-api-access-2287d\") pod \"network-check-source-6f78d59f99-cbhrt\" (UID: \"1611b730-6163-4776-bd2d-3d10c2cc9a88\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cbhrt"
Nov 03 19:31:27.858605 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.858296    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/eb488ece-2ac3-499f-84c1-73534fcbb20b-nginx-conf\") pod \"networking-console-plugin-86c478bb88-zpgl6\" (UID: \"eb488ece-2ac3-499f-84c1-73534fcbb20b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zpgl6"
Nov 03 19:31:27.858605 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.858307    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktkfm\" (UniqueName: \"kubernetes.io/projected/df9f9c4a-d054-42cf-b17c-0d23acd0459b-kube-api-access-ktkfm\") pod \"kas-connection-checker-685895c7c9-xgbkd\" (UID: \"df9f9c4a-d054-42cf-b17c-0d23acd0459b\") " pod="kube-system/kas-connection-checker-685895c7c9-xgbkd"
Nov 03 19:31:27.858605 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.858334    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6f215b42-0f09-46a0-bc98-392c0fa909c9-serving-cert\") pod \"console-operator-7cd56b8dbc-gzf8k\" (UID: \"6f215b42-0f09-46a0-bc98-392c0fa909c9\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gzf8k"
Nov 03 19:31:27.858605 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.858364    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjtq4\" (UniqueName: \"kubernetes.io/projected/6f215b42-0f09-46a0-bc98-392c0fa909c9-kube-api-access-qjtq4\") pod \"console-operator-7cd56b8dbc-gzf8k\" (UID: \"6f215b42-0f09-46a0-bc98-392c0fa909c9\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gzf8k"
Nov 03 19:31:27.858605 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.858337    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/96c016f7-9098-4a39-9e72-e54da374d8fc-registry-certificates\") pod \"image-registry-d74787d9c-nn7sk\" (UID: \"96c016f7-9098-4a39-9e72-e54da374d8fc\") " pod="openshift-image-registry/image-registry-d74787d9c-nn7sk"
Nov 03 19:31:27.858605 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.858391    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/96c016f7-9098-4a39-9e72-e54da374d8fc-registry-tls\") pod \"image-registry-d74787d9c-nn7sk\" (UID: \"96c016f7-9098-4a39-9e72-e54da374d8fc\") " pod="openshift-image-registry/image-registry-d74787d9c-nn7sk"
Nov 03 19:31:27.858605 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.858427    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wk2r\" (UniqueName: \"kubernetes.io/projected/56e21184-e568-46bc-8e90-204684d6c366-kube-api-access-6wk2r\") pod \"node-resolver-p2p8d\" (UID: \"56e21184-e568-46bc-8e90-204684d6c366\") " pod="openshift-dns/node-resolver-p2p8d"
Nov 03 19:31:27.858605 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.858465    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/a023911d-3669-43e1-ac77-8a986f3715d6-default-certificate\") pod \"router-default-569d4c45f-7pszr\" (UID: \"a023911d-3669-43e1-ac77-8a986f3715d6\") " pod="openshift-ingress/router-default-569d4c45f-7pszr"
Nov 03 19:31:27.858605 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.858496    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1753fd31-8933-4567-8146-07f3fd2d10a5-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-7hcf8\" (UID: \"1753fd31-8933-4567-8146-07f3fd2d10a5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7hcf8"
Nov 03 19:31:27.858605 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.858565    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p288b\" (UniqueName: \"kubernetes.io/projected/f97b063e-8403-46bd-9716-72411589a494-kube-api-access-p288b\") pod \"kas-connection-checker-685895c7c9-wtgvw\" (UID: \"f97b063e-8403-46bd-9716-72411589a494\") " pod="kube-system/kas-connection-checker-685895c7c9-wtgvw"
Nov 03 19:31:27.858605 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.858591    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9fb8c33c-fe15-4daa-a01b-4cb2fc7eae7b-config\") pod \"service-ca-operator-544d749c8c-mh2tn\" (UID: \"9fb8c33c-fe15-4daa-a01b-4cb2fc7eae7b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mh2tn"
Nov 03 19:31:27.858605 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.858608    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/56e21184-e568-46bc-8e90-204684d6c366-hosts-file\") pod \"node-resolver-p2p8d\" (UID: \"56e21184-e568-46bc-8e90-204684d6c366\") " pod="openshift-dns/node-resolver-p2p8d"
Nov 03 19:31:27.859078 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.858626    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/a023911d-3669-43e1-ac77-8a986f3715d6-stats-auth\") pod \"router-default-569d4c45f-7pszr\" (UID: \"a023911d-3669-43e1-ac77-8a986f3715d6\") " pod="openshift-ingress/router-default-569d4c45f-7pszr"
Nov 03 19:31:27.859078 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.858651    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6f215b42-0f09-46a0-bc98-392c0fa909c9-trusted-ca\") pod \"console-operator-7cd56b8dbc-gzf8k\" (UID: \"6f215b42-0f09-46a0-bc98-392c0fa909c9\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gzf8k"
Nov 03 19:31:27.859078 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.858779    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/96c016f7-9098-4a39-9e72-e54da374d8fc-bound-sa-token\") pod \"image-registry-d74787d9c-nn7sk\" (UID: \"96c016f7-9098-4a39-9e72-e54da374d8fc\") " pod="openshift-image-registry/image-registry-d74787d9c-nn7sk"
Nov 03 19:31:27.859078 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.858798    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/1753fd31-8933-4567-8146-07f3fd2d10a5-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-7hcf8\" (UID: \"1753fd31-8933-4567-8146-07f3fd2d10a5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7hcf8"
Nov 03 19:31:27.859078 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.858816    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n7xj\" (UniqueName: \"kubernetes.io/projected/1753fd31-8933-4567-8146-07f3fd2d10a5-kube-api-access-6n7xj\") pod \"cluster-samples-operator-6ddb6c8b94-7hcf8\" (UID: \"1753fd31-8933-4567-8146-07f3fd2d10a5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7hcf8"
Nov 03 19:31:27.859078 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.858829    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9fb8c33c-fe15-4daa-a01b-4cb2fc7eae7b-config\") pod \"service-ca-operator-544d749c8c-mh2tn\" (UID: \"9fb8c33c-fe15-4daa-a01b-4cb2fc7eae7b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mh2tn"
Nov 03 19:31:27.859078 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.858836    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/96c016f7-9098-4a39-9e72-e54da374d8fc-trusted-ca\") pod \"image-registry-d74787d9c-nn7sk\" (UID: \"96c016f7-9098-4a39-9e72-e54da374d8fc\") " pod="openshift-image-registry/image-registry-d74787d9c-nn7sk"
Nov 03 19:31:27.859078 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.859044    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/1753fd31-8933-4567-8146-07f3fd2d10a5-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-7hcf8\" (UID: \"1753fd31-8933-4567-8146-07f3fd2d10a5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7hcf8"
Nov 03 19:31:27.859078 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.859059    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6f215b42-0f09-46a0-bc98-392c0fa909c9-trusted-ca\") pod \"console-operator-7cd56b8dbc-gzf8k\" (UID: \"6f215b42-0f09-46a0-bc98-392c0fa909c9\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gzf8k"
Nov 03 19:31:27.859316 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.859152    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/96c016f7-9098-4a39-9e72-e54da374d8fc-trusted-ca\") pod \"image-registry-d74787d9c-nn7sk\" (UID: \"96c016f7-9098-4a39-9e72-e54da374d8fc\") " pod="openshift-image-registry/image-registry-d74787d9c-nn7sk"
Nov 03 19:31:27.864026 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.863999    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftr5x\" (UniqueName: \"kubernetes.io/projected/a023911d-3669-43e1-ac77-8a986f3715d6-kube-api-access-ftr5x\") pod \"router-default-569d4c45f-7pszr\" (UID: \"a023911d-3669-43e1-ac77-8a986f3715d6\") " pod="openshift-ingress/router-default-569d4c45f-7pszr"
Nov 03 19:31:27.864113 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.864062    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdw57\" (UniqueName: \"kubernetes.io/projected/f89ef12d-9c60-4538-af9c-f97453bbc855-kube-api-access-xdw57\") pod \"volume-data-source-validator-58c9f5f46d-tx285\" (UID: \"f89ef12d-9c60-4538-af9c-f97453bbc855\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-tx285"
Nov 03 19:31:27.864113 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.864003    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f75xn\" (UniqueName: \"kubernetes.io/projected/b46f51b2-436b-4967-8879-a16b58930591-kube-api-access-f75xn\") pod \"network-check-target-njg27\" (UID: \"b46f51b2-436b-4967-8879-a16b58930591\") " pod="openshift-network-diagnostics/network-check-target-njg27"
Nov 03 19:31:27.864113 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.864105    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wk2r\" (UniqueName: \"kubernetes.io/projected/56e21184-e568-46bc-8e90-204684d6c366-kube-api-access-6wk2r\") pod \"node-resolver-p2p8d\" (UID: \"56e21184-e568-46bc-8e90-204684d6c366\") " pod="openshift-dns/node-resolver-p2p8d"
Nov 03 19:31:27.864282 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.864239    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtqhk\" (UniqueName: \"kubernetes.io/projected/8fa315db-e48d-4ac5-960a-23750974d431-kube-api-access-jtqhk\") pod \"kas-connection-checker-685895c7c9-42xn2\" (UID: \"8fa315db-e48d-4ac5-960a-23750974d431\") " pod="kube-system/kas-connection-checker-685895c7c9-42xn2"
Nov 03 19:31:27.864375 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.864357    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfg8j\" (UniqueName: \"kubernetes.io/projected/e10dc749-bc71-456e-a315-6581add01a7b-kube-api-access-nfg8j\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hxlrb\" (UID: \"e10dc749-bc71-456e-a315-6581add01a7b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hxlrb"
Nov 03 19:31:27.864580 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.864556    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz8xk\" (UniqueName: \"kubernetes.io/projected/e2b752d9-88c4-4dee-8b0c-7e9ac252adb3-kube-api-access-xz8xk\") pod \"cluster-monitoring-operator-884576c57-kfdsx\" (UID: \"e2b752d9-88c4-4dee-8b0c-7e9ac252adb3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kfdsx"
Nov 03 19:31:27.864918 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.864900    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktkfm\" (UniqueName: \"kubernetes.io/projected/df9f9c4a-d054-42cf-b17c-0d23acd0459b-kube-api-access-ktkfm\") pod \"kas-connection-checker-685895c7c9-xgbkd\" (UID: \"df9f9c4a-d054-42cf-b17c-0d23acd0459b\") " pod="kube-system/kas-connection-checker-685895c7c9-xgbkd"
Nov 03 19:31:27.864984 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.864928    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mh9g\" (UniqueName: \"kubernetes.io/projected/3880f3d3-aad1-43c0-b704-01374aad4f78-kube-api-access-8mh9g\") pod \"insights-operator-7c4986d449-8kh6f\" (UID: \"3880f3d3-aad1-43c0-b704-01374aad4f78\") " pod="openshift-insights/insights-operator-7c4986d449-8kh6f"
Nov 03 19:31:27.865073 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.865054    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p288b\" (UniqueName: \"kubernetes.io/projected/f97b063e-8403-46bd-9716-72411589a494-kube-api-access-p288b\") pod \"kas-connection-checker-685895c7c9-wtgvw\" (UID: \"f97b063e-8403-46bd-9716-72411589a494\") " pod="kube-system/kas-connection-checker-685895c7c9-wtgvw"
Nov 03 19:31:27.865128 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.865109    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q68w5\" (UniqueName: \"kubernetes.io/projected/9fb8c33c-fe15-4daa-a01b-4cb2fc7eae7b-kube-api-access-q68w5\") pod \"service-ca-operator-544d749c8c-mh2tn\" (UID: \"9fb8c33c-fe15-4daa-a01b-4cb2fc7eae7b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mh2tn"
Nov 03 19:31:27.865221 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.865208    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2287d\" (UniqueName: \"kubernetes.io/projected/1611b730-6163-4776-bd2d-3d10c2cc9a88-kube-api-access-2287d\") pod \"network-check-source-6f78d59f99-cbhrt\" (UID: \"1611b730-6163-4776-bd2d-3d10c2cc9a88\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cbhrt"
Nov 03 19:31:27.865251 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.865211    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w42k\" (UniqueName: \"kubernetes.io/projected/96c016f7-9098-4a39-9e72-e54da374d8fc-kube-api-access-4w42k\") pod \"image-registry-d74787d9c-nn7sk\" (UID: \"96c016f7-9098-4a39-9e72-e54da374d8fc\") " pod="openshift-image-registry/image-registry-d74787d9c-nn7sk"
Nov 03 19:31:27.865517 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.865501    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjtq4\" (UniqueName: \"kubernetes.io/projected/6f215b42-0f09-46a0-bc98-392c0fa909c9-kube-api-access-qjtq4\") pod \"console-operator-7cd56b8dbc-gzf8k\" (UID: \"6f215b42-0f09-46a0-bc98-392c0fa909c9\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gzf8k"
Nov 03 19:31:27.866171 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.866152    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/96c016f7-9098-4a39-9e72-e54da374d8fc-bound-sa-token\") pod \"image-registry-d74787d9c-nn7sk\" (UID: \"96c016f7-9098-4a39-9e72-e54da374d8fc\") " pod="openshift-image-registry/image-registry-d74787d9c-nn7sk"
Nov 03 19:31:27.866463 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.866447    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n7xj\" (UniqueName: \"kubernetes.io/projected/1753fd31-8933-4567-8146-07f3fd2d10a5-kube-api-access-6n7xj\") pod \"cluster-samples-operator-6ddb6c8b94-7hcf8\" (UID: \"1753fd31-8933-4567-8146-07f3fd2d10a5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7hcf8"
Nov 03 19:31:27.882306 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.882276    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-p2p8d_56e21184-e568-46bc-8e90-204684d6c366/dns-node-resolver/0.log"
Nov 03 19:31:27.916253 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.916207    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 19:31:27.916467 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.916446    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6p8wh\""
Nov 03 19:31:27.916602 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.916583    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-bh6bb\""
Nov 03 19:31:27.916658 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.916610    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9h8df\""
Nov 03 19:31:27.916737 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.916692    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 19:31:27.916737 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.916705    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-vgldx\""
Nov 03 19:31:27.916737 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.916719    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 19:31:27.916878 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.916767    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-gbtrh\""
Nov 03 19:31:27.916878 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.916775    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Nov 03 19:31:27.916878 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.916809    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Nov 03 19:31:27.916878 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.916823    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-skb8s\""
Nov 03 19:31:27.917054 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.916876    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-z89rv\""
Nov 03 19:31:27.917112 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.917066    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-mxrmf\""
Nov 03 19:31:27.917163 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.917129    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 19:31:27.917163 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.917129    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-kqxn2\""
Nov 03 19:31:27.917163 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.917158    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-glk7q\""
Nov 03 19:31:27.917267 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.917172    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 19:31:27.917304 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.917294    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-brhwf\""
Nov 03 19:31:27.917304 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.917300    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 19:31:27.917442 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.917426    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-77jlf\""
Nov 03 19:31:27.917504 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.917481    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Nov 03 19:31:27.917504 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.917484    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-cpz4k\""
Nov 03 19:31:27.917504 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.917484    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Nov 03 19:31:27.917603 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.917519    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Nov 03 19:31:27.917603 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.917527    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Nov 03 19:31:27.917702 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.917625    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-dspcg\""
Nov 03 19:31:27.917771 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.917754    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 03 19:31:27.917822 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.917762    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 03 19:31:27.917822 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.917791    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-sh2j9\""
Nov 03 19:31:27.917822 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.917795    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-wrft7\""
Nov 03 19:31:27.917822 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.917817    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Nov 03 19:31:27.917985 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.917834    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-wllhj\""
Nov 03 19:31:27.917985 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.917835    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-k6x7r\""
Nov 03 19:31:27.917985 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.917871    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-8fvqg\""
Nov 03 19:31:27.917985 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.917887    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pq7nf\""
Nov 03 19:31:27.917985 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.917887    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Nov 03 19:31:27.917985 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.917909    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 19:31:27.917985 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.917902    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Nov 03 19:31:27.918165 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.917915    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-n46sn\""
Nov 03 19:31:27.918165 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.917919    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 03 19:31:27.918165 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.917938    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-2lfv2\""
Nov 03 19:31:27.918165 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.917974    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-b925m\""
Nov 03 19:31:27.918165 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.918026    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 19:31:27.918165 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.918095    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-nblsj\""
Nov 03 19:31:27.918165 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.918130    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-rprdn\""
Nov 03 19:31:27.918349 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.918177    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Nov 03 19:31:27.918349 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.918205    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tdq2h\""
Nov 03 19:31:27.918349 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.918228    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-khl78\""
Nov 03 19:31:27.918469 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.918358    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/96c016f7-9098-4a39-9e72-e54da374d8fc-image-registry-private-configuration\") pod \"image-registry-d74787d9c-nn7sk\" (UID: \"96c016f7-9098-4a39-9e72-e54da374d8fc\") " pod="openshift-image-registry/image-registry-d74787d9c-nn7sk"
Nov 03 19:31:27.918541 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.918515    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6f215b42-0f09-46a0-bc98-392c0fa909c9-serving-cert\") pod \"console-operator-7cd56b8dbc-gzf8k\" (UID: \"6f215b42-0f09-46a0-bc98-392c0fa909c9\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gzf8k"
Nov 03 19:31:27.918980 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.918960    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/a023911d-3669-43e1-ac77-8a986f3715d6-stats-auth\") pod \"router-default-569d4c45f-7pszr\" (UID: \"a023911d-3669-43e1-ac77-8a986f3715d6\") " pod="openshift-ingress/router-default-569d4c45f-7pszr"
Nov 03 19:31:27.919059 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.918963    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/a023911d-3669-43e1-ac77-8a986f3715d6-default-certificate\") pod \"router-default-569d4c45f-7pszr\" (UID: \"a023911d-3669-43e1-ac77-8a986f3715d6\") " pod="openshift-ingress/router-default-569d4c45f-7pszr"
Nov 03 19:31:27.927423 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.927388    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/96c016f7-9098-4a39-9e72-e54da374d8fc-installation-pull-secrets\") pod \"image-registry-d74787d9c-nn7sk\" (UID: \"96c016f7-9098-4a39-9e72-e54da374d8fc\") " pod="openshift-image-registry/image-registry-d74787d9c-nn7sk"
Nov 03 19:31:27.927507 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.927439    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9fb8c33c-fe15-4daa-a01b-4cb2fc7eae7b-serving-cert\") pod \"service-ca-operator-544d749c8c-mh2tn\" (UID: \"9fb8c33c-fe15-4daa-a01b-4cb2fc7eae7b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mh2tn"
Nov 03 19:31:27.927587 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.927574    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e10dc749-bc71-456e-a315-6581add01a7b-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hxlrb\" (UID: \"e10dc749-bc71-456e-a315-6581add01a7b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hxlrb"
Nov 03 19:31:27.927825 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.927810    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3880f3d3-aad1-43c0-b704-01374aad4f78-serving-cert\") pod \"insights-operator-7c4986d449-8kh6f\" (UID: \"3880f3d3-aad1-43c0-b704-01374aad4f78\") " pod="openshift-insights/insights-operator-7c4986d449-8kh6f"
Nov 03 19:31:27.939445 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.939417    4921 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-222.us-east-2.compute.internal"
Nov 03 19:31:27.939557 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.939540    4921 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-222.us-east-2.compute.internal"
Nov 03 19:31:27.942483 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.942466    4921 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 19:31:27.959863 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.959830    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:31:27.960002 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.959869    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/38a6add1-b051-450b-bb96-8ace8a0ccff3-config-volume\") pod \"dns-default-wj8mp\" (UID: \"38a6add1-b051-450b-bb96-8ace8a0ccff3\") " pod="openshift-dns/dns-default-wj8mp"
Nov 03 19:31:27.960002 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.959891    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b797bdb0-b354-4dfc-a50a-3c60ecacbf71-system-cni-dir\") pod \"multus-additional-cni-plugins-g67bt\" (UID: \"b797bdb0-b354-4dfc-a50a-3c60ecacbf71\") " pod="openshift-multus/multus-additional-cni-plugins-g67bt"
Nov 03 19:31:27.960002 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.959907    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/eee821c4-6fb0-4f84-a58d-23dc2ff77720-dbus\") pod \"global-pull-secret-syncer-hrxdt\" (UID: \"eee821c4-6fb0-4f84-a58d-23dc2ff77720\") " pod="kube-system/global-pull-secret-syncer-hrxdt"
Nov 03 19:31:27.960002 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.959968    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25-etc-sysconfig\") pod \"tuned-l98pn\" (UID: \"0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25\") " pod="openshift-cluster-node-tuning-operator/tuned-l98pn"
Nov 03 19:31:27.960202 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.960029    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5pj6\" (UniqueName: \"kubernetes.io/projected/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25-kube-api-access-w5pj6\") pod \"tuned-l98pn\" (UID: \"0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25\") " pod="openshift-cluster-node-tuning-operator/tuned-l98pn"
Nov 03 19:31:27.960202 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.960091    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/653db542-8702-4eaa-abc1-a3bb866163c6-multus-daemon-config\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:31:27.960202 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.960173    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/38a6add1-b051-450b-bb96-8ace8a0ccff3-config-volume\") pod \"dns-default-wj8mp\" (UID: \"38a6add1-b051-450b-bb96-8ace8a0ccff3\") " pod="openshift-dns/dns-default-wj8mp"
Nov 03 19:31:27.960321 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.960250    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/eee821c4-6fb0-4f84-a58d-23dc2ff77720-kubelet-config\") pod \"global-pull-secret-syncer-hrxdt\" (UID: \"eee821c4-6fb0-4f84-a58d-23dc2ff77720\") " pod="kube-system/global-pull-secret-syncer-hrxdt"
Nov 03 19:31:27.960321 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.960283    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a023911d-3669-43e1-ac77-8a986f3715d6-service-ca-bundle\") pod \"router-default-569d4c45f-7pszr\" (UID: \"a023911d-3669-43e1-ac77-8a986f3715d6\") " pod="openshift-ingress/router-default-569d4c45f-7pszr"
Nov 03 19:31:27.960404 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.960347    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b797bdb0-b354-4dfc-a50a-3c60ecacbf71-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g67bt\" (UID: \"b797bdb0-b354-4dfc-a50a-3c60ecacbf71\") " pod="openshift-multus/multus-additional-cni-plugins-g67bt"
Nov 03 19:31:27.960461 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.960421    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/10137ba2-4beb-433e-b6df-199738351c61-data-volume\") pod \"insights-runtime-extractor-cvll4\" (UID: \"10137ba2-4beb-433e-b6df-199738351c61\") " pod="openshift-insights/insights-runtime-extractor-cvll4"
Nov 03 19:31:27.960461 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.960449    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-run-ovn\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:31:27.960461 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.960453    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/653db542-8702-4eaa-abc1-a3bb866163c6-multus-daemon-config\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:31:27.960606 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.960476    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eee821c4-6fb0-4f84-a58d-23dc2ff77720-original-pull-secret\") pod \"global-pull-secret-syncer-hrxdt\" (UID: \"eee821c4-6fb0-4f84-a58d-23dc2ff77720\") " pod="kube-system/global-pull-secret-syncer-hrxdt"
Nov 03 19:31:27.960606 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.960525    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25-run\") pod \"tuned-l98pn\" (UID: \"0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25\") " pod="openshift-cluster-node-tuning-operator/tuned-l98pn"
Nov 03 19:31:27.960606 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.960554    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-trusted-ca\") pod \"image-registry-c5c44f4d-dv455\" (UID: \"d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca\") " pod="openshift-image-registry/image-registry-c5c44f4d-dv455"
Nov 03 19:31:27.960606 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.960557    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/10137ba2-4beb-433e-b6df-199738351c61-data-volume\") pod \"insights-runtime-extractor-cvll4\" (UID: \"10137ba2-4beb-433e-b6df-199738351c61\") " pod="openshift-insights/insights-runtime-extractor-cvll4"
Nov 03 19:31:27.960606 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.960581    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b797bdb0-b354-4dfc-a50a-3c60ecacbf71-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g67bt\" (UID: \"b797bdb0-b354-4dfc-a50a-3c60ecacbf71\") " pod="openshift-multus/multus-additional-cni-plugins-g67bt"
Nov 03 19:31:27.960606 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.960581    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce-sys-fs\") pod \"aws-ebs-csi-driver-node-mcxcv\" (UID: \"8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mcxcv"
Nov 03 19:31:27.960907 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.960624    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-host-run-ovn-kubernetes\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:31:27.960907 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.960690    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/545f3c9a-2ac6-44a3-be0b-cf71b1592d42-serviceca\") pod \"node-ca-vv2s2\" (UID: \"545f3c9a-2ac6-44a3-be0b-cf71b1592d42\") " pod="openshift-image-registry/node-ca-vv2s2"
Nov 03 19:31:27.960907 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.960818    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25-lib-modules\") pod \"tuned-l98pn\" (UID: \"0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25\") " pod="openshift-cluster-node-tuning-operator/tuned-l98pn"
Nov 03 19:31:27.960907 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.960821    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eee821c4-6fb0-4f84-a58d-23dc2ff77720-original-pull-secret\") pod \"global-pull-secret-syncer-hrxdt\" (UID: \"eee821c4-6fb0-4f84-a58d-23dc2ff77720\") " pod="kube-system/global-pull-secret-syncer-hrxdt"
Nov 03 19:31:27.960907 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.960860    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce-device-dir\") pod \"aws-ebs-csi-driver-node-mcxcv\" (UID: \"8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mcxcv"
Nov 03 19:31:27.960907 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.960885    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtt8c\" (UniqueName: \"kubernetes.io/projected/8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce-kube-api-access-mtt8c\") pod \"aws-ebs-csi-driver-node-mcxcv\" (UID: \"8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mcxcv"
Nov 03 19:31:27.960907 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.960889    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/545f3c9a-2ac6-44a3-be0b-cf71b1592d42-serviceca\") pod \"node-ca-vv2s2\" (UID: \"545f3c9a-2ac6-44a3-be0b-cf71b1592d42\") " pod="openshift-image-registry/node-ca-vv2s2"
Nov 03 19:31:27.960907 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.960901    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lgzq\" (UniqueName: \"kubernetes.io/projected/220ed31c-1577-4c4b-ab5b-e812fd63d193-kube-api-access-5lgzq\") pod \"migrator-66df5d496b-5l649\" (UID: \"220ed31c-1577-4c4b-ab5b-e812fd63d193\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5l649"
Nov 03 19:31:27.961253 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.960942    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b797bdb0-b354-4dfc-a50a-3c60ecacbf71-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g67bt\" (UID: \"b797bdb0-b354-4dfc-a50a-3c60ecacbf71\") " pod="openshift-multus/multus-additional-cni-plugins-g67bt"
Nov 03 19:31:27.961253 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.960953    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a023911d-3669-43e1-ac77-8a986f3715d6-service-ca-bundle\") pod \"router-default-569d4c45f-7pszr\" (UID: \"a023911d-3669-43e1-ac77-8a986f3715d6\") " pod="openshift-ingress/router-default-569d4c45f-7pszr"
Nov 03 19:31:27.961253 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.960970    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38a6add1-b051-450b-bb96-8ace8a0ccff3-tmp-dir\") pod \"dns-default-wj8mp\" (UID: \"38a6add1-b051-450b-bb96-8ace8a0ccff3\") " pod="openshift-dns/dns-default-wj8mp"
Nov 03 19:31:27.961253 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.961048    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38a6add1-b051-450b-bb96-8ace8a0ccff3-tmp-dir\") pod \"dns-default-wj8mp\" (UID: \"38a6add1-b051-450b-bb96-8ace8a0ccff3\") " pod="openshift-dns/dns-default-wj8mp"
Nov 03 19:31:27.961411 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.961089    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-run-openvswitch\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:31:27.961465 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.961443    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1c00e787-a35d-4ac4-95a4-2546e29602aa-agent-certs\") pod \"konnectivity-agent-jwrp9\" (UID: \"1c00e787-a35d-4ac4-95a4-2546e29602aa\") " pod="kube-system/konnectivity-agent-jwrp9"
Nov 03 19:31:27.961519 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.961490    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/653db542-8702-4eaa-abc1-a3bb866163c6-host-run-k8s-cni-cncf-io\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:31:27.961579 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.961509    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b797bdb0-b354-4dfc-a50a-3c60ecacbf71-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g67bt\" (UID: \"b797bdb0-b354-4dfc-a50a-3c60ecacbf71\") " pod="openshift-multus/multus-additional-cni-plugins-g67bt"
Nov 03 19:31:27.961579 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.961525    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-host-cni-bin\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:31:27.961579 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.961568    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e2b752d9-88c4-4dee-8b0c-7e9ac252adb3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kfdsx\" (UID: \"e2b752d9-88c4-4dee-8b0c-7e9ac252adb3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kfdsx"
Nov 03 19:31:27.961740 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.961595    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/653db542-8702-4eaa-abc1-a3bb866163c6-multus-conf-dir\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:31:27.961740 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.961628    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-host-kubelet\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:31:27.961867 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.961801    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgtvt\" (UniqueName: \"kubernetes.io/projected/790607e1-d57c-4622-9a72-1af902ab1d85-kube-api-access-hgtvt\") pod \"network-metrics-daemon-rdr5d\" (UID: \"790607e1-d57c-4622-9a72-1af902ab1d85\") " pod="openshift-multus/network-metrics-daemon-rdr5d"
Nov 03 19:31:27.961867 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.961837    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/653db542-8702-4eaa-abc1-a3bb866163c6-host-var-lib-cni-bin\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:31:27.961867 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.961861    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/545f3c9a-2ac6-44a3-be0b-cf71b1592d42-host\") pod \"node-ca-vv2s2\" (UID: \"545f3c9a-2ac6-44a3-be0b-cf71b1592d42\") " pod="openshift-image-registry/node-ca-vv2s2"
Nov 03 19:31:27.961992 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.961881    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-trusted-ca\") pod \"image-registry-c5c44f4d-dv455\" (UID: \"d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca\") " pod="openshift-image-registry/image-registry-c5c44f4d-dv455"
Nov 03 19:31:27.961992 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.961891    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25-etc-modprobe-d\") pod \"tuned-l98pn\" (UID: \"0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25\") " pod="openshift-cluster-node-tuning-operator/tuned-l98pn"
Nov 03 19:31:27.961992 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.961903    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1c00e787-a35d-4ac4-95a4-2546e29602aa-agent-certs\") pod \"konnectivity-agent-jwrp9\" (UID: \"1c00e787-a35d-4ac4-95a4-2546e29602aa\") " pod="kube-system/konnectivity-agent-jwrp9"
Nov 03 19:31:27.961992 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.961954    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25-etc-kubernetes\") pod \"tuned-l98pn\" (UID: \"0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25\") " pod="openshift-cluster-node-tuning-operator/tuned-l98pn"
Nov 03 19:31:27.962165 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.962017    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a023911d-3669-43e1-ac77-8a986f3715d6-metrics-certs\") pod \"router-default-569d4c45f-7pszr\" (UID: \"a023911d-3669-43e1-ac77-8a986f3715d6\") " pod="openshift-ingress/router-default-569d4c45f-7pszr"
Nov 03 19:31:27.962165 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:31:27.962007    4921 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 19:31:27.962165 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.962054    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25-host\") pod \"tuned-l98pn\" (UID: \"0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25\") " pod="openshift-cluster-node-tuning-operator/tuned-l98pn"
Nov 03 19:31:27.962165 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.962082    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/653db542-8702-4eaa-abc1-a3bb866163c6-cni-binary-copy\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:31:27.962165 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:31:27.962120    4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2b752d9-88c4-4dee-8b0c-7e9ac252adb3-cluster-monitoring-operator-tls podName:e2b752d9-88c4-4dee-8b0c-7e9ac252adb3 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:28.462093003 +0000 UTC m=+1.706058476 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e2b752d9-88c4-4dee-8b0c-7e9ac252adb3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-kfdsx" (UID: "e2b752d9-88c4-4dee-8b0c-7e9ac252adb3") : secret "cluster-monitoring-operator-tls" not found
Nov 03 19:31:27.962394 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.962172    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/653db542-8702-4eaa-abc1-a3bb866163c6-host-var-lib-cni-multus\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:31:27.962394 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.962208    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-installation-pull-secrets\") pod \"image-registry-c5c44f4d-dv455\" (UID: \"d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca\") " pod="openshift-image-registry/image-registry-c5c44f4d-dv455"
Nov 03 19:31:27.962394 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.962245    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g8fh\" (UniqueName: \"kubernetes.io/projected/8b5e1b09-699d-4db8-9eb0-0b71bdda858f-kube-api-access-2g8fh\") pod \"iptables-alerter-wkrtt\" (UID: \"8b5e1b09-699d-4db8-9eb0-0b71bdda858f\") " pod="openshift-network-operator/iptables-alerter-wkrtt"
Nov 03 19:31:27.962394 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.962270    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce-registration-dir\") pod \"aws-ebs-csi-driver-node-mcxcv\" (UID: \"8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mcxcv"
Nov 03 19:31:27.962394 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.962305    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/eb488ece-2ac3-499f-84c1-73534fcbb20b-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zpgl6\" (UID: \"eb488ece-2ac3-499f-84c1-73534fcbb20b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zpgl6"
Nov 03 19:31:27.962613 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.962577    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b797bdb0-b354-4dfc-a50a-3c60ecacbf71-cnibin\") pod \"multus-additional-cni-plugins-g67bt\" (UID: \"b797bdb0-b354-4dfc-a50a-3c60ecacbf71\") " pod="openshift-multus/multus-additional-cni-plugins-g67bt"
Nov 03 19:31:27.962613 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:31:27.962584    4921 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 03 19:31:27.962733 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.962617    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25-etc-sysctl-conf\") pod \"tuned-l98pn\" (UID: \"0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25\") " pod="openshift-cluster-node-tuning-operator/tuned-l98pn"
Nov 03 19:31:27.962733 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.962646    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d0ab7c64-a85f-4713-a270-f0cf711fcee5-env-overrides\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:31:27.962733 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:31:27.962664    4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a023911d-3669-43e1-ac77-8a986f3715d6-metrics-certs podName:a023911d-3669-43e1-ac77-8a986f3715d6 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:28.462646873 +0000 UTC m=+1.706612334 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a023911d-3669-43e1-ac77-8a986f3715d6-metrics-certs") pod "router-default-569d4c45f-7pszr" (UID: "a023911d-3669-43e1-ac77-8a986f3715d6") : secret "router-metrics-certs-default" not found
Nov 03 19:31:27.962733 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.962719    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/96c016f7-9098-4a39-9e72-e54da374d8fc-registry-tls\") pod \"image-registry-d74787d9c-nn7sk\" (UID: \"96c016f7-9098-4a39-9e72-e54da374d8fc\") " pod="openshift-image-registry/image-registry-d74787d9c-nn7sk"
Nov 03 19:31:27.962919 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.962759    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/10137ba2-4beb-433e-b6df-199738351c61-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cvll4\" (UID: \"10137ba2-4beb-433e-b6df-199738351c61\") " pod="openshift-insights/insights-runtime-extractor-cvll4"
Nov 03 19:31:27.962919 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.962790    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/653db542-8702-4eaa-abc1-a3bb866163c6-host-run-multus-certs\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:31:27.962919 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.962856    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d0ab7c64-a85f-4713-a270-f0cf711fcee5-env-overrides\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:31:27.963046 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.962928    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpwxv\" (UniqueName: \"kubernetes.io/projected/653db542-8702-4eaa-abc1-a3bb866163c6-kube-api-access-mpwxv\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:31:27.963046 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.962962    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d0ab7c64-a85f-4713-a270-f0cf711fcee5-ovnkube-script-lib\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:31:27.963046 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.962997    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/38a6add1-b051-450b-bb96-8ace8a0ccff3-metrics-tls\") pod \"dns-default-wj8mp\" (UID: \"38a6add1-b051-450b-bb96-8ace8a0ccff3\") " pod="openshift-dns/dns-default-wj8mp"
Nov 03 19:31:27.963046 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.963003    4921 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 19:31:27.963046 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.963028    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-registry-tls\") pod \"image-registry-c5c44f4d-dv455\" (UID: \"d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca\") " pod="openshift-image-registry/image-registry-c5c44f4d-dv455"
Nov 03 19:31:27.963274 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.963058    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/653db542-8702-4eaa-abc1-a3bb866163c6-host-run-netns\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:31:27.963274 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.963089    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-etc-openvswitch\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:31:27.963274 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.963129    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngpxm\" (UniqueName: \"kubernetes.io/projected/38a6add1-b051-450b-bb96-8ace8a0ccff3-kube-api-access-ngpxm\") pod \"dns-default-wj8mp\" (UID: \"38a6add1-b051-450b-bb96-8ace8a0ccff3\") " pod="openshift-dns/dns-default-wj8mp"
Nov 03 19:31:27.963274 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.963171    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1c00e787-a35d-4ac4-95a4-2546e29602aa-konnectivity-ca\") pod \"konnectivity-agent-jwrp9\" (UID: \"1c00e787-a35d-4ac4-95a4-2546e29602aa\") " pod="kube-system/konnectivity-agent-jwrp9"
Nov 03 19:31:27.963450 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.963395    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mcxcv\" (UID: \"8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mcxcv"
Nov 03 19:31:27.963450 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.963430    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b797bdb0-b354-4dfc-a50a-3c60ecacbf71-cni-binary-copy\") pod \"multus-additional-cni-plugins-g67bt\" (UID: \"b797bdb0-b354-4dfc-a50a-3c60ecacbf71\") " pod="openshift-multus/multus-additional-cni-plugins-g67bt"
Nov 03 19:31:27.963553 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.963466    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55hqg\" (UniqueName: \"kubernetes.io/projected/545f3c9a-2ac6-44a3-be0b-cf71b1592d42-kube-api-access-55hqg\") pod \"node-ca-vv2s2\" (UID: \"545f3c9a-2ac6-44a3-be0b-cf71b1592d42\") " pod="openshift-image-registry/node-ca-vv2s2"
Nov 03 19:31:27.963553 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.963501    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25-etc-sysctl-d\") pod \"tuned-l98pn\" (UID: \"0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25\") " pod="openshift-cluster-node-tuning-operator/tuned-l98pn"
Nov 03 19:31:27.963553 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.963532    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/653db542-8702-4eaa-abc1-a3bb866163c6-multus-socket-dir-parent\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:31:27.963763 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.963565    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8b5e1b09-699d-4db8-9eb0-0b71bdda858f-iptables-alerter-script\") pod \"iptables-alerter-wkrtt\" (UID: \"8b5e1b09-699d-4db8-9eb0-0b71bdda858f\") " pod="openshift-network-operator/iptables-alerter-wkrtt"
Nov 03 19:31:27.963763 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.963592    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8b5e1b09-699d-4db8-9eb0-0b71bdda858f-host-slash\") pod \"iptables-alerter-wkrtt\" (UID: \"8b5e1b09-699d-4db8-9eb0-0b71bdda858f\") " pod="openshift-network-operator/iptables-alerter-wkrtt"
Nov 03 19:31:27.963763 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.963625    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/10137ba2-4beb-433e-b6df-199738351c61-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cvll4\" (UID: \"10137ba2-4beb-433e-b6df-199738351c61\") " pod="openshift-insights/insights-runtime-extractor-cvll4"
Nov 03 19:31:27.963763 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.963658    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d0ab7c64-a85f-4713-a270-f0cf711fcee5-ovn-node-metrics-cert\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:31:27.963763 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.963707    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/543b8778-baa4-4e09-b697-68e550e42942-cert\") pod \"ingress-canary-j7m2z\" (UID: \"543b8778-baa4-4e09-b697-68e550e42942\") " pod="openshift-ingress-canary/ingress-canary-j7m2z"
Nov 03 19:31:27.963763 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.963725    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d0ab7c64-a85f-4713-a270-f0cf711fcee5-ovnkube-script-lib\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:31:27.963763 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.963734    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-ca-trust-extracted\") pod \"image-registry-c5c44f4d-dv455\" (UID: \"d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca\") " pod="openshift-image-registry/image-registry-c5c44f4d-dv455"
Nov 03 19:31:27.964094 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.963771    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/10137ba2-4beb-433e-b6df-199738351c61-crio-socket\") pod \"insights-runtime-extractor-cvll4\" (UID: \"10137ba2-4beb-433e-b6df-199738351c61\") " pod="openshift-insights/insights-runtime-extractor-cvll4"
Nov 03 19:31:27.964150 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.964122    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-ca-trust-extracted\") pod \"image-registry-c5c44f4d-dv455\" (UID: \"d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca\") " pod="openshift-image-registry/image-registry-c5c44f4d-dv455"
Nov 03 19:31:27.964298 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.964275    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/653db542-8702-4eaa-abc1-a3bb866163c6-cni-binary-copy\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:31:27.964351 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.964302    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-installation-pull-secrets\") pod \"image-registry-c5c44f4d-dv455\" (UID: \"d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca\") " pod="openshift-image-registry/image-registry-c5c44f4d-dv455"
Nov 03 19:31:27.964351 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.964307    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1c00e787-a35d-4ac4-95a4-2546e29602aa-konnectivity-ca\") pod \"konnectivity-agent-jwrp9\" (UID: \"1c00e787-a35d-4ac4-95a4-2546e29602aa\") " pod="kube-system/konnectivity-agent-jwrp9"
Nov 03 19:31:27.964351 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.964333    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/653db542-8702-4eaa-abc1-a3bb866163c6-host-var-lib-kubelet\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:31:27.964490 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.964373    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-registry-certificates\") pod \"image-registry-c5c44f4d-dv455\" (UID: \"d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca\") " pod="openshift-image-registry/image-registry-c5c44f4d-dv455"
Nov 03 19:31:27.964490 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.964407    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gm62\" (UniqueName: \"kubernetes.io/projected/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-kube-api-access-8gm62\") pod \"image-registry-c5c44f4d-dv455\" (UID: \"d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca\") " pod="openshift-image-registry/image-registry-c5c44f4d-dv455"
Nov 03 19:31:27.964490 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.964440    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce-etc-selinux\") pod \"aws-ebs-csi-driver-node-mcxcv\" (UID: \"8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mcxcv"
Nov 03 19:31:27.964490 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.964473    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s4qj\" (UniqueName: \"kubernetes.io/projected/10137ba2-4beb-433e-b6df-199738351c61-kube-api-access-2s4qj\") pod \"insights-runtime-extractor-cvll4\" (UID: \"10137ba2-4beb-433e-b6df-199738351c61\") " pod="openshift-insights/insights-runtime-extractor-cvll4"
Nov 03 19:31:27.964699 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.964554    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-bound-sa-token\") pod \"image-registry-c5c44f4d-dv455\" (UID: \"d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca\") " pod="openshift-image-registry/image-registry-c5c44f4d-dv455"
Nov 03 19:31:27.964699 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.964591    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d0ab7c64-a85f-4713-a270-f0cf711fcee5-ovnkube-config\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:31:27.964699 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.964654    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/653db542-8702-4eaa-abc1-a3bb866163c6-cnibin\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:31:27.964861 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.964704    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj2lf\" (UniqueName: \"kubernetes.io/projected/543b8778-baa4-4e09-b697-68e550e42942-kube-api-access-gj2lf\") pod \"ingress-canary-j7m2z\" (UID: \"543b8778-baa4-4e09-b697-68e550e42942\") " pod="openshift-ingress-canary/ingress-canary-j7m2z"
Nov 03 19:31:27.964861 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.964719    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d0ab7c64-a85f-4713-a270-f0cf711fcee5-ovn-node-metrics-cert\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:31:27.964861 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.964732    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/56e21184-e568-46bc-8e90-204684d6c366-hosts-file\") pod \"node-resolver-p2p8d\" (UID: \"56e21184-e568-46bc-8e90-204684d6c366\") " pod="openshift-dns/node-resolver-p2p8d"
Nov 03 19:31:27.965002 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.964909    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25-etc-tuned\") pod \"tuned-l98pn\" (UID: \"0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25\") " pod="openshift-cluster-node-tuning-operator/tuned-l98pn"
Nov 03 19:31:27.965002 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.964914    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b797bdb0-b354-4dfc-a50a-3c60ecacbf71-cni-binary-copy\") pod \"multus-additional-cni-plugins-g67bt\" (UID: \"b797bdb0-b354-4dfc-a50a-3c60ecacbf71\") " pod="openshift-multus/multus-additional-cni-plugins-g67bt"
Nov 03 19:31:27.965002 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.964957    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/653db542-8702-4eaa-abc1-a3bb866163c6-system-cni-dir\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:31:27.965002 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.964994    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/790607e1-d57c-4622-9a72-1af902ab1d85-metrics-certs\") pod \"network-metrics-daemon-rdr5d\" (UID: \"790607e1-d57c-4622-9a72-1af902ab1d85\") " pod="openshift-multus/network-metrics-daemon-rdr5d"
Nov 03 19:31:27.965186 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.965002    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25-etc-tuned\") pod \"tuned-l98pn\" (UID: \"0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25\") " pod="openshift-cluster-node-tuning-operator/tuned-l98pn"
Nov 03 19:31:27.965186 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.965027    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce-socket-dir\") pod \"aws-ebs-csi-driver-node-mcxcv\" (UID: \"8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mcxcv"
Nov 03 19:31:27.965186 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.965057    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25-etc-systemd\") pod \"tuned-l98pn\" (UID: \"0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25\") " pod="openshift-cluster-node-tuning-operator/tuned-l98pn"
Nov 03 19:31:27.965186 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.965122    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25-var-lib-kubelet\") pod \"tuned-l98pn\" (UID: \"0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25\") " pod="openshift-cluster-node-tuning-operator/tuned-l98pn"
Nov 03 19:31:27.965186 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.965155    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-host-cni-netd\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:31:27.965186 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.965179    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/56e21184-e568-46bc-8e90-204684d6c366-hosts-file\") pod \"node-resolver-p2p8d\" (UID: \"56e21184-e568-46bc-8e90-204684d6c366\") " pod="openshift-dns/node-resolver-p2p8d"
Nov 03 19:31:27.965453 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.965232    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d0ab7c64-a85f-4713-a270-f0cf711fcee5-ovnkube-config\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:31:27.965453 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:31:27.965262    4921 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 19:31:27.965453 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:31:27.965283    4921 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d74787d9c-nn7sk: secret "image-registry-tls" not found
Nov 03 19:31:27.965453 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:31:27.965388    4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/96c016f7-9098-4a39-9e72-e54da374d8fc-registry-tls podName:96c016f7-9098-4a39-9e72-e54da374d8fc nodeName:}" failed. No retries permitted until 2026-11-03 19:31:28.465371181 +0000 UTC m=+1.709336658 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/96c016f7-9098-4a39-9e72-e54da374d8fc-registry-tls") pod "image-registry-d74787d9c-nn7sk" (UID: "96c016f7-9098-4a39-9e72-e54da374d8fc") : secret "image-registry-tls" not found
Nov 03 19:31:27.965938 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.965625    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8b5e1b09-699d-4db8-9eb0-0b71bdda858f-iptables-alerter-script\") pod \"iptables-alerter-wkrtt\" (UID: \"8b5e1b09-699d-4db8-9eb0-0b71bdda858f\") " pod="openshift-network-operator/iptables-alerter-wkrtt"
Nov 03 19:31:27.965938 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.965684    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-registry-certificates\") pod \"image-registry-c5c44f4d-dv455\" (UID: \"d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca\") " pod="openshift-image-registry/image-registry-c5c44f4d-dv455"
Nov 03 19:31:27.965938 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.965715    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr684\" (UniqueName: \"kubernetes.io/projected/b797bdb0-b354-4dfc-a50a-3c60ecacbf71-kube-api-access-xr684\") pod \"multus-additional-cni-plugins-g67bt\" (UID: \"b797bdb0-b354-4dfc-a50a-3c60ecacbf71\") " pod="openshift-multus/multus-additional-cni-plugins-g67bt"
Nov 03 19:31:27.965938 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.965907    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/543b8778-baa4-4e09-b697-68e550e42942-cert\") pod \"ingress-canary-j7m2z\" (UID: \"543b8778-baa4-4e09-b697-68e550e42942\") " pod="openshift-ingress-canary/ingress-canary-j7m2z"
Nov 03 19:31:27.967221 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.967197    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25-sys\") pod \"tuned-l98pn\" (UID: \"0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25\") " pod="openshift-cluster-node-tuning-operator/tuned-l98pn"
Nov 03 19:31:27.967645 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.967620    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-systemd-units\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:31:27.967777 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.967735    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-run-systemd\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:31:27.967837 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.967786    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/653db542-8702-4eaa-abc1-a3bb866163c6-etc-kubernetes\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:31:27.967913 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.967889    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-host-slash\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:31:27.968003 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.967928    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-node-log\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:31:27.968003 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.967953    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/eb488ece-2ac3-499f-84c1-73534fcbb20b-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zpgl6\" (UID: \"eb488ece-2ac3-499f-84c1-73534fcbb20b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zpgl6"
Nov 03 19:31:27.968003 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.967991    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-var-lib-openvswitch\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:31:27.968117 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.968054    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-image-registry-private-configuration\") pod \"image-registry-c5c44f4d-dv455\" (UID: \"d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca\") " pod="openshift-image-registry/image-registry-c5c44f4d-dv455"
Nov 03 19:31:27.968187 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.968168    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b797bdb0-b354-4dfc-a50a-3c60ecacbf71-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g67bt\" (UID: \"b797bdb0-b354-4dfc-a50a-3c60ecacbf71\") " pod="openshift-multus/multus-additional-cni-plugins-g67bt"
Nov 03 19:31:27.968232 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.968214    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-log-socket\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:31:27.968267 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.968243    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc8v6\" (UniqueName: \"kubernetes.io/projected/d0ab7c64-a85f-4713-a270-f0cf711fcee5-kube-api-access-jc8v6\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:31:27.968267 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.968256    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-image-registry-private-configuration\") pod \"image-registry-c5c44f4d-dv455\" (UID: \"d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca\") " pod="openshift-image-registry/image-registry-c5c44f4d-dv455"
Nov 03 19:31:27.968344 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.968284    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1753fd31-8933-4567-8146-07f3fd2d10a5-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-7hcf8\" (UID: \"1753fd31-8933-4567-8146-07f3fd2d10a5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7hcf8"
Nov 03 19:31:27.968410 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.968393    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-host-run-netns\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:31:27.968522 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.968503    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25-tmp\") pod \"tuned-l98pn\" (UID: \"0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25\") " pod="openshift-cluster-node-tuning-operator/tuned-l98pn"
Nov 03 19:31:27.968592 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.968576    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/653db542-8702-4eaa-abc1-a3bb866163c6-multus-cni-dir\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:31:27.968643 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.968609    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/653db542-8702-4eaa-abc1-a3bb866163c6-hostroot\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:31:27.968643 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.968626    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25-tmp\") pod \"tuned-l98pn\" (UID: \"0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25\") " pod="openshift-cluster-node-tuning-operator/tuned-l98pn"
Nov 03 19:31:27.970697 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.970680    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1753fd31-8933-4567-8146-07f3fd2d10a5-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-7hcf8\" (UID: \"1753fd31-8933-4567-8146-07f3fd2d10a5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7hcf8"
Nov 03 19:31:27.971804 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.971783    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lgzq\" (UniqueName: \"kubernetes.io/projected/220ed31c-1577-4c4b-ab5b-e812fd63d193-kube-api-access-5lgzq\") pod \"migrator-66df5d496b-5l649\" (UID: \"220ed31c-1577-4c4b-ab5b-e812fd63d193\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5l649"
Nov 03 19:31:27.972129 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.972109    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgtvt\" (UniqueName: \"kubernetes.io/projected/790607e1-d57c-4622-9a72-1af902ab1d85-kube-api-access-hgtvt\") pod \"network-metrics-daemon-rdr5d\" (UID: \"790607e1-d57c-4622-9a72-1af902ab1d85\") " pod="openshift-multus/network-metrics-daemon-rdr5d"
Nov 03 19:31:27.972223 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.972205    4921 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 19:31:27.972301 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:31:27.972244    4921 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-222.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-222.us-east-2.compute.internal"
Nov 03 19:31:27.972301 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.972205    4921 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 19:31:27.972401 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:31:27.972350    4921 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-222.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-222.us-east-2.compute.internal"
Nov 03 19:31:27.972762 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.972742    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5pj6\" (UniqueName: \"kubernetes.io/projected/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25-kube-api-access-w5pj6\") pod \"tuned-l98pn\" (UID: \"0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25\") " pod="openshift-cluster-node-tuning-operator/tuned-l98pn"
Nov 03 19:31:27.973711 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.973693    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g8fh\" (UniqueName: \"kubernetes.io/projected/8b5e1b09-699d-4db8-9eb0-0b71bdda858f-kube-api-access-2g8fh\") pod \"iptables-alerter-wkrtt\" (UID: \"8b5e1b09-699d-4db8-9eb0-0b71bdda858f\") " pod="openshift-network-operator/iptables-alerter-wkrtt"
Nov 03 19:31:27.973765 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.973697    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtt8c\" (UniqueName: \"kubernetes.io/projected/8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce-kube-api-access-mtt8c\") pod \"aws-ebs-csi-driver-node-mcxcv\" (UID: \"8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mcxcv"
Nov 03 19:31:27.974627 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.974612    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngpxm\" (UniqueName: \"kubernetes.io/projected/38a6add1-b051-450b-bb96-8ace8a0ccff3-kube-api-access-ngpxm\") pod \"dns-default-wj8mp\" (UID: \"38a6add1-b051-450b-bb96-8ace8a0ccff3\") " pod="openshift-dns/dns-default-wj8mp"
Nov 03 19:31:27.976708 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.976691    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr684\" (UniqueName: \"kubernetes.io/projected/b797bdb0-b354-4dfc-a50a-3c60ecacbf71-kube-api-access-xr684\") pod \"multus-additional-cni-plugins-g67bt\" (UID: \"b797bdb0-b354-4dfc-a50a-3c60ecacbf71\") " pod="openshift-multus/multus-additional-cni-plugins-g67bt"
Nov 03 19:31:27.985956 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.985934    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55hqg\" (UniqueName: \"kubernetes.io/projected/545f3c9a-2ac6-44a3-be0b-cf71b1592d42-kube-api-access-55hqg\") pod \"node-ca-vv2s2\" (UID: \"545f3c9a-2ac6-44a3-be0b-cf71b1592d42\") " pod="openshift-image-registry/node-ca-vv2s2"
Nov 03 19:31:27.985956 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.985947    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s4qj\" (UniqueName: \"kubernetes.io/projected/10137ba2-4beb-433e-b6df-199738351c61-kube-api-access-2s4qj\") pod \"insights-runtime-extractor-cvll4\" (UID: \"10137ba2-4beb-433e-b6df-199738351c61\") " pod="openshift-insights/insights-runtime-extractor-cvll4"
Nov 03 19:31:27.986116 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.985934    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc8v6\" (UniqueName: \"kubernetes.io/projected/d0ab7c64-a85f-4713-a270-f0cf711fcee5-kube-api-access-jc8v6\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:31:27.987517 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.987500    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-bound-sa-token\") pod \"image-registry-c5c44f4d-dv455\" (UID: \"d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca\") " pod="openshift-image-registry/image-registry-c5c44f4d-dv455"
Nov 03 19:31:27.987575 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.987560    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpwxv\" (UniqueName: \"kubernetes.io/projected/653db542-8702-4eaa-abc1-a3bb866163c6-kube-api-access-mpwxv\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:31:27.987789 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.987772    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj2lf\" (UniqueName: \"kubernetes.io/projected/543b8778-baa4-4e09-b697-68e550e42942-kube-api-access-gj2lf\") pod \"ingress-canary-j7m2z\" (UID: \"543b8778-baa4-4e09-b697-68e550e42942\") " pod="openshift-ingress-canary/ingress-canary-j7m2z"
Nov 03 19:31:27.988784 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:27.988768    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gm62\" (UniqueName: \"kubernetes.io/projected/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-kube-api-access-8gm62\") pod \"image-registry-c5c44f4d-dv455\" (UID: \"d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca\") " pod="openshift-image-registry/image-registry-c5c44f4d-dv455"
Nov 03 19:31:28.069830 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.069789    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25-etc-systemd\") pod \"tuned-l98pn\" (UID: \"0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25\") " pod="openshift-cluster-node-tuning-operator/tuned-l98pn"
Nov 03 19:31:28.069830 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.069827    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25-var-lib-kubelet\") pod \"tuned-l98pn\" (UID: \"0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25\") " pod="openshift-cluster-node-tuning-operator/tuned-l98pn"
Nov 03 19:31:28.070031 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.069844    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-host-cni-netd\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:31:28.070031 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.069864    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25-sys\") pod \"tuned-l98pn\" (UID: \"0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25\") " pod="openshift-cluster-node-tuning-operator/tuned-l98pn"
Nov 03 19:31:28.070031 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.069879    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-systemd-units\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:31:28.070031 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.069899    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-run-systemd\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:31:28.070031 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.069923    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/653db542-8702-4eaa-abc1-a3bb866163c6-etc-kubernetes\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:31:28.070031 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.069927    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25-etc-systemd\") pod \"tuned-l98pn\" (UID: \"0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25\") " pod="openshift-cluster-node-tuning-operator/tuned-l98pn"
Nov 03 19:31:28.070031 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.069929    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-host-cni-netd\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:31:28.070031 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.069929    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25-var-lib-kubelet\") pod \"tuned-l98pn\" (UID: \"0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25\") " pod="openshift-cluster-node-tuning-operator/tuned-l98pn"
Nov 03 19:31:28.070312 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.069946    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-host-slash\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:31:28.070312 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.070061    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-node-log\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:31:28.070312 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.070078    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-var-lib-openvswitch\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:31:28.070312 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.069956    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25-sys\") pod \"tuned-l98pn\" (UID: \"0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25\") " pod="openshift-cluster-node-tuning-operator/tuned-l98pn"
Nov 03 19:31:28.070312 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.070100    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b797bdb0-b354-4dfc-a50a-3c60ecacbf71-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g67bt\" (UID: \"b797bdb0-b354-4dfc-a50a-3c60ecacbf71\") " pod="openshift-multus/multus-additional-cni-plugins-g67bt"
Nov 03 19:31:28.070312 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.069985    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/653db542-8702-4eaa-abc1-a3bb866163c6-etc-kubernetes\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:31:28.070312 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.069974    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-run-systemd\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:31:28.070312 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.070120    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-node-log\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:31:28.070312 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.070106    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-var-lib-openvswitch\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:31:28.070312 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.069976    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-host-slash\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:31:28.070312 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.069962    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-systemd-units\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:31:28.070312 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.070146    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-log-socket\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:31:28.070312 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.070190    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b797bdb0-b354-4dfc-a50a-3c60ecacbf71-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g67bt\" (UID: \"b797bdb0-b354-4dfc-a50a-3c60ecacbf71\") " pod="openshift-multus/multus-additional-cni-plugins-g67bt"
Nov 03 19:31:28.070312 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.070215    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-host-run-netns\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:31:28.070312 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.070224    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-log-socket\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:31:28.070312 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.070254    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-host-run-netns\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:31:28.070312 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.070276    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/653db542-8702-4eaa-abc1-a3bb866163c6-multus-cni-dir\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:31:28.070312 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.070300    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/653db542-8702-4eaa-abc1-a3bb866163c6-hostroot\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:31:28.070312 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.070316    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:31:28.071001 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.070341    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b797bdb0-b354-4dfc-a50a-3c60ecacbf71-system-cni-dir\") pod \"multus-additional-cni-plugins-g67bt\" (UID: \"b797bdb0-b354-4dfc-a50a-3c60ecacbf71\") " pod="openshift-multus/multus-additional-cni-plugins-g67bt"
Nov 03 19:31:28.071001 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.070358    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/653db542-8702-4eaa-abc1-a3bb866163c6-hostroot\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:31:28.071001 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.070377    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/653db542-8702-4eaa-abc1-a3bb866163c6-multus-cni-dir\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:31:28.071001 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.070402    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/eee821c4-6fb0-4f84-a58d-23dc2ff77720-dbus\") pod \"global-pull-secret-syncer-hrxdt\" (UID: \"eee821c4-6fb0-4f84-a58d-23dc2ff77720\") " pod="kube-system/global-pull-secret-syncer-hrxdt"
Nov 03 19:31:28.071001 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.070422    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25-etc-sysconfig\") pod \"tuned-l98pn\" (UID: \"0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25\") " pod="openshift-cluster-node-tuning-operator/tuned-l98pn"
Nov 03 19:31:28.071001 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.070438    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b797bdb0-b354-4dfc-a50a-3c60ecacbf71-system-cni-dir\") pod \"multus-additional-cni-plugins-g67bt\" (UID: \"b797bdb0-b354-4dfc-a50a-3c60ecacbf71\") " pod="openshift-multus/multus-additional-cni-plugins-g67bt"
Nov 03 19:31:28.071001 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.070480    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/eee821c4-6fb0-4f84-a58d-23dc2ff77720-dbus\") pod \"global-pull-secret-syncer-hrxdt\" (UID: \"eee821c4-6fb0-4f84-a58d-23dc2ff77720\") " pod="kube-system/global-pull-secret-syncer-hrxdt"
Nov 03 19:31:28.071001 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.070508    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25-etc-sysconfig\") pod \"tuned-l98pn\" (UID: \"0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25\") " pod="openshift-cluster-node-tuning-operator/tuned-l98pn"
Nov 03 19:31:28.071001 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.070519    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/eee821c4-6fb0-4f84-a58d-23dc2ff77720-kubelet-config\") pod \"global-pull-secret-syncer-hrxdt\" (UID: \"eee821c4-6fb0-4f84-a58d-23dc2ff77720\") " pod="kube-system/global-pull-secret-syncer-hrxdt"
Nov 03 19:31:28.071001 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.070524    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:31:28.071001 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.070570    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-run-ovn\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:31:28.071001 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.070581    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/eee821c4-6fb0-4f84-a58d-23dc2ff77720-kubelet-config\") pod \"global-pull-secret-syncer-hrxdt\" (UID: \"eee821c4-6fb0-4f84-a58d-23dc2ff77720\") " pod="kube-system/global-pull-secret-syncer-hrxdt"
Nov 03 19:31:28.071001 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.070598    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-run-ovn\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:31:28.071001 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.070615    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25-run\") pod \"tuned-l98pn\" (UID: \"0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25\") " pod="openshift-cluster-node-tuning-operator/tuned-l98pn"
Nov 03 19:31:28.071001 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.070632    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce-sys-fs\") pod \"aws-ebs-csi-driver-node-mcxcv\" (UID: \"8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mcxcv"
Nov 03 19:31:28.071001 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.070647    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-host-run-ovn-kubernetes\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:31:28.071001 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.070662    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25-lib-modules\") pod \"tuned-l98pn\" (UID: \"0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25\") " pod="openshift-cluster-node-tuning-operator/tuned-l98pn"
Nov 03 19:31:28.071001 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.070718    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce-sys-fs\") pod \"aws-ebs-csi-driver-node-mcxcv\" (UID: \"8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mcxcv"
Nov 03 19:31:28.071001 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.070723    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25-run\") pod \"tuned-l98pn\" (UID: \"0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25\") " pod="openshift-cluster-node-tuning-operator/tuned-l98pn"
Nov 03 19:31:28.071001 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.070726    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce-device-dir\") pod \"aws-ebs-csi-driver-node-mcxcv\" (UID: \"8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mcxcv"
Nov 03 19:31:28.071001 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.070723    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-host-run-ovn-kubernetes\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:31:28.071001 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.070760    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-run-openvswitch\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:31:28.071001 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.070779    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/653db542-8702-4eaa-abc1-a3bb866163c6-host-run-k8s-cni-cncf-io\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:31:28.071001 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.070799    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-host-cni-bin\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:31:28.071001 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.070806    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25-lib-modules\") pod \"tuned-l98pn\" (UID: \"0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25\") " pod="openshift-cluster-node-tuning-operator/tuned-l98pn"
Nov 03 19:31:28.071001 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.070826    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce-device-dir\") pod \"aws-ebs-csi-driver-node-mcxcv\" (UID: \"8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mcxcv"
Nov 03 19:31:28.071001 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.070826    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-run-openvswitch\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:31:28.071001 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.070836    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/653db542-8702-4eaa-abc1-a3bb866163c6-host-run-k8s-cni-cncf-io\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:31:28.071001 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.070854    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/653db542-8702-4eaa-abc1-a3bb866163c6-multus-conf-dir\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:31:28.071001 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.070849    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-host-cni-bin\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:31:28.071001 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.070880    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-host-kubelet\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:31:28.071001 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.070893    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/653db542-8702-4eaa-abc1-a3bb866163c6-multus-conf-dir\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:31:28.071001 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.070899    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/653db542-8702-4eaa-abc1-a3bb866163c6-host-var-lib-cni-bin\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:31:28.071001 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.070928    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/653db542-8702-4eaa-abc1-a3bb866163c6-host-var-lib-cni-bin\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:31:28.071001 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.070928    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-host-kubelet\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:31:28.071001 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.070939    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/545f3c9a-2ac6-44a3-be0b-cf71b1592d42-host\") pod \"node-ca-vv2s2\" (UID: \"545f3c9a-2ac6-44a3-be0b-cf71b1592d42\") " pod="openshift-image-registry/node-ca-vv2s2"
Nov 03 19:31:28.071001 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.070955    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25-etc-modprobe-d\") pod \"tuned-l98pn\" (UID: \"0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25\") " pod="openshift-cluster-node-tuning-operator/tuned-l98pn"
Nov 03 19:31:28.071001 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.070969    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25-etc-kubernetes\") pod \"tuned-l98pn\" (UID: \"0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25\") " pod="openshift-cluster-node-tuning-operator/tuned-l98pn"
Nov 03 19:31:28.071001 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.070999    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25-host\") pod \"tuned-l98pn\" (UID: \"0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25\") " pod="openshift-cluster-node-tuning-operator/tuned-l98pn"
Nov 03 19:31:28.071001 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.071021    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/653db542-8702-4eaa-abc1-a3bb866163c6-host-var-lib-cni-multus\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:31:28.072354 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.071035    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/545f3c9a-2ac6-44a3-be0b-cf71b1592d42-host\") pod \"node-ca-vv2s2\" (UID: \"545f3c9a-2ac6-44a3-be0b-cf71b1592d42\") " pod="openshift-image-registry/node-ca-vv2s2"
Nov 03 19:31:28.072354 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.071047    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25-etc-modprobe-d\") pod \"tuned-l98pn\" (UID: \"0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25\") " pod="openshift-cluster-node-tuning-operator/tuned-l98pn"
Nov 03 19:31:28.072354 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.071046    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce-registration-dir\") pod \"aws-ebs-csi-driver-node-mcxcv\" (UID: \"8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mcxcv"
Nov 03 19:31:28.072354 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.071074    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25-host\") pod \"tuned-l98pn\" (UID: \"0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25\") " pod="openshift-cluster-node-tuning-operator/tuned-l98pn"
Nov 03 19:31:28.072354 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.071086    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/653db542-8702-4eaa-abc1-a3bb866163c6-host-var-lib-cni-multus\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:31:28.072354 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.071095    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce-registration-dir\") pod \"aws-ebs-csi-driver-node-mcxcv\" (UID: \"8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mcxcv"
Nov 03 19:31:28.072354 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.071062    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25-etc-kubernetes\") pod \"tuned-l98pn\" (UID: \"0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25\") " pod="openshift-cluster-node-tuning-operator/tuned-l98pn"
Nov 03 19:31:28.072354 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.071105    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b797bdb0-b354-4dfc-a50a-3c60ecacbf71-cnibin\") pod \"multus-additional-cni-plugins-g67bt\" (UID: \"b797bdb0-b354-4dfc-a50a-3c60ecacbf71\") " pod="openshift-multus/multus-additional-cni-plugins-g67bt"
Nov 03 19:31:28.072354 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.071138    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25-etc-sysctl-conf\") pod \"tuned-l98pn\" (UID: \"0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25\") " pod="openshift-cluster-node-tuning-operator/tuned-l98pn"
Nov 03 19:31:28.072354 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.071142    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b797bdb0-b354-4dfc-a50a-3c60ecacbf71-cnibin\") pod \"multus-additional-cni-plugins-g67bt\" (UID: \"b797bdb0-b354-4dfc-a50a-3c60ecacbf71\") " pod="openshift-multus/multus-additional-cni-plugins-g67bt"
Nov 03 19:31:28.072354 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.071171    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/10137ba2-4beb-433e-b6df-199738351c61-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cvll4\" (UID: \"10137ba2-4beb-433e-b6df-199738351c61\") " pod="openshift-insights/insights-runtime-extractor-cvll4"
Nov 03 19:31:28.072354 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.071189    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/653db542-8702-4eaa-abc1-a3bb866163c6-host-run-multus-certs\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:31:28.072354 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.071206    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/38a6add1-b051-450b-bb96-8ace8a0ccff3-metrics-tls\") pod \"dns-default-wj8mp\" (UID: \"38a6add1-b051-450b-bb96-8ace8a0ccff3\") " pod="openshift-dns/dns-default-wj8mp"
Nov 03 19:31:28.072354 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.071229    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-registry-tls\") pod \"image-registry-c5c44f4d-dv455\" (UID: \"d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca\") " pod="openshift-image-registry/image-registry-c5c44f4d-dv455"
Nov 03 19:31:28.072354 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.071267    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/653db542-8702-4eaa-abc1-a3bb866163c6-host-run-multus-certs\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:31:28.072354 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.071275    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25-etc-sysctl-conf\") pod \"tuned-l98pn\" (UID: \"0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25\") " pod="openshift-cluster-node-tuning-operator/tuned-l98pn"
Nov 03 19:31:28.072354 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.071297    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/653db542-8702-4eaa-abc1-a3bb866163c6-host-run-netns\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:31:28.072354 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:31:28.071323    4921 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 19:31:28.072354 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.071326    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-etc-openvswitch\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:31:28.072354 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.071352    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/653db542-8702-4eaa-abc1-a3bb866163c6-host-run-netns\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:31:28.072354 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.071355    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mcxcv\" (UID: \"8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mcxcv"
Nov 03 19:31:28.072354 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:31:28.071323    4921 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 19:31:28.072354 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.071373    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d0ab7c64-a85f-4713-a270-f0cf711fcee5-etc-openvswitch\") pod \"ovnkube-node-k5nbr\" (UID: \"d0ab7c64-a85f-4713-a270-f0cf711fcee5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:31:28.072354 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:31:28.071384    4921 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c5c44f4d-dv455: secret "image-registry-tls" not found
Nov 03 19:31:28.072354 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.071398    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mcxcv\" (UID: \"8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mcxcv"
Nov 03 19:31:28.072354 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:31:28.071373    4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38a6add1-b051-450b-bb96-8ace8a0ccff3-metrics-tls podName:38a6add1-b051-450b-bb96-8ace8a0ccff3 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:28.571357104 +0000 UTC m=+1.815322550 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/38a6add1-b051-450b-bb96-8ace8a0ccff3-metrics-tls") pod "dns-default-wj8mp" (UID: "38a6add1-b051-450b-bb96-8ace8a0ccff3") : secret "dns-default-metrics-tls" not found
Nov 03 19:31:28.072354 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:31:28.071458    4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-registry-tls podName:d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca nodeName:}" failed. No retries permitted until 2026-11-03 19:31:28.571443153 +0000 UTC m=+1.815408605 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-registry-tls") pod "image-registry-c5c44f4d-dv455" (UID: "d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca") : secret "image-registry-tls" not found
Nov 03 19:31:28.072354 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.071474    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25-etc-sysctl-d\") pod \"tuned-l98pn\" (UID: \"0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25\") " pod="openshift-cluster-node-tuning-operator/tuned-l98pn"
Nov 03 19:31:28.072354 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.071497    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/653db542-8702-4eaa-abc1-a3bb866163c6-multus-socket-dir-parent\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:31:28.072354 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.071522    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8b5e1b09-699d-4db8-9eb0-0b71bdda858f-host-slash\") pod \"iptables-alerter-wkrtt\" (UID: \"8b5e1b09-699d-4db8-9eb0-0b71bdda858f\") " pod="openshift-network-operator/iptables-alerter-wkrtt"
Nov 03 19:31:28.072354 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.071546    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/10137ba2-4beb-433e-b6df-199738351c61-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cvll4\" (UID: \"10137ba2-4beb-433e-b6df-199738351c61\") " pod="openshift-insights/insights-runtime-extractor-cvll4"
Nov 03 19:31:28.072354 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.071584    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8b5e1b09-699d-4db8-9eb0-0b71bdda858f-host-slash\") pod \"iptables-alerter-wkrtt\" (UID: \"8b5e1b09-699d-4db8-9eb0-0b71bdda858f\") " pod="openshift-network-operator/iptables-alerter-wkrtt"
Nov 03 19:31:28.072354 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.071590    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/10137ba2-4beb-433e-b6df-199738351c61-crio-socket\") pod \"insights-runtime-extractor-cvll4\" (UID: \"10137ba2-4beb-433e-b6df-199738351c61\") " pod="openshift-insights/insights-runtime-extractor-cvll4"
Nov 03 19:31:28.072354 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.071596    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/653db542-8702-4eaa-abc1-a3bb866163c6-multus-socket-dir-parent\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:31:28.072354 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.071593    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25-etc-sysctl-d\") pod \"tuned-l98pn\" (UID: \"0dc3e7bd-dc1d-4f77-b7c6-86733b40ac25\") " pod="openshift-cluster-node-tuning-operator/tuned-l98pn"
Nov 03 19:31:28.072354 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.071643    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/10137ba2-4beb-433e-b6df-199738351c61-crio-socket\") pod \"insights-runtime-extractor-cvll4\" (UID: \"10137ba2-4beb-433e-b6df-199738351c61\") " pod="openshift-insights/insights-runtime-extractor-cvll4"
Nov 03 19:31:28.072354 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:31:28.071656    4921 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 19:31:28.072354 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.071656    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/653db542-8702-4eaa-abc1-a3bb866163c6-host-var-lib-kubelet\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:31:28.072354 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.071706    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/653db542-8702-4eaa-abc1-a3bb866163c6-host-var-lib-kubelet\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:31:28.072354 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.071714    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce-etc-selinux\") pod \"aws-ebs-csi-driver-node-mcxcv\" (UID: \"8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mcxcv"
Nov 03 19:31:28.072354 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:31:28.071724    4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10137ba2-4beb-433e-b6df-199738351c61-insights-runtime-extractor-tls podName:10137ba2-4beb-433e-b6df-199738351c61 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:28.571709037 +0000 UTC m=+1.815674497 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/10137ba2-4beb-433e-b6df-199738351c61-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cvll4" (UID: "10137ba2-4beb-433e-b6df-199738351c61") : secret "insights-runtime-extractor-tls" not found
Nov 03 19:31:28.072354 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.071762    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/653db542-8702-4eaa-abc1-a3bb866163c6-cnibin\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:31:28.072354 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.071771    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/10137ba2-4beb-433e-b6df-199738351c61-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cvll4\" (UID: \"10137ba2-4beb-433e-b6df-199738351c61\") " pod="openshift-insights/insights-runtime-extractor-cvll4"
Nov 03 19:31:28.072354 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.071762    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce-etc-selinux\") pod \"aws-ebs-csi-driver-node-mcxcv\" (UID: \"8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mcxcv"
Nov 03 19:31:28.072354 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.071791    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/653db542-8702-4eaa-abc1-a3bb866163c6-system-cni-dir\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:31:28.072354 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.071803    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/653db542-8702-4eaa-abc1-a3bb866163c6-cnibin\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:31:28.072354 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.071818    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/790607e1-d57c-4622-9a72-1af902ab1d85-metrics-certs\") pod \"network-metrics-daemon-rdr5d\" (UID: \"790607e1-d57c-4622-9a72-1af902ab1d85\") " pod="openshift-multus/network-metrics-daemon-rdr5d"
Nov 03 19:31:28.072354 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.071837    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce-socket-dir\") pod \"aws-ebs-csi-driver-node-mcxcv\" (UID: \"8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mcxcv"
Nov 03 19:31:28.072354 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.071837    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/653db542-8702-4eaa-abc1-a3bb866163c6-system-cni-dir\") pod \"multus-9ck9p\" (UID: \"653db542-8702-4eaa-abc1-a3bb866163c6\") " pod="openshift-multus/multus-9ck9p"
Nov 03 19:31:28.072354 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:31:28.071921    4921 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 03 19:31:28.072354 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.071923    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce-socket-dir\") pod \"aws-ebs-csi-driver-node-mcxcv\" (UID: \"8ec67cf7-b089-40ac-9a9e-eb2e2ea400ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mcxcv"
Nov 03 19:31:28.072354 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:31:28.071959    4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/790607e1-d57c-4622-9a72-1af902ab1d85-metrics-certs podName:790607e1-d57c-4622-9a72-1af902ab1d85 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:28.571947425 +0000 UTC m=+1.815912870 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/790607e1-d57c-4622-9a72-1af902ab1d85-metrics-certs") pod "network-metrics-daemon-rdr5d" (UID: "790607e1-d57c-4622-9a72-1af902ab1d85") : secret "metrics-daemon-secret" not found
Nov 03 19:31:28.135461 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.135374    4921 scope.go:122] "RemoveContainer" containerID="3895cd529be0be6af57557f5b4638c0b5aa21cd6a6d699b224ff7bffcb1bd8e3"
Nov 03 19:31:28.138301 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.138281    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7hcf8"
Nov 03 19:31:28.141441 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.141422    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-zpgl6"
Nov 03 19:31:28.432745 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.432651    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7hcf8"]
Nov 03 19:31:28.468480 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.468446    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-zpgl6"]
Nov 03 19:31:28.471764 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:28.471730    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb488ece_2ac3_499f_84c1_73534fcbb20b.slice/crio-e32e4afdbde76329d78fda4930f7e0ffc06a15fa89238b6a7fb749855adf4aee WatchSource:0}: Error finding container e32e4afdbde76329d78fda4930f7e0ffc06a15fa89238b6a7fb749855adf4aee: Status 404 returned error can't find the container with id e32e4afdbde76329d78fda4930f7e0ffc06a15fa89238b6a7fb749855adf4aee
Nov 03 19:31:28.473908 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.473888    4921 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 19:31:28.475058 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.475038    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/96c016f7-9098-4a39-9e72-e54da374d8fc-registry-tls\") pod \"image-registry-d74787d9c-nn7sk\" (UID: \"96c016f7-9098-4a39-9e72-e54da374d8fc\") " pod="openshift-image-registry/image-registry-d74787d9c-nn7sk"
Nov 03 19:31:28.475190 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:31:28.475176    4921 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 19:31:28.475254 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:31:28.475194    4921 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d74787d9c-nn7sk: secret "image-registry-tls" not found
Nov 03 19:31:28.475254 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.475212    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e2b752d9-88c4-4dee-8b0c-7e9ac252adb3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kfdsx\" (UID: \"e2b752d9-88c4-4dee-8b0c-7e9ac252adb3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kfdsx"
Nov 03 19:31:28.475254 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:31:28.475247    4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/96c016f7-9098-4a39-9e72-e54da374d8fc-registry-tls podName:96c016f7-9098-4a39-9e72-e54da374d8fc nodeName:}" failed. No retries permitted until 2026-11-03 19:31:29.475228249 +0000 UTC m=+2.719193694 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/96c016f7-9098-4a39-9e72-e54da374d8fc-registry-tls") pod "image-registry-d74787d9c-nn7sk" (UID: "96c016f7-9098-4a39-9e72-e54da374d8fc") : secret "image-registry-tls" not found
Nov 03 19:31:28.475408 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.475301    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a023911d-3669-43e1-ac77-8a986f3715d6-metrics-certs\") pod \"router-default-569d4c45f-7pszr\" (UID: \"a023911d-3669-43e1-ac77-8a986f3715d6\") " pod="openshift-ingress/router-default-569d4c45f-7pszr"
Nov 03 19:31:28.475408 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:31:28.475308    4921 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 19:31:28.475530 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:31:28.475407    4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2b752d9-88c4-4dee-8b0c-7e9ac252adb3-cluster-monitoring-operator-tls podName:e2b752d9-88c4-4dee-8b0c-7e9ac252adb3 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:29.475394252 +0000 UTC m=+2.719359697 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e2b752d9-88c4-4dee-8b0c-7e9ac252adb3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-kfdsx" (UID: "e2b752d9-88c4-4dee-8b0c-7e9ac252adb3") : secret "cluster-monitoring-operator-tls" not found
Nov 03 19:31:28.475530 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:31:28.475416    4921 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 03 19:31:28.475530 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:31:28.475459    4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a023911d-3669-43e1-ac77-8a986f3715d6-metrics-certs podName:a023911d-3669-43e1-ac77-8a986f3715d6 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:29.47544788 +0000 UTC m=+2.719413324 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a023911d-3669-43e1-ac77-8a986f3715d6-metrics-certs") pod "router-default-569d4c45f-7pszr" (UID: "a023911d-3669-43e1-ac77-8a986f3715d6") : secret "router-metrics-certs-default" not found
Nov 03 19:31:28.576208 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.576173    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/38a6add1-b051-450b-bb96-8ace8a0ccff3-metrics-tls\") pod \"dns-default-wj8mp\" (UID: \"38a6add1-b051-450b-bb96-8ace8a0ccff3\") " pod="openshift-dns/dns-default-wj8mp"
Nov 03 19:31:28.576208 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.576209    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-registry-tls\") pod \"image-registry-c5c44f4d-dv455\" (UID: \"d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca\") " pod="openshift-image-registry/image-registry-c5c44f4d-dv455"
Nov 03 19:31:28.576417 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.576232    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/10137ba2-4beb-433e-b6df-199738351c61-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cvll4\" (UID: \"10137ba2-4beb-433e-b6df-199738351c61\") " pod="openshift-insights/insights-runtime-extractor-cvll4"
Nov 03 19:31:28.576417 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.576273    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/790607e1-d57c-4622-9a72-1af902ab1d85-metrics-certs\") pod \"network-metrics-daemon-rdr5d\" (UID: \"790607e1-d57c-4622-9a72-1af902ab1d85\") " pod="openshift-multus/network-metrics-daemon-rdr5d"
Nov 03 19:31:28.576417 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:31:28.576349    4921 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 19:31:28.576417 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:31:28.576376    4921 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 03 19:31:28.576417 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:31:28.576376    4921 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 19:31:28.576417 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:31:28.576352    4921 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 19:31:28.576417 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:31:28.576413    4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38a6add1-b051-450b-bb96-8ace8a0ccff3-metrics-tls podName:38a6add1-b051-450b-bb96-8ace8a0ccff3 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:29.576399408 +0000 UTC m=+2.820364886 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/38a6add1-b051-450b-bb96-8ace8a0ccff3-metrics-tls") pod "dns-default-wj8mp" (UID: "38a6add1-b051-450b-bb96-8ace8a0ccff3") : secret "dns-default-metrics-tls" not found
Nov 03 19:31:28.576686 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:31:28.576413    4921 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c5c44f4d-dv455: secret "image-registry-tls" not found
Nov 03 19:31:28.576686 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:31:28.576431    4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/790607e1-d57c-4622-9a72-1af902ab1d85-metrics-certs podName:790607e1-d57c-4622-9a72-1af902ab1d85 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:29.576422434 +0000 UTC m=+2.820387879 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/790607e1-d57c-4622-9a72-1af902ab1d85-metrics-certs") pod "network-metrics-daemon-rdr5d" (UID: "790607e1-d57c-4622-9a72-1af902ab1d85") : secret "metrics-daemon-secret" not found
Nov 03 19:31:28.576686 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:31:28.576478    4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10137ba2-4beb-433e-b6df-199738351c61-insights-runtime-extractor-tls podName:10137ba2-4beb-433e-b6df-199738351c61 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:29.576470173 +0000 UTC m=+2.820435618 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/10137ba2-4beb-433e-b6df-199738351c61-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cvll4" (UID: "10137ba2-4beb-433e-b6df-199738351c61") : secret "insights-runtime-extractor-tls" not found
Nov 03 19:31:28.576686 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:31:28.576491    4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-registry-tls podName:d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca nodeName:}" failed. No retries permitted until 2026-11-03 19:31:29.57648573 +0000 UTC m=+2.820451173 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-registry-tls") pod "image-registry-c5c44f4d-dv455" (UID: "d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca") : secret "image-registry-tls" not found
Nov 03 19:31:28.879016 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.878989    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vv2s2_545f3c9a-2ac6-44a3-be0b-cf71b1592d42/node-ca/0.log"
Nov 03 19:31:28.943576 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.943534    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-zpgl6" event={"ID":"eb488ece-2ac3-499f-84c1-73534fcbb20b","Type":"ContainerStarted","Data":"e32e4afdbde76329d78fda4930f7e0ffc06a15fa89238b6a7fb749855adf4aee"}
Nov 03 19:31:28.944647 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.944621    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7hcf8" event={"ID":"1753fd31-8933-4567-8146-07f3fd2d10a5","Type":"ContainerStarted","Data":"c11c0480fc18db960ce59787e0319ab68034eefa31d4ea9129b1210d38edff41"}
Nov 03 19:31:28.946039 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.946021    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gzf8k_6f215b42-0f09-46a0-bc98-392c0fa909c9/console-operator/2.log"
Nov 03 19:31:28.946409 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.946393    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gzf8k_6f215b42-0f09-46a0-bc98-392c0fa909c9/console-operator/1.log"
Nov 03 19:31:28.946483 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.946428    4921 generic.go:356] "Generic (PLEG): container finished" podID="6f215b42-0f09-46a0-bc98-392c0fa909c9" containerID="2486b5467a6fc2b3c65ed9a9942d79836f6e99e7cb7ab904167391b30843d756" exitCode=255
Nov 03 19:31:28.946560 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.946500    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-gzf8k" event={"ID":"6f215b42-0f09-46a0-bc98-392c0fa909c9","Type":"ContainerDied","Data":"2486b5467a6fc2b3c65ed9a9942d79836f6e99e7cb7ab904167391b30843d756"}
Nov 03 19:31:28.946560 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.946544    4921 scope.go:122] "RemoveContainer" containerID="3895cd529be0be6af57557f5b4638c0b5aa21cd6a6d699b224ff7bffcb1bd8e3"
Nov 03 19:31:28.947053 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:28.947037    4921 scope.go:122] "RemoveContainer" containerID="2486b5467a6fc2b3c65ed9a9942d79836f6e99e7cb7ab904167391b30843d756"
Nov 03 19:31:28.947206 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:31:28.947189    4921 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-gzf8k_openshift-console-operator(6f215b42-0f09-46a0-bc98-392c0fa909c9)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-gzf8k" podUID="6f215b42-0f09-46a0-bc98-392c0fa909c9"
Nov 03 19:31:29.485115 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:29.485071    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/96c016f7-9098-4a39-9e72-e54da374d8fc-registry-tls\") pod \"image-registry-d74787d9c-nn7sk\" (UID: \"96c016f7-9098-4a39-9e72-e54da374d8fc\") " pod="openshift-image-registry/image-registry-d74787d9c-nn7sk"
Nov 03 19:31:29.485287 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:29.485226    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e2b752d9-88c4-4dee-8b0c-7e9ac252adb3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kfdsx\" (UID: \"e2b752d9-88c4-4dee-8b0c-7e9ac252adb3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kfdsx"
Nov 03 19:31:29.485287 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:31:29.485250    4921 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 19:31:29.485287 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:29.485267    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a023911d-3669-43e1-ac77-8a986f3715d6-metrics-certs\") pod \"router-default-569d4c45f-7pszr\" (UID: \"a023911d-3669-43e1-ac77-8a986f3715d6\") " pod="openshift-ingress/router-default-569d4c45f-7pszr"
Nov 03 19:31:29.485287 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:31:29.485275    4921 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d74787d9c-nn7sk: secret "image-registry-tls" not found
Nov 03 19:31:29.485495 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:31:29.485343    4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/96c016f7-9098-4a39-9e72-e54da374d8fc-registry-tls podName:96c016f7-9098-4a39-9e72-e54da374d8fc nodeName:}" failed. No retries permitted until 2026-11-03 19:31:31.485322656 +0000 UTC m=+4.729288118 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/96c016f7-9098-4a39-9e72-e54da374d8fc-registry-tls") pod "image-registry-d74787d9c-nn7sk" (UID: "96c016f7-9098-4a39-9e72-e54da374d8fc") : secret "image-registry-tls" not found
Nov 03 19:31:29.485495 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:31:29.485399    4921 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 03 19:31:29.485495 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:31:29.485448    4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a023911d-3669-43e1-ac77-8a986f3715d6-metrics-certs podName:a023911d-3669-43e1-ac77-8a986f3715d6 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:31.485435937 +0000 UTC m=+4.729401395 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a023911d-3669-43e1-ac77-8a986f3715d6-metrics-certs") pod "router-default-569d4c45f-7pszr" (UID: "a023911d-3669-43e1-ac77-8a986f3715d6") : secret "router-metrics-certs-default" not found
Nov 03 19:31:29.485636 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:31:29.485511    4921 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 19:31:29.485636 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:31:29.485548    4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2b752d9-88c4-4dee-8b0c-7e9ac252adb3-cluster-monitoring-operator-tls podName:e2b752d9-88c4-4dee-8b0c-7e9ac252adb3 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:31.485537009 +0000 UTC m=+4.729502458 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e2b752d9-88c4-4dee-8b0c-7e9ac252adb3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-kfdsx" (UID: "e2b752d9-88c4-4dee-8b0c-7e9ac252adb3") : secret "cluster-monitoring-operator-tls" not found
Nov 03 19:31:29.585924 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:29.585886    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/38a6add1-b051-450b-bb96-8ace8a0ccff3-metrics-tls\") pod \"dns-default-wj8mp\" (UID: \"38a6add1-b051-450b-bb96-8ace8a0ccff3\") " pod="openshift-dns/dns-default-wj8mp"
Nov 03 19:31:29.586100 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:29.585933    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-registry-tls\") pod \"image-registry-c5c44f4d-dv455\" (UID: \"d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca\") " pod="openshift-image-registry/image-registry-c5c44f4d-dv455"
Nov 03 19:31:29.586100 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:29.585969    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/10137ba2-4beb-433e-b6df-199738351c61-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cvll4\" (UID: \"10137ba2-4beb-433e-b6df-199738351c61\") " pod="openshift-insights/insights-runtime-extractor-cvll4"
Nov 03 19:31:29.586100 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:29.586015    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/790607e1-d57c-4622-9a72-1af902ab1d85-metrics-certs\") pod \"network-metrics-daemon-rdr5d\" (UID: \"790607e1-d57c-4622-9a72-1af902ab1d85\") " pod="openshift-multus/network-metrics-daemon-rdr5d"
Nov 03 19:31:29.586100 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:31:29.586049    4921 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 19:31:29.586265 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:31:29.586111    4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38a6add1-b051-450b-bb96-8ace8a0ccff3-metrics-tls podName:38a6add1-b051-450b-bb96-8ace8a0ccff3 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:31.586093925 +0000 UTC m=+4.830059388 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/38a6add1-b051-450b-bb96-8ace8a0ccff3-metrics-tls") pod "dns-default-wj8mp" (UID: "38a6add1-b051-450b-bb96-8ace8a0ccff3") : secret "dns-default-metrics-tls" not found
Nov 03 19:31:29.586265 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:31:29.586130    4921 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 03 19:31:29.586265 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:31:29.586135    4921 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 19:31:29.586265 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:31:29.586161    4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/790607e1-d57c-4622-9a72-1af902ab1d85-metrics-certs podName:790607e1-d57c-4622-9a72-1af902ab1d85 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:31.586151668 +0000 UTC m=+4.830117124 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/790607e1-d57c-4622-9a72-1af902ab1d85-metrics-certs") pod "network-metrics-daemon-rdr5d" (UID: "790607e1-d57c-4622-9a72-1af902ab1d85") : secret "metrics-daemon-secret" not found
Nov 03 19:31:29.586265 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:31:29.586206    4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10137ba2-4beb-433e-b6df-199738351c61-insights-runtime-extractor-tls podName:10137ba2-4beb-433e-b6df-199738351c61 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:31.586189324 +0000 UTC m=+4.830154784 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/10137ba2-4beb-433e-b6df-199738351c61-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cvll4" (UID: "10137ba2-4beb-433e-b6df-199738351c61") : secret "insights-runtime-extractor-tls" not found
Nov 03 19:31:29.586465 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:31:29.586286    4921 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 19:31:29.586465 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:31:29.586299    4921 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c5c44f4d-dv455: secret "image-registry-tls" not found
Nov 03 19:31:29.586465 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:31:29.586343    4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-registry-tls podName:d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca nodeName:}" failed. No retries permitted until 2026-11-03 19:31:31.586329932 +0000 UTC m=+4.830295381 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-registry-tls") pod "image-registry-c5c44f4d-dv455" (UID: "d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca") : secret "image-registry-tls" not found
Nov 03 19:31:29.678960 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:29.678931    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-j7m2z_543b8778-baa4-4e09-b697-68e550e42942/serve-healthcheck-canary/0.log"
Nov 03 19:31:29.879131 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:29.879042    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-5l649_220ed31c-1577-4c4b-ab5b-e812fd63d193/migrator/0.log"
Nov 03 19:31:29.951762 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:29.951738    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gzf8k_6f215b42-0f09-46a0-bc98-392c0fa909c9/console-operator/2.log"
Nov 03 19:31:30.280383 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:30.280351    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-hxlrb_e10dc749-bc71-456e-a315-6581add01a7b/kube-storage-version-migrator-operator/0.log"
Nov 03 19:31:30.957404 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:30.957368    4921 generic.go:356] "Generic (PLEG): container finished" podID="9fb8c33c-fe15-4daa-a01b-4cb2fc7eae7b" containerID="d3620807293dfc2435cd5f84c226d2eb54765fd456022147d2ce289bc117fc51" exitCode=0
Nov 03 19:31:30.957862 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:30.957440    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mh2tn" event={"ID":"9fb8c33c-fe15-4daa-a01b-4cb2fc7eae7b","Type":"ContainerDied","Data":"d3620807293dfc2435cd5f84c226d2eb54765fd456022147d2ce289bc117fc51"}
Nov 03 19:31:30.957945 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:30.957865    4921 scope.go:122] "RemoveContainer" containerID="d3620807293dfc2435cd5f84c226d2eb54765fd456022147d2ce289bc117fc51"
Nov 03 19:31:30.959147 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:30.959111    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-zpgl6" event={"ID":"eb488ece-2ac3-499f-84c1-73534fcbb20b","Type":"ContainerStarted","Data":"fef87f4ab740ba91a316af1e3df535c84886508482c442b58e9388d5ab929aab"}
Nov 03 19:31:30.960785 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:30.960764    4921 generic.go:356] "Generic (PLEG): container finished" podID="3880f3d3-aad1-43c0-b704-01374aad4f78" containerID="b7ce8ebe35d2285a3f7ef842541ab9676c44e1038b05db62a869ad1d327de641" exitCode=0
Nov 03 19:31:30.960882 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:30.960797    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-8kh6f" event={"ID":"3880f3d3-aad1-43c0-b704-01374aad4f78","Type":"ContainerDied","Data":"b7ce8ebe35d2285a3f7ef842541ab9676c44e1038b05db62a869ad1d327de641"}
Nov 03 19:31:30.961089 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:30.961073    4921 scope.go:122] "RemoveContainer" containerID="b7ce8ebe35d2285a3f7ef842541ab9676c44e1038b05db62a869ad1d327de641"
Nov 03 19:31:31.417473 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:31.417440    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-njg27"
Nov 03 19:31:31.505044 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:31.505006    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a023911d-3669-43e1-ac77-8a986f3715d6-metrics-certs\") pod \"router-default-569d4c45f-7pszr\" (UID: \"a023911d-3669-43e1-ac77-8a986f3715d6\") " pod="openshift-ingress/router-default-569d4c45f-7pszr"
Nov 03 19:31:31.505205 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:31.505054    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/96c016f7-9098-4a39-9e72-e54da374d8fc-registry-tls\") pod \"image-registry-d74787d9c-nn7sk\" (UID: \"96c016f7-9098-4a39-9e72-e54da374d8fc\") " pod="openshift-image-registry/image-registry-d74787d9c-nn7sk"
Nov 03 19:31:31.505205 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:31:31.505150    4921 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 19:31:31.505205 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:31:31.505161    4921 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d74787d9c-nn7sk: secret "image-registry-tls" not found
Nov 03 19:31:31.505205 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:31:31.505157    4921 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 03 19:31:31.505348 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:31:31.505211    4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/96c016f7-9098-4a39-9e72-e54da374d8fc-registry-tls podName:96c016f7-9098-4a39-9e72-e54da374d8fc nodeName:}" failed. No retries permitted until 2026-11-03 19:31:35.50519802 +0000 UTC m=+8.749163466 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/96c016f7-9098-4a39-9e72-e54da374d8fc-registry-tls") pod "image-registry-d74787d9c-nn7sk" (UID: "96c016f7-9098-4a39-9e72-e54da374d8fc") : secret "image-registry-tls" not found
Nov 03 19:31:31.505348 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:31:31.505282    4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a023911d-3669-43e1-ac77-8a986f3715d6-metrics-certs podName:a023911d-3669-43e1-ac77-8a986f3715d6 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:35.505263758 +0000 UTC m=+8.749229201 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a023911d-3669-43e1-ac77-8a986f3715d6-metrics-certs") pod "router-default-569d4c45f-7pszr" (UID: "a023911d-3669-43e1-ac77-8a986f3715d6") : secret "router-metrics-certs-default" not found
Nov 03 19:31:31.505452 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:31.505387    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e2b752d9-88c4-4dee-8b0c-7e9ac252adb3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kfdsx\" (UID: \"e2b752d9-88c4-4dee-8b0c-7e9ac252adb3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kfdsx"
Nov 03 19:31:31.505507 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:31:31.505472    4921 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 19:31:31.505552 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:31:31.505508    4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2b752d9-88c4-4dee-8b0c-7e9ac252adb3-cluster-monitoring-operator-tls podName:e2b752d9-88c4-4dee-8b0c-7e9ac252adb3 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:35.505499907 +0000 UTC m=+8.749465350 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e2b752d9-88c4-4dee-8b0c-7e9ac252adb3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-kfdsx" (UID: "e2b752d9-88c4-4dee-8b0c-7e9ac252adb3") : secret "cluster-monitoring-operator-tls" not found
Nov 03 19:31:31.606887 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:31.606847    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/38a6add1-b051-450b-bb96-8ace8a0ccff3-metrics-tls\") pod \"dns-default-wj8mp\" (UID: \"38a6add1-b051-450b-bb96-8ace8a0ccff3\") " pod="openshift-dns/dns-default-wj8mp"
Nov 03 19:31:31.606887 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:31.606884    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-registry-tls\") pod \"image-registry-c5c44f4d-dv455\" (UID: \"d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca\") " pod="openshift-image-registry/image-registry-c5c44f4d-dv455"
Nov 03 19:31:31.607110 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:31.606907    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/10137ba2-4beb-433e-b6df-199738351c61-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cvll4\" (UID: \"10137ba2-4beb-433e-b6df-199738351c61\") " pod="openshift-insights/insights-runtime-extractor-cvll4"
Nov 03 19:31:31.607110 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:31.606981    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/790607e1-d57c-4622-9a72-1af902ab1d85-metrics-certs\") pod \"network-metrics-daemon-rdr5d\" (UID: \"790607e1-d57c-4622-9a72-1af902ab1d85\") " pod="openshift-multus/network-metrics-daemon-rdr5d"
Nov 03 19:31:31.607110 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:31:31.607029    4921 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 19:31:31.607110 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:31:31.607092    4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38a6add1-b051-450b-bb96-8ace8a0ccff3-metrics-tls podName:38a6add1-b051-450b-bb96-8ace8a0ccff3 nodeName:}" failed. No retries permitted until 2026-11-03 19:31:35.607072954 +0000 UTC m=+8.851038398 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/38a6add1-b051-450b-bb96-8ace8a0ccff3-metrics-tls") pod "dns-default-wj8mp" (UID: "38a6add1-b051-450b-bb96-8ace8a0ccff3") : secret "dns-default-metrics-tls" not found
Nov 03 19:31:31.607110 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:31:31.607110    4921 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 19:31:31.607367 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:31:31.607122    4921 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c5c44f4d-dv455: secret "image-registry-tls" not found
Nov 03 19:31:31.607367 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:31:31.607159    4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-registry-tls podName:d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca nodeName:}" failed. No retries permitted until 2026-11-03 19:31:35.607148101 +0000 UTC m=+8.851113560 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-registry-tls") pod "image-registry-c5c44f4d-dv455" (UID: "d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca") : secret "image-registry-tls" not found
Nov 03 19:31:31.609297 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:31.609273    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/10137ba2-4beb-433e-b6df-199738351c61-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cvll4\" (UID: \"10137ba2-4beb-433e-b6df-199738351c61\") " pod="openshift-insights/insights-runtime-extractor-cvll4"
Nov 03 19:31:31.609297 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:31.609294    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/790607e1-d57c-4622-9a72-1af902ab1d85-metrics-certs\") pod \"network-metrics-daemon-rdr5d\" (UID: \"790607e1-d57c-4622-9a72-1af902ab1d85\") " pod="openshift-multus/network-metrics-daemon-rdr5d"
Nov 03 19:31:31.746983 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:31.746951    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rdr5d"
Nov 03 19:31:31.756059 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:31.756030    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-cvll4"
Nov 03 19:31:31.966728 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:31.966694    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-8kh6f" event={"ID":"3880f3d3-aad1-43c0-b704-01374aad4f78","Type":"ContainerStarted","Data":"2a86f477e3c4f116e4a2246fe55a4a6f8aa9318da6ac3455a9dc979101b9f50f"}
Nov 03 19:31:31.968503 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:31.968481    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mh2tn" event={"ID":"9fb8c33c-fe15-4daa-a01b-4cb2fc7eae7b","Type":"ContainerStarted","Data":"27b70e5aaced0b33ea18df23777c97e03822a0581abca23c5c327579d2e2f3c1"}
Nov 03 19:31:31.970172 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:31.970149    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7hcf8" event={"ID":"1753fd31-8933-4567-8146-07f3fd2d10a5","Type":"ContainerStarted","Data":"cc8784f4ac5671464db2f2b2bf3c4ed576b31d495895376e380ec621b68361be"}
Nov 03 19:31:31.970264 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:31.970196    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-7hcf8" event={"ID":"1753fd31-8933-4567-8146-07f3fd2d10a5","Type":"ContainerStarted","Data":"9d5aa95d0322a38e9a98f09a4db000a91f71b6db23ead2049916e6fa5fa6b729"}
Nov 03 19:31:32.015445 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:32.015375    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rdr5d"]
Nov 03 19:31:32.020555 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:32.020524    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod790607e1_d57c_4622_9a72_1af902ab1d85.slice/crio-691e27cbb769bc4ca1bff92e4ebe3a7f55df1b2bd2a772bdd3202da5cc67ceca WatchSource:0}: Error finding container 691e27cbb769bc4ca1bff92e4ebe3a7f55df1b2bd2a772bdd3202da5cc67ceca: Status 404 returned error can't find the container with id 691e27cbb769bc4ca1bff92e4ebe3a7f55df1b2bd2a772bdd3202da5cc67ceca
Nov 03 19:31:32.037634 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:32.037603    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-cvll4"]
Nov 03 19:31:32.040825 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:32.040796    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10137ba2_4beb_433e_b6df_199738351c61.slice/crio-7ceb4832cebde6ed16f89662cdebe001d4545badc70fde6f5e9dc2b8bbad807a WatchSource:0}: Error finding container 7ceb4832cebde6ed16f89662cdebe001d4545badc70fde6f5e9dc2b8bbad807a: Status 404 returned error can't find the container with id 7ceb4832cebde6ed16f89662cdebe001d4545badc70fde6f5e9dc2b8bbad807a
Nov 03 19:31:32.950032 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:32.949995    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-gzf8k"
Nov 03 19:31:32.950434 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:32.950419    4921 scope.go:122] "RemoveContainer" containerID="2486b5467a6fc2b3c65ed9a9942d79836f6e99e7cb7ab904167391b30843d756"
Nov 03 19:31:32.950648 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:31:32.950627    4921 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-gzf8k_openshift-console-operator(6f215b42-0f09-46a0-bc98-392c0fa909c9)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-gzf8k" podUID="6f215b42-0f09-46a0-bc98-392c0fa909c9"
Nov 03 19:31:32.978015 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:32.977955    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rdr5d" event={"ID":"790607e1-d57c-4622-9a72-1af902ab1d85","Type":"ContainerStarted","Data":"691e27cbb769bc4ca1bff92e4ebe3a7f55df1b2bd2a772bdd3202da5cc67ceca"}
Nov 03 19:31:32.980203 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:32.980172    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cvll4" event={"ID":"10137ba2-4beb-433e-b6df-199738351c61","Type":"ContainerStarted","Data":"fc0f3e0caeb8938f9e2f03dce0ab279f21edcc6cbf497ee8ff092f594d20f32c"}
Nov 03 19:31:32.980345 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:32.980237    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cvll4" event={"ID":"10137ba2-4beb-433e-b6df-199738351c61","Type":"ContainerStarted","Data":"51573f8e7b901bfa0d7efbfbfeafd59cf0b81c12463d9f5bfd485910a3626249"}
Nov 03 19:31:32.980345 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:32.980257    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cvll4" event={"ID":"10137ba2-4beb-433e-b6df-199738351c61","Type":"ContainerStarted","Data":"7ceb4832cebde6ed16f89662cdebe001d4545badc70fde6f5e9dc2b8bbad807a"}
Nov 03 19:31:33.985714 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:33.985660    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rdr5d" event={"ID":"790607e1-d57c-4622-9a72-1af902ab1d85","Type":"ContainerStarted","Data":"f56b87e94be4102f4dedc621a23c4028857be3bde50657106126b40d4f6adde0"}
Nov 03 19:31:33.986131 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:33.985762    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rdr5d" event={"ID":"790607e1-d57c-4622-9a72-1af902ab1d85","Type":"ContainerStarted","Data":"ef0db3792a7c1a84cebd7059a442a1cc1f9b675a407c9e4032ada512f6e916d5"}
Nov 03 19:31:34.751711 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:34.751681    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:31:35.545281 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:35.545243    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e2b752d9-88c4-4dee-8b0c-7e9ac252adb3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kfdsx\" (UID: \"e2b752d9-88c4-4dee-8b0c-7e9ac252adb3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kfdsx"
Nov 03 19:31:35.545683 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:35.545289    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a023911d-3669-43e1-ac77-8a986f3715d6-metrics-certs\") pod \"router-default-569d4c45f-7pszr\" (UID: \"a023911d-3669-43e1-ac77-8a986f3715d6\") " pod="openshift-ingress/router-default-569d4c45f-7pszr"
Nov 03 19:31:35.545683 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:35.545410    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/96c016f7-9098-4a39-9e72-e54da374d8fc-registry-tls\") pod \"image-registry-d74787d9c-nn7sk\" (UID: \"96c016f7-9098-4a39-9e72-e54da374d8fc\") " pod="openshift-image-registry/image-registry-d74787d9c-nn7sk"
Nov 03 19:31:35.547791 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:35.547768    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/96c016f7-9098-4a39-9e72-e54da374d8fc-registry-tls\") pod \"image-registry-d74787d9c-nn7sk\" (UID: \"96c016f7-9098-4a39-9e72-e54da374d8fc\") " pod="openshift-image-registry/image-registry-d74787d9c-nn7sk"
Nov 03 19:31:35.547976 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:35.547953    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a023911d-3669-43e1-ac77-8a986f3715d6-metrics-certs\") pod \"router-default-569d4c45f-7pszr\" (UID: \"a023911d-3669-43e1-ac77-8a986f3715d6\") " pod="openshift-ingress/router-default-569d4c45f-7pszr"
Nov 03 19:31:35.548015 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:35.547953    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e2b752d9-88c4-4dee-8b0c-7e9ac252adb3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kfdsx\" (UID: \"e2b752d9-88c4-4dee-8b0c-7e9ac252adb3\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kfdsx"
Nov 03 19:31:35.636997 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:35.636909    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-569d4c45f-7pszr"
Nov 03 19:31:35.636997 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:35.636975    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kfdsx"
Nov 03 19:31:35.641774 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:35.641751    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d74787d9c-nn7sk"
Nov 03 19:31:35.646730 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:35.646708    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/38a6add1-b051-450b-bb96-8ace8a0ccff3-metrics-tls\") pod \"dns-default-wj8mp\" (UID: \"38a6add1-b051-450b-bb96-8ace8a0ccff3\") " pod="openshift-dns/dns-default-wj8mp"
Nov 03 19:31:35.646829 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:35.646735    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-registry-tls\") pod \"image-registry-c5c44f4d-dv455\" (UID: \"d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca\") " pod="openshift-image-registry/image-registry-c5c44f4d-dv455"
Nov 03 19:31:35.649352 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:35.649330    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/38a6add1-b051-450b-bb96-8ace8a0ccff3-metrics-tls\") pod \"dns-default-wj8mp\" (UID: \"38a6add1-b051-450b-bb96-8ace8a0ccff3\") " pod="openshift-dns/dns-default-wj8mp"
Nov 03 19:31:35.649543 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:35.649521    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-registry-tls\") pod \"image-registry-c5c44f4d-dv455\" (UID: \"d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca\") " pod="openshift-image-registry/image-registry-c5c44f4d-dv455"
Nov 03 19:31:35.652365 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:35.652336    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c5c44f4d-dv455"
Nov 03 19:31:35.652610 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:35.652591    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wj8mp"
Nov 03 19:31:35.931230 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:35.931143    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-kfdsx"]
Nov 03 19:31:35.931978 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:35.931915    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-569d4c45f-7pszr"]
Nov 03 19:31:35.933747 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:35.933725    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-d74787d9c-nn7sk"]
Nov 03 19:31:35.936483 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:35.936458    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2b752d9_88c4_4dee_8b0c_7e9ac252adb3.slice/crio-8fe922163d741ccd6b0a90e9d5eb3cdc614b278614fd33a613aa6acf3db777dd WatchSource:0}: Error finding container 8fe922163d741ccd6b0a90e9d5eb3cdc614b278614fd33a613aa6acf3db777dd: Status 404 returned error can't find the container with id 8fe922163d741ccd6b0a90e9d5eb3cdc614b278614fd33a613aa6acf3db777dd
Nov 03 19:31:35.937172 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:35.937151    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda023911d_3669_43e1_ac77_8a986f3715d6.slice/crio-d6b199e62d9302e520b8484bdfc7acf801d5f1e152895a651ca0f576185c7cc7 WatchSource:0}: Error finding container d6b199e62d9302e520b8484bdfc7acf801d5f1e152895a651ca0f576185c7cc7: Status 404 returned error can't find the container with id d6b199e62d9302e520b8484bdfc7acf801d5f1e152895a651ca0f576185c7cc7
Nov 03 19:31:35.937836 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:35.937817    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96c016f7_9098_4a39_9e72_e54da374d8fc.slice/crio-34e274469bc0110e779b44dbe51d88dd9e86372a8b1e978b8b598768722f15a6 WatchSource:0}: Error finding container 34e274469bc0110e779b44dbe51d88dd9e86372a8b1e978b8b598768722f15a6: Status 404 returned error can't find the container with id 34e274469bc0110e779b44dbe51d88dd9e86372a8b1e978b8b598768722f15a6
Nov 03 19:31:35.947337 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:35.947311    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wj8mp"]
Nov 03 19:31:35.953185 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:35.953159    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38a6add1_b051_450b_bb96_8ace8a0ccff3.slice/crio-d3c08561e0956cae70552ad436b8cb3e942d6278494918f595ab1234ba24d042 WatchSource:0}: Error finding container d3c08561e0956cae70552ad436b8cb3e942d6278494918f595ab1234ba24d042: Status 404 returned error can't find the container with id d3c08561e0956cae70552ad436b8cb3e942d6278494918f595ab1234ba24d042
Nov 03 19:31:35.962412 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:35.962361    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-c5c44f4d-dv455"]
Nov 03 19:31:35.971582 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:35.971562    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3e8dd07_ea2a_46dc_9ea6_e0bd949698ca.slice/crio-415be167d9d95e070ae4a1d50c1e46128de973c7ca7089b2423b7c88b6be19c7 WatchSource:0}: Error finding container 415be167d9d95e070ae4a1d50c1e46128de973c7ca7089b2423b7c88b6be19c7: Status 404 returned error can't find the container with id 415be167d9d95e070ae4a1d50c1e46128de973c7ca7089b2423b7c88b6be19c7
Nov 03 19:31:35.998745 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:35.998413    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cvll4" event={"ID":"10137ba2-4beb-433e-b6df-199738351c61","Type":"ContainerStarted","Data":"2d59f3686239d5304bdcd611206b37f95f88373bf5181b1f5d2e97131627be2a"}
Nov 03 19:31:36.000713 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:36.000688    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d74787d9c-nn7sk" event={"ID":"96c016f7-9098-4a39-9e72-e54da374d8fc","Type":"ContainerStarted","Data":"34e274469bc0110e779b44dbe51d88dd9e86372a8b1e978b8b598768722f15a6"}
Nov 03 19:31:36.002249 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:36.002139    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kfdsx" event={"ID":"e2b752d9-88c4-4dee-8b0c-7e9ac252adb3","Type":"ContainerStarted","Data":"8fe922163d741ccd6b0a90e9d5eb3cdc614b278614fd33a613aa6acf3db777dd"}
Nov 03 19:31:36.003381 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:36.003327    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c5c44f4d-dv455" event={"ID":"d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca","Type":"ContainerStarted","Data":"415be167d9d95e070ae4a1d50c1e46128de973c7ca7089b2423b7c88b6be19c7"}
Nov 03 19:31:36.004866 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:36.004839    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wj8mp" event={"ID":"38a6add1-b051-450b-bb96-8ace8a0ccff3","Type":"ContainerStarted","Data":"d3c08561e0956cae70552ad436b8cb3e942d6278494918f595ab1234ba24d042"}
Nov 03 19:31:36.006272 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:36.006248    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-569d4c45f-7pszr" event={"ID":"a023911d-3669-43e1-ac77-8a986f3715d6","Type":"ContainerStarted","Data":"d6b199e62d9302e520b8484bdfc7acf801d5f1e152895a651ca0f576185c7cc7"}
Nov 03 19:31:37.013849 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:37.013798    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d74787d9c-nn7sk" event={"ID":"96c016f7-9098-4a39-9e72-e54da374d8fc","Type":"ContainerStarted","Data":"d5c57951c085459f9357db844566a803a7cdbfb75eb1704fc9863a12f59dcc8d"}
Nov 03 19:31:37.014295 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:37.013929    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-d74787d9c-nn7sk"
Nov 03 19:31:37.016838 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:37.015697    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c5c44f4d-dv455" event={"ID":"d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca","Type":"ContainerStarted","Data":"a0383f4dede1c6b4e6e8bf6daed883b7a0cf26a07299f94861ab7fb22175293b"}
Nov 03 19:31:37.019258 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:37.018785    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-569d4c45f-7pszr" event={"ID":"a023911d-3669-43e1-ac77-8a986f3715d6","Type":"ContainerStarted","Data":"c05526fe633dfb2a8187778297684e7bb6eacd04dab908ba97f3d657b5e1f195"}
Nov 03 19:31:37.359953 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:37.359870    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:31:37.360179 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:37.360108    4921 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 19:31:37.637468 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:37.637382    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-569d4c45f-7pszr"
Nov 03 19:31:37.640743 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:37.640692    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-569d4c45f-7pszr"
Nov 03 19:31:37.977488 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:37.977454    4921 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-gzf8k"
Nov 03 19:31:37.977980 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:37.977951    4921 scope.go:122] "RemoveContainer" containerID="2486b5467a6fc2b3c65ed9a9942d79836f6e99e7cb7ab904167391b30843d756"
Nov 03 19:31:37.978202 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:31:37.978178    4921 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-gzf8k_openshift-console-operator(6f215b42-0f09-46a0-bc98-392c0fa909c9)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-gzf8k" podUID="6f215b42-0f09-46a0-bc98-392c0fa909c9"
Nov 03 19:31:38.026268 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:38.026241    4921 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 19:31:38.026847 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:38.026288    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kfdsx" event={"ID":"e2b752d9-88c4-4dee-8b0c-7e9ac252adb3","Type":"ContainerStarted","Data":"7d403af8763e68a07c331ccb44c3ac9851a0c763af1403669fa75de25895b9ff"}
Nov 03 19:31:38.399977 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:38.399947    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-jwrp9"
Nov 03 19:31:38.881058 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:38.881029    4921 scope.go:122] "RemoveContainer" containerID="2486b5467a6fc2b3c65ed9a9942d79836f6e99e7cb7ab904167391b30843d756"
Nov 03 19:31:38.914625 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:38.914600    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k5nbr"
Nov 03 19:31:39.031491 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:39.031448    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wj8mp" event={"ID":"38a6add1-b051-450b-bb96-8ace8a0ccff3","Type":"ContainerStarted","Data":"79529a0cfabd4d6a5fd415cde20c533bb9482e131ddf65fa69feb1cadec3233d"}
Nov 03 19:31:39.031970 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:39.031639    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-wj8mp"
Nov 03 19:31:39.031970 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:39.031680    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wj8mp" event={"ID":"38a6add1-b051-450b-bb96-8ace8a0ccff3","Type":"ContainerStarted","Data":"6378a8bd166b8cb20f0bf7afbb3218e2edabe44f1385dcfb72febb8ae5953e86"}
Nov 03 19:31:39.033463 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:39.033443    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gzf8k_6f215b42-0f09-46a0-bc98-392c0fa909c9/console-operator/2.log"
Nov 03 19:31:39.033589 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:39.033557    4921 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 19:31:39.033589 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:39.033559    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-gzf8k" event={"ID":"6f215b42-0f09-46a0-bc98-392c0fa909c9","Type":"ContainerStarted","Data":"6fdad68961d08376b3f336b8fe8ff953ef1976976ded6c054c479de5bb316110"}
Nov 03 19:31:39.034033 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:39.034011    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-gzf8k"
Nov 03 19:31:39.329136 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:39.329100    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-gzf8k"
Nov 03 19:31:39.816162 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:39.816133    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-d74787d9c-nn7sk"]
Nov 03 19:31:39.986847 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:39.986810    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-569d4c45f-7pszr"
Nov 03 19:31:39.988218 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:39.988193    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-569d4c45f-7pszr"
Nov 03 19:31:44.531292 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:44.531252    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wj8mp_38a6add1-b051-450b-bb96-8ace8a0ccff3/dns/0.log"
Nov 03 19:31:44.730684 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:44.730652    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wj8mp_38a6add1-b051-450b-bb96-8ace8a0ccff3/kube-rbac-proxy/0.log"
Nov 03 19:31:45.131167 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:45.131139    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-p2p8d_56e21184-e568-46bc-8e90-204684d6c366/dns-node-resolver/0.log"
Nov 03 19:31:45.235687 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:45.235643    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-c5c44f4d-dv455"
Nov 03 19:31:45.731916 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:45.731869    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-c5c44f4d-dv455_d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca/registry/0.log"
Nov 03 19:31:45.930569 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:45.930539    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-d74787d9c-nn7sk_96c016f7-9098-4a39-9e72-e54da374d8fc/registry/0.log"
Nov 03 19:31:46.330757 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:46.330730    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vv2s2_545f3c9a-2ac6-44a3-be0b-cf71b1592d42/node-ca/0.log"
Nov 03 19:31:46.731487 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:46.731444    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-569d4c45f-7pszr_a023911d-3669-43e1-ac77-8a986f3715d6/router/0.log"
Nov 03 19:31:47.330654 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:47.330629    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-j7m2z_543b8778-baa4-4e09-b697-68e550e42942/serve-healthcheck-canary/0.log"
Nov 03 19:31:47.733316 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:47.733278    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-2bg7x"]
Nov 03 19:31:47.739861 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:47.739838    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2bg7x"
Nov 03 19:31:47.751421 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:47.751395    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zffgv\""
Nov 03 19:31:47.751624 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:47.751395    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 03 19:31:47.751624 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:47.751472    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 03 19:31:47.816057 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:47.816026    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 03 19:31:47.816057 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:47.816029    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 03 19:31:47.854162 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:47.854124    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/131f2758-bb6e-4e73-979c-2db443bf6b05-root\") pod \"node-exporter-2bg7x\" (UID: \"131f2758-bb6e-4e73-979c-2db443bf6b05\") " pod="openshift-monitoring/node-exporter-2bg7x"
Nov 03 19:31:47.854162 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:47.854163    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/131f2758-bb6e-4e73-979c-2db443bf6b05-node-exporter-accelerators-collector-config\") pod \"node-exporter-2bg7x\" (UID: \"131f2758-bb6e-4e73-979c-2db443bf6b05\") " pod="openshift-monitoring/node-exporter-2bg7x"
Nov 03 19:31:47.854390 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:47.854184    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/131f2758-bb6e-4e73-979c-2db443bf6b05-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2bg7x\" (UID: \"131f2758-bb6e-4e73-979c-2db443bf6b05\") " pod="openshift-monitoring/node-exporter-2bg7x"
Nov 03 19:31:47.854390 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:47.854201    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/131f2758-bb6e-4e73-979c-2db443bf6b05-node-exporter-wtmp\") pod \"node-exporter-2bg7x\" (UID: \"131f2758-bb6e-4e73-979c-2db443bf6b05\") " pod="openshift-monitoring/node-exporter-2bg7x"
Nov 03 19:31:47.854390 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:47.854335    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/131f2758-bb6e-4e73-979c-2db443bf6b05-sys\") pod \"node-exporter-2bg7x\" (UID: \"131f2758-bb6e-4e73-979c-2db443bf6b05\") " pod="openshift-monitoring/node-exporter-2bg7x"
Nov 03 19:31:47.854390 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:47.854370    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/131f2758-bb6e-4e73-979c-2db443bf6b05-node-exporter-textfile\") pod \"node-exporter-2bg7x\" (UID: \"131f2758-bb6e-4e73-979c-2db443bf6b05\") " pod="openshift-monitoring/node-exporter-2bg7x"
Nov 03 19:31:47.854558 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:47.854395    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/131f2758-bb6e-4e73-979c-2db443bf6b05-node-exporter-tls\") pod \"node-exporter-2bg7x\" (UID: \"131f2758-bb6e-4e73-979c-2db443bf6b05\") " pod="openshift-monitoring/node-exporter-2bg7x"
Nov 03 19:31:47.854558 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:47.854458    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flcmv\" (UniqueName: \"kubernetes.io/projected/131f2758-bb6e-4e73-979c-2db443bf6b05-kube-api-access-flcmv\") pod \"node-exporter-2bg7x\" (UID: \"131f2758-bb6e-4e73-979c-2db443bf6b05\") " pod="openshift-monitoring/node-exporter-2bg7x"
Nov 03 19:31:47.854558 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:47.854518    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/131f2758-bb6e-4e73-979c-2db443bf6b05-metrics-client-ca\") pod \"node-exporter-2bg7x\" (UID: \"131f2758-bb6e-4e73-979c-2db443bf6b05\") " pod="openshift-monitoring/node-exporter-2bg7x"
Nov 03 19:31:47.956059 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:47.956020    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/131f2758-bb6e-4e73-979c-2db443bf6b05-sys\") pod \"node-exporter-2bg7x\" (UID: \"131f2758-bb6e-4e73-979c-2db443bf6b05\") " pod="openshift-monitoring/node-exporter-2bg7x"
Nov 03 19:31:47.956059 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:47.956057    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/131f2758-bb6e-4e73-979c-2db443bf6b05-node-exporter-textfile\") pod \"node-exporter-2bg7x\" (UID: \"131f2758-bb6e-4e73-979c-2db443bf6b05\") " pod="openshift-monitoring/node-exporter-2bg7x"
Nov 03 19:31:47.956310 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:47.956088    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/131f2758-bb6e-4e73-979c-2db443bf6b05-node-exporter-tls\") pod \"node-exporter-2bg7x\" (UID: \"131f2758-bb6e-4e73-979c-2db443bf6b05\") " pod="openshift-monitoring/node-exporter-2bg7x"
Nov 03 19:31:47.956310 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:47.956114    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flcmv\" (UniqueName: \"kubernetes.io/projected/131f2758-bb6e-4e73-979c-2db443bf6b05-kube-api-access-flcmv\") pod \"node-exporter-2bg7x\" (UID: \"131f2758-bb6e-4e73-979c-2db443bf6b05\") " pod="openshift-monitoring/node-exporter-2bg7x"
Nov 03 19:31:47.956310 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:47.956126    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/131f2758-bb6e-4e73-979c-2db443bf6b05-sys\") pod \"node-exporter-2bg7x\" (UID: \"131f2758-bb6e-4e73-979c-2db443bf6b05\") " pod="openshift-monitoring/node-exporter-2bg7x"
Nov 03 19:31:47.956310 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:47.956140    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/131f2758-bb6e-4e73-979c-2db443bf6b05-metrics-client-ca\") pod \"node-exporter-2bg7x\" (UID: \"131f2758-bb6e-4e73-979c-2db443bf6b05\") " pod="openshift-monitoring/node-exporter-2bg7x"
Nov 03 19:31:47.956310 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:47.956210    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/131f2758-bb6e-4e73-979c-2db443bf6b05-root\") pod \"node-exporter-2bg7x\" (UID: \"131f2758-bb6e-4e73-979c-2db443bf6b05\") " pod="openshift-monitoring/node-exporter-2bg7x"
Nov 03 19:31:47.956310 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:47.956237    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/131f2758-bb6e-4e73-979c-2db443bf6b05-node-exporter-accelerators-collector-config\") pod \"node-exporter-2bg7x\" (UID: \"131f2758-bb6e-4e73-979c-2db443bf6b05\") " pod="openshift-monitoring/node-exporter-2bg7x"
Nov 03 19:31:47.956310 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:47.956271    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/131f2758-bb6e-4e73-979c-2db443bf6b05-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2bg7x\" (UID: \"131f2758-bb6e-4e73-979c-2db443bf6b05\") " pod="openshift-monitoring/node-exporter-2bg7x"
Nov 03 19:31:47.956310 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:47.956298    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/131f2758-bb6e-4e73-979c-2db443bf6b05-node-exporter-wtmp\") pod \"node-exporter-2bg7x\" (UID: \"131f2758-bb6e-4e73-979c-2db443bf6b05\") " pod="openshift-monitoring/node-exporter-2bg7x"
Nov 03 19:31:47.956687 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:47.956326    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/131f2758-bb6e-4e73-979c-2db443bf6b05-root\") pod \"node-exporter-2bg7x\" (UID: \"131f2758-bb6e-4e73-979c-2db443bf6b05\") " pod="openshift-monitoring/node-exporter-2bg7x"
Nov 03 19:31:47.956687 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:47.956452    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/131f2758-bb6e-4e73-979c-2db443bf6b05-node-exporter-wtmp\") pod \"node-exporter-2bg7x\" (UID: \"131f2758-bb6e-4e73-979c-2db443bf6b05\") " pod="openshift-monitoring/node-exporter-2bg7x"
Nov 03 19:31:47.956687 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:47.956478    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/131f2758-bb6e-4e73-979c-2db443bf6b05-node-exporter-textfile\") pod \"node-exporter-2bg7x\" (UID: \"131f2758-bb6e-4e73-979c-2db443bf6b05\") " pod="openshift-monitoring/node-exporter-2bg7x"
Nov 03 19:31:47.956836 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:47.956779    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/131f2758-bb6e-4e73-979c-2db443bf6b05-metrics-client-ca\") pod \"node-exporter-2bg7x\" (UID: \"131f2758-bb6e-4e73-979c-2db443bf6b05\") " pod="openshift-monitoring/node-exporter-2bg7x"
Nov 03 19:31:47.956836 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:47.956818    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/131f2758-bb6e-4e73-979c-2db443bf6b05-node-exporter-accelerators-collector-config\") pod \"node-exporter-2bg7x\" (UID: \"131f2758-bb6e-4e73-979c-2db443bf6b05\") " pod="openshift-monitoring/node-exporter-2bg7x"
Nov 03 19:31:47.958610 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:47.958581    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/131f2758-bb6e-4e73-979c-2db443bf6b05-node-exporter-tls\") pod \"node-exporter-2bg7x\" (UID: \"131f2758-bb6e-4e73-979c-2db443bf6b05\") " pod="openshift-monitoring/node-exporter-2bg7x"
Nov 03 19:31:47.958738 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:47.958620    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/131f2758-bb6e-4e73-979c-2db443bf6b05-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2bg7x\" (UID: \"131f2758-bb6e-4e73-979c-2db443bf6b05\") " pod="openshift-monitoring/node-exporter-2bg7x"
Nov 03 19:31:47.962783 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:47.962761    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flcmv\" (UniqueName: \"kubernetes.io/projected/131f2758-bb6e-4e73-979c-2db443bf6b05-kube-api-access-flcmv\") pod \"node-exporter-2bg7x\" (UID: \"131f2758-bb6e-4e73-979c-2db443bf6b05\") " pod="openshift-monitoring/node-exporter-2bg7x"
Nov 03 19:31:48.058046 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:48.057954    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2bg7x"
Nov 03 19:31:48.072307 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:31:48.072278    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod131f2758_bb6e_4e73_979c_2db443bf6b05.slice/crio-62323c3e2b94011548ba0e4019da6c03af74c27120c143bdb06c50d08ae8d1f8 WatchSource:0}: Error finding container 62323c3e2b94011548ba0e4019da6c03af74c27120c143bdb06c50d08ae8d1f8: Status 404 returned error can't find the container with id 62323c3e2b94011548ba0e4019da6c03af74c27120c143bdb06c50d08ae8d1f8
Nov 03 19:31:49.042175 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:49.042136    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-wj8mp"
Nov 03 19:31:49.075437 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:49.075351    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2bg7x" event={"ID":"131f2758-bb6e-4e73-979c-2db443bf6b05","Type":"ContainerStarted","Data":"62323c3e2b94011548ba0e4019da6c03af74c27120c143bdb06c50d08ae8d1f8"}
Nov 03 19:31:50.080830 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:50.080791    4921 generic.go:356] "Generic (PLEG): container finished" podID="131f2758-bb6e-4e73-979c-2db443bf6b05" containerID="ea95e8638db09756939183484f196b744c683e6f8b05cbb8f8963050cb36df9f" exitCode=0
Nov 03 19:31:50.081340 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:50.080854    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2bg7x" event={"ID":"131f2758-bb6e-4e73-979c-2db443bf6b05","Type":"ContainerDied","Data":"ea95e8638db09756939183484f196b744c683e6f8b05cbb8f8963050cb36df9f"}
Nov 03 19:31:51.086649 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:51.086610    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2bg7x" event={"ID":"131f2758-bb6e-4e73-979c-2db443bf6b05","Type":"ContainerStarted","Data":"cfd64aadf1942456630554c59995040081565d2ea9b52ba884cd753c91968d6b"}
Nov 03 19:31:51.087061 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:51.086687    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2bg7x" event={"ID":"131f2758-bb6e-4e73-979c-2db443bf6b05","Type":"ContainerStarted","Data":"6ffcb3873a84e1ecd16354e8c06052040cdc5f9f985204da686524c4a3639e17"}
Nov 03 19:31:51.103880 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:51.103818    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-2bg7x" podStartSLOduration=2.848878869 podStartE2EDuration="4.103802107s" totalImagesPullingTime="1.186487378s" totalInitContainerRuntime="68.43586ms" isStatefulPod=true podCreationTimestamp="2026-11-03 19:31:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:31:51.102049723 +0000 UTC m=+24.346015188" watchObservedRunningTime="2026-11-03 19:31:51.103802107 +0000 UTC m=+24.347767571"
Nov 03 19:31:51.420622 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:51.420546    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-njg27"
Nov 03 19:31:52.758467 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:52.758428    4921 patch_prober.go:29] interesting pod/image-registry-c5c44f4d-dv455 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Nov 03 19:31:52.758882 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:52.758490    4921 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-c5c44f4d-dv455" podUID="d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 19:31:55.220225 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:55.220189    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-c5c44f4d-dv455"]
Nov 03 19:31:55.224479 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:55.224427    4921 patch_prober.go:29] interesting pod/image-registry-c5c44f4d-dv455 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Nov 03 19:31:55.224692 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:55.224487    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-c5c44f4d-dv455" podUID="d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 19:31:59.823357 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:31:59.823322    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-d74787d9c-nn7sk"
Nov 03 19:32:02.930244 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:32:02.930203    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 19:32:04.838480 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:32:04.838435    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-d74787d9c-nn7sk" podUID="96c016f7-9098-4a39-9e72-e54da374d8fc" containerName="registry" containerID="cri-o://d5c57951c085459f9357db844566a803a7cdbfb75eb1704fc9863a12f59dcc8d" gracePeriod=30
Nov 03 19:32:05.071732 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:32:05.071706    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d74787d9c-nn7sk"
Nov 03 19:32:05.106719 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:32:05.106624    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96c016f7-9098-4a39-9e72-e54da374d8fc-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/96c016f7-9098-4a39-9e72-e54da374d8fc-bound-sa-token\") pod \"96c016f7-9098-4a39-9e72-e54da374d8fc\" (UID: \"96c016f7-9098-4a39-9e72-e54da374d8fc\") "
Nov 03 19:32:05.106871 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:32:05.106712    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96c016f7-9098-4a39-9e72-e54da374d8fc-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/96c016f7-9098-4a39-9e72-e54da374d8fc-image-registry-private-configuration\") pod \"96c016f7-9098-4a39-9e72-e54da374d8fc\" (UID: \"96c016f7-9098-4a39-9e72-e54da374d8fc\") "
Nov 03 19:32:05.106871 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:32:05.106761    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96c016f7-9098-4a39-9e72-e54da374d8fc-kube-api-access-4w42k\" (UniqueName: \"kubernetes.io/projected/96c016f7-9098-4a39-9e72-e54da374d8fc-kube-api-access-4w42k\") pod \"96c016f7-9098-4a39-9e72-e54da374d8fc\" (UID: \"96c016f7-9098-4a39-9e72-e54da374d8fc\") "
Nov 03 19:32:05.106871 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:32:05.106793    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96c016f7-9098-4a39-9e72-e54da374d8fc-registry-tls\" (UniqueName: \"kubernetes.io/projected/96c016f7-9098-4a39-9e72-e54da374d8fc-registry-tls\") pod \"96c016f7-9098-4a39-9e72-e54da374d8fc\" (UID: \"96c016f7-9098-4a39-9e72-e54da374d8fc\") "
Nov 03 19:32:05.106871 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:32:05.106862    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96c016f7-9098-4a39-9e72-e54da374d8fc-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/96c016f7-9098-4a39-9e72-e54da374d8fc-ca-trust-extracted\") pod \"96c016f7-9098-4a39-9e72-e54da374d8fc\" (UID: \"96c016f7-9098-4a39-9e72-e54da374d8fc\") "
Nov 03 19:32:05.107088 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:32:05.106925    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96c016f7-9098-4a39-9e72-e54da374d8fc-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/96c016f7-9098-4a39-9e72-e54da374d8fc-registry-certificates\") pod \"96c016f7-9098-4a39-9e72-e54da374d8fc\" (UID: \"96c016f7-9098-4a39-9e72-e54da374d8fc\") "
Nov 03 19:32:05.107088 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:32:05.106974    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96c016f7-9098-4a39-9e72-e54da374d8fc-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/96c016f7-9098-4a39-9e72-e54da374d8fc-trusted-ca\") pod \"96c016f7-9098-4a39-9e72-e54da374d8fc\" (UID: \"96c016f7-9098-4a39-9e72-e54da374d8fc\") "
Nov 03 19:32:05.107088 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:32:05.107007    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96c016f7-9098-4a39-9e72-e54da374d8fc-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/96c016f7-9098-4a39-9e72-e54da374d8fc-installation-pull-secrets\") pod \"96c016f7-9098-4a39-9e72-e54da374d8fc\" (UID: \"96c016f7-9098-4a39-9e72-e54da374d8fc\") "
Nov 03 19:32:05.107623 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:32:05.107583    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96c016f7-9098-4a39-9e72-e54da374d8fc-trusted-ca" pod "96c016f7-9098-4a39-9e72-e54da374d8fc" (UID: "96c016f7-9098-4a39-9e72-e54da374d8fc"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:32:05.107964 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:32:05.107935    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96c016f7-9098-4a39-9e72-e54da374d8fc-registry-certificates" pod "96c016f7-9098-4a39-9e72-e54da374d8fc" (UID: "96c016f7-9098-4a39-9e72-e54da374d8fc"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:32:05.109656 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:32:05.109625    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96c016f7-9098-4a39-9e72-e54da374d8fc-registry-tls" pod "96c016f7-9098-4a39-9e72-e54da374d8fc" (UID: "96c016f7-9098-4a39-9e72-e54da374d8fc"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:32:05.109919 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:32:05.109873    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96c016f7-9098-4a39-9e72-e54da374d8fc-kube-api-access-4w42k" pod "96c016f7-9098-4a39-9e72-e54da374d8fc" (UID: "96c016f7-9098-4a39-9e72-e54da374d8fc"). InnerVolumeSpecName "kube-api-access-4w42k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:32:05.110110 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:32:05.109942    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96c016f7-9098-4a39-9e72-e54da374d8fc-installation-pull-secrets" pod "96c016f7-9098-4a39-9e72-e54da374d8fc" (UID: "96c016f7-9098-4a39-9e72-e54da374d8fc"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:32:05.110110 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:32:05.109973    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96c016f7-9098-4a39-9e72-e54da374d8fc-bound-sa-token" pod "96c016f7-9098-4a39-9e72-e54da374d8fc" (UID: "96c016f7-9098-4a39-9e72-e54da374d8fc"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:32:05.110233 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:32:05.110106    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96c016f7-9098-4a39-9e72-e54da374d8fc-image-registry-private-configuration" pod "96c016f7-9098-4a39-9e72-e54da374d8fc" (UID: "96c016f7-9098-4a39-9e72-e54da374d8fc"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:32:05.115108 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:32:05.115075    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96c016f7-9098-4a39-9e72-e54da374d8fc-ca-trust-extracted" pod "96c016f7-9098-4a39-9e72-e54da374d8fc" (UID: "96c016f7-9098-4a39-9e72-e54da374d8fc"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:32:05.145623 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:32:05.145591    4921 generic.go:356] "Generic (PLEG): container finished" podID="96c016f7-9098-4a39-9e72-e54da374d8fc" containerID="d5c57951c085459f9357db844566a803a7cdbfb75eb1704fc9863a12f59dcc8d" exitCode=0
Nov 03 19:32:05.145853 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:32:05.145643    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d74787d9c-nn7sk"
Nov 03 19:32:05.145853 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:32:05.145644    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d74787d9c-nn7sk" event={"ID":"96c016f7-9098-4a39-9e72-e54da374d8fc","Type":"ContainerDied","Data":"d5c57951c085459f9357db844566a803a7cdbfb75eb1704fc9863a12f59dcc8d"}
Nov 03 19:32:05.145853 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:32:05.145775    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d74787d9c-nn7sk" event={"ID":"96c016f7-9098-4a39-9e72-e54da374d8fc","Type":"ContainerDied","Data":"34e274469bc0110e779b44dbe51d88dd9e86372a8b1e978b8b598768722f15a6"}
Nov 03 19:32:05.145853 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:32:05.145807    4921 scope.go:122] "RemoveContainer" containerID="d5c57951c085459f9357db844566a803a7cdbfb75eb1704fc9863a12f59dcc8d"
Nov 03 19:32:05.154468 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:32:05.154443    4921 scope.go:122] "RemoveContainer" containerID="d5c57951c085459f9357db844566a803a7cdbfb75eb1704fc9863a12f59dcc8d"
Nov 03 19:32:05.154767 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:32:05.154746    4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5c57951c085459f9357db844566a803a7cdbfb75eb1704fc9863a12f59dcc8d\": container with ID starting with d5c57951c085459f9357db844566a803a7cdbfb75eb1704fc9863a12f59dcc8d not found: ID does not exist" containerID="d5c57951c085459f9357db844566a803a7cdbfb75eb1704fc9863a12f59dcc8d"
Nov 03 19:32:05.154846 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:32:05.154782    4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5c57951c085459f9357db844566a803a7cdbfb75eb1704fc9863a12f59dcc8d"} err="failed to get container status \"d5c57951c085459f9357db844566a803a7cdbfb75eb1704fc9863a12f59dcc8d\": rpc error: code = NotFound desc = could not find container \"d5c57951c085459f9357db844566a803a7cdbfb75eb1704fc9863a12f59dcc8d\": container with ID starting with d5c57951c085459f9357db844566a803a7cdbfb75eb1704fc9863a12f59dcc8d not found: ID does not exist"
Nov 03 19:32:05.166016 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:32:05.165985    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-d74787d9c-nn7sk"]
Nov 03 19:32:05.168980 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:32:05.168954    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-d74787d9c-nn7sk"]
Nov 03 19:32:05.208269 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:32:05.208233    4921 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/96c016f7-9098-4a39-9e72-e54da374d8fc-ca-trust-extracted\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:32:05.208269 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:32:05.208267    4921 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/96c016f7-9098-4a39-9e72-e54da374d8fc-registry-certificates\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:32:05.208448 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:32:05.208281    4921 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/96c016f7-9098-4a39-9e72-e54da374d8fc-trusted-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:32:05.208448 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:32:05.208290    4921 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/96c016f7-9098-4a39-9e72-e54da374d8fc-installation-pull-secrets\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:32:05.208448 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:32:05.208299    4921 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/96c016f7-9098-4a39-9e72-e54da374d8fc-bound-sa-token\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:32:05.208448 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:32:05.208308    4921 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/96c016f7-9098-4a39-9e72-e54da374d8fc-image-registry-private-configuration\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:32:05.208448 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:32:05.208317    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4w42k\" (UniqueName: \"kubernetes.io/projected/96c016f7-9098-4a39-9e72-e54da374d8fc-kube-api-access-4w42k\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:32:05.208448 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:32:05.208325    4921 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/96c016f7-9098-4a39-9e72-e54da374d8fc-registry-tls\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:32:05.224729 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:32:05.224700    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-c5c44f4d-dv455"
Nov 03 19:32:06.885842 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:32:06.885770    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96c016f7-9098-4a39-9e72-e54da374d8fc" path="/var/lib/kubelet/pods/96c016f7-9098-4a39-9e72-e54da374d8fc/volumes"
Nov 03 19:32:20.239925 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:32:20.239862    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-c5c44f4d-dv455" podUID="d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca" containerName="registry" containerID="cri-o://a0383f4dede1c6b4e6e8bf6daed883b7a0cf26a07299f94861ab7fb22175293b" gracePeriod=30
Nov 03 19:32:20.489338 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:32:20.489312    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c5c44f4d-dv455"
Nov 03 19:32:20.526081 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:32:20.525989    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-image-registry-private-configuration\") pod \"d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca\" (UID: \"d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca\") "
Nov 03 19:32:20.526081 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:32:20.526040    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-registry-certificates\") pod \"d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca\" (UID: \"d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca\") "
Nov 03 19:32:20.526081 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:32:20.526068    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-registry-tls\" (UniqueName: \"kubernetes.io/projected/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-registry-tls\") pod \"d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca\" (UID: \"d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca\") "
Nov 03 19:32:20.526354 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:32:20.526238    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-kube-api-access-8gm62\" (UniqueName: \"kubernetes.io/projected/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-kube-api-access-8gm62\") pod \"d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca\" (UID: \"d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca\") "
Nov 03 19:32:20.526354 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:32:20.526295    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-bound-sa-token\") pod \"d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca\" (UID: \"d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca\") "
Nov 03 19:32:20.526468 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:32:20.526385    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-trusted-ca\") pod \"d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca\" (UID: \"d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca\") "
Nov 03 19:32:20.526468 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:32:20.526417    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-ca-trust-extracted\") pod \"d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca\" (UID: \"d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca\") "
Nov 03 19:32:20.526573 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:32:20.526472    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-installation-pull-secrets\") pod \"d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca\" (UID: \"d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca\") "
Nov 03 19:32:20.526573 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:32:20.526503    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-registry-certificates" pod "d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca" (UID: "d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:32:20.527033 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:32:20.526706    4921 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-registry-certificates\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:32:20.527033 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:32:20.526913    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-trusted-ca" pod "d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca" (UID: "d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:32:20.529264 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:32:20.529232    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-registry-tls" pod "d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca" (UID: "d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:32:20.529488 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:32:20.529459    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-bound-sa-token" pod "d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca" (UID: "d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:32:20.529555 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:32:20.529494    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-installation-pull-secrets" pod "d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca" (UID: "d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:32:20.529602 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:32:20.529553    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-kube-api-access-8gm62" pod "d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca" (UID: "d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca"). InnerVolumeSpecName "kube-api-access-8gm62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:32:20.529637 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:32:20.529606    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-image-registry-private-configuration" pod "d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca" (UID: "d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:32:20.533854 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:32:20.533830    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-ca-trust-extracted" pod "d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca" (UID: "d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:32:20.627094 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:32:20.627052    4921 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-trusted-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:32:20.627094 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:32:20.627091    4921 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-ca-trust-extracted\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:32:20.627094 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:32:20.627102    4921 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-installation-pull-secrets\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:32:20.627327 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:32:20.627114    4921 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-image-registry-private-configuration\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:32:20.627327 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:32:20.627126    4921 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-registry-tls\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:32:20.627327 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:32:20.627135    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gm62\" (UniqueName: \"kubernetes.io/projected/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-kube-api-access-8gm62\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:32:20.627327 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:32:20.627143    4921 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca-bound-sa-token\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:32:21.208484 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:32:21.208397    4921 generic.go:356] "Generic (PLEG): container finished" podID="d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca" containerID="a0383f4dede1c6b4e6e8bf6daed883b7a0cf26a07299f94861ab7fb22175293b" exitCode=0
Nov 03 19:32:21.208484 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:32:21.208451    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c5c44f4d-dv455"
Nov 03 19:32:21.208741 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:32:21.208479    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c5c44f4d-dv455" event={"ID":"d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca","Type":"ContainerDied","Data":"a0383f4dede1c6b4e6e8bf6daed883b7a0cf26a07299f94861ab7fb22175293b"}
Nov 03 19:32:21.208741 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:32:21.208538    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c5c44f4d-dv455" event={"ID":"d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca","Type":"ContainerDied","Data":"415be167d9d95e070ae4a1d50c1e46128de973c7ca7089b2423b7c88b6be19c7"}
Nov 03 19:32:21.208741 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:32:21.208559    4921 scope.go:122] "RemoveContainer" containerID="a0383f4dede1c6b4e6e8bf6daed883b7a0cf26a07299f94861ab7fb22175293b"
Nov 03 19:32:21.218929 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:32:21.218902    4921 scope.go:122] "RemoveContainer" containerID="a0383f4dede1c6b4e6e8bf6daed883b7a0cf26a07299f94861ab7fb22175293b"
Nov 03 19:32:21.219271 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:32:21.219251    4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0383f4dede1c6b4e6e8bf6daed883b7a0cf26a07299f94861ab7fb22175293b\": container with ID starting with a0383f4dede1c6b4e6e8bf6daed883b7a0cf26a07299f94861ab7fb22175293b not found: ID does not exist" containerID="a0383f4dede1c6b4e6e8bf6daed883b7a0cf26a07299f94861ab7fb22175293b"
Nov 03 19:32:21.219338 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:32:21.219283    4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0383f4dede1c6b4e6e8bf6daed883b7a0cf26a07299f94861ab7fb22175293b"} err="failed to get container status \"a0383f4dede1c6b4e6e8bf6daed883b7a0cf26a07299f94861ab7fb22175293b\": rpc error: code = NotFound desc = could not find container \"a0383f4dede1c6b4e6e8bf6daed883b7a0cf26a07299f94861ab7fb22175293b\": container with ID starting with a0383f4dede1c6b4e6e8bf6daed883b7a0cf26a07299f94861ab7fb22175293b not found: ID does not exist"
Nov 03 19:32:21.222520 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:32:21.222486    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-c5c44f4d-dv455"]
Nov 03 19:32:21.226194 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:32:21.226168    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-c5c44f4d-dv455"]
Nov 03 19:32:22.884878 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:32:22.884846    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca" path="/var/lib/kubelet/pods/d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca/volumes"
Nov 03 19:32:32.920624 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:32:32.920587    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 19:32:44.292116 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:32:44.292078    4921 generic.go:356] "Generic (PLEG): container finished" podID="e10dc749-bc71-456e-a315-6581add01a7b" containerID="1c87bff726a20edd63a0001b79174fd2bfc95b7add8a7f63b480949c9844f331" exitCode=0
Nov 03 19:32:44.292626 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:32:44.292151    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hxlrb" event={"ID":"e10dc749-bc71-456e-a315-6581add01a7b","Type":"ContainerDied","Data":"1c87bff726a20edd63a0001b79174fd2bfc95b7add8a7f63b480949c9844f331"}
Nov 03 19:32:44.292626 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:32:44.292489    4921 scope.go:122] "RemoveContainer" containerID="1c87bff726a20edd63a0001b79174fd2bfc95b7add8a7f63b480949c9844f331"
Nov 03 19:32:45.297104 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:32:45.297069    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hxlrb" event={"ID":"e10dc749-bc71-456e-a315-6581add01a7b","Type":"ContainerStarted","Data":"0e36c38dbf37ed579ea00c97d702da6e703404443a7778a8fb6ebe79b7254110"}
Nov 03 19:33:01.355880 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:33:01.355845    4921 generic.go:356] "Generic (PLEG): container finished" podID="3880f3d3-aad1-43c0-b704-01374aad4f78" containerID="2a86f477e3c4f116e4a2246fe55a4a6f8aa9318da6ac3455a9dc979101b9f50f" exitCode=0
Nov 03 19:33:01.356314 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:33:01.355923    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-8kh6f" event={"ID":"3880f3d3-aad1-43c0-b704-01374aad4f78","Type":"ContainerDied","Data":"2a86f477e3c4f116e4a2246fe55a4a6f8aa9318da6ac3455a9dc979101b9f50f"}
Nov 03 19:33:01.356314 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:33:01.355983    4921 scope.go:122] "RemoveContainer" containerID="b7ce8ebe35d2285a3f7ef842541ab9676c44e1038b05db62a869ad1d327de641"
Nov 03 19:33:01.356314 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:33:01.356258    4921 scope.go:122] "RemoveContainer" containerID="2a86f477e3c4f116e4a2246fe55a4a6f8aa9318da6ac3455a9dc979101b9f50f"
Nov 03 19:33:01.356507 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:33:01.356486    4921 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"insights-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=insights-operator pod=insights-operator-7c4986d449-8kh6f_openshift-insights(3880f3d3-aad1-43c0-b704-01374aad4f78)\"" pod="openshift-insights/insights-operator-7c4986d449-8kh6f" podUID="3880f3d3-aad1-43c0-b704-01374aad4f78"
Nov 03 19:33:02.919702 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:33:02.919652    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 19:33:11.881118 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:33:11.881084    4921 scope.go:122] "RemoveContainer" containerID="2a86f477e3c4f116e4a2246fe55a4a6f8aa9318da6ac3455a9dc979101b9f50f"
Nov 03 19:33:12.397188 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:33:12.397145    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-8kh6f" event={"ID":"3880f3d3-aad1-43c0-b704-01374aad4f78","Type":"ContainerStarted","Data":"1f457a26f52421749011cbf846b0a46af5a1100fe6a3ea0537dadc6ad9f9bfa6"}
Nov 03 19:33:32.920362 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:33:32.920325    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 19:34:02.919858 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:34:02.919821    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 19:34:32.919569 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:34:32.919525    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 19:35:02.919040 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:35:02.919000    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 19:35:32.918055 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:35:32.917971    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 19:35:47.005857 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:47.005818    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj255d"]
Nov 03 19:35:47.008240 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:47.006141    4921 cpu_manager.go:414] "containerMap: removing container" podUID="d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca" containerName="registry"
Nov 03 19:35:47.008240 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:47.006153    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca" containerName="registry"
Nov 03 19:35:47.008240 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:47.006168    4921 cpu_manager.go:414] "containerMap: removing container" podUID="96c016f7-9098-4a39-9e72-e54da374d8fc" containerName="registry"
Nov 03 19:35:47.008240 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:47.006174    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96c016f7-9098-4a39-9e72-e54da374d8fc" containerName="registry"
Nov 03 19:35:47.008240 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:47.006235    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3e8dd07-ea2a-46dc-9ea6-e0bd949698ca" containerName="registry"
Nov 03 19:35:47.008240 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:47.006243    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="96c016f7-9098-4a39-9e72-e54da374d8fc" containerName="registry"
Nov 03 19:35:47.009171 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:47.009154    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj255d"
Nov 03 19:35:47.011872 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:47.011851    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Nov 03 19:35:47.012346 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:47.012322    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Nov 03 19:35:47.012465 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:47.012369    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-2xm6j\""
Nov 03 19:35:47.027985 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:47.027947    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj255d"]
Nov 03 19:35:47.078865 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:47.078827    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/01cf4a61-867e-41d3-b0b7-2be0d700458b-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj255d\" (UID: \"01cf4a61-867e-41d3-b0b7-2be0d700458b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj255d"
Nov 03 19:35:47.079065 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:47.078900    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gmj5\" (UniqueName: \"kubernetes.io/projected/01cf4a61-867e-41d3-b0b7-2be0d700458b-kube-api-access-5gmj5\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj255d\" (UID: \"01cf4a61-867e-41d3-b0b7-2be0d700458b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj255d"
Nov 03 19:35:47.079065 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:47.078997    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/01cf4a61-867e-41d3-b0b7-2be0d700458b-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj255d\" (UID: \"01cf4a61-867e-41d3-b0b7-2be0d700458b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj255d"
Nov 03 19:35:47.180211 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:47.180176    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/01cf4a61-867e-41d3-b0b7-2be0d700458b-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj255d\" (UID: \"01cf4a61-867e-41d3-b0b7-2be0d700458b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj255d"
Nov 03 19:35:47.180395 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:47.180232    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gmj5\" (UniqueName: \"kubernetes.io/projected/01cf4a61-867e-41d3-b0b7-2be0d700458b-kube-api-access-5gmj5\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj255d\" (UID: \"01cf4a61-867e-41d3-b0b7-2be0d700458b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj255d"
Nov 03 19:35:47.180395 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:47.180269    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/01cf4a61-867e-41d3-b0b7-2be0d700458b-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj255d\" (UID: \"01cf4a61-867e-41d3-b0b7-2be0d700458b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj255d"
Nov 03 19:35:47.180686 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:47.180647    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/01cf4a61-867e-41d3-b0b7-2be0d700458b-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj255d\" (UID: \"01cf4a61-867e-41d3-b0b7-2be0d700458b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj255d"
Nov 03 19:35:47.180729 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:47.180660    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/01cf4a61-867e-41d3-b0b7-2be0d700458b-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj255d\" (UID: \"01cf4a61-867e-41d3-b0b7-2be0d700458b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj255d"
Nov 03 19:35:47.188790 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:47.188765    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vpk8d"]
Nov 03 19:35:47.191612 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:47.191590    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gmj5\" (UniqueName: \"kubernetes.io/projected/01cf4a61-867e-41d3-b0b7-2be0d700458b-kube-api-access-5gmj5\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj255d\" (UID: \"01cf4a61-867e-41d3-b0b7-2be0d700458b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj255d"
Nov 03 19:35:47.192493 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:47.192478    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vpk8d"
Nov 03 19:35:47.203230 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:47.203193    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vpk8d"]
Nov 03 19:35:47.274167 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:47.274083    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-cplc4\""
Nov 03 19:35:47.281403 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:47.281374    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ba24656-710d-4547-9b5c-293b76fabd06-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vpk8d\" (UID: \"2ba24656-710d-4547-9b5c-293b76fabd06\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vpk8d"
Nov 03 19:35:47.281464 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:47.281422    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xvpq\" (UniqueName: \"kubernetes.io/projected/2ba24656-710d-4547-9b5c-293b76fabd06-kube-api-access-7xvpq\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vpk8d\" (UID: \"2ba24656-710d-4547-9b5c-293b76fabd06\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vpk8d"
Nov 03 19:35:47.281527 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:47.281514    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2ba24656-710d-4547-9b5c-293b76fabd06-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vpk8d\" (UID: \"2ba24656-710d-4547-9b5c-293b76fabd06\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vpk8d"
Nov 03 19:35:47.324619 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:47.324589    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj255d"
Nov 03 19:35:47.382889 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:47.382853    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2ba24656-710d-4547-9b5c-293b76fabd06-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vpk8d\" (UID: \"2ba24656-710d-4547-9b5c-293b76fabd06\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vpk8d"
Nov 03 19:35:47.383051 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:47.382939    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ba24656-710d-4547-9b5c-293b76fabd06-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vpk8d\" (UID: \"2ba24656-710d-4547-9b5c-293b76fabd06\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vpk8d"
Nov 03 19:35:47.383051 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:47.382980    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xvpq\" (UniqueName: \"kubernetes.io/projected/2ba24656-710d-4547-9b5c-293b76fabd06-kube-api-access-7xvpq\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vpk8d\" (UID: \"2ba24656-710d-4547-9b5c-293b76fabd06\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vpk8d"
Nov 03 19:35:47.383316 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:47.383292    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2ba24656-710d-4547-9b5c-293b76fabd06-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vpk8d\" (UID: \"2ba24656-710d-4547-9b5c-293b76fabd06\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vpk8d"
Nov 03 19:35:47.383396 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:47.383327    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ba24656-710d-4547-9b5c-293b76fabd06-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vpk8d\" (UID: \"2ba24656-710d-4547-9b5c-293b76fabd06\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vpk8d"
Nov 03 19:35:47.397041 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:47.395912    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xvpq\" (UniqueName: \"kubernetes.io/projected/2ba24656-710d-4547-9b5c-293b76fabd06-kube-api-access-7xvpq\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vpk8d\" (UID: \"2ba24656-710d-4547-9b5c-293b76fabd06\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vpk8d"
Nov 03 19:35:47.513338 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:47.513299    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vpk8d"
Nov 03 19:35:47.595044 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:47.595009    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj255d"]
Nov 03 19:35:47.598204 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:35:47.598166    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01cf4a61_867e_41d3_b0b7_2be0d700458b.slice/crio-d1f9ceb7014d0fcf442a25a66fe6851808417f60d6115a79523091b04656c27a WatchSource:0}: Error finding container d1f9ceb7014d0fcf442a25a66fe6851808417f60d6115a79523091b04656c27a: Status 404 returned error can't find the container with id d1f9ceb7014d0fcf442a25a66fe6851808417f60d6115a79523091b04656c27a
Nov 03 19:35:47.783982 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:47.783906    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vpk8d"]
Nov 03 19:35:47.787206 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:35:47.787178    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ba24656_710d_4547_9b5c_293b76fabd06.slice/crio-a36db7ba4464961285483896b5ced0485d3b1d987a83417bee45dd02b02def79 WatchSource:0}: Error finding container a36db7ba4464961285483896b5ced0485d3b1d987a83417bee45dd02b02def79: Status 404 returned error can't find the container with id a36db7ba4464961285483896b5ced0485d3b1d987a83417bee45dd02b02def79
Nov 03 19:35:47.938808 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:47.938770    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj255d" event={"ID":"01cf4a61-867e-41d3-b0b7-2be0d700458b","Type":"ContainerStarted","Data":"d1f9ceb7014d0fcf442a25a66fe6851808417f60d6115a79523091b04656c27a"}
Nov 03 19:35:47.939913 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:47.939889    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vpk8d" event={"ID":"2ba24656-710d-4547-9b5c-293b76fabd06","Type":"ContainerStarted","Data":"a36db7ba4464961285483896b5ced0485d3b1d987a83417bee45dd02b02def79"}
Nov 03 19:35:50.973817 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:50.973776    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449cmzc"]
Nov 03 19:35:50.977783 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:50.977755    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449cmzc"
Nov 03 19:35:50.983277 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:50.983247    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-4xpwm\""
Nov 03 19:35:50.985600 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:50.985576    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449cmzc"]
Nov 03 19:35:51.116922 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:51.116662    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2e086d0a-690d-4411-8fca-3f8f3c5f8bb5-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449cmzc\" (UID: \"2e086d0a-690d-4411-8fca-3f8f3c5f8bb5\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449cmzc"
Nov 03 19:35:51.116922 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:51.116733    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wknr\" (UniqueName: \"kubernetes.io/projected/2e086d0a-690d-4411-8fca-3f8f3c5f8bb5-kube-api-access-2wknr\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449cmzc\" (UID: \"2e086d0a-690d-4411-8fca-3f8f3c5f8bb5\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449cmzc"
Nov 03 19:35:51.116922 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:51.116808    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2e086d0a-690d-4411-8fca-3f8f3c5f8bb5-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449cmzc\" (UID: \"2e086d0a-690d-4411-8fca-3f8f3c5f8bb5\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449cmzc"
Nov 03 19:35:51.217451 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:51.217415    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2e086d0a-690d-4411-8fca-3f8f3c5f8bb5-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449cmzc\" (UID: \"2e086d0a-690d-4411-8fca-3f8f3c5f8bb5\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449cmzc"
Nov 03 19:35:51.217646 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:51.217463    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wknr\" (UniqueName: \"kubernetes.io/projected/2e086d0a-690d-4411-8fca-3f8f3c5f8bb5-kube-api-access-2wknr\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449cmzc\" (UID: \"2e086d0a-690d-4411-8fca-3f8f3c5f8bb5\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449cmzc"
Nov 03 19:35:51.217646 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:51.217553    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2e086d0a-690d-4411-8fca-3f8f3c5f8bb5-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449cmzc\" (UID: \"2e086d0a-690d-4411-8fca-3f8f3c5f8bb5\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449cmzc"
Nov 03 19:35:51.218035 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:51.218013    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2e086d0a-690d-4411-8fca-3f8f3c5f8bb5-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449cmzc\" (UID: \"2e086d0a-690d-4411-8fca-3f8f3c5f8bb5\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449cmzc"
Nov 03 19:35:51.218118 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:51.218047    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2e086d0a-690d-4411-8fca-3f8f3c5f8bb5-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449cmzc\" (UID: \"2e086d0a-690d-4411-8fca-3f8f3c5f8bb5\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449cmzc"
Nov 03 19:35:51.229489 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:51.229446    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wknr\" (UniqueName: \"kubernetes.io/projected/2e086d0a-690d-4411-8fca-3f8f3c5f8bb5-kube-api-access-2wknr\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449cmzc\" (UID: \"2e086d0a-690d-4411-8fca-3f8f3c5f8bb5\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449cmzc"
Nov 03 19:35:51.299403 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:51.299364    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449cmzc"
Nov 03 19:35:51.586809 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:51.586734    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449cmzc"]
Nov 03 19:35:51.592408 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:35:51.592373    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e086d0a_690d_4411_8fca_3f8f3c5f8bb5.slice/crio-c27b29d2f223044c4247322418ca00fa55b104f64016d37c0ee732f6911c4013 WatchSource:0}: Error finding container c27b29d2f223044c4247322418ca00fa55b104f64016d37c0ee732f6911c4013: Status 404 returned error can't find the container with id c27b29d2f223044c4247322418ca00fa55b104f64016d37c0ee732f6911c4013
Nov 03 19:35:51.960173 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:51.960087    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449cmzc" event={"ID":"2e086d0a-690d-4411-8fca-3f8f3c5f8bb5","Type":"ContainerStarted","Data":"c27b29d2f223044c4247322418ca00fa55b104f64016d37c0ee732f6911c4013"}
Nov 03 19:35:52.044750 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:52.044703    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ashw75"]
Nov 03 19:35:52.049697 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:52.049649    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ashw75"
Nov 03 19:35:52.057739 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:52.057710    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ashw75"]
Nov 03 19:35:52.059724 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:52.059118    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-64tbf\""
Nov 03 19:35:52.127133 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:52.127093    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwvn4\" (UniqueName: \"kubernetes.io/projected/28d6b5eb-53bd-4b4c-9474-7d5fb948b879-kube-api-access-nwvn4\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ashw75\" (UID: \"28d6b5eb-53bd-4b4c-9474-7d5fb948b879\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ashw75"
Nov 03 19:35:52.127327 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:52.127178    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/28d6b5eb-53bd-4b4c-9474-7d5fb948b879-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ashw75\" (UID: \"28d6b5eb-53bd-4b4c-9474-7d5fb948b879\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ashw75"
Nov 03 19:35:52.127327 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:52.127216    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/28d6b5eb-53bd-4b4c-9474-7d5fb948b879-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ashw75\" (UID: \"28d6b5eb-53bd-4b4c-9474-7d5fb948b879\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ashw75"
Nov 03 19:35:52.145747 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:52.145710    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78jhbr"]
Nov 03 19:35:52.149552 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:52.149526    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78jhbr"
Nov 03 19:35:52.153466 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:52.153441    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-9h24s\""
Nov 03 19:35:52.158364 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:52.158305    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78jhbr"]
Nov 03 19:35:52.228205 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:52.228168    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwvn4\" (UniqueName: \"kubernetes.io/projected/28d6b5eb-53bd-4b4c-9474-7d5fb948b879-kube-api-access-nwvn4\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ashw75\" (UID: \"28d6b5eb-53bd-4b4c-9474-7d5fb948b879\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ashw75"
Nov 03 19:35:52.228369 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:52.228238    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/43a925de-112e-49dc-8fa9-6e37978a907c-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78jhbr\" (UID: \"43a925de-112e-49dc-8fa9-6e37978a907c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78jhbr"
Nov 03 19:35:52.228369 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:52.228279    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/28d6b5eb-53bd-4b4c-9474-7d5fb948b879-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ashw75\" (UID: \"28d6b5eb-53bd-4b4c-9474-7d5fb948b879\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ashw75"
Nov 03 19:35:52.228369 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:52.228318    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/28d6b5eb-53bd-4b4c-9474-7d5fb948b879-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ashw75\" (UID: \"28d6b5eb-53bd-4b4c-9474-7d5fb948b879\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ashw75"
Nov 03 19:35:52.228369 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:52.228347    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgpsk\" (UniqueName: \"kubernetes.io/projected/43a925de-112e-49dc-8fa9-6e37978a907c-kube-api-access-sgpsk\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78jhbr\" (UID: \"43a925de-112e-49dc-8fa9-6e37978a907c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78jhbr"
Nov 03 19:35:52.228505 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:52.228394    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/43a925de-112e-49dc-8fa9-6e37978a907c-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78jhbr\" (UID: \"43a925de-112e-49dc-8fa9-6e37978a907c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78jhbr"
Nov 03 19:35:52.229175 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:52.229145    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/28d6b5eb-53bd-4b4c-9474-7d5fb948b879-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ashw75\" (UID: \"28d6b5eb-53bd-4b4c-9474-7d5fb948b879\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ashw75"
Nov 03 19:35:52.229416 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:52.229398    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/28d6b5eb-53bd-4b4c-9474-7d5fb948b879-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ashw75\" (UID: \"28d6b5eb-53bd-4b4c-9474-7d5fb948b879\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ashw75"
Nov 03 19:35:52.238863 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:52.238801    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwvn4\" (UniqueName: \"kubernetes.io/projected/28d6b5eb-53bd-4b4c-9474-7d5fb948b879-kube-api-access-nwvn4\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ashw75\" (UID: \"28d6b5eb-53bd-4b4c-9474-7d5fb948b879\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ashw75"
Nov 03 19:35:52.241510 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:52.239690    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm2c9c"]
Nov 03 19:35:52.244043 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:52.243995    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm2c9c"
Nov 03 19:35:52.251009 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:52.249326    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-v6vh6\""
Nov 03 19:35:52.251009 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:52.250954    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm2c9c"]
Nov 03 19:35:52.329270 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:52.329225    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/43a925de-112e-49dc-8fa9-6e37978a907c-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78jhbr\" (UID: \"43a925de-112e-49dc-8fa9-6e37978a907c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78jhbr"
Nov 03 19:35:52.329447 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:52.329292    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/db169534-8aca-4878-81b2-176a18d7053d-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm2c9c\" (UID: \"db169534-8aca-4878-81b2-176a18d7053d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm2c9c"
Nov 03 19:35:52.329447 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:52.329331    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cpqg\" (UniqueName: \"kubernetes.io/projected/db169534-8aca-4878-81b2-176a18d7053d-kube-api-access-8cpqg\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm2c9c\" (UID: \"db169534-8aca-4878-81b2-176a18d7053d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm2c9c"
Nov 03 19:35:52.329535 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:52.329454    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/43a925de-112e-49dc-8fa9-6e37978a907c-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78jhbr\" (UID: \"43a925de-112e-49dc-8fa9-6e37978a907c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78jhbr"
Nov 03 19:35:52.329535 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:52.329504    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/db169534-8aca-4878-81b2-176a18d7053d-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm2c9c\" (UID: \"db169534-8aca-4878-81b2-176a18d7053d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm2c9c"
Nov 03 19:35:52.329614 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:52.329556    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgpsk\" (UniqueName: \"kubernetes.io/projected/43a925de-112e-49dc-8fa9-6e37978a907c-kube-api-access-sgpsk\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78jhbr\" (UID: \"43a925de-112e-49dc-8fa9-6e37978a907c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78jhbr"
Nov 03 19:35:52.329703 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:52.329660    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/43a925de-112e-49dc-8fa9-6e37978a907c-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78jhbr\" (UID: \"43a925de-112e-49dc-8fa9-6e37978a907c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78jhbr"
Nov 03 19:35:52.329953 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:52.329932    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/43a925de-112e-49dc-8fa9-6e37978a907c-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78jhbr\" (UID: \"43a925de-112e-49dc-8fa9-6e37978a907c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78jhbr"
Nov 03 19:35:52.338912 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:52.338863    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgpsk\" (UniqueName: \"kubernetes.io/projected/43a925de-112e-49dc-8fa9-6e37978a907c-kube-api-access-sgpsk\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78jhbr\" (UID: \"43a925de-112e-49dc-8fa9-6e37978a907c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78jhbr"
Nov 03 19:35:52.340316 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:52.340287    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6s6jjd"]
Nov 03 19:35:52.344149 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:52.344128    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6s6jjd"
Nov 03 19:35:52.349144 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:52.349120    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-l8jk4\""
Nov 03 19:35:52.352912 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:52.352219    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6s6jjd"]
Nov 03 19:35:52.373317 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:52.373287    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ashw75"
Nov 03 19:35:52.430277 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:52.430233    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24dc9\" (UniqueName: \"kubernetes.io/projected/48e24ffe-d9a9-4f6c-98d7-0f1b2bf80378-kube-api-access-24dc9\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6s6jjd\" (UID: \"48e24ffe-d9a9-4f6c-98d7-0f1b2bf80378\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6s6jjd"
Nov 03 19:35:52.430457 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:52.430309    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/48e24ffe-d9a9-4f6c-98d7-0f1b2bf80378-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6s6jjd\" (UID: \"48e24ffe-d9a9-4f6c-98d7-0f1b2bf80378\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6s6jjd"
Nov 03 19:35:52.430457 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:52.430345    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/48e24ffe-d9a9-4f6c-98d7-0f1b2bf80378-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6s6jjd\" (UID: \"48e24ffe-d9a9-4f6c-98d7-0f1b2bf80378\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6s6jjd"
Nov 03 19:35:52.430457 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:52.430383    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/db169534-8aca-4878-81b2-176a18d7053d-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm2c9c\" (UID: \"db169534-8aca-4878-81b2-176a18d7053d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm2c9c"
Nov 03 19:35:52.430457 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:52.430419    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cpqg\" (UniqueName: \"kubernetes.io/projected/db169534-8aca-4878-81b2-176a18d7053d-kube-api-access-8cpqg\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm2c9c\" (UID: \"db169534-8aca-4878-81b2-176a18d7053d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm2c9c"
Nov 03 19:35:52.430599 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:52.430474    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/db169534-8aca-4878-81b2-176a18d7053d-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm2c9c\" (UID: \"db169534-8aca-4878-81b2-176a18d7053d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm2c9c"
Nov 03 19:35:52.430935 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:52.430914    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/db169534-8aca-4878-81b2-176a18d7053d-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm2c9c\" (UID: \"db169534-8aca-4878-81b2-176a18d7053d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm2c9c"
Nov 03 19:35:52.431162 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:52.431134    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/db169534-8aca-4878-81b2-176a18d7053d-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm2c9c\" (UID: \"db169534-8aca-4878-81b2-176a18d7053d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm2c9c"
Nov 03 19:35:52.444108 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:52.443358    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cpqg\" (UniqueName: \"kubernetes.io/projected/db169534-8aca-4878-81b2-176a18d7053d-kube-api-access-8cpqg\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm2c9c\" (UID: \"db169534-8aca-4878-81b2-176a18d7053d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm2c9c"
Nov 03 19:35:52.479888 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:52.479541    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78jhbr"
Nov 03 19:35:52.531917 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:52.531881    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24dc9\" (UniqueName: \"kubernetes.io/projected/48e24ffe-d9a9-4f6c-98d7-0f1b2bf80378-kube-api-access-24dc9\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6s6jjd\" (UID: \"48e24ffe-d9a9-4f6c-98d7-0f1b2bf80378\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6s6jjd"
Nov 03 19:35:52.532129 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:52.531942    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/48e24ffe-d9a9-4f6c-98d7-0f1b2bf80378-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6s6jjd\" (UID: \"48e24ffe-d9a9-4f6c-98d7-0f1b2bf80378\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6s6jjd"
Nov 03 19:35:52.532129 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:52.531973    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/48e24ffe-d9a9-4f6c-98d7-0f1b2bf80378-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6s6jjd\" (UID: \"48e24ffe-d9a9-4f6c-98d7-0f1b2bf80378\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6s6jjd"
Nov 03 19:35:52.532396 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:52.532376    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/48e24ffe-d9a9-4f6c-98d7-0f1b2bf80378-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6s6jjd\" (UID: \"48e24ffe-d9a9-4f6c-98d7-0f1b2bf80378\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6s6jjd"
Nov 03 19:35:52.533775 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:52.533738    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/48e24ffe-d9a9-4f6c-98d7-0f1b2bf80378-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6s6jjd\" (UID: \"48e24ffe-d9a9-4f6c-98d7-0f1b2bf80378\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6s6jjd"
Nov 03 19:35:52.541014 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:52.540968    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24dc9\" (UniqueName: \"kubernetes.io/projected/48e24ffe-d9a9-4f6c-98d7-0f1b2bf80378-kube-api-access-24dc9\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6s6jjd\" (UID: \"48e24ffe-d9a9-4f6c-98d7-0f1b2bf80378\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6s6jjd"
Nov 03 19:35:52.574693 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:52.572306    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm2c9c"
Nov 03 19:35:52.672470 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:52.672441    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6s6jjd"
Nov 03 19:35:52.673384 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:52.673354    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ashw75"]
Nov 03 19:35:52.673860 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:35:52.673826    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28d6b5eb_53bd_4b4c_9474_7d5fb948b879.slice/crio-19b6d905c3219eaacd5579dfaca023c2547f0cd3b54001891e5761a19236ad35 WatchSource:0}: Error finding container 19b6d905c3219eaacd5579dfaca023c2547f0cd3b54001891e5761a19236ad35: Status 404 returned error can't find the container with id 19b6d905c3219eaacd5579dfaca023c2547f0cd3b54001891e5761a19236ad35
Nov 03 19:35:52.794258 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:52.792998    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78jhbr"]
Nov 03 19:35:52.796247 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:35:52.796206    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43a925de_112e_49dc_8fa9_6e37978a907c.slice/crio-07ad0bf918f1de3d1929f4a99ecd352a1a5a84e0e62529f7c920ce78379a5fb9 WatchSource:0}: Error finding container 07ad0bf918f1de3d1929f4a99ecd352a1a5a84e0e62529f7c920ce78379a5fb9: Status 404 returned error can't find the container with id 07ad0bf918f1de3d1929f4a99ecd352a1a5a84e0e62529f7c920ce78379a5fb9
Nov 03 19:35:52.858880 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:52.858835    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm2c9c"]
Nov 03 19:35:52.863951 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:35:52.863910    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb169534_8aca_4878_81b2_176a18d7053d.slice/crio-921e5a0b1dce71b257efdf08ce1d23cda1f815ce1e7450ab53e868ff8237d7ec WatchSource:0}: Error finding container 921e5a0b1dce71b257efdf08ce1d23cda1f815ce1e7450ab53e868ff8237d7ec: Status 404 returned error can't find the container with id 921e5a0b1dce71b257efdf08ce1d23cda1f815ce1e7450ab53e868ff8237d7ec
Nov 03 19:35:52.967050 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:52.967005    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm2c9c" event={"ID":"db169534-8aca-4878-81b2-176a18d7053d","Type":"ContainerStarted","Data":"921e5a0b1dce71b257efdf08ce1d23cda1f815ce1e7450ab53e868ff8237d7ec"}
Nov 03 19:35:52.968877 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:52.968801    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78jhbr" event={"ID":"43a925de-112e-49dc-8fa9-6e37978a907c","Type":"ContainerStarted","Data":"07ad0bf918f1de3d1929f4a99ecd352a1a5a84e0e62529f7c920ce78379a5fb9"}
Nov 03 19:35:52.968982 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:52.968949    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6s6jjd"]
Nov 03 19:35:52.971520 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:52.971481    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ashw75" event={"ID":"28d6b5eb-53bd-4b4c-9474-7d5fb948b879","Type":"ContainerStarted","Data":"19b6d905c3219eaacd5579dfaca023c2547f0cd3b54001891e5761a19236ad35"}
Nov 03 19:35:52.972478 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:35:52.972449    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48e24ffe_d9a9_4f6c_98d7_0f1b2bf80378.slice/crio-7b0b318ab1d41b1ceadf417847e16a716e15548524b5a2d87178306febd72a56 WatchSource:0}: Error finding container 7b0b318ab1d41b1ceadf417847e16a716e15548524b5a2d87178306febd72a56: Status 404 returned error can't find the container with id 7b0b318ab1d41b1ceadf417847e16a716e15548524b5a2d87178306febd72a56
Nov 03 19:35:53.356523 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:53.356484    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zl5bx"]
Nov 03 19:35:53.379857 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:53.379822    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zl5bx"]
Nov 03 19:35:53.380033 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:53.379947    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zl5bx"
Nov 03 19:35:53.385874 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:53.384955    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-7lc4h\""
Nov 03 19:35:53.385874 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:53.385844    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 03 19:35:53.431556 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:53.430511    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-vr74f"]
Nov 03 19:35:53.436105 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:53.434742    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-vr74f"
Nov 03 19:35:53.443622 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:53.442267    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-7xlsg\""
Nov 03 19:35:53.457517 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:53.457435    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-vr74f"]
Nov 03 19:35:53.546372 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:53.546303    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/788abbb1-9316-46b4-8b47-a75066db292b-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-vr74f\" (UID: \"788abbb1-9316-46b4-8b47-a75066db292b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-vr74f"
Nov 03 19:35:53.546591 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:53.546389    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0e23fd0-6cce-402f-9e00-0f7da02ed53d-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-zl5bx\" (UID: \"d0e23fd0-6cce-402f-9e00-0f7da02ed53d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zl5bx"
Nov 03 19:35:53.546591 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:53.546423    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/788abbb1-9316-46b4-8b47-a75066db292b-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-vr74f\" (UID: \"788abbb1-9316-46b4-8b47-a75066db292b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-vr74f"
Nov 03 19:35:53.546591 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:53.546470    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/788abbb1-9316-46b4-8b47-a75066db292b-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-vr74f\" (UID: \"788abbb1-9316-46b4-8b47-a75066db292b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-vr74f"
Nov 03 19:35:53.546591 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:53.546499    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/788abbb1-9316-46b4-8b47-a75066db292b-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-vr74f\" (UID: \"788abbb1-9316-46b4-8b47-a75066db292b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-vr74f"
Nov 03 19:35:53.546591 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:53.546527    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7wdn\" (UniqueName: \"kubernetes.io/projected/d0e23fd0-6cce-402f-9e00-0f7da02ed53d-kube-api-access-c7wdn\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-zl5bx\" (UID: \"d0e23fd0-6cce-402f-9e00-0f7da02ed53d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zl5bx"
Nov 03 19:35:53.546591 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:53.546583    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/788abbb1-9316-46b4-8b47-a75066db292b-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-vr74f\" (UID: \"788abbb1-9316-46b4-8b47-a75066db292b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-vr74f"
Nov 03 19:35:53.546924 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:53.546613    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/788abbb1-9316-46b4-8b47-a75066db292b-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-vr74f\" (UID: \"788abbb1-9316-46b4-8b47-a75066db292b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-vr74f"
Nov 03 19:35:53.546924 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:53.546653    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/788abbb1-9316-46b4-8b47-a75066db292b-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-vr74f\" (UID: \"788abbb1-9316-46b4-8b47-a75066db292b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-vr74f"
Nov 03 19:35:53.546924 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:53.546706    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0e23fd0-6cce-402f-9e00-0f7da02ed53d-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-zl5bx\" (UID: \"d0e23fd0-6cce-402f-9e00-0f7da02ed53d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zl5bx"
Nov 03 19:35:53.546924 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:53.546741    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0e23fd0-6cce-402f-9e00-0f7da02ed53d-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-zl5bx\" (UID: \"d0e23fd0-6cce-402f-9e00-0f7da02ed53d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zl5bx"
Nov 03 19:35:53.546924 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:53.546771    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0e23fd0-6cce-402f-9e00-0f7da02ed53d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-zl5bx\" (UID: \"d0e23fd0-6cce-402f-9e00-0f7da02ed53d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zl5bx"
Nov 03 19:35:53.546924 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:53.546796    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0e23fd0-6cce-402f-9e00-0f7da02ed53d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-zl5bx\" (UID: \"d0e23fd0-6cce-402f-9e00-0f7da02ed53d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zl5bx"
Nov 03 19:35:53.546924 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:53.546826    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0e23fd0-6cce-402f-9e00-0f7da02ed53d-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-zl5bx\" (UID: \"d0e23fd0-6cce-402f-9e00-0f7da02ed53d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zl5bx"
Nov 03 19:35:53.546924 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:53.546857    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/788abbb1-9316-46b4-8b47-a75066db292b-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-vr74f\" (UID: \"788abbb1-9316-46b4-8b47-a75066db292b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-vr74f"
Nov 03 19:35:53.546924 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:53.546881    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0e23fd0-6cce-402f-9e00-0f7da02ed53d-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-zl5bx\" (UID: \"d0e23fd0-6cce-402f-9e00-0f7da02ed53d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zl5bx"
Nov 03 19:35:53.547420 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:53.546927    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf2qz\" (UniqueName: \"kubernetes.io/projected/788abbb1-9316-46b4-8b47-a75066db292b-kube-api-access-xf2qz\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-vr74f\" (UID: \"788abbb1-9316-46b4-8b47-a75066db292b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-vr74f"
Nov 03 19:35:53.547420 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:53.546971    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0e23fd0-6cce-402f-9e00-0f7da02ed53d-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-zl5bx\" (UID: \"d0e23fd0-6cce-402f-9e00-0f7da02ed53d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zl5bx"
Nov 03 19:35:53.650748 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:53.648059    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0e23fd0-6cce-402f-9e00-0f7da02ed53d-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-zl5bx\" (UID: \"d0e23fd0-6cce-402f-9e00-0f7da02ed53d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zl5bx"
Nov 03 19:35:53.650748 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:53.648124    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf2qz\" (UniqueName: \"kubernetes.io/projected/788abbb1-9316-46b4-8b47-a75066db292b-kube-api-access-xf2qz\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-vr74f\" (UID: \"788abbb1-9316-46b4-8b47-a75066db292b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-vr74f"
Nov 03 19:35:53.650748 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:53.648164    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0e23fd0-6cce-402f-9e00-0f7da02ed53d-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-zl5bx\" (UID: \"d0e23fd0-6cce-402f-9e00-0f7da02ed53d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zl5bx"
Nov 03 19:35:53.650748 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:53.648203    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/788abbb1-9316-46b4-8b47-a75066db292b-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-vr74f\" (UID: \"788abbb1-9316-46b4-8b47-a75066db292b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-vr74f"
Nov 03 19:35:53.650748 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:53.648229    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0e23fd0-6cce-402f-9e00-0f7da02ed53d-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-zl5bx\" (UID: \"d0e23fd0-6cce-402f-9e00-0f7da02ed53d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zl5bx"
Nov 03 19:35:53.650748 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:53.648250    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/788abbb1-9316-46b4-8b47-a75066db292b-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-vr74f\" (UID: \"788abbb1-9316-46b4-8b47-a75066db292b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-vr74f"
Nov 03 19:35:53.650748 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:53.648266    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/788abbb1-9316-46b4-8b47-a75066db292b-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-vr74f\" (UID: \"788abbb1-9316-46b4-8b47-a75066db292b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-vr74f"
Nov 03 19:35:53.650748 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:53.648282    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/788abbb1-9316-46b4-8b47-a75066db292b-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-vr74f\" (UID: \"788abbb1-9316-46b4-8b47-a75066db292b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-vr74f"
Nov 03 19:35:53.650748 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:53.648298    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7wdn\" (UniqueName: \"kubernetes.io/projected/d0e23fd0-6cce-402f-9e00-0f7da02ed53d-kube-api-access-c7wdn\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-zl5bx\" (UID: \"d0e23fd0-6cce-402f-9e00-0f7da02ed53d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zl5bx"
Nov 03 19:35:53.650748 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:53.648326    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/788abbb1-9316-46b4-8b47-a75066db292b-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-vr74f\" (UID: \"788abbb1-9316-46b4-8b47-a75066db292b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-vr74f"
Nov 03 19:35:53.650748 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:53.648346    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/788abbb1-9316-46b4-8b47-a75066db292b-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-vr74f\" (UID: \"788abbb1-9316-46b4-8b47-a75066db292b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-vr74f"
Nov 03 19:35:53.650748 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:53.648369    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/788abbb1-9316-46b4-8b47-a75066db292b-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-vr74f\" (UID: \"788abbb1-9316-46b4-8b47-a75066db292b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-vr74f"
Nov 03 19:35:53.650748 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:53.648385    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0e23fd0-6cce-402f-9e00-0f7da02ed53d-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-zl5bx\" (UID: \"d0e23fd0-6cce-402f-9e00-0f7da02ed53d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zl5bx"
Nov 03 19:35:53.650748 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:53.648408    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0e23fd0-6cce-402f-9e00-0f7da02ed53d-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-zl5bx\" (UID: \"d0e23fd0-6cce-402f-9e00-0f7da02ed53d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zl5bx"
Nov 03 19:35:53.650748 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:53.648432    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0e23fd0-6cce-402f-9e00-0f7da02ed53d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-zl5bx\" (UID: \"d0e23fd0-6cce-402f-9e00-0f7da02ed53d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zl5bx"
Nov 03 19:35:53.650748 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:53.648447    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0e23fd0-6cce-402f-9e00-0f7da02ed53d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-zl5bx\" (UID: \"d0e23fd0-6cce-402f-9e00-0f7da02ed53d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zl5bx"
Nov 03 19:35:53.650748 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:53.648465    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0e23fd0-6cce-402f-9e00-0f7da02ed53d-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-zl5bx\" (UID: \"d0e23fd0-6cce-402f-9e00-0f7da02ed53d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zl5bx"
Nov 03 19:35:53.650748 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:53.648483    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/788abbb1-9316-46b4-8b47-a75066db292b-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-vr74f\" (UID: \"788abbb1-9316-46b4-8b47-a75066db292b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-vr74f"
Nov 03 19:35:53.650748 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:53.648919    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/788abbb1-9316-46b4-8b47-a75066db292b-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-vr74f\" (UID: \"788abbb1-9316-46b4-8b47-a75066db292b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-vr74f"
Nov 03 19:35:53.650748 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:53.649921    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/788abbb1-9316-46b4-8b47-a75066db292b-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-vr74f\" (UID: \"788abbb1-9316-46b4-8b47-a75066db292b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-vr74f"
Nov 03 19:35:53.650748 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:53.650148    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/788abbb1-9316-46b4-8b47-a75066db292b-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-vr74f\" (UID: \"788abbb1-9316-46b4-8b47-a75066db292b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-vr74f"
Nov 03 19:35:53.650748 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:53.650362    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0e23fd0-6cce-402f-9e00-0f7da02ed53d-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-zl5bx\" (UID: \"d0e23fd0-6cce-402f-9e00-0f7da02ed53d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zl5bx"
Nov 03 19:35:53.653549 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:53.652232    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/788abbb1-9316-46b4-8b47-a75066db292b-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-vr74f\" (UID: \"788abbb1-9316-46b4-8b47-a75066db292b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-vr74f"
Nov 03 19:35:53.653549 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:53.652941    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0e23fd0-6cce-402f-9e00-0f7da02ed53d-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-zl5bx\" (UID: \"d0e23fd0-6cce-402f-9e00-0f7da02ed53d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zl5bx"
Nov 03 19:35:53.653549 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:53.653143    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0e23fd0-6cce-402f-9e00-0f7da02ed53d-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-zl5bx\" (UID: \"d0e23fd0-6cce-402f-9e00-0f7da02ed53d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zl5bx"
Nov 03 19:35:53.653549 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:53.653478    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/788abbb1-9316-46b4-8b47-a75066db292b-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-vr74f\" (UID: \"788abbb1-9316-46b4-8b47-a75066db292b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-vr74f"
Nov 03 19:35:53.653927 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:53.653882    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0e23fd0-6cce-402f-9e00-0f7da02ed53d-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-zl5bx\" (UID: \"d0e23fd0-6cce-402f-9e00-0f7da02ed53d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zl5bx"
Nov 03 19:35:53.654250 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:53.654216    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0e23fd0-6cce-402f-9e00-0f7da02ed53d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-zl5bx\" (UID: \"d0e23fd0-6cce-402f-9e00-0f7da02ed53d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zl5bx"
Nov 03 19:35:53.658616 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:53.658571    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0e23fd0-6cce-402f-9e00-0f7da02ed53d-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-zl5bx\" (UID: \"d0e23fd0-6cce-402f-9e00-0f7da02ed53d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zl5bx"
Nov 03 19:35:53.659230 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:53.659206    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/788abbb1-9316-46b4-8b47-a75066db292b-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-vr74f\" (UID: \"788abbb1-9316-46b4-8b47-a75066db292b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-vr74f"
Nov 03 19:35:53.668994 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:53.668541    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0e23fd0-6cce-402f-9e00-0f7da02ed53d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-zl5bx\" (UID: \"d0e23fd0-6cce-402f-9e00-0f7da02ed53d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zl5bx"
Nov 03 19:35:53.668994 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:53.668548    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/788abbb1-9316-46b4-8b47-a75066db292b-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-vr74f\" (UID: \"788abbb1-9316-46b4-8b47-a75066db292b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-vr74f"
Nov 03 19:35:53.691618 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:53.691454    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/788abbb1-9316-46b4-8b47-a75066db292b-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-vr74f\" (UID: \"788abbb1-9316-46b4-8b47-a75066db292b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-vr74f"
Nov 03 19:35:53.691618 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:53.691578    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf2qz\" (UniqueName: \"kubernetes.io/projected/788abbb1-9316-46b4-8b47-a75066db292b-kube-api-access-xf2qz\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-vr74f\" (UID: \"788abbb1-9316-46b4-8b47-a75066db292b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-vr74f"
Nov 03 19:35:53.695384 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:53.695357    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7wdn\" (UniqueName: \"kubernetes.io/projected/d0e23fd0-6cce-402f-9e00-0f7da02ed53d-kube-api-access-c7wdn\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-zl5bx\" (UID: \"d0e23fd0-6cce-402f-9e00-0f7da02ed53d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zl5bx"
Nov 03 19:35:53.702121 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:53.702042    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0e23fd0-6cce-402f-9e00-0f7da02ed53d-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-zl5bx\" (UID: \"d0e23fd0-6cce-402f-9e00-0f7da02ed53d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zl5bx"
Nov 03 19:35:53.761748 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:53.761291    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-vr74f"
Nov 03 19:35:53.983786 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:53.983692    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6s6jjd" event={"ID":"48e24ffe-d9a9-4f6c-98d7-0f1b2bf80378","Type":"ContainerStarted","Data":"7b0b318ab1d41b1ceadf417847e16a716e15548524b5a2d87178306febd72a56"}
Nov 03 19:35:54.000010 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:53.999630    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zl5bx"
Nov 03 19:35:54.076963 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:54.076923    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-vr74f"]
Nov 03 19:35:54.309920 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:54.309692    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zl5bx"]
Nov 03 19:35:54.313805 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:35:54.313775    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0e23fd0_6cce_402f_9e00_0f7da02ed53d.slice/crio-332fae60c82be13a4e5a36152d228397a9f63498a79cc4adda9ede3a11ab9e9e WatchSource:0}: Error finding container 332fae60c82be13a4e5a36152d228397a9f63498a79cc4adda9ede3a11ab9e9e: Status 404 returned error can't find the container with id 332fae60c82be13a4e5a36152d228397a9f63498a79cc4adda9ede3a11ab9e9e
Nov 03 19:35:54.628692 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:54.626487    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c499nz"]
Nov 03 19:35:54.635415 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:54.634643    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c499nz"
Nov 03 19:35:54.640982 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:54.640900    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c499nz"]
Nov 03 19:35:54.761926 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:54.761597    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f96k7\" (UniqueName: \"kubernetes.io/projected/e853cbd6-5e8c-4c73-98ab-d183e941a549-kube-api-access-f96k7\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c499nz\" (UID: \"e853cbd6-5e8c-4c73-98ab-d183e941a549\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c499nz"
Nov 03 19:35:54.761926 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:54.761687    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e853cbd6-5e8c-4c73-98ab-d183e941a549-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c499nz\" (UID: \"e853cbd6-5e8c-4c73-98ab-d183e941a549\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c499nz"
Nov 03 19:35:54.761926 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:54.761765    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e853cbd6-5e8c-4c73-98ab-d183e941a549-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c499nz\" (UID: \"e853cbd6-5e8c-4c73-98ab-d183e941a549\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c499nz"
Nov 03 19:35:54.867567 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:54.864725    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e853cbd6-5e8c-4c73-98ab-d183e941a549-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c499nz\" (UID: \"e853cbd6-5e8c-4c73-98ab-d183e941a549\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c499nz"
Nov 03 19:35:54.867567 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:54.864737    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e853cbd6-5e8c-4c73-98ab-d183e941a549-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c499nz\" (UID: \"e853cbd6-5e8c-4c73-98ab-d183e941a549\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c499nz"
Nov 03 19:35:54.867567 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:54.864837    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f96k7\" (UniqueName: \"kubernetes.io/projected/e853cbd6-5e8c-4c73-98ab-d183e941a549-kube-api-access-f96k7\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c499nz\" (UID: \"e853cbd6-5e8c-4c73-98ab-d183e941a549\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c499nz"
Nov 03 19:35:54.867567 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:54.864924    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e853cbd6-5e8c-4c73-98ab-d183e941a549-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c499nz\" (UID: \"e853cbd6-5e8c-4c73-98ab-d183e941a549\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c499nz"
Nov 03 19:35:54.867567 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:54.865256    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e853cbd6-5e8c-4c73-98ab-d183e941a549-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c499nz\" (UID: \"e853cbd6-5e8c-4c73-98ab-d183e941a549\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c499nz"
Nov 03 19:35:54.867567 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:54.865692    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lg74p"]
Nov 03 19:35:54.870122 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:54.870099    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lg74p"
Nov 03 19:35:54.879471 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:54.879037    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lg74p"]
Nov 03 19:35:54.885220 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:54.885191    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f96k7\" (UniqueName: \"kubernetes.io/projected/e853cbd6-5e8c-4c73-98ab-d183e941a549-kube-api-access-f96k7\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c499nz\" (UID: \"e853cbd6-5e8c-4c73-98ab-d183e941a549\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c499nz"
Nov 03 19:35:54.962046 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:54.961768    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c499nz"
Nov 03 19:35:54.965875 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:54.965602    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3760839d-09c5-43db-a771-3b196a6d1abb-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lg74p\" (UID: \"3760839d-09c5-43db-a771-3b196a6d1abb\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lg74p"
Nov 03 19:35:54.965875 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:54.965636    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8dm5\" (UniqueName: \"kubernetes.io/projected/3760839d-09c5-43db-a771-3b196a6d1abb-kube-api-access-t8dm5\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lg74p\" (UID: \"3760839d-09c5-43db-a771-3b196a6d1abb\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lg74p"
Nov 03 19:35:54.965875 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:54.965800    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3760839d-09c5-43db-a771-3b196a6d1abb-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lg74p\" (UID: \"3760839d-09c5-43db-a771-3b196a6d1abb\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lg74p"
Nov 03 19:35:54.994570 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:54.994497    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-vr74f" event={"ID":"788abbb1-9316-46b4-8b47-a75066db292b","Type":"ContainerStarted","Data":"35b00bbc54ac254da1c81822da3f25efb31a8b30c8e75c133edd0c350b165f62"}
Nov 03 19:35:54.996274 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:54.996208    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zl5bx" event={"ID":"d0e23fd0-6cce-402f-9e00-0f7da02ed53d","Type":"ContainerStarted","Data":"332fae60c82be13a4e5a36152d228397a9f63498a79cc4adda9ede3a11ab9e9e"}
Nov 03 19:35:55.068742 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:55.068449    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3760839d-09c5-43db-a771-3b196a6d1abb-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lg74p\" (UID: \"3760839d-09c5-43db-a771-3b196a6d1abb\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lg74p"
Nov 03 19:35:55.068742 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:55.068499    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8dm5\" (UniqueName: \"kubernetes.io/projected/3760839d-09c5-43db-a771-3b196a6d1abb-kube-api-access-t8dm5\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lg74p\" (UID: \"3760839d-09c5-43db-a771-3b196a6d1abb\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lg74p"
Nov 03 19:35:55.068742 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:55.068570    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3760839d-09c5-43db-a771-3b196a6d1abb-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lg74p\" (UID: \"3760839d-09c5-43db-a771-3b196a6d1abb\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lg74p"
Nov 03 19:35:55.086281 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:55.076162    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3760839d-09c5-43db-a771-3b196a6d1abb-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lg74p\" (UID: \"3760839d-09c5-43db-a771-3b196a6d1abb\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lg74p"
Nov 03 19:35:55.086281 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:55.076872    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3760839d-09c5-43db-a771-3b196a6d1abb-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lg74p\" (UID: \"3760839d-09c5-43db-a771-3b196a6d1abb\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lg74p"
Nov 03 19:35:55.099514 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:55.099459    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8dm5\" (UniqueName: \"kubernetes.io/projected/3760839d-09c5-43db-a771-3b196a6d1abb-kube-api-access-t8dm5\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lg74p\" (UID: \"3760839d-09c5-43db-a771-3b196a6d1abb\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lg74p"
Nov 03 19:35:55.204045 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:55.203761    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lg74p"
Nov 03 19:35:55.269692 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:55.269553    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c499nz"]
Nov 03 19:35:55.548021 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:55.547759    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lg74p"]
Nov 03 19:35:56.025817 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:56.025726    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lg74p" event={"ID":"3760839d-09c5-43db-a771-3b196a6d1abb","Type":"ContainerStarted","Data":"4bc6895518cf7adc86f9696ad9f48006291a39a5ff157b696deeafdd622a9277"}
Nov 03 19:35:56.038216 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:56.038132    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c499nz" event={"ID":"e853cbd6-5e8c-4c73-98ab-d183e941a549","Type":"ContainerStarted","Data":"73c9655701210aba71ae634ed1de5eb48aba3651ce64827e73879c6c430cba3d"}
Nov 03 19:35:57.056871 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:57.056834    4921 generic.go:356] "Generic (PLEG): container finished" podID="2ba24656-710d-4547-9b5c-293b76fabd06" containerID="6e7596a89bc2fb32830cb4ce35d8527e0d0f4821c9b8954123901a89bc083543" exitCode=0
Nov 03 19:35:57.057351 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:57.056886    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vpk8d" event={"ID":"2ba24656-710d-4547-9b5c-293b76fabd06","Type":"ContainerDied","Data":"6e7596a89bc2fb32830cb4ce35d8527e0d0f4821c9b8954123901a89bc083543"}
Nov 03 19:35:58.094114 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:58.093961    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vpk8d" event={"ID":"2ba24656-710d-4547-9b5c-293b76fabd06","Type":"ContainerStarted","Data":"292a70c00a366a879c83323000eadee3d2249b212819235e5a8e8dd81ee28494"}
Nov 03 19:35:59.107303 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:59.107262    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj255d" event={"ID":"01cf4a61-867e-41d3-b0b7-2be0d700458b","Type":"ContainerStarted","Data":"816947448acf0f2f86e1358a41338f35ccb56621a54732682d55219455d2aada"}
Nov 03 19:35:59.115387 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:59.115194    4921 generic.go:356] "Generic (PLEG): container finished" podID="2ba24656-710d-4547-9b5c-293b76fabd06" containerID="292a70c00a366a879c83323000eadee3d2249b212819235e5a8e8dd81ee28494" exitCode=0
Nov 03 19:35:59.115387 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:35:59.115283    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vpk8d" event={"ID":"2ba24656-710d-4547-9b5c-293b76fabd06","Type":"ContainerDied","Data":"292a70c00a366a879c83323000eadee3d2249b212819235e5a8e8dd81ee28494"}
Nov 03 19:36:00.134359 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:00.134314    4921 generic.go:356] "Generic (PLEG): container finished" podID="01cf4a61-867e-41d3-b0b7-2be0d700458b" containerID="816947448acf0f2f86e1358a41338f35ccb56621a54732682d55219455d2aada" exitCode=0
Nov 03 19:36:00.134842 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:00.134469    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj255d" event={"ID":"01cf4a61-867e-41d3-b0b7-2be0d700458b","Type":"ContainerDied","Data":"816947448acf0f2f86e1358a41338f35ccb56621a54732682d55219455d2aada"}
Nov 03 19:36:02.998979 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:36:02.998800    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 19:36:03.187552 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:03.187514    4921 generic.go:356] "Generic (PLEG): container finished" podID="2e086d0a-690d-4411-8fca-3f8f3c5f8bb5" containerID="479808b49458033c1a355fd029547fb70c05b4ec3273bb17c38253bfd5285493" exitCode=0
Nov 03 19:36:03.187833 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:03.187687    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449cmzc" event={"ID":"2e086d0a-690d-4411-8fca-3f8f3c5f8bb5","Type":"ContainerDied","Data":"479808b49458033c1a355fd029547fb70c05b4ec3273bb17c38253bfd5285493"}
Nov 03 19:36:03.929746 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:03.929707    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:36:03.929877 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:03.929782    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:36:03.929877 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:03.929815    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:36:04.178046 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:04.178003    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:36:04.178349 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:04.178098    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:36:04.178349 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:04.178142    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:36:04.217142 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:04.216225    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-vr74f" event={"ID":"788abbb1-9316-46b4-8b47-a75066db292b","Type":"ContainerStarted","Data":"0b0f7b81dfb3098bc412ecf01150977ec47c8aa5b3c38b5afed9bc0b679a7ea8"}
Nov 03 19:36:04.229983 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:04.229934    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ashw75" event={"ID":"28d6b5eb-53bd-4b4c-9474-7d5fb948b879","Type":"ContainerStarted","Data":"f001c447cc36891f7cbf1c819d0f0bc1165ba15dd3ffb1f540cd1e9f699d6f01"}
Nov 03 19:36:04.238733 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:04.237095    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-vr74f" podStartSLOduration=1.3956748700000001 podStartE2EDuration="11.237077362s" totalImagesPullingTime="9.841402492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:35:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:36:04.236053453 +0000 UTC m=+277.480018931" watchObservedRunningTime="2026-11-03 19:36:04.237077362 +0000 UTC m=+277.481042828"
Nov 03 19:36:04.762202 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:04.762166    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-vr74f"
Nov 03 19:36:05.241883 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:05.241843    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zl5bx" event={"ID":"d0e23fd0-6cce-402f-9e00-0f7da02ed53d","Type":"ContainerStarted","Data":"bf56445ec8fad51d29e43cc5de28a72b3b7ee5ad6e6941ff946ecd62b8f32eaa"}
Nov 03 19:36:05.247934 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:05.247054    4921 generic.go:356] "Generic (PLEG): container finished" podID="48e24ffe-d9a9-4f6c-98d7-0f1b2bf80378" containerID="2050111de608c8acd651b8cf14457d29ac71e562ff35e1399617ff474f5b64b5" exitCode=0
Nov 03 19:36:05.247934 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:05.247153    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6s6jjd" event={"ID":"48e24ffe-d9a9-4f6c-98d7-0f1b2bf80378","Type":"ContainerDied","Data":"2050111de608c8acd651b8cf14457d29ac71e562ff35e1399617ff474f5b64b5"}
Nov 03 19:36:05.251774 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:05.251001    4921 generic.go:356] "Generic (PLEG): container finished" podID="db169534-8aca-4878-81b2-176a18d7053d" containerID="e9cc0d0f8e2f67f5b5198d09b92e3fa64cc26384c4f762f82fb9e5482b3b010c" exitCode=0
Nov 03 19:36:05.251774 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:05.251067    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm2c9c" event={"ID":"db169534-8aca-4878-81b2-176a18d7053d","Type":"ContainerDied","Data":"e9cc0d0f8e2f67f5b5198d09b92e3fa64cc26384c4f762f82fb9e5482b3b010c"}
Nov 03 19:36:05.255538 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:05.255515    4921 generic.go:356] "Generic (PLEG): container finished" podID="43a925de-112e-49dc-8fa9-6e37978a907c" containerID="4928dbae004f456d93bc1bbf2ab9d307ca9e1c66e55e2aeb5a8a0618b1ebcd6b" exitCode=0
Nov 03 19:36:05.255746 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:05.255727    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78jhbr" event={"ID":"43a925de-112e-49dc-8fa9-6e37978a907c","Type":"ContainerDied","Data":"4928dbae004f456d93bc1bbf2ab9d307ca9e1c66e55e2aeb5a8a0618b1ebcd6b"}
Nov 03 19:36:05.257949 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:05.257923    4921 generic.go:356] "Generic (PLEG): container finished" podID="28d6b5eb-53bd-4b4c-9474-7d5fb948b879" containerID="f001c447cc36891f7cbf1c819d0f0bc1165ba15dd3ffb1f540cd1e9f699d6f01" exitCode=0
Nov 03 19:36:05.258090 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:05.258065    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ashw75" event={"ID":"28d6b5eb-53bd-4b4c-9474-7d5fb948b879","Type":"ContainerDied","Data":"f001c447cc36891f7cbf1c819d0f0bc1165ba15dd3ffb1f540cd1e9f699d6f01"}
Nov 03 19:36:05.263364 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:05.262380    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zl5bx" podStartSLOduration=2.405506366 podStartE2EDuration="12.262361386s" totalImagesPullingTime="9.85685502s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:35:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:36:05.260867942 +0000 UTC m=+278.504833419" watchObservedRunningTime="2026-11-03 19:36:05.262361386 +0000 UTC m=+278.506326852"
Nov 03 19:36:05.767743 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:05.767659    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-vr74f"
Nov 03 19:36:06.000424 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:06.000294    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zl5bx"
Nov 03 19:36:06.006369 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:06.006193    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zl5bx"
Nov 03 19:36:06.264841 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:06.264805    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj255d" event={"ID":"01cf4a61-867e-41d3-b0b7-2be0d700458b","Type":"ContainerStarted","Data":"ebd480cd314cc9967e00b993d931c1e6f1997ea8364647bf62cea8eee0564b89"}
Nov 03 19:36:06.265948 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:06.265871    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-vr74f"
Nov 03 19:36:06.266079 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:06.265984    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zl5bx"
Nov 03 19:36:06.266453 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:06.266429    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-vr74f"
Nov 03 19:36:06.266545 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:06.266525    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zl5bx"
Nov 03 19:36:07.272285 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:07.272257    4921 generic.go:356] "Generic (PLEG): container finished" podID="01cf4a61-867e-41d3-b0b7-2be0d700458b" containerID="ebd480cd314cc9967e00b993d931c1e6f1997ea8364647bf62cea8eee0564b89" exitCode=0
Nov 03 19:36:07.272645 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:07.272349    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj255d" event={"ID":"01cf4a61-867e-41d3-b0b7-2be0d700458b","Type":"ContainerDied","Data":"ebd480cd314cc9967e00b993d931c1e6f1997ea8364647bf62cea8eee0564b89"}
Nov 03 19:36:07.274394 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:07.274358    4921 generic.go:356] "Generic (PLEG): container finished" podID="3760839d-09c5-43db-a771-3b196a6d1abb" containerID="a59a57fece0f3acbb3a1b15a3453dc63c3bf02f2d631a2930f246f2dc57b45f7" exitCode=0
Nov 03 19:36:07.274462 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:07.274391    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lg74p" event={"ID":"3760839d-09c5-43db-a771-3b196a6d1abb","Type":"ContainerDied","Data":"a59a57fece0f3acbb3a1b15a3453dc63c3bf02f2d631a2930f246f2dc57b45f7"}
Nov 03 19:36:07.276812 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:07.276769    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449cmzc" event={"ID":"2e086d0a-690d-4411-8fca-3f8f3c5f8bb5","Type":"ContainerStarted","Data":"3f40a06a3dae35869bfd57595e372aaaee7dc2a66fab71961acb55c0c6168cda"}
Nov 03 19:36:07.278385 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:07.278360    4921 generic.go:356] "Generic (PLEG): container finished" podID="e853cbd6-5e8c-4c73-98ab-d183e941a549" containerID="7938d4b3082f04295a1561e03b5228e813d9b810fc66cd775bbf0a1106aa562b" exitCode=0
Nov 03 19:36:07.278466 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:07.278446    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c499nz" event={"ID":"e853cbd6-5e8c-4c73-98ab-d183e941a549","Type":"ContainerDied","Data":"7938d4b3082f04295a1561e03b5228e813d9b810fc66cd775bbf0a1106aa562b"}
Nov 03 19:36:07.280452 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:07.280431    4921 generic.go:356] "Generic (PLEG): container finished" podID="48e24ffe-d9a9-4f6c-98d7-0f1b2bf80378" containerID="a6acdea1f1ee7d1f1ef2373ae9f9912de79f4bf56da7d29192357a39875a4500" exitCode=0
Nov 03 19:36:07.280547 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:07.280518    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6s6jjd" event={"ID":"48e24ffe-d9a9-4f6c-98d7-0f1b2bf80378","Type":"ContainerDied","Data":"a6acdea1f1ee7d1f1ef2373ae9f9912de79f4bf56da7d29192357a39875a4500"}
Nov 03 19:36:07.282738 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:07.282653    4921 generic.go:356] "Generic (PLEG): container finished" podID="db169534-8aca-4878-81b2-176a18d7053d" containerID="13eec075d34f6d1542775ae871183e2e1dd3a5f8fd6d6e2193e7ff35705bc138" exitCode=0
Nov 03 19:36:07.282829 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:07.282807    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm2c9c" event={"ID":"db169534-8aca-4878-81b2-176a18d7053d","Type":"ContainerDied","Data":"13eec075d34f6d1542775ae871183e2e1dd3a5f8fd6d6e2193e7ff35705bc138"}
Nov 03 19:36:07.286112 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:07.286072    4921 generic.go:356] "Generic (PLEG): container finished" podID="43a925de-112e-49dc-8fa9-6e37978a907c" containerID="be99510886ffcf55a917e4a436abe8ebc10ae021494a89652797546c30e05fbd" exitCode=0
Nov 03 19:36:07.286363 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:07.286200    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78jhbr" event={"ID":"43a925de-112e-49dc-8fa9-6e37978a907c","Type":"ContainerDied","Data":"be99510886ffcf55a917e4a436abe8ebc10ae021494a89652797546c30e05fbd"}
Nov 03 19:36:08.296565 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:08.295435    4921 generic.go:356] "Generic (PLEG): container finished" podID="3760839d-09c5-43db-a771-3b196a6d1abb" containerID="00857654967d287813dd746e49b82da59c9fac7cf9ddb3fbe798ef4aa6b8438f" exitCode=0
Nov 03 19:36:08.296565 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:08.295553    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lg74p" event={"ID":"3760839d-09c5-43db-a771-3b196a6d1abb","Type":"ContainerDied","Data":"00857654967d287813dd746e49b82da59c9fac7cf9ddb3fbe798ef4aa6b8438f"}
Nov 03 19:36:08.308939 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:08.308641    4921 generic.go:356] "Generic (PLEG): container finished" podID="2e086d0a-690d-4411-8fca-3f8f3c5f8bb5" containerID="3f40a06a3dae35869bfd57595e372aaaee7dc2a66fab71961acb55c0c6168cda" exitCode=0
Nov 03 19:36:08.308939 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:08.308733    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449cmzc" event={"ID":"2e086d0a-690d-4411-8fca-3f8f3c5f8bb5","Type":"ContainerDied","Data":"3f40a06a3dae35869bfd57595e372aaaee7dc2a66fab71961acb55c0c6168cda"}
Nov 03 19:36:08.315092 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:08.315062    4921 generic.go:356] "Generic (PLEG): container finished" podID="e853cbd6-5e8c-4c73-98ab-d183e941a549" containerID="fdd8de5f58abaa03430ec54b386a977fbc52c95010b568d96a9b6db315abe2a3" exitCode=0
Nov 03 19:36:08.316069 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:08.316048    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c499nz" event={"ID":"e853cbd6-5e8c-4c73-98ab-d183e941a549","Type":"ContainerDied","Data":"fdd8de5f58abaa03430ec54b386a977fbc52c95010b568d96a9b6db315abe2a3"}
Nov 03 19:36:14.401731 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:14.400516    4921 generic.go:356] "Generic (PLEG): container finished" podID="28d6b5eb-53bd-4b4c-9474-7d5fb948b879" containerID="bc67f36405e93c8652edaab3561301885a92e6ae16ea30185e99b06d04f30628" exitCode=0
Nov 03 19:36:14.401731 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:14.400641    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ashw75" event={"ID":"28d6b5eb-53bd-4b4c-9474-7d5fb948b879","Type":"ContainerDied","Data":"bc67f36405e93c8652edaab3561301885a92e6ae16ea30185e99b06d04f30628"}
Nov 03 19:36:16.426194 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:16.426153    4921 generic.go:356] "Generic (PLEG): container finished" podID="2ba24656-710d-4547-9b5c-293b76fabd06" containerID="ced7cddfc9aef8946d4d57b78a1ad177ca2b85e3ca4b319e81c3d2c7988b9941" exitCode=0
Nov 03 19:36:16.426867 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:16.426338    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vpk8d" event={"ID":"2ba24656-710d-4547-9b5c-293b76fabd06","Type":"ContainerDied","Data":"ced7cddfc9aef8946d4d57b78a1ad177ca2b85e3ca4b319e81c3d2c7988b9941"}
Nov 03 19:36:17.440182 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:17.439220    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj255d" event={"ID":"01cf4a61-867e-41d3-b0b7-2be0d700458b","Type":"ContainerStarted","Data":"25c74a88207411a3d4dbb652f0b6a93b1377e21b66a4c6e15c4467ccafb15cb6"}
Nov 03 19:36:17.445218 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:17.445191    4921 generic.go:356] "Generic (PLEG): container finished" podID="db169534-8aca-4878-81b2-176a18d7053d" containerID="78d0e8e12e50d5a4c01eef1d3d0f63c82b0ef2c9fec3d8d0fbc091487de259ce" exitCode=0
Nov 03 19:36:17.445564 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:17.445410    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm2c9c" event={"ID":"db169534-8aca-4878-81b2-176a18d7053d","Type":"ContainerDied","Data":"78d0e8e12e50d5a4c01eef1d3d0f63c82b0ef2c9fec3d8d0fbc091487de259ce"}
Nov 03 19:36:17.451979 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:17.451404    4921 generic.go:356] "Generic (PLEG): container finished" podID="43a925de-112e-49dc-8fa9-6e37978a907c" containerID="3ceed6b7a7f8be435ea6cb20667b1a9a705e584b20aa74d9d124fc50a3bba57f" exitCode=0
Nov 03 19:36:17.451979 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:17.451897    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78jhbr" event={"ID":"43a925de-112e-49dc-8fa9-6e37978a907c","Type":"ContainerDied","Data":"3ceed6b7a7f8be435ea6cb20667b1a9a705e584b20aa74d9d124fc50a3bba57f"}
Nov 03 19:36:17.485484 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:17.485135    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj255d" podStartSLOduration=4.157502392 podStartE2EDuration="31.485114332s" totalImagesPullingTime="27.32761194s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:35:46 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:36:17.459211164 +0000 UTC m=+290.703176630" watchObservedRunningTime="2026-11-03 19:36:17.485114332 +0000 UTC m=+290.729079798"
Nov 03 19:36:17.629706 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:17.629571    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vpk8d"
Nov 03 19:36:17.719142 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:17.719106    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ba24656-710d-4547-9b5c-293b76fabd06-kube-api-access-7xvpq\" (UniqueName: \"kubernetes.io/projected/2ba24656-710d-4547-9b5c-293b76fabd06-kube-api-access-7xvpq\") pod \"2ba24656-710d-4547-9b5c-293b76fabd06\" (UID: \"2ba24656-710d-4547-9b5c-293b76fabd06\") "
Nov 03 19:36:17.719391 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:17.719218    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ba24656-710d-4547-9b5c-293b76fabd06-util\" (UniqueName: \"kubernetes.io/empty-dir/2ba24656-710d-4547-9b5c-293b76fabd06-util\") pod \"2ba24656-710d-4547-9b5c-293b76fabd06\" (UID: \"2ba24656-710d-4547-9b5c-293b76fabd06\") "
Nov 03 19:36:17.719391 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:17.719278    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ba24656-710d-4547-9b5c-293b76fabd06-bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ba24656-710d-4547-9b5c-293b76fabd06-bundle\") pod \"2ba24656-710d-4547-9b5c-293b76fabd06\" (UID: \"2ba24656-710d-4547-9b5c-293b76fabd06\") "
Nov 03 19:36:17.722730 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:17.720288    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ba24656-710d-4547-9b5c-293b76fabd06-bundle" pod "2ba24656-710d-4547-9b5c-293b76fabd06" (UID: "2ba24656-710d-4547-9b5c-293b76fabd06"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:36:17.722730 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:17.722436    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ba24656-710d-4547-9b5c-293b76fabd06-kube-api-access-7xvpq" pod "2ba24656-710d-4547-9b5c-293b76fabd06" (UID: "2ba24656-710d-4547-9b5c-293b76fabd06"). InnerVolumeSpecName "kube-api-access-7xvpq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:36:17.726089 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:17.726058    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ba24656-710d-4547-9b5c-293b76fabd06-util" pod "2ba24656-710d-4547-9b5c-293b76fabd06" (UID: "2ba24656-710d-4547-9b5c-293b76fabd06"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:36:17.820409 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:17.820382    4921 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ba24656-710d-4547-9b5c-293b76fabd06-bundle\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:36:17.820567 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:17.820552    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xvpq\" (UniqueName: \"kubernetes.io/projected/2ba24656-710d-4547-9b5c-293b76fabd06-kube-api-access-7xvpq\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:36:17.820691 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:17.820659    4921 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2ba24656-710d-4547-9b5c-293b76fabd06-util\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:36:18.459623 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:18.459546    4921 generic.go:356] "Generic (PLEG): container finished" podID="01cf4a61-867e-41d3-b0b7-2be0d700458b" containerID="25c74a88207411a3d4dbb652f0b6a93b1377e21b66a4c6e15c4467ccafb15cb6" exitCode=0
Nov 03 19:36:18.460092 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:18.459628    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj255d" event={"ID":"01cf4a61-867e-41d3-b0b7-2be0d700458b","Type":"ContainerDied","Data":"25c74a88207411a3d4dbb652f0b6a93b1377e21b66a4c6e15c4467ccafb15cb6"}
Nov 03 19:36:18.462251 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:18.462222    4921 generic.go:356] "Generic (PLEG): container finished" podID="3760839d-09c5-43db-a771-3b196a6d1abb" containerID="5e27f88dd091d75ad861a4927e01f5a9f918075d2b4d2e29f1681612bda896b4" exitCode=0
Nov 03 19:36:18.462374 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:18.462354    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lg74p" event={"ID":"3760839d-09c5-43db-a771-3b196a6d1abb","Type":"ContainerDied","Data":"5e27f88dd091d75ad861a4927e01f5a9f918075d2b4d2e29f1681612bda896b4"}
Nov 03 19:36:18.465655 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:18.465632    4921 generic.go:356] "Generic (PLEG): container finished" podID="2e086d0a-690d-4411-8fca-3f8f3c5f8bb5" containerID="3022bbe0555eea807e1f71f81a0a79eaf526e432016c2cc6799f5e93e4ea989c" exitCode=0
Nov 03 19:36:18.465781 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:18.465697    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449cmzc" event={"ID":"2e086d0a-690d-4411-8fca-3f8f3c5f8bb5","Type":"ContainerDied","Data":"3022bbe0555eea807e1f71f81a0a79eaf526e432016c2cc6799f5e93e4ea989c"}
Nov 03 19:36:18.468198 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:18.468172    4921 generic.go:356] "Generic (PLEG): container finished" podID="e853cbd6-5e8c-4c73-98ab-d183e941a549" containerID="58dba7d3d1170be5824386e7cf594983d879e108603166cc4763af1854835487" exitCode=0
Nov 03 19:36:18.468295 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:18.468210    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c499nz" event={"ID":"e853cbd6-5e8c-4c73-98ab-d183e941a549","Type":"ContainerDied","Data":"58dba7d3d1170be5824386e7cf594983d879e108603166cc4763af1854835487"}
Nov 03 19:36:18.470252 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:18.470233    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vpk8d" event={"ID":"2ba24656-710d-4547-9b5c-293b76fabd06","Type":"ContainerDied","Data":"a36db7ba4464961285483896b5ced0485d3b1d987a83417bee45dd02b02def79"}
Nov 03 19:36:18.470330 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:18.470271    4921 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a36db7ba4464961285483896b5ced0485d3b1d987a83417bee45dd02b02def79"
Nov 03 19:36:18.470390 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:18.470361    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vpk8d"
Nov 03 19:36:18.475470 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:18.475442    4921 generic.go:356] "Generic (PLEG): container finished" podID="48e24ffe-d9a9-4f6c-98d7-0f1b2bf80378" containerID="4c1562f9a3844c9b1e486f01f4961ad7694e45552b5b994b35ed2a00f3eba58d" exitCode=0
Nov 03 19:36:18.475929 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:18.475907    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6s6jjd" event={"ID":"48e24ffe-d9a9-4f6c-98d7-0f1b2bf80378","Type":"ContainerDied","Data":"4c1562f9a3844c9b1e486f01f4961ad7694e45552b5b994b35ed2a00f3eba58d"}
Nov 03 19:36:18.626688 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:18.626645    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78jhbr"
Nov 03 19:36:18.660624 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:18.660594    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm2c9c"
Nov 03 19:36:18.732188 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:18.732152    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43a925de-112e-49dc-8fa9-6e37978a907c-kube-api-access-sgpsk\" (UniqueName: \"kubernetes.io/projected/43a925de-112e-49dc-8fa9-6e37978a907c-kube-api-access-sgpsk\") pod \"43a925de-112e-49dc-8fa9-6e37978a907c\" (UID: \"43a925de-112e-49dc-8fa9-6e37978a907c\") "
Nov 03 19:36:18.732425 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:18.732203    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db169534-8aca-4878-81b2-176a18d7053d-util\" (UniqueName: \"kubernetes.io/empty-dir/db169534-8aca-4878-81b2-176a18d7053d-util\") pod \"db169534-8aca-4878-81b2-176a18d7053d\" (UID: \"db169534-8aca-4878-81b2-176a18d7053d\") "
Nov 03 19:36:18.732425 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:18.732291    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db169534-8aca-4878-81b2-176a18d7053d-bundle\" (UniqueName: \"kubernetes.io/empty-dir/db169534-8aca-4878-81b2-176a18d7053d-bundle\") pod \"db169534-8aca-4878-81b2-176a18d7053d\" (UID: \"db169534-8aca-4878-81b2-176a18d7053d\") "
Nov 03 19:36:18.732425 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:18.732334    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43a925de-112e-49dc-8fa9-6e37978a907c-util\" (UniqueName: \"kubernetes.io/empty-dir/43a925de-112e-49dc-8fa9-6e37978a907c-util\") pod \"43a925de-112e-49dc-8fa9-6e37978a907c\" (UID: \"43a925de-112e-49dc-8fa9-6e37978a907c\") "
Nov 03 19:36:18.732425 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:18.732361    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43a925de-112e-49dc-8fa9-6e37978a907c-bundle\" (UniqueName: \"kubernetes.io/empty-dir/43a925de-112e-49dc-8fa9-6e37978a907c-bundle\") pod \"43a925de-112e-49dc-8fa9-6e37978a907c\" (UID: \"43a925de-112e-49dc-8fa9-6e37978a907c\") "
Nov 03 19:36:18.732425 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:18.732418    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db169534-8aca-4878-81b2-176a18d7053d-kube-api-access-8cpqg\" (UniqueName: \"kubernetes.io/projected/db169534-8aca-4878-81b2-176a18d7053d-kube-api-access-8cpqg\") pod \"db169534-8aca-4878-81b2-176a18d7053d\" (UID: \"db169534-8aca-4878-81b2-176a18d7053d\") "
Nov 03 19:36:18.733043 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:18.733010    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db169534-8aca-4878-81b2-176a18d7053d-bundle" pod "db169534-8aca-4878-81b2-176a18d7053d" (UID: "db169534-8aca-4878-81b2-176a18d7053d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:36:18.734447 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:18.734418    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43a925de-112e-49dc-8fa9-6e37978a907c-bundle" pod "43a925de-112e-49dc-8fa9-6e37978a907c" (UID: "43a925de-112e-49dc-8fa9-6e37978a907c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:36:18.734659 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:18.734633    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db169534-8aca-4878-81b2-176a18d7053d-kube-api-access-8cpqg" pod "db169534-8aca-4878-81b2-176a18d7053d" (UID: "db169534-8aca-4878-81b2-176a18d7053d"). InnerVolumeSpecName "kube-api-access-8cpqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:36:18.735006 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:18.734989    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43a925de-112e-49dc-8fa9-6e37978a907c-kube-api-access-sgpsk" pod "43a925de-112e-49dc-8fa9-6e37978a907c" (UID: "43a925de-112e-49dc-8fa9-6e37978a907c"). InnerVolumeSpecName "kube-api-access-sgpsk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:36:18.738988 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:18.738964    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43a925de-112e-49dc-8fa9-6e37978a907c-util" pod "43a925de-112e-49dc-8fa9-6e37978a907c" (UID: "43a925de-112e-49dc-8fa9-6e37978a907c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:36:18.739085 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:18.739066    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db169534-8aca-4878-81b2-176a18d7053d-util" pod "db169534-8aca-4878-81b2-176a18d7053d" (UID: "db169534-8aca-4878-81b2-176a18d7053d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:36:18.833880 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:18.833825    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgpsk\" (UniqueName: \"kubernetes.io/projected/43a925de-112e-49dc-8fa9-6e37978a907c-kube-api-access-sgpsk\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:36:18.833880 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:18.833862    4921 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/db169534-8aca-4878-81b2-176a18d7053d-util\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:36:18.833880 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:18.833876    4921 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/db169534-8aca-4878-81b2-176a18d7053d-bundle\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:36:18.833880 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:18.833888    4921 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/43a925de-112e-49dc-8fa9-6e37978a907c-util\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:36:18.834193 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:18.833896    4921 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/43a925de-112e-49dc-8fa9-6e37978a907c-bundle\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:36:18.834193 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:18.833904    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cpqg\" (UniqueName: \"kubernetes.io/projected/db169534-8aca-4878-81b2-176a18d7053d-kube-api-access-8cpqg\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:36:19.483941 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:19.483906    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78jhbr"
Nov 03 19:36:19.484414 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:19.483910    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78jhbr" event={"ID":"43a925de-112e-49dc-8fa9-6e37978a907c","Type":"ContainerDied","Data":"07ad0bf918f1de3d1929f4a99ecd352a1a5a84e0e62529f7c920ce78379a5fb9"}
Nov 03 19:36:19.484414 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:19.484048    4921 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07ad0bf918f1de3d1929f4a99ecd352a1a5a84e0e62529f7c920ce78379a5fb9"
Nov 03 19:36:19.486305 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:19.486257    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm2c9c" event={"ID":"db169534-8aca-4878-81b2-176a18d7053d","Type":"ContainerDied","Data":"921e5a0b1dce71b257efdf08ce1d23cda1f815ce1e7450ab53e868ff8237d7ec"}
Nov 03 19:36:19.486305 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:19.486301    4921 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="921e5a0b1dce71b257efdf08ce1d23cda1f815ce1e7450ab53e868ff8237d7ec"
Nov 03 19:36:19.487074 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:19.486975    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cm2c9c"
Nov 03 19:36:19.675350 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:19.675323    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lg74p"
Nov 03 19:36:19.750723 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:19.743705    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3760839d-09c5-43db-a771-3b196a6d1abb-kube-api-access-t8dm5\" (UniqueName: \"kubernetes.io/projected/3760839d-09c5-43db-a771-3b196a6d1abb-kube-api-access-t8dm5\") pod \"3760839d-09c5-43db-a771-3b196a6d1abb\" (UID: \"3760839d-09c5-43db-a771-3b196a6d1abb\") "
Nov 03 19:36:19.750723 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:19.743770    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3760839d-09c5-43db-a771-3b196a6d1abb-util\" (UniqueName: \"kubernetes.io/empty-dir/3760839d-09c5-43db-a771-3b196a6d1abb-util\") pod \"3760839d-09c5-43db-a771-3b196a6d1abb\" (UID: \"3760839d-09c5-43db-a771-3b196a6d1abb\") "
Nov 03 19:36:19.750723 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:19.743823    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3760839d-09c5-43db-a771-3b196a6d1abb-bundle\" (UniqueName: \"kubernetes.io/empty-dir/3760839d-09c5-43db-a771-3b196a6d1abb-bundle\") pod \"3760839d-09c5-43db-a771-3b196a6d1abb\" (UID: \"3760839d-09c5-43db-a771-3b196a6d1abb\") "
Nov 03 19:36:19.750723 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:19.744523    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3760839d-09c5-43db-a771-3b196a6d1abb-bundle" pod "3760839d-09c5-43db-a771-3b196a6d1abb" (UID: "3760839d-09c5-43db-a771-3b196a6d1abb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:36:19.750723 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:19.746152    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3760839d-09c5-43db-a771-3b196a6d1abb-kube-api-access-t8dm5" pod "3760839d-09c5-43db-a771-3b196a6d1abb" (UID: "3760839d-09c5-43db-a771-3b196a6d1abb"). InnerVolumeSpecName "kube-api-access-t8dm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:36:19.750723 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:19.750463    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj255d"
Nov 03 19:36:19.751080 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:19.751057    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3760839d-09c5-43db-a771-3b196a6d1abb-util" pod "3760839d-09c5-43db-a771-3b196a6d1abb" (UID: "3760839d-09c5-43db-a771-3b196a6d1abb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:36:19.760331 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:19.760304    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449cmzc"
Nov 03 19:36:19.763962 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:19.763941    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c499nz"
Nov 03 19:36:19.770458 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:19.770434    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6s6jjd"
Nov 03 19:36:19.844567 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:19.844538    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e853cbd6-5e8c-4c73-98ab-d183e941a549-kube-api-access-f96k7\" (UniqueName: \"kubernetes.io/projected/e853cbd6-5e8c-4c73-98ab-d183e941a549-kube-api-access-f96k7\") pod \"e853cbd6-5e8c-4c73-98ab-d183e941a549\" (UID: \"e853cbd6-5e8c-4c73-98ab-d183e941a549\") "
Nov 03 19:36:19.844759 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:19.844593    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e086d0a-690d-4411-8fca-3f8f3c5f8bb5-util\" (UniqueName: \"kubernetes.io/empty-dir/2e086d0a-690d-4411-8fca-3f8f3c5f8bb5-util\") pod \"2e086d0a-690d-4411-8fca-3f8f3c5f8bb5\" (UID: \"2e086d0a-690d-4411-8fca-3f8f3c5f8bb5\") "
Nov 03 19:36:19.844759 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:19.844627    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48e24ffe-d9a9-4f6c-98d7-0f1b2bf80378-kube-api-access-24dc9\" (UniqueName: \"kubernetes.io/projected/48e24ffe-d9a9-4f6c-98d7-0f1b2bf80378-kube-api-access-24dc9\") pod \"48e24ffe-d9a9-4f6c-98d7-0f1b2bf80378\" (UID: \"48e24ffe-d9a9-4f6c-98d7-0f1b2bf80378\") "
Nov 03 19:36:19.844759 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:19.844657    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01cf4a61-867e-41d3-b0b7-2be0d700458b-kube-api-access-5gmj5\" (UniqueName: \"kubernetes.io/projected/01cf4a61-867e-41d3-b0b7-2be0d700458b-kube-api-access-5gmj5\") pod \"01cf4a61-867e-41d3-b0b7-2be0d700458b\" (UID: \"01cf4a61-867e-41d3-b0b7-2be0d700458b\") "
Nov 03 19:36:19.844759 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:19.844712    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48e24ffe-d9a9-4f6c-98d7-0f1b2bf80378-util\" (UniqueName: \"kubernetes.io/empty-dir/48e24ffe-d9a9-4f6c-98d7-0f1b2bf80378-util\") pod \"48e24ffe-d9a9-4f6c-98d7-0f1b2bf80378\" (UID: \"48e24ffe-d9a9-4f6c-98d7-0f1b2bf80378\") "
Nov 03 19:36:19.844759 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:19.844741    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e086d0a-690d-4411-8fca-3f8f3c5f8bb5-kube-api-access-2wknr\" (UniqueName: \"kubernetes.io/projected/2e086d0a-690d-4411-8fca-3f8f3c5f8bb5-kube-api-access-2wknr\") pod \"2e086d0a-690d-4411-8fca-3f8f3c5f8bb5\" (UID: \"2e086d0a-690d-4411-8fca-3f8f3c5f8bb5\") "
Nov 03 19:36:19.845033 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:19.844784    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e086d0a-690d-4411-8fca-3f8f3c5f8bb5-bundle\" (UniqueName: \"kubernetes.io/empty-dir/2e086d0a-690d-4411-8fca-3f8f3c5f8bb5-bundle\") pod \"2e086d0a-690d-4411-8fca-3f8f3c5f8bb5\" (UID: \"2e086d0a-690d-4411-8fca-3f8f3c5f8bb5\") "
Nov 03 19:36:19.845033 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:19.844809    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e853cbd6-5e8c-4c73-98ab-d183e941a549-bundle\" (UniqueName: \"kubernetes.io/empty-dir/e853cbd6-5e8c-4c73-98ab-d183e941a549-bundle\") pod \"e853cbd6-5e8c-4c73-98ab-d183e941a549\" (UID: \"e853cbd6-5e8c-4c73-98ab-d183e941a549\") "
Nov 03 19:36:19.845033 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:19.844853    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01cf4a61-867e-41d3-b0b7-2be0d700458b-util\" (UniqueName: \"kubernetes.io/empty-dir/01cf4a61-867e-41d3-b0b7-2be0d700458b-util\") pod \"01cf4a61-867e-41d3-b0b7-2be0d700458b\" (UID: \"01cf4a61-867e-41d3-b0b7-2be0d700458b\") "
Nov 03 19:36:19.845033 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:19.844888    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48e24ffe-d9a9-4f6c-98d7-0f1b2bf80378-bundle\" (UniqueName: \"kubernetes.io/empty-dir/48e24ffe-d9a9-4f6c-98d7-0f1b2bf80378-bundle\") pod \"48e24ffe-d9a9-4f6c-98d7-0f1b2bf80378\" (UID: \"48e24ffe-d9a9-4f6c-98d7-0f1b2bf80378\") "
Nov 03 19:36:19.845033 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:19.844918    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e853cbd6-5e8c-4c73-98ab-d183e941a549-util\" (UniqueName: \"kubernetes.io/empty-dir/e853cbd6-5e8c-4c73-98ab-d183e941a549-util\") pod \"e853cbd6-5e8c-4c73-98ab-d183e941a549\" (UID: \"e853cbd6-5e8c-4c73-98ab-d183e941a549\") "
Nov 03 19:36:19.845033 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:19.844967    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01cf4a61-867e-41d3-b0b7-2be0d700458b-bundle\" (UniqueName: \"kubernetes.io/empty-dir/01cf4a61-867e-41d3-b0b7-2be0d700458b-bundle\") pod \"01cf4a61-867e-41d3-b0b7-2be0d700458b\" (UID: \"01cf4a61-867e-41d3-b0b7-2be0d700458b\") "
Nov 03 19:36:19.845309 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:19.845252    4921 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3760839d-09c5-43db-a771-3b196a6d1abb-bundle\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:36:19.845309 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:19.845279    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8dm5\" (UniqueName: \"kubernetes.io/projected/3760839d-09c5-43db-a771-3b196a6d1abb-kube-api-access-t8dm5\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:36:19.845309 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:19.845294    4921 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3760839d-09c5-43db-a771-3b196a6d1abb-util\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:36:19.846803 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:19.846270    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48e24ffe-d9a9-4f6c-98d7-0f1b2bf80378-bundle" pod "48e24ffe-d9a9-4f6c-98d7-0f1b2bf80378" (UID: "48e24ffe-d9a9-4f6c-98d7-0f1b2bf80378"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:36:19.846803 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:19.846318    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01cf4a61-867e-41d3-b0b7-2be0d700458b-bundle" pod "01cf4a61-867e-41d3-b0b7-2be0d700458b" (UID: "01cf4a61-867e-41d3-b0b7-2be0d700458b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:36:19.847993 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:19.847964    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e086d0a-690d-4411-8fca-3f8f3c5f8bb5-bundle" pod "2e086d0a-690d-4411-8fca-3f8f3c5f8bb5" (UID: "2e086d0a-690d-4411-8fca-3f8f3c5f8bb5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:36:19.848104 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:19.848033    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48e24ffe-d9a9-4f6c-98d7-0f1b2bf80378-kube-api-access-24dc9" pod "48e24ffe-d9a9-4f6c-98d7-0f1b2bf80378" (UID: "48e24ffe-d9a9-4f6c-98d7-0f1b2bf80378"). InnerVolumeSpecName "kube-api-access-24dc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:36:19.848178 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:19.848159    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e853cbd6-5e8c-4c73-98ab-d183e941a549-bundle" pod "e853cbd6-5e8c-4c73-98ab-d183e941a549" (UID: "e853cbd6-5e8c-4c73-98ab-d183e941a549"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:36:19.848341 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:19.848307    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e086d0a-690d-4411-8fca-3f8f3c5f8bb5-kube-api-access-2wknr" pod "2e086d0a-690d-4411-8fca-3f8f3c5f8bb5" (UID: "2e086d0a-690d-4411-8fca-3f8f3c5f8bb5"). InnerVolumeSpecName "kube-api-access-2wknr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:36:19.849337 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:19.849304    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01cf4a61-867e-41d3-b0b7-2be0d700458b-kube-api-access-5gmj5" pod "01cf4a61-867e-41d3-b0b7-2be0d700458b" (UID: "01cf4a61-867e-41d3-b0b7-2be0d700458b"). InnerVolumeSpecName "kube-api-access-5gmj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:36:19.851739 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:19.851711    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e853cbd6-5e8c-4c73-98ab-d183e941a549-util" pod "e853cbd6-5e8c-4c73-98ab-d183e941a549" (UID: "e853cbd6-5e8c-4c73-98ab-d183e941a549"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:36:19.853689 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:19.853645    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01cf4a61-867e-41d3-b0b7-2be0d700458b-util" pod "01cf4a61-867e-41d3-b0b7-2be0d700458b" (UID: "01cf4a61-867e-41d3-b0b7-2be0d700458b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:36:19.853993 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:19.853976    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e086d0a-690d-4411-8fca-3f8f3c5f8bb5-util" pod "2e086d0a-690d-4411-8fca-3f8f3c5f8bb5" (UID: "2e086d0a-690d-4411-8fca-3f8f3c5f8bb5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:36:19.854530 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:19.854508    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48e24ffe-d9a9-4f6c-98d7-0f1b2bf80378-util" pod "48e24ffe-d9a9-4f6c-98d7-0f1b2bf80378" (UID: "48e24ffe-d9a9-4f6c-98d7-0f1b2bf80378"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:36:19.856135 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:19.856114    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e853cbd6-5e8c-4c73-98ab-d183e941a549-kube-api-access-f96k7" pod "e853cbd6-5e8c-4c73-98ab-d183e941a549" (UID: "e853cbd6-5e8c-4c73-98ab-d183e941a549"). InnerVolumeSpecName "kube-api-access-f96k7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:36:19.946620 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:19.946579    4921 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2e086d0a-690d-4411-8fca-3f8f3c5f8bb5-bundle\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:36:19.946620 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:19.946617    4921 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e853cbd6-5e8c-4c73-98ab-d183e941a549-bundle\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:36:19.946620 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:19.946626    4921 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/01cf4a61-867e-41d3-b0b7-2be0d700458b-util\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:36:19.946911 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:19.946633    4921 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/48e24ffe-d9a9-4f6c-98d7-0f1b2bf80378-bundle\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:36:19.946911 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:19.946642    4921 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e853cbd6-5e8c-4c73-98ab-d183e941a549-util\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:36:19.946911 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:19.946649    4921 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/01cf4a61-867e-41d3-b0b7-2be0d700458b-bundle\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:36:19.946911 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:19.946658    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f96k7\" (UniqueName: \"kubernetes.io/projected/e853cbd6-5e8c-4c73-98ab-d183e941a549-kube-api-access-f96k7\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:36:19.946911 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:19.946678    4921 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2e086d0a-690d-4411-8fca-3f8f3c5f8bb5-util\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:36:19.946911 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:19.946686    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24dc9\" (UniqueName: \"kubernetes.io/projected/48e24ffe-d9a9-4f6c-98d7-0f1b2bf80378-kube-api-access-24dc9\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:36:19.946911 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:19.946700    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gmj5\" (UniqueName: \"kubernetes.io/projected/01cf4a61-867e-41d3-b0b7-2be0d700458b-kube-api-access-5gmj5\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:36:19.946911 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:19.946714    4921 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/48e24ffe-d9a9-4f6c-98d7-0f1b2bf80378-util\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:36:19.946911 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:19.946727    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wknr\" (UniqueName: \"kubernetes.io/projected/2e086d0a-690d-4411-8fca-3f8f3c5f8bb5-kube-api-access-2wknr\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:36:20.492856 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:20.492827    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lg74p"
Nov 03 19:36:20.492856 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:20.492848    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lg74p" event={"ID":"3760839d-09c5-43db-a771-3b196a6d1abb","Type":"ContainerDied","Data":"4bc6895518cf7adc86f9696ad9f48006291a39a5ff157b696deeafdd622a9277"}
Nov 03 19:36:20.493328 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:20.492898    4921 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4bc6895518cf7adc86f9696ad9f48006291a39a5ff157b696deeafdd622a9277"
Nov 03 19:36:20.494888 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:20.494864    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449cmzc"
Nov 03 19:36:20.494987 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:20.494862    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6449cmzc" event={"ID":"2e086d0a-690d-4411-8fca-3f8f3c5f8bb5","Type":"ContainerDied","Data":"c27b29d2f223044c4247322418ca00fa55b104f64016d37c0ee732f6911c4013"}
Nov 03 19:36:20.495046 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:20.494990    4921 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c27b29d2f223044c4247322418ca00fa55b104f64016d37c0ee732f6911c4013"
Nov 03 19:36:20.496839 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:20.496812    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c499nz"
Nov 03 19:36:20.496839 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:20.496827    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c499nz" event={"ID":"e853cbd6-5e8c-4c73-98ab-d183e941a549","Type":"ContainerDied","Data":"73c9655701210aba71ae634ed1de5eb48aba3651ce64827e73879c6c430cba3d"}
Nov 03 19:36:20.496977 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:20.496862    4921 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73c9655701210aba71ae634ed1de5eb48aba3651ce64827e73879c6c430cba3d"
Nov 03 19:36:20.498917 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:20.498893    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6s6jjd"
Nov 03 19:36:20.499007 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:20.498912    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6s6jjd" event={"ID":"48e24ffe-d9a9-4f6c-98d7-0f1b2bf80378","Type":"ContainerDied","Data":"7b0b318ab1d41b1ceadf417847e16a716e15548524b5a2d87178306febd72a56"}
Nov 03 19:36:20.499007 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:20.498943    4921 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b0b318ab1d41b1ceadf417847e16a716e15548524b5a2d87178306febd72a56"
Nov 03 19:36:20.501118 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:20.501095    4921 generic.go:356] "Generic (PLEG): container finished" podID="28d6b5eb-53bd-4b4c-9474-7d5fb948b879" containerID="0d661377c2b2eea9fab0480225c70943ec451e19118824fbb8d635f17325b9a3" exitCode=0
Nov 03 19:36:20.501220 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:20.501190    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ashw75" event={"ID":"28d6b5eb-53bd-4b4c-9474-7d5fb948b879","Type":"ContainerDied","Data":"0d661377c2b2eea9fab0480225c70943ec451e19118824fbb8d635f17325b9a3"}
Nov 03 19:36:20.503806 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:20.503649    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj255d" event={"ID":"01cf4a61-867e-41d3-b0b7-2be0d700458b","Type":"ContainerDied","Data":"d1f9ceb7014d0fcf442a25a66fe6851808417f60d6115a79523091b04656c27a"}
Nov 03 19:36:20.503806 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:20.503703    4921 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1f9ceb7014d0fcf442a25a66fe6851808417f60d6115a79523091b04656c27a"
Nov 03 19:36:20.503806 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:20.503770    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj255d"
Nov 03 19:36:21.634039 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:21.634011    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ashw75"
Nov 03 19:36:21.660359 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:21.660329    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28d6b5eb-53bd-4b4c-9474-7d5fb948b879-kube-api-access-nwvn4\" (UniqueName: \"kubernetes.io/projected/28d6b5eb-53bd-4b4c-9474-7d5fb948b879-kube-api-access-nwvn4\") pod \"28d6b5eb-53bd-4b4c-9474-7d5fb948b879\" (UID: \"28d6b5eb-53bd-4b4c-9474-7d5fb948b879\") "
Nov 03 19:36:21.660547 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:21.660387    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28d6b5eb-53bd-4b4c-9474-7d5fb948b879-util\" (UniqueName: \"kubernetes.io/empty-dir/28d6b5eb-53bd-4b4c-9474-7d5fb948b879-util\") pod \"28d6b5eb-53bd-4b4c-9474-7d5fb948b879\" (UID: \"28d6b5eb-53bd-4b4c-9474-7d5fb948b879\") "
Nov 03 19:36:21.660547 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:21.660525    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28d6b5eb-53bd-4b4c-9474-7d5fb948b879-bundle\" (UniqueName: \"kubernetes.io/empty-dir/28d6b5eb-53bd-4b4c-9474-7d5fb948b879-bundle\") pod \"28d6b5eb-53bd-4b4c-9474-7d5fb948b879\" (UID: \"28d6b5eb-53bd-4b4c-9474-7d5fb948b879\") "
Nov 03 19:36:21.661168 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:21.661146    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28d6b5eb-53bd-4b4c-9474-7d5fb948b879-bundle" pod "28d6b5eb-53bd-4b4c-9474-7d5fb948b879" (UID: "28d6b5eb-53bd-4b4c-9474-7d5fb948b879"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:36:21.662613 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:21.662582    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28d6b5eb-53bd-4b4c-9474-7d5fb948b879-kube-api-access-nwvn4" pod "28d6b5eb-53bd-4b4c-9474-7d5fb948b879" (UID: "28d6b5eb-53bd-4b4c-9474-7d5fb948b879"). InnerVolumeSpecName "kube-api-access-nwvn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:36:21.665525 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:21.665504    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28d6b5eb-53bd-4b4c-9474-7d5fb948b879-util" pod "28d6b5eb-53bd-4b4c-9474-7d5fb948b879" (UID: "28d6b5eb-53bd-4b4c-9474-7d5fb948b879"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:36:21.762214 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:21.762123    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwvn4\" (UniqueName: \"kubernetes.io/projected/28d6b5eb-53bd-4b4c-9474-7d5fb948b879-kube-api-access-nwvn4\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:36:21.762214 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:21.762156    4921 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/28d6b5eb-53bd-4b4c-9474-7d5fb948b879-util\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:36:21.762214 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:21.762174    4921 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/28d6b5eb-53bd-4b4c-9474-7d5fb948b879-bundle\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:36:22.514537 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:22.514504    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ashw75"
Nov 03 19:36:22.514765 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:22.514505    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ashw75" event={"ID":"28d6b5eb-53bd-4b4c-9474-7d5fb948b879","Type":"ContainerDied","Data":"19b6d905c3219eaacd5579dfaca023c2547f0cd3b54001891e5761a19236ad35"}
Nov 03 19:36:22.514765 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:22.514623    4921 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19b6d905c3219eaacd5579dfaca023c2547f0cd3b54001891e5761a19236ad35"
Nov 03 19:36:26.910991 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:26.910961    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gzf8k_6f215b42-0f09-46a0-bc98-392c0fa909c9/console-operator/2.log"
Nov 03 19:36:26.911487 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:26.911045    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gzf8k_6f215b42-0f09-46a0-bc98-392c0fa909c9/console-operator/2.log"
Nov 03 19:36:32.920301 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:36:32.920259    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 19:36:32.961900 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:32.961865    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-n6kg5"]
Nov 03 19:36:32.962189 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:32.962176    4921 cpu_manager.go:414] "containerMap: removing container" podUID="e853cbd6-5e8c-4c73-98ab-d183e941a549" containerName="pull"
Nov 03 19:36:32.962237 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:32.962191    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e853cbd6-5e8c-4c73-98ab-d183e941a549" containerName="pull"
Nov 03 19:36:32.962237 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:32.962205    4921 cpu_manager.go:414] "containerMap: removing container" podUID="01cf4a61-867e-41d3-b0b7-2be0d700458b" containerName="pull"
Nov 03 19:36:32.962237 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:32.962210    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01cf4a61-867e-41d3-b0b7-2be0d700458b" containerName="pull"
Nov 03 19:36:32.962237 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:32.962217    4921 cpu_manager.go:414] "containerMap: removing container" podUID="2e086d0a-690d-4411-8fca-3f8f3c5f8bb5" containerName="extract"
Nov 03 19:36:32.962237 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:32.962222    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e086d0a-690d-4411-8fca-3f8f3c5f8bb5" containerName="extract"
Nov 03 19:36:32.962237 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:32.962232    4921 cpu_manager.go:414] "containerMap: removing container" podUID="48e24ffe-d9a9-4f6c-98d7-0f1b2bf80378" containerName="util"
Nov 03 19:36:32.962237 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:32.962237    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48e24ffe-d9a9-4f6c-98d7-0f1b2bf80378" containerName="util"
Nov 03 19:36:32.962440 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:32.962244    4921 cpu_manager.go:414] "containerMap: removing container" podUID="db169534-8aca-4878-81b2-176a18d7053d" containerName="pull"
Nov 03 19:36:32.962440 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:32.962248    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db169534-8aca-4878-81b2-176a18d7053d" containerName="pull"
Nov 03 19:36:32.962440 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:32.962256    4921 cpu_manager.go:414] "containerMap: removing container" podUID="e853cbd6-5e8c-4c73-98ab-d183e941a549" containerName="util"
Nov 03 19:36:32.962440 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:32.962260    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e853cbd6-5e8c-4c73-98ab-d183e941a549" containerName="util"
Nov 03 19:36:32.962440 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:32.962267    4921 cpu_manager.go:414] "containerMap: removing container" podUID="db169534-8aca-4878-81b2-176a18d7053d" containerName="util"
Nov 03 19:36:32.962440 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:32.962271    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db169534-8aca-4878-81b2-176a18d7053d" containerName="util"
Nov 03 19:36:32.962440 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:32.962276    4921 cpu_manager.go:414] "containerMap: removing container" podUID="48e24ffe-d9a9-4f6c-98d7-0f1b2bf80378" containerName="extract"
Nov 03 19:36:32.962440 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:32.962280    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48e24ffe-d9a9-4f6c-98d7-0f1b2bf80378" containerName="extract"
Nov 03 19:36:32.962440 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:32.962286    4921 cpu_manager.go:414] "containerMap: removing container" podUID="3760839d-09c5-43db-a771-3b196a6d1abb" containerName="extract"
Nov 03 19:36:32.962440 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:32.962291    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3760839d-09c5-43db-a771-3b196a6d1abb" containerName="extract"
Nov 03 19:36:32.962440 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:32.962299    4921 cpu_manager.go:414] "containerMap: removing container" podUID="db169534-8aca-4878-81b2-176a18d7053d" containerName="extract"
Nov 03 19:36:32.962440 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:32.962303    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db169534-8aca-4878-81b2-176a18d7053d" containerName="extract"
Nov 03 19:36:32.962440 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:32.962311    4921 cpu_manager.go:414] "containerMap: removing container" podUID="28d6b5eb-53bd-4b4c-9474-7d5fb948b879" containerName="extract"
Nov 03 19:36:32.962440 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:32.962315    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28d6b5eb-53bd-4b4c-9474-7d5fb948b879" containerName="extract"
Nov 03 19:36:32.962440 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:32.962323    4921 cpu_manager.go:414] "containerMap: removing container" podUID="3760839d-09c5-43db-a771-3b196a6d1abb" containerName="util"
Nov 03 19:36:32.962440 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:32.962327    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3760839d-09c5-43db-a771-3b196a6d1abb" containerName="util"
Nov 03 19:36:32.962440 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:32.962333    4921 cpu_manager.go:414] "containerMap: removing container" podUID="2e086d0a-690d-4411-8fca-3f8f3c5f8bb5" containerName="pull"
Nov 03 19:36:32.962440 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:32.962337    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e086d0a-690d-4411-8fca-3f8f3c5f8bb5" containerName="pull"
Nov 03 19:36:32.962440 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:32.962342    4921 cpu_manager.go:414] "containerMap: removing container" podUID="3760839d-09c5-43db-a771-3b196a6d1abb" containerName="pull"
Nov 03 19:36:32.962440 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:32.962345    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3760839d-09c5-43db-a771-3b196a6d1abb" containerName="pull"
Nov 03 19:36:32.962440 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:32.962350    4921 cpu_manager.go:414] "containerMap: removing container" podUID="2ba24656-710d-4547-9b5c-293b76fabd06" containerName="extract"
Nov 03 19:36:32.962440 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:32.962354    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ba24656-710d-4547-9b5c-293b76fabd06" containerName="extract"
Nov 03 19:36:32.962440 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:32.962362    4921 cpu_manager.go:414] "containerMap: removing container" podUID="2ba24656-710d-4547-9b5c-293b76fabd06" containerName="pull"
Nov 03 19:36:32.962440 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:32.962366    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ba24656-710d-4547-9b5c-293b76fabd06" containerName="pull"
Nov 03 19:36:32.962440 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:32.962372    4921 cpu_manager.go:414] "containerMap: removing container" podUID="28d6b5eb-53bd-4b4c-9474-7d5fb948b879" containerName="pull"
Nov 03 19:36:32.962440 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:32.962376    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28d6b5eb-53bd-4b4c-9474-7d5fb948b879" containerName="pull"
Nov 03 19:36:32.962440 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:32.962388    4921 cpu_manager.go:414] "containerMap: removing container" podUID="2e086d0a-690d-4411-8fca-3f8f3c5f8bb5" containerName="util"
Nov 03 19:36:32.962440 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:32.962392    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e086d0a-690d-4411-8fca-3f8f3c5f8bb5" containerName="util"
Nov 03 19:36:32.962440 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:32.962399    4921 cpu_manager.go:414] "containerMap: removing container" podUID="43a925de-112e-49dc-8fa9-6e37978a907c" containerName="util"
Nov 03 19:36:32.962440 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:32.962403    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43a925de-112e-49dc-8fa9-6e37978a907c" containerName="util"
Nov 03 19:36:32.962440 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:32.962409    4921 cpu_manager.go:414] "containerMap: removing container" podUID="43a925de-112e-49dc-8fa9-6e37978a907c" containerName="pull"
Nov 03 19:36:32.962440 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:32.962413    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43a925de-112e-49dc-8fa9-6e37978a907c" containerName="pull"
Nov 03 19:36:32.962440 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:32.962419    4921 cpu_manager.go:414] "containerMap: removing container" podUID="48e24ffe-d9a9-4f6c-98d7-0f1b2bf80378" containerName="pull"
Nov 03 19:36:32.962440 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:32.962424    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48e24ffe-d9a9-4f6c-98d7-0f1b2bf80378" containerName="pull"
Nov 03 19:36:32.962440 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:32.962433    4921 cpu_manager.go:414] "containerMap: removing container" podUID="2ba24656-710d-4547-9b5c-293b76fabd06" containerName="util"
Nov 03 19:36:32.962440 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:32.962438    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ba24656-710d-4547-9b5c-293b76fabd06" containerName="util"
Nov 03 19:36:32.962440 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:32.962444    4921 cpu_manager.go:414] "containerMap: removing container" podUID="43a925de-112e-49dc-8fa9-6e37978a907c" containerName="extract"
Nov 03 19:36:32.962440 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:32.962448    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43a925de-112e-49dc-8fa9-6e37978a907c" containerName="extract"
Nov 03 19:36:32.962440 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:32.962456    4921 cpu_manager.go:414] "containerMap: removing container" podUID="28d6b5eb-53bd-4b4c-9474-7d5fb948b879" containerName="util"
Nov 03 19:36:32.962440 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:32.962460    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28d6b5eb-53bd-4b4c-9474-7d5fb948b879" containerName="util"
Nov 03 19:36:32.963560 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:32.962467    4921 cpu_manager.go:414] "containerMap: removing container" podUID="01cf4a61-867e-41d3-b0b7-2be0d700458b" containerName="util"
Nov 03 19:36:32.963560 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:32.962472    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01cf4a61-867e-41d3-b0b7-2be0d700458b" containerName="util"
Nov 03 19:36:32.963560 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:32.962478    4921 cpu_manager.go:414] "containerMap: removing container" podUID="01cf4a61-867e-41d3-b0b7-2be0d700458b" containerName="extract"
Nov 03 19:36:32.963560 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:32.962482    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01cf4a61-867e-41d3-b0b7-2be0d700458b" containerName="extract"
Nov 03 19:36:32.963560 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:32.962488    4921 cpu_manager.go:414] "containerMap: removing container" podUID="e853cbd6-5e8c-4c73-98ab-d183e941a549" containerName="extract"
Nov 03 19:36:32.963560 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:32.962492    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e853cbd6-5e8c-4c73-98ab-d183e941a549" containerName="extract"
Nov 03 19:36:32.963560 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:32.962536    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="db169534-8aca-4878-81b2-176a18d7053d" containerName="extract"
Nov 03 19:36:32.963560 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:32.962542    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="28d6b5eb-53bd-4b4c-9474-7d5fb948b879" containerName="extract"
Nov 03 19:36:32.963560 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:32.962548    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ba24656-710d-4547-9b5c-293b76fabd06" containerName="extract"
Nov 03 19:36:32.963560 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:32.962555    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="01cf4a61-867e-41d3-b0b7-2be0d700458b" containerName="extract"
Nov 03 19:36:32.963560 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:32.962561    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="43a925de-112e-49dc-8fa9-6e37978a907c" containerName="extract"
Nov 03 19:36:32.963560 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:32.962568    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="e853cbd6-5e8c-4c73-98ab-d183e941a549" containerName="extract"
Nov 03 19:36:32.963560 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:32.962573    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e086d0a-690d-4411-8fca-3f8f3c5f8bb5" containerName="extract"
Nov 03 19:36:32.963560 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:32.962580    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="48e24ffe-d9a9-4f6c-98d7-0f1b2bf80378" containerName="extract"
Nov 03 19:36:32.963560 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:32.962586    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="3760839d-09c5-43db-a771-3b196a6d1abb" containerName="extract"
Nov 03 19:36:32.965267 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:32.965248    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-n6kg5"
Nov 03 19:36:32.969175 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:32.969150    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Nov 03 19:36:32.969399 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:32.969381    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-tjspm\""
Nov 03 19:36:32.969476 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:32.969399    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Nov 03 19:36:32.977602 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:32.977579    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-n6kg5"]
Nov 03 19:36:33.046690 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:33.046638    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/971744d1-9edf-49cc-9b85-5c3ce81fa136-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-n6kg5\" (UID: \"971744d1-9edf-49cc-9b85-5c3ce81fa136\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-n6kg5"
Nov 03 19:36:33.046861 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:33.046784    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw982\" (UniqueName: \"kubernetes.io/projected/971744d1-9edf-49cc-9b85-5c3ce81fa136-kube-api-access-bw982\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-n6kg5\" (UID: \"971744d1-9edf-49cc-9b85-5c3ce81fa136\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-n6kg5"
Nov 03 19:36:33.148100 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:33.148062    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw982\" (UniqueName: \"kubernetes.io/projected/971744d1-9edf-49cc-9b85-5c3ce81fa136-kube-api-access-bw982\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-n6kg5\" (UID: \"971744d1-9edf-49cc-9b85-5c3ce81fa136\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-n6kg5"
Nov 03 19:36:33.148274 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:33.148111    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/971744d1-9edf-49cc-9b85-5c3ce81fa136-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-n6kg5\" (UID: \"971744d1-9edf-49cc-9b85-5c3ce81fa136\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-n6kg5"
Nov 03 19:36:33.148422 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:33.148407    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/971744d1-9edf-49cc-9b85-5c3ce81fa136-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-n6kg5\" (UID: \"971744d1-9edf-49cc-9b85-5c3ce81fa136\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-n6kg5"
Nov 03 19:36:33.161478 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:33.161447    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw982\" (UniqueName: \"kubernetes.io/projected/971744d1-9edf-49cc-9b85-5c3ce81fa136-kube-api-access-bw982\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-n6kg5\" (UID: \"971744d1-9edf-49cc-9b85-5c3ce81fa136\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-n6kg5"
Nov 03 19:36:33.275471 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:33.275432    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-n6kg5"
Nov 03 19:36:33.556156 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:33.556127    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-n6kg5"]
Nov 03 19:36:33.558901 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:36:33.558872    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod971744d1_9edf_49cc_9b85_5c3ce81fa136.slice/crio-54e5c41140d3a976f228a5d0d7f760233606f392f63d04a52631fcead1a53860 WatchSource:0}: Error finding container 54e5c41140d3a976f228a5d0d7f760233606f392f63d04a52631fcead1a53860: Status 404 returned error can't find the container with id 54e5c41140d3a976f228a5d0d7f760233606f392f63d04a52631fcead1a53860
Nov 03 19:36:33.562118 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:33.562097    4921 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 19:36:34.568723 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:34.568685    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-n6kg5" event={"ID":"971744d1-9edf-49cc-9b85-5c3ce81fa136","Type":"ContainerStarted","Data":"54e5c41140d3a976f228a5d0d7f760233606f392f63d04a52631fcead1a53860"}
Nov 03 19:36:34.807228 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:34.807192    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-rxrg2"]
Nov 03 19:36:34.810170 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:34.810101    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-rxrg2"
Nov 03 19:36:34.822090 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:34.822018    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 03 19:36:34.822223 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:34.822194    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 03 19:36:34.823545 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:34.823524    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 03 19:36:34.823650 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:34.823586    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 03 19:36:34.823752 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:34.823736    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-dpmf9\""
Nov 03 19:36:34.823836 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:34.823816    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 03 19:36:34.831580 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:34.831549    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-rxrg2"]
Nov 03 19:36:34.862771 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:34.862733    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a1b5e79c-2e6b-44d7-8562-d2e9240be700-tmp\") pod \"opendatahub-operator-controller-manager-bd9746b54-rxrg2\" (UID: \"a1b5e79c-2e6b-44d7-8562-d2e9240be700\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-rxrg2"
Nov 03 19:36:34.862955 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:34.862843    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a1b5e79c-2e6b-44d7-8562-d2e9240be700-webhook-cert\") pod \"opendatahub-operator-controller-manager-bd9746b54-rxrg2\" (UID: \"a1b5e79c-2e6b-44d7-8562-d2e9240be700\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-rxrg2"
Nov 03 19:36:34.862955 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:34.862883    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a1b5e79c-2e6b-44d7-8562-d2e9240be700-metrics-cert\") pod \"opendatahub-operator-controller-manager-bd9746b54-rxrg2\" (UID: \"a1b5e79c-2e6b-44d7-8562-d2e9240be700\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-rxrg2"
Nov 03 19:36:34.863052 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:34.862955    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a1b5e79c-2e6b-44d7-8562-d2e9240be700-apiservice-cert\") pod \"opendatahub-operator-controller-manager-bd9746b54-rxrg2\" (UID: \"a1b5e79c-2e6b-44d7-8562-d2e9240be700\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-rxrg2"
Nov 03 19:36:34.863096 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:34.863069    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a1b5e79c-2e6b-44d7-8562-d2e9240be700-manifests\") pod \"opendatahub-operator-controller-manager-bd9746b54-rxrg2\" (UID: \"a1b5e79c-2e6b-44d7-8562-d2e9240be700\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-rxrg2"
Nov 03 19:36:34.863135 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:34.863103    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k97k5\" (UniqueName: \"kubernetes.io/projected/a1b5e79c-2e6b-44d7-8562-d2e9240be700-kube-api-access-k97k5\") pod \"opendatahub-operator-controller-manager-bd9746b54-rxrg2\" (UID: \"a1b5e79c-2e6b-44d7-8562-d2e9240be700\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-rxrg2"
Nov 03 19:36:34.964138 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:34.963612    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a1b5e79c-2e6b-44d7-8562-d2e9240be700-tmp\") pod \"opendatahub-operator-controller-manager-bd9746b54-rxrg2\" (UID: \"a1b5e79c-2e6b-44d7-8562-d2e9240be700\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-rxrg2"
Nov 03 19:36:34.964138 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:34.963701    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a1b5e79c-2e6b-44d7-8562-d2e9240be700-webhook-cert\") pod \"opendatahub-operator-controller-manager-bd9746b54-rxrg2\" (UID: \"a1b5e79c-2e6b-44d7-8562-d2e9240be700\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-rxrg2"
Nov 03 19:36:34.964138 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:34.963902    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a1b5e79c-2e6b-44d7-8562-d2e9240be700-metrics-cert\") pod \"opendatahub-operator-controller-manager-bd9746b54-rxrg2\" (UID: \"a1b5e79c-2e6b-44d7-8562-d2e9240be700\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-rxrg2"
Nov 03 19:36:34.964138 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:34.963933    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a1b5e79c-2e6b-44d7-8562-d2e9240be700-apiservice-cert\") pod \"opendatahub-operator-controller-manager-bd9746b54-rxrg2\" (UID: \"a1b5e79c-2e6b-44d7-8562-d2e9240be700\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-rxrg2"
Nov 03 19:36:34.964138 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:34.963985    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a1b5e79c-2e6b-44d7-8562-d2e9240be700-manifests\") pod \"opendatahub-operator-controller-manager-bd9746b54-rxrg2\" (UID: \"a1b5e79c-2e6b-44d7-8562-d2e9240be700\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-rxrg2"
Nov 03 19:36:34.964138 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:34.964004    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k97k5\" (UniqueName: \"kubernetes.io/projected/a1b5e79c-2e6b-44d7-8562-d2e9240be700-kube-api-access-k97k5\") pod \"opendatahub-operator-controller-manager-bd9746b54-rxrg2\" (UID: \"a1b5e79c-2e6b-44d7-8562-d2e9240be700\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-rxrg2"
Nov 03 19:36:34.964595 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:34.964425    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a1b5e79c-2e6b-44d7-8562-d2e9240be700-manifests\") pod \"opendatahub-operator-controller-manager-bd9746b54-rxrg2\" (UID: \"a1b5e79c-2e6b-44d7-8562-d2e9240be700\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-rxrg2"
Nov 03 19:36:34.966744 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:34.966717    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a1b5e79c-2e6b-44d7-8562-d2e9240be700-tmp\") pod \"opendatahub-operator-controller-manager-bd9746b54-rxrg2\" (UID: \"a1b5e79c-2e6b-44d7-8562-d2e9240be700\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-rxrg2"
Nov 03 19:36:34.966848 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:34.966817    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a1b5e79c-2e6b-44d7-8562-d2e9240be700-webhook-cert\") pod \"opendatahub-operator-controller-manager-bd9746b54-rxrg2\" (UID: \"a1b5e79c-2e6b-44d7-8562-d2e9240be700\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-rxrg2"
Nov 03 19:36:34.966898 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:34.966847    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a1b5e79c-2e6b-44d7-8562-d2e9240be700-apiservice-cert\") pod \"opendatahub-operator-controller-manager-bd9746b54-rxrg2\" (UID: \"a1b5e79c-2e6b-44d7-8562-d2e9240be700\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-rxrg2"
Nov 03 19:36:34.966898 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:34.966885    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a1b5e79c-2e6b-44d7-8562-d2e9240be700-metrics-cert\") pod \"opendatahub-operator-controller-manager-bd9746b54-rxrg2\" (UID: \"a1b5e79c-2e6b-44d7-8562-d2e9240be700\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-rxrg2"
Nov 03 19:36:34.971378 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:34.971351    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k97k5\" (UniqueName: \"kubernetes.io/projected/a1b5e79c-2e6b-44d7-8562-d2e9240be700-kube-api-access-k97k5\") pod \"opendatahub-operator-controller-manager-bd9746b54-rxrg2\" (UID: \"a1b5e79c-2e6b-44d7-8562-d2e9240be700\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-rxrg2"
Nov 03 19:36:35.126803 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:35.126715    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-rxrg2"
Nov 03 19:36:35.424930 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:35.424815    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-rxrg2"]
Nov 03 19:36:35.428799 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:36:35.428770    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1b5e79c_2e6b_44d7_8562_d2e9240be700.slice/crio-b5ac9e708a3b3c6164a41fc5fd621e7f532750d375dbb38565ae9e00a0e7d4c2 WatchSource:0}: Error finding container b5ac9e708a3b3c6164a41fc5fd621e7f532750d375dbb38565ae9e00a0e7d4c2: Status 404 returned error can't find the container with id b5ac9e708a3b3c6164a41fc5fd621e7f532750d375dbb38565ae9e00a0e7d4c2
Nov 03 19:36:35.575790 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:35.575753    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-rxrg2" event={"ID":"a1b5e79c-2e6b-44d7-8562-d2e9240be700","Type":"ContainerStarted","Data":"b5ac9e708a3b3c6164a41fc5fd621e7f532750d375dbb38565ae9e00a0e7d4c2"}
Nov 03 19:36:36.583608 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:36.583564    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-n6kg5" event={"ID":"971744d1-9edf-49cc-9b85-5c3ce81fa136","Type":"ContainerStarted","Data":"8f9dce9bc71462a16e0bc447030f28e1169cb5ebdf2b1aacd7f5005f4aaf00f4"}
Nov 03 19:36:38.557035 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:38.556986    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-n6kg5" podStartSLOduration=4.234104743 podStartE2EDuration="6.55696951s" totalImagesPullingTime="2.322864767s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:36:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:36:36.606829091 +0000 UTC m=+309.850794582" watchObservedRunningTime="2026-11-03 19:36:38.55696951 +0000 UTC m=+311.800934975"
Nov 03 19:36:38.557414 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:38.557170    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-6mrqr"]
Nov 03 19:36:38.561874 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:38.561854    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6mrqr"
Nov 03 19:36:38.575005 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:38.574980    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Nov 03 19:36:38.575133 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:38.575113    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Nov 03 19:36:38.575265 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:38.575247    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-p2jfg\""
Nov 03 19:36:38.575869 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:38.575851    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-6mrqr"]
Nov 03 19:36:38.595922 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:38.595882    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-rxrg2" event={"ID":"a1b5e79c-2e6b-44d7-8562-d2e9240be700","Type":"ContainerStarted","Data":"b0fe3cc537c3685634527e9a40213d458bb924f56fb0fb616354de581e1ac266"}
Nov 03 19:36:38.699929 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:38.699845    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/73136ab5-fad4-49ac-a39c-26abd533c8a1-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-6mrqr\" (UID: \"73136ab5-fad4-49ac-a39c-26abd533c8a1\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6mrqr"
Nov 03 19:36:38.699929 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:38.699904    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv285\" (UniqueName: \"kubernetes.io/projected/73136ab5-fad4-49ac-a39c-26abd533c8a1-kube-api-access-rv285\") pod \"cert-manager-webhook-5dbbbf75cc-6mrqr\" (UID: \"73136ab5-fad4-49ac-a39c-26abd533c8a1\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6mrqr"
Nov 03 19:36:38.801021 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:38.800987    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv285\" (UniqueName: \"kubernetes.io/projected/73136ab5-fad4-49ac-a39c-26abd533c8a1-kube-api-access-rv285\") pod \"cert-manager-webhook-5dbbbf75cc-6mrqr\" (UID: \"73136ab5-fad4-49ac-a39c-26abd533c8a1\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6mrqr"
Nov 03 19:36:38.801184 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:38.801049    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/73136ab5-fad4-49ac-a39c-26abd533c8a1-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-6mrqr\" (UID: \"73136ab5-fad4-49ac-a39c-26abd533c8a1\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6mrqr"
Nov 03 19:36:38.812689 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:38.812636    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv285\" (UniqueName: \"kubernetes.io/projected/73136ab5-fad4-49ac-a39c-26abd533c8a1-kube-api-access-rv285\") pod \"cert-manager-webhook-5dbbbf75cc-6mrqr\" (UID: \"73136ab5-fad4-49ac-a39c-26abd533c8a1\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6mrqr"
Nov 03 19:36:38.812960 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:38.812942    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/73136ab5-fad4-49ac-a39c-26abd533c8a1-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-6mrqr\" (UID: \"73136ab5-fad4-49ac-a39c-26abd533c8a1\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6mrqr"
Nov 03 19:36:38.888793 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:38.888764    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6mrqr"
Nov 03 19:36:39.179225 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:39.179197    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-6mrqr"]
Nov 03 19:36:39.181544 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:36:39.181515    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod73136ab5_fad4_49ac_a39c_26abd533c8a1.slice/crio-a94bede0d9b73e6d37c88a1be40042a021348195b5cf4769b7c5e2c84ad52a1b WatchSource:0}: Error finding container a94bede0d9b73e6d37c88a1be40042a021348195b5cf4769b7c5e2c84ad52a1b: Status 404 returned error can't find the container with id a94bede0d9b73e6d37c88a1be40042a021348195b5cf4769b7c5e2c84ad52a1b
Nov 03 19:36:39.601861 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:39.601821    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6mrqr" event={"ID":"73136ab5-fad4-49ac-a39c-26abd533c8a1","Type":"ContainerStarted","Data":"a94bede0d9b73e6d37c88a1be40042a021348195b5cf4769b7c5e2c84ad52a1b"}
Nov 03 19:36:39.603367 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:39.603342    4921 generic.go:356] "Generic (PLEG): container finished" podID="a1b5e79c-2e6b-44d7-8562-d2e9240be700" containerID="b0fe3cc537c3685634527e9a40213d458bb924f56fb0fb616354de581e1ac266" exitCode=0
Nov 03 19:36:39.603475 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:39.603395    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-rxrg2" event={"ID":"a1b5e79c-2e6b-44d7-8562-d2e9240be700","Type":"ContainerDied","Data":"b0fe3cc537c3685634527e9a40213d458bb924f56fb0fb616354de581e1ac266"}
Nov 03 19:36:40.611929 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:40.611888    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-rxrg2" event={"ID":"a1b5e79c-2e6b-44d7-8562-d2e9240be700","Type":"ContainerStarted","Data":"b852598f078ef58ae926f611c601b384a3ad77be4d69f6c126b353ec83ea1f9d"}
Nov 03 19:36:40.612343 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:40.612001    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-rxrg2"
Nov 03 19:36:40.699419 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:40.699364    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-rxrg2" podStartSLOduration=2.717859669 podStartE2EDuration="6.699346465s" totalImagesPullingTime="3.45143591s" totalInitContainerRuntime="530.050886ms" isStatefulPod=false podCreationTimestamp="2026-11-03 19:36:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:36:40.695661277 +0000 UTC m=+313.939626766" watchObservedRunningTime="2026-11-03 19:36:40.699346465 +0000 UTC m=+313.943311930"
Nov 03 19:36:42.624016 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:42.623977    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6mrqr" event={"ID":"73136ab5-fad4-49ac-a39c-26abd533c8a1","Type":"ContainerStarted","Data":"2990551cebbabb10fa88d74f120957eabe4f79993e937931df25899d0eb6307d"}
Nov 03 19:36:42.624403 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:42.624030    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6mrqr"
Nov 03 19:36:42.650097 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:42.650036    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6mrqr" podStartSLOduration=1.920082117 podStartE2EDuration="4.65002137s" totalImagesPullingTime="2.729939253s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:36:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:36:42.648245013 +0000 UTC m=+315.892210478" watchObservedRunningTime="2026-11-03 19:36:42.65002137 +0000 UTC m=+315.893986834"
Nov 03 19:36:46.894697 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:46.894645    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-6sfqz"]
Nov 03 19:36:46.898587 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:46.898566    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-6sfqz"
Nov 03 19:36:46.905706 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:46.905660    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 03 19:36:46.905854 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:46.905705    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 03 19:36:46.905854 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:46.905781    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-zxf5c\""
Nov 03 19:36:46.907136 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:46.907115    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-6sfqz"]
Nov 03 19:36:46.972400 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:46.972356    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n75mb\" (UniqueName: \"kubernetes.io/projected/0eff05fe-81c6-4ced-adfe-e426333985a8-kube-api-access-n75mb\") pod \"limitador-operator-controller-manager-5778bb4449-6sfqz\" (UID: \"0eff05fe-81c6-4ced-adfe-e426333985a8\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-6sfqz"
Nov 03 19:36:47.072952 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:47.072918    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n75mb\" (UniqueName: \"kubernetes.io/projected/0eff05fe-81c6-4ced-adfe-e426333985a8-kube-api-access-n75mb\") pod \"limitador-operator-controller-manager-5778bb4449-6sfqz\" (UID: \"0eff05fe-81c6-4ced-adfe-e426333985a8\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-6sfqz"
Nov 03 19:36:47.086274 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:47.086242    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n75mb\" (UniqueName: \"kubernetes.io/projected/0eff05fe-81c6-4ced-adfe-e426333985a8-kube-api-access-n75mb\") pod \"limitador-operator-controller-manager-5778bb4449-6sfqz\" (UID: \"0eff05fe-81c6-4ced-adfe-e426333985a8\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-6sfqz"
Nov 03 19:36:47.211372 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:47.211278    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-6sfqz"
Nov 03 19:36:47.490043 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:47.489876    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-6sfqz"]
Nov 03 19:36:47.492890 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:36:47.492855    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eff05fe_81c6_4ced_adfe_e426333985a8.slice/crio-f5edfe27524951a59b4b18f7672b8d795f2f283437b05517e803dea37b46d5a2 WatchSource:0}: Error finding container f5edfe27524951a59b4b18f7672b8d795f2f283437b05517e803dea37b46d5a2: Status 404 returned error can't find the container with id f5edfe27524951a59b4b18f7672b8d795f2f283437b05517e803dea37b46d5a2
Nov 03 19:36:47.648899 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:47.648860    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-6sfqz" event={"ID":"0eff05fe-81c6-4ced-adfe-e426333985a8","Type":"ContainerStarted","Data":"f5edfe27524951a59b4b18f7672b8d795f2f283437b05517e803dea37b46d5a2"}
Nov 03 19:36:48.630795 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:48.630763    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6mrqr"
Nov 03 19:36:50.666196 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:50.666162    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-6sfqz" event={"ID":"0eff05fe-81c6-4ced-adfe-e426333985a8","Type":"ContainerStarted","Data":"e1e17118f1ea1a5cfbe91ba66161d7de92214a74bce3ebea1c2c940d69f571da"}
Nov 03 19:36:50.666596 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:50.666287    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-6sfqz"
Nov 03 19:36:50.680487 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:50.680436    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-6sfqz" podStartSLOduration=1.690360447 podStartE2EDuration="4.680421861s" totalImagesPullingTime="2.990061414s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:36:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:36:50.680224802 +0000 UTC m=+323.924190268" watchObservedRunningTime="2026-11-03 19:36:50.680421861 +0000 UTC m=+323.924387323"
Nov 03 19:36:51.620717 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:51.620662    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-bd9746b54-rxrg2"
Nov 03 19:36:54.019175 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:54.019139    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-cmrkh"]
Nov 03 19:36:54.022956 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:54.022938    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-cmrkh"
Nov 03 19:36:54.031592 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:54.031563    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-z646t\""
Nov 03 19:36:54.035418 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:54.035392    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-cmrkh"]
Nov 03 19:36:54.138268 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:54.138219    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/399879e5-816a-40a3-a6de-5574cecfc489-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-cmrkh\" (UID: \"399879e5-816a-40a3-a6de-5574cecfc489\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-cmrkh"
Nov 03 19:36:54.138461 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:54.138366    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlhm8\" (UniqueName: \"kubernetes.io/projected/399879e5-816a-40a3-a6de-5574cecfc489-kube-api-access-jlhm8\") pod \"kuadrant-operator-controller-manager-cf8469d7b-cmrkh\" (UID: \"399879e5-816a-40a3-a6de-5574cecfc489\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-cmrkh"
Nov 03 19:36:54.239159 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:54.239121    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlhm8\" (UniqueName: \"kubernetes.io/projected/399879e5-816a-40a3-a6de-5574cecfc489-kube-api-access-jlhm8\") pod \"kuadrant-operator-controller-manager-cf8469d7b-cmrkh\" (UID: \"399879e5-816a-40a3-a6de-5574cecfc489\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-cmrkh"
Nov 03 19:36:54.239355 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:54.239177    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/399879e5-816a-40a3-a6de-5574cecfc489-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-cmrkh\" (UID: \"399879e5-816a-40a3-a6de-5574cecfc489\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-cmrkh"
Nov 03 19:36:54.239506 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:54.239487    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/399879e5-816a-40a3-a6de-5574cecfc489-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-cmrkh\" (UID: \"399879e5-816a-40a3-a6de-5574cecfc489\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-cmrkh"
Nov 03 19:36:54.247045 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:54.247018    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlhm8\" (UniqueName: \"kubernetes.io/projected/399879e5-816a-40a3-a6de-5574cecfc489-kube-api-access-jlhm8\") pod \"kuadrant-operator-controller-manager-cf8469d7b-cmrkh\" (UID: \"399879e5-816a-40a3-a6de-5574cecfc489\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-cmrkh"
Nov 03 19:36:54.336078 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:54.335978    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-cmrkh"
Nov 03 19:36:54.618459 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:54.618428    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-cmrkh"]
Nov 03 19:36:54.621237 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:36:54.621210    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod399879e5_816a_40a3_a6de_5574cecfc489.slice/crio-fa82b981b3e41938e379f44c3f1c26a1a69a7c08b51e2c2f8d66e4490f20921e WatchSource:0}: Error finding container fa82b981b3e41938e379f44c3f1c26a1a69a7c08b51e2c2f8d66e4490f20921e: Status 404 returned error can't find the container with id fa82b981b3e41938e379f44c3f1c26a1a69a7c08b51e2c2f8d66e4490f20921e
Nov 03 19:36:54.696037 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:54.696004    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-cmrkh" event={"ID":"399879e5-816a-40a3-a6de-5574cecfc489","Type":"ContainerStarted","Data":"fa82b981b3e41938e379f44c3f1c26a1a69a7c08b51e2c2f8d66e4490f20921e"}
Nov 03 19:36:57.263004 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:57.262969    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-5gsgr"]
Nov 03 19:36:57.266565 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:57.266542    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-5gsgr"
Nov 03 19:36:57.275461 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:57.275434    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Nov 03 19:36:57.275567 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:57.275487    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-qsh6s\""
Nov 03 19:36:57.278661 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:57.278635    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-5gsgr"]
Nov 03 19:36:57.368442 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:57.368368    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c62v\" (UniqueName: \"kubernetes.io/projected/8ac562db-3ea3-4b98-8189-d758dd0f500a-kube-api-access-4c62v\") pod \"dns-operator-controller-manager-8557cbdb84-5gsgr\" (UID: \"8ac562db-3ea3-4b98-8189-d758dd0f500a\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-5gsgr"
Nov 03 19:36:57.469367 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:57.469317    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c62v\" (UniqueName: \"kubernetes.io/projected/8ac562db-3ea3-4b98-8189-d758dd0f500a-kube-api-access-4c62v\") pod \"dns-operator-controller-manager-8557cbdb84-5gsgr\" (UID: \"8ac562db-3ea3-4b98-8189-d758dd0f500a\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-5gsgr"
Nov 03 19:36:57.482343 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:57.482311    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c62v\" (UniqueName: \"kubernetes.io/projected/8ac562db-3ea3-4b98-8189-d758dd0f500a-kube-api-access-4c62v\") pod \"dns-operator-controller-manager-8557cbdb84-5gsgr\" (UID: \"8ac562db-3ea3-4b98-8189-d758dd0f500a\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-5gsgr"
Nov 03 19:36:57.582009 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:57.581968    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-5gsgr"
Nov 03 19:36:57.876359 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:57.876251    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-5gsgr"]
Nov 03 19:36:57.879160 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:36:57.879126    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ac562db_3ea3_4b98_8189_d758dd0f500a.slice/crio-538cd1b97bf5d5b198736322388f91d7fc59dfb9a917c0e1be38a8a16b4a2258 WatchSource:0}: Error finding container 538cd1b97bf5d5b198736322388f91d7fc59dfb9a917c0e1be38a8a16b4a2258: Status 404 returned error can't find the container with id 538cd1b97bf5d5b198736322388f91d7fc59dfb9a917c0e1be38a8a16b4a2258
Nov 03 19:36:58.719889 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:58.719846    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-5gsgr" event={"ID":"8ac562db-3ea3-4b98-8189-d758dd0f500a","Type":"ContainerStarted","Data":"538cd1b97bf5d5b198736322388f91d7fc59dfb9a917c0e1be38a8a16b4a2258"}
Nov 03 19:36:58.721807 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:58.721778    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-cmrkh" event={"ID":"399879e5-816a-40a3-a6de-5574cecfc489","Type":"ContainerStarted","Data":"11ee121b86681eb9fd7d857203e0d7eb4e4493c1696bbf9a7a094c1bb36ef2fa"}
Nov 03 19:36:58.722023 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:58.721989    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-cmrkh"
Nov 03 19:36:58.754622 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:58.754560    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-cmrkh" podStartSLOduration=1.784812827 podStartE2EDuration="5.754539603s" totalImagesPullingTime="3.969726776s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:36:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:36:58.752412546 +0000 UTC m=+331.996378010" watchObservedRunningTime="2026-11-03 19:36:58.754539603 +0000 UTC m=+331.998505070"
Nov 03 19:36:59.728427 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:59.728389    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-5gsgr" event={"ID":"8ac562db-3ea3-4b98-8189-d758dd0f500a","Type":"ContainerStarted","Data":"59135566fd881a85f3f168d49d3a7e11c80c91f4dcd0336c40038879979cc559"}
Nov 03 19:36:59.728860 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:59.728479    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-5gsgr"
Nov 03 19:36:59.743047 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:36:59.742990    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-5gsgr" podStartSLOduration=1.041384219 podStartE2EDuration="2.742974884s" totalImagesPullingTime="1.701590665s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:36:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:36:59.742017942 +0000 UTC m=+332.985983406" watchObservedRunningTime="2026-11-03 19:36:59.742974884 +0000 UTC m=+332.986940349"
Nov 03 19:37:00.091823 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:00.091786    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-4nlfd"]
Nov 03 19:37:00.095327 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:00.095305    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-4nlfd"
Nov 03 19:37:00.101657 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:00.101626    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-9wdkh\""
Nov 03 19:37:00.104134 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:00.104102    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-4nlfd"]
Nov 03 19:37:00.196675 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:00.196624    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9498c\" (UniqueName: \"kubernetes.io/projected/6c22cc72-92e3-42a2-9a82-ecc3973562f8-kube-api-access-9498c\") pod \"authorino-operator-7b7bb4bc77-4nlfd\" (UID: \"6c22cc72-92e3-42a2-9a82-ecc3973562f8\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-4nlfd"
Nov 03 19:37:00.297517 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:00.297473    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9498c\" (UniqueName: \"kubernetes.io/projected/6c22cc72-92e3-42a2-9a82-ecc3973562f8-kube-api-access-9498c\") pod \"authorino-operator-7b7bb4bc77-4nlfd\" (UID: \"6c22cc72-92e3-42a2-9a82-ecc3973562f8\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-4nlfd"
Nov 03 19:37:00.304849 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:00.304817    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9498c\" (UniqueName: \"kubernetes.io/projected/6c22cc72-92e3-42a2-9a82-ecc3973562f8-kube-api-access-9498c\") pod \"authorino-operator-7b7bb4bc77-4nlfd\" (UID: \"6c22cc72-92e3-42a2-9a82-ecc3973562f8\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-4nlfd"
Nov 03 19:37:00.409864 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:00.409776    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-4nlfd"
Nov 03 19:37:00.692357 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:00.692173    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-4nlfd"]
Nov 03 19:37:00.694850 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:37:00.694821    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c22cc72_92e3_42a2_9a82_ecc3973562f8.slice/crio-31b078aef7ae5466de1a095aa60bb7e25867cbb1c97aa60a9dcaee86d0bdaf84 WatchSource:0}: Error finding container 31b078aef7ae5466de1a095aa60bb7e25867cbb1c97aa60a9dcaee86d0bdaf84: Status 404 returned error can't find the container with id 31b078aef7ae5466de1a095aa60bb7e25867cbb1c97aa60a9dcaee86d0bdaf84
Nov 03 19:37:00.734697 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:00.734634    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-4nlfd" event={"ID":"6c22cc72-92e3-42a2-9a82-ecc3973562f8","Type":"ContainerStarted","Data":"31b078aef7ae5466de1a095aa60bb7e25867cbb1c97aa60a9dcaee86d0bdaf84"}
Nov 03 19:37:01.682200 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:01.682168    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-6sfqz"
Nov 03 19:37:02.748297 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:02.748257    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-4nlfd" event={"ID":"6c22cc72-92e3-42a2-9a82-ecc3973562f8","Type":"ContainerStarted","Data":"862d00c55633ba21f389fdf5aa112e6f3b42574469012218deda1d03ad8ffed1"}
Nov 03 19:37:02.748732 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:02.748389    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-4nlfd"
Nov 03 19:37:02.767935 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:02.767877    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-4nlfd" podStartSLOduration=1.237740054 podStartE2EDuration="2.767862496s" totalImagesPullingTime="1.530122442s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:37:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:37:02.765546453 +0000 UTC m=+336.009511917" watchObservedRunningTime="2026-11-03 19:37:02.767862496 +0000 UTC m=+336.011827960"
Nov 03 19:37:02.919705 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:37:02.919648    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 19:37:08.973108 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:08.973071    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57rcpnr"]
Nov 03 19:37:09.039131 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:09.039101    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57rcpnr"]
Nov 03 19:37:09.039269 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:09.039206    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57rcpnr"
Nov 03 19:37:09.042732 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:09.042705    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-4qmb7\""
Nov 03 19:37:09.074748 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:09.074717    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36946052-eb8d-448d-8907-db1ab6888bf3-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b57rcpnr\" (UID: \"36946052-eb8d-448d-8907-db1ab6888bf3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57rcpnr"
Nov 03 19:37:09.074893 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:09.074771    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36946052-eb8d-448d-8907-db1ab6888bf3-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b57rcpnr\" (UID: \"36946052-eb8d-448d-8907-db1ab6888bf3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57rcpnr"
Nov 03 19:37:09.074893 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:09.074856    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36946052-eb8d-448d-8907-db1ab6888bf3-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b57rcpnr\" (UID: \"36946052-eb8d-448d-8907-db1ab6888bf3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57rcpnr"
Nov 03 19:37:09.074998 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:09.074892    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36946052-eb8d-448d-8907-db1ab6888bf3-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b57rcpnr\" (UID: \"36946052-eb8d-448d-8907-db1ab6888bf3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57rcpnr"
Nov 03 19:37:09.074998 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:09.074923    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tx8s\" (UniqueName: \"kubernetes.io/projected/36946052-eb8d-448d-8907-db1ab6888bf3-kube-api-access-5tx8s\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b57rcpnr\" (UID: \"36946052-eb8d-448d-8907-db1ab6888bf3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57rcpnr"
Nov 03 19:37:09.074998 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:09.074962    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36946052-eb8d-448d-8907-db1ab6888bf3-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b57rcpnr\" (UID: \"36946052-eb8d-448d-8907-db1ab6888bf3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57rcpnr"
Nov 03 19:37:09.074998 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:09.074990    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36946052-eb8d-448d-8907-db1ab6888bf3-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b57rcpnr\" (UID: \"36946052-eb8d-448d-8907-db1ab6888bf3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57rcpnr"
Nov 03 19:37:09.075161 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:09.075075    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36946052-eb8d-448d-8907-db1ab6888bf3-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b57rcpnr\" (UID: \"36946052-eb8d-448d-8907-db1ab6888bf3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57rcpnr"
Nov 03 19:37:09.075161 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:09.075109    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36946052-eb8d-448d-8907-db1ab6888bf3-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b57rcpnr\" (UID: \"36946052-eb8d-448d-8907-db1ab6888bf3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57rcpnr"
Nov 03 19:37:09.175976 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:09.175936    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36946052-eb8d-448d-8907-db1ab6888bf3-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b57rcpnr\" (UID: \"36946052-eb8d-448d-8907-db1ab6888bf3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57rcpnr"
Nov 03 19:37:09.175976 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:09.175984    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36946052-eb8d-448d-8907-db1ab6888bf3-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b57rcpnr\" (UID: \"36946052-eb8d-448d-8907-db1ab6888bf3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57rcpnr"
Nov 03 19:37:09.176232 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:09.176021    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36946052-eb8d-448d-8907-db1ab6888bf3-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b57rcpnr\" (UID: \"36946052-eb8d-448d-8907-db1ab6888bf3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57rcpnr"
Nov 03 19:37:09.176232 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:09.176091    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36946052-eb8d-448d-8907-db1ab6888bf3-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b57rcpnr\" (UID: \"36946052-eb8d-448d-8907-db1ab6888bf3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57rcpnr"
Nov 03 19:37:09.176232 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:09.176204    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36946052-eb8d-448d-8907-db1ab6888bf3-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b57rcpnr\" (UID: \"36946052-eb8d-448d-8907-db1ab6888bf3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57rcpnr"
Nov 03 19:37:09.176380 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:09.176234    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36946052-eb8d-448d-8907-db1ab6888bf3-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b57rcpnr\" (UID: \"36946052-eb8d-448d-8907-db1ab6888bf3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57rcpnr"
Nov 03 19:37:09.176380 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:09.176262    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tx8s\" (UniqueName: \"kubernetes.io/projected/36946052-eb8d-448d-8907-db1ab6888bf3-kube-api-access-5tx8s\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b57rcpnr\" (UID: \"36946052-eb8d-448d-8907-db1ab6888bf3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57rcpnr"
Nov 03 19:37:09.176380 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:09.176294    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36946052-eb8d-448d-8907-db1ab6888bf3-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b57rcpnr\" (UID: \"36946052-eb8d-448d-8907-db1ab6888bf3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57rcpnr"
Nov 03 19:37:09.176380 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:09.176331    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36946052-eb8d-448d-8907-db1ab6888bf3-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b57rcpnr\" (UID: \"36946052-eb8d-448d-8907-db1ab6888bf3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57rcpnr"
Nov 03 19:37:09.176578 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:09.176408    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36946052-eb8d-448d-8907-db1ab6888bf3-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b57rcpnr\" (UID: \"36946052-eb8d-448d-8907-db1ab6888bf3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57rcpnr"
Nov 03 19:37:09.176578 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:09.176437    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36946052-eb8d-448d-8907-db1ab6888bf3-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b57rcpnr\" (UID: \"36946052-eb8d-448d-8907-db1ab6888bf3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57rcpnr"
Nov 03 19:37:09.176720 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:09.176661    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36946052-eb8d-448d-8907-db1ab6888bf3-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b57rcpnr\" (UID: \"36946052-eb8d-448d-8907-db1ab6888bf3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57rcpnr"
Nov 03 19:37:09.176720 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:09.176656    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36946052-eb8d-448d-8907-db1ab6888bf3-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b57rcpnr\" (UID: \"36946052-eb8d-448d-8907-db1ab6888bf3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57rcpnr"
Nov 03 19:37:09.176864 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:09.176843    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36946052-eb8d-448d-8907-db1ab6888bf3-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b57rcpnr\" (UID: \"36946052-eb8d-448d-8907-db1ab6888bf3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57rcpnr"
Nov 03 19:37:09.178586 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:09.178559    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36946052-eb8d-448d-8907-db1ab6888bf3-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b57rcpnr\" (UID: \"36946052-eb8d-448d-8907-db1ab6888bf3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57rcpnr"
Nov 03 19:37:09.178797 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:09.178781    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36946052-eb8d-448d-8907-db1ab6888bf3-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b57rcpnr\" (UID: \"36946052-eb8d-448d-8907-db1ab6888bf3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57rcpnr"
Nov 03 19:37:09.183397 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:09.183376    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tx8s\" (UniqueName: \"kubernetes.io/projected/36946052-eb8d-448d-8907-db1ab6888bf3-kube-api-access-5tx8s\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b57rcpnr\" (UID: \"36946052-eb8d-448d-8907-db1ab6888bf3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57rcpnr"
Nov 03 19:37:09.183722 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:09.183703    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36946052-eb8d-448d-8907-db1ab6888bf3-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b57rcpnr\" (UID: \"36946052-eb8d-448d-8907-db1ab6888bf3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57rcpnr"
Nov 03 19:37:09.377900 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:09.377864    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57rcpnr"
Nov 03 19:37:09.660903 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:09.660878    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57rcpnr"]
Nov 03 19:37:09.663125 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:37:09.663097    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36946052_eb8d_448d_8907_db1ab6888bf3.slice/crio-282c989272e3a28cfb0d3c45b4471c799c22491527b4784b3b3b8660d5965829 WatchSource:0}: Error finding container 282c989272e3a28cfb0d3c45b4471c799c22491527b4784b3b3b8660d5965829: Status 404 returned error can't find the container with id 282c989272e3a28cfb0d3c45b4471c799c22491527b4784b3b3b8660d5965829
Nov 03 19:37:09.665759 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:09.665722    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:37:09.665819 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:09.665808    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:37:09.665870 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:09.665849    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:37:09.730417 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:09.730387    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-cmrkh"
Nov 03 19:37:09.788660 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:09.788619    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57rcpnr" event={"ID":"36946052-eb8d-448d-8907-db1ab6888bf3","Type":"ContainerStarted","Data":"a8f05ef064c2f66e23d98a529692ba94fae3e9f025de1de7d6863210cdc1b96a"}
Nov 03 19:37:09.788861 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:09.788696    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57rcpnr" event={"ID":"36946052-eb8d-448d-8907-db1ab6888bf3","Type":"ContainerStarted","Data":"282c989272e3a28cfb0d3c45b4471c799c22491527b4784b3b3b8660d5965829"}
Nov 03 19:37:09.811034 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:09.810977    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57rcpnr" podStartSLOduration=1.8109579340000002 podStartE2EDuration="1.810957934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:37:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:37:09.809494715 +0000 UTC m=+343.053460191" watchObservedRunningTime="2026-11-03 19:37:09.810957934 +0000 UTC m=+343.054923402"
Nov 03 19:37:10.378930 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:10.378895    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57rcpnr"
Nov 03 19:37:10.383557 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:10.383536    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57rcpnr"
Nov 03 19:37:10.737679 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:10.737639    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-5gsgr"
Nov 03 19:37:10.794682 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:10.794637    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57rcpnr"
Nov 03 19:37:10.795568 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:10.795543    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57rcpnr"
Nov 03 19:37:12.669180 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:12.669144    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-9dhcg"]
Nov 03 19:37:12.671678 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:12.671650    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9dhcg"
Nov 03 19:37:12.674609 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:12.674587    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 03 19:37:12.675295 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:12.675278    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 03 19:37:12.675295 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:12.675289    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-7jvn9\""
Nov 03 19:37:12.676975 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:12.676945    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\""
Nov 03 19:37:12.686098 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:12.685690    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-9dhcg"]
Nov 03 19:37:12.706719 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:12.706687    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4a679f4b-2bbf-42b8-96a1-a3b6469a43ce-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-9dhcg\" (UID: \"4a679f4b-2bbf-42b8-96a1-a3b6469a43ce\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9dhcg"
Nov 03 19:37:12.706868 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:12.706821    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4a679f4b-2bbf-42b8-96a1-a3b6469a43ce-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-9dhcg\" (UID: \"4a679f4b-2bbf-42b8-96a1-a3b6469a43ce\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9dhcg"
Nov 03 19:37:12.706868 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:12.706861    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4a679f4b-2bbf-42b8-96a1-a3b6469a43ce-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-9dhcg\" (UID: \"4a679f4b-2bbf-42b8-96a1-a3b6469a43ce\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9dhcg"
Nov 03 19:37:12.706956 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:12.706885    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ct5jq\" (UniqueName: \"kubernetes.io/projected/4a679f4b-2bbf-42b8-96a1-a3b6469a43ce-kube-api-access-ct5jq\") pod \"kserve-module-controller-manager-7566cf5fc8-9dhcg\" (UID: \"4a679f4b-2bbf-42b8-96a1-a3b6469a43ce\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9dhcg"
Nov 03 19:37:12.807881 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:12.807847    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4a679f4b-2bbf-42b8-96a1-a3b6469a43ce-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-9dhcg\" (UID: \"4a679f4b-2bbf-42b8-96a1-a3b6469a43ce\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9dhcg"
Nov 03 19:37:12.808063 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:12.807928    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4a679f4b-2bbf-42b8-96a1-a3b6469a43ce-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-9dhcg\" (UID: \"4a679f4b-2bbf-42b8-96a1-a3b6469a43ce\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9dhcg"
Nov 03 19:37:12.808063 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:12.807953    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4a679f4b-2bbf-42b8-96a1-a3b6469a43ce-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-9dhcg\" (UID: \"4a679f4b-2bbf-42b8-96a1-a3b6469a43ce\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9dhcg"
Nov 03 19:37:12.808063 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:12.807983    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct5jq\" (UniqueName: \"kubernetes.io/projected/4a679f4b-2bbf-42b8-96a1-a3b6469a43ce-kube-api-access-ct5jq\") pod \"kserve-module-controller-manager-7566cf5fc8-9dhcg\" (UID: \"4a679f4b-2bbf-42b8-96a1-a3b6469a43ce\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9dhcg"
Nov 03 19:37:12.808277 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:12.808254    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4a679f4b-2bbf-42b8-96a1-a3b6469a43ce-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-9dhcg\" (UID: \"4a679f4b-2bbf-42b8-96a1-a3b6469a43ce\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9dhcg"
Nov 03 19:37:12.808403 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:12.808278    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4a679f4b-2bbf-42b8-96a1-a3b6469a43ce-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-9dhcg\" (UID: \"4a679f4b-2bbf-42b8-96a1-a3b6469a43ce\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9dhcg"
Nov 03 19:37:12.808836 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:12.808813    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4a679f4b-2bbf-42b8-96a1-a3b6469a43ce-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-9dhcg\" (UID: \"4a679f4b-2bbf-42b8-96a1-a3b6469a43ce\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9dhcg"
Nov 03 19:37:12.815122 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:12.815101    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct5jq\" (UniqueName: \"kubernetes.io/projected/4a679f4b-2bbf-42b8-96a1-a3b6469a43ce-kube-api-access-ct5jq\") pod \"kserve-module-controller-manager-7566cf5fc8-9dhcg\" (UID: \"4a679f4b-2bbf-42b8-96a1-a3b6469a43ce\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9dhcg"
Nov 03 19:37:12.991298 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:12.991264    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9dhcg"
Nov 03 19:37:13.267463 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:13.267408    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-9dhcg"]
Nov 03 19:37:13.269936 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:37:13.269910    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a679f4b_2bbf_42b8_96a1_a3b6469a43ce.slice/crio-d00ea47eb72be782fde3497dc19fc4963a06edb4b7241560abd58b1fc123a6b5 WatchSource:0}: Error finding container d00ea47eb72be782fde3497dc19fc4963a06edb4b7241560abd58b1fc123a6b5: Status 404 returned error can't find the container with id d00ea47eb72be782fde3497dc19fc4963a06edb4b7241560abd58b1fc123a6b5
Nov 03 19:37:13.756591 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:13.756549    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-4nlfd"
Nov 03 19:37:13.812971 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:13.812934    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9dhcg" event={"ID":"4a679f4b-2bbf-42b8-96a1-a3b6469a43ce","Type":"ContainerStarted","Data":"d00ea47eb72be782fde3497dc19fc4963a06edb4b7241560abd58b1fc123a6b5"}
Nov 03 19:37:15.825133 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:15.825050    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9dhcg" event={"ID":"4a679f4b-2bbf-42b8-96a1-a3b6469a43ce","Type":"ContainerStarted","Data":"3e33e87624fd1cf10f690c18cda9f1f78a1766f779c55456847abce4930101e4"}
Nov 03 19:37:15.825520 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:15.825190    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9dhcg"
Nov 03 19:37:15.844959 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:15.844903    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9dhcg" podStartSLOduration=1.586423833 podStartE2EDuration="3.84488901s" totalImagesPullingTime="2.258465177s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:37:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:37:15.842619306 +0000 UTC m=+349.086584772" watchObservedRunningTime="2026-11-03 19:37:15.84488901 +0000 UTC m=+349.088854475"
Nov 03 19:37:26.911079 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:26.911035    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-cmrkh"]
Nov 03 19:37:26.911705 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:26.911302    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-cmrkh" podUID="399879e5-816a-40a3-a6de-5574cecfc489" containerName="manager" containerID="cri-o://11ee121b86681eb9fd7d857203e0d7eb4e4493c1696bbf9a7a094c1bb36ef2fa" gracePeriod=10
Nov 03 19:37:27.170822 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:27.170762    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-cmrkh"
Nov 03 19:37:27.234714 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:27.234654    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/399879e5-816a-40a3-a6de-5574cecfc489-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/399879e5-816a-40a3-a6de-5574cecfc489-extensions-socket-volume\") pod \"399879e5-816a-40a3-a6de-5574cecfc489\" (UID: \"399879e5-816a-40a3-a6de-5574cecfc489\") "
Nov 03 19:37:27.234886 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:27.234749    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/399879e5-816a-40a3-a6de-5574cecfc489-kube-api-access-jlhm8\" (UniqueName: \"kubernetes.io/projected/399879e5-816a-40a3-a6de-5574cecfc489-kube-api-access-jlhm8\") pod \"399879e5-816a-40a3-a6de-5574cecfc489\" (UID: \"399879e5-816a-40a3-a6de-5574cecfc489\") "
Nov 03 19:37:27.235047 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:27.235023    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/399879e5-816a-40a3-a6de-5574cecfc489-extensions-socket-volume" pod "399879e5-816a-40a3-a6de-5574cecfc489" (UID: "399879e5-816a-40a3-a6de-5574cecfc489"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:37:27.237035 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:27.237006    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/399879e5-816a-40a3-a6de-5574cecfc489-kube-api-access-jlhm8" pod "399879e5-816a-40a3-a6de-5574cecfc489" (UID: "399879e5-816a-40a3-a6de-5574cecfc489"). InnerVolumeSpecName "kube-api-access-jlhm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:37:27.337799 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:27.337730    4921 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/399879e5-816a-40a3-a6de-5574cecfc489-extensions-socket-volume\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:27.337799 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:27.337771    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlhm8\" (UniqueName: \"kubernetes.io/projected/399879e5-816a-40a3-a6de-5574cecfc489-kube-api-access-jlhm8\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:27.890571 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:27.890532    4921 generic.go:356] "Generic (PLEG): container finished" podID="399879e5-816a-40a3-a6de-5574cecfc489" containerID="11ee121b86681eb9fd7d857203e0d7eb4e4493c1696bbf9a7a094c1bb36ef2fa" exitCode=0
Nov 03 19:37:27.890760 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:27.890580    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-cmrkh"
Nov 03 19:37:27.890760 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:27.890614    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-cmrkh" event={"ID":"399879e5-816a-40a3-a6de-5574cecfc489","Type":"ContainerDied","Data":"11ee121b86681eb9fd7d857203e0d7eb4e4493c1696bbf9a7a094c1bb36ef2fa"}
Nov 03 19:37:27.890760 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:27.890694    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-cmrkh" event={"ID":"399879e5-816a-40a3-a6de-5574cecfc489","Type":"ContainerDied","Data":"fa82b981b3e41938e379f44c3f1c26a1a69a7c08b51e2c2f8d66e4490f20921e"}
Nov 03 19:37:27.890760 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:27.890721    4921 scope.go:122] "RemoveContainer" containerID="11ee121b86681eb9fd7d857203e0d7eb4e4493c1696bbf9a7a094c1bb36ef2fa"
Nov 03 19:37:27.900331 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:27.900307    4921 scope.go:122] "RemoveContainer" containerID="11ee121b86681eb9fd7d857203e0d7eb4e4493c1696bbf9a7a094c1bb36ef2fa"
Nov 03 19:37:27.900591 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:37:27.900575    4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11ee121b86681eb9fd7d857203e0d7eb4e4493c1696bbf9a7a094c1bb36ef2fa\": container with ID starting with 11ee121b86681eb9fd7d857203e0d7eb4e4493c1696bbf9a7a094c1bb36ef2fa not found: ID does not exist" containerID="11ee121b86681eb9fd7d857203e0d7eb4e4493c1696bbf9a7a094c1bb36ef2fa"
Nov 03 19:37:27.900654 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:27.900600    4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11ee121b86681eb9fd7d857203e0d7eb4e4493c1696bbf9a7a094c1bb36ef2fa"} err="failed to get container status \"11ee121b86681eb9fd7d857203e0d7eb4e4493c1696bbf9a7a094c1bb36ef2fa\": rpc error: code = NotFound desc = could not find container \"11ee121b86681eb9fd7d857203e0d7eb4e4493c1696bbf9a7a094c1bb36ef2fa\": container with ID starting with 11ee121b86681eb9fd7d857203e0d7eb4e4493c1696bbf9a7a094c1bb36ef2fa not found: ID does not exist"
Nov 03 19:37:27.910713 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:27.910681    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-cmrkh"]
Nov 03 19:37:27.915803 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:27.915774    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-cmrkh"]
Nov 03 19:37:28.893800 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:28.893766    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="399879e5-816a-40a3-a6de-5574cecfc489" path="/var/lib/kubelet/pods/399879e5-816a-40a3-a6de-5574cecfc489/volumes"
Nov 03 19:37:32.920827 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:37:32.920793    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 19:37:36.832020 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:36.831985    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9dhcg"
Nov 03 19:37:43.643232 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:43.643196    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-h2bzs"]
Nov 03 19:37:43.643619 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:43.643573    4921 cpu_manager.go:414] "containerMap: removing container" podUID="399879e5-816a-40a3-a6de-5574cecfc489" containerName="manager"
Nov 03 19:37:43.643619 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:43.643584    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="399879e5-816a-40a3-a6de-5574cecfc489" containerName="manager"
Nov 03 19:37:43.643712 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:43.643644    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="399879e5-816a-40a3-a6de-5574cecfc489" containerName="manager"
Nov 03 19:37:43.646746 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:43.646729    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-h2bzs"
Nov 03 19:37:43.652135 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:43.652112    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Nov 03 19:37:43.652713 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:43.652695    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-j2gnf\""
Nov 03 19:37:43.656289 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:43.656263    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-h2bzs"]
Nov 03 19:37:43.735255 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:43.735220    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-h2bzs"]
Nov 03 19:37:43.781824 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:43.781784    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgm5p\" (UniqueName: \"kubernetes.io/projected/8f1bb47f-e0d7-410e-a09c-0f88ba6d138a-kube-api-access-rgm5p\") pod \"limitador-limitador-74dfcfb894-h2bzs\" (UID: \"8f1bb47f-e0d7-410e-a09c-0f88ba6d138a\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-h2bzs"
Nov 03 19:37:43.781999 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:43.781841    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8f1bb47f-e0d7-410e-a09c-0f88ba6d138a-config-file\") pod \"limitador-limitador-74dfcfb894-h2bzs\" (UID: \"8f1bb47f-e0d7-410e-a09c-0f88ba6d138a\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-h2bzs"
Nov 03 19:37:43.882641 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:43.882601    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgm5p\" (UniqueName: \"kubernetes.io/projected/8f1bb47f-e0d7-410e-a09c-0f88ba6d138a-kube-api-access-rgm5p\") pod \"limitador-limitador-74dfcfb894-h2bzs\" (UID: \"8f1bb47f-e0d7-410e-a09c-0f88ba6d138a\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-h2bzs"
Nov 03 19:37:43.882820 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:43.882654    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8f1bb47f-e0d7-410e-a09c-0f88ba6d138a-config-file\") pod \"limitador-limitador-74dfcfb894-h2bzs\" (UID: \"8f1bb47f-e0d7-410e-a09c-0f88ba6d138a\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-h2bzs"
Nov 03 19:37:43.883378 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:43.883358    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8f1bb47f-e0d7-410e-a09c-0f88ba6d138a-config-file\") pod \"limitador-limitador-74dfcfb894-h2bzs\" (UID: \"8f1bb47f-e0d7-410e-a09c-0f88ba6d138a\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-h2bzs"
Nov 03 19:37:43.890447 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:43.890418    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgm5p\" (UniqueName: \"kubernetes.io/projected/8f1bb47f-e0d7-410e-a09c-0f88ba6d138a-kube-api-access-rgm5p\") pod \"limitador-limitador-74dfcfb894-h2bzs\" (UID: \"8f1bb47f-e0d7-410e-a09c-0f88ba6d138a\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-h2bzs"
Nov 03 19:37:43.961211 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:43.961116    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-h2bzs"
Nov 03 19:37:44.243992 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:44.243948    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-h2bzs"]
Nov 03 19:37:44.246485 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:37:44.246450    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f1bb47f_e0d7_410e_a09c_0f88ba6d138a.slice/crio-604113f6794a2fc8365fb347d07631974f42fb0ed9636423a9a6dcd7c6a17e94 WatchSource:0}: Error finding container 604113f6794a2fc8365fb347d07631974f42fb0ed9636423a9a6dcd7c6a17e94: Status 404 returned error can't find the container with id 604113f6794a2fc8365fb347d07631974f42fb0ed9636423a9a6dcd7c6a17e94
Nov 03 19:37:44.410416 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:44.410385    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-d4wgr"]
Nov 03 19:37:44.414948 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:44.414926    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-d4wgr"
Nov 03 19:37:44.418647 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:44.418614    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-gnxfw\""
Nov 03 19:37:44.421334 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:44.421308    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-d4wgr"]
Nov 03 19:37:44.488258 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:44.488218    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thln2\" (UniqueName: \"kubernetes.io/projected/39d57b35-653d-4314-8de9-c9bbfbfefb16-kube-api-access-thln2\") pod \"authorino-57f7f8659f-d4wgr\" (UID: \"39d57b35-653d-4314-8de9-c9bbfbfefb16\") " pod="kuadrant-system/authorino-57f7f8659f-d4wgr"
Nov 03 19:37:44.589610 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:44.589530    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thln2\" (UniqueName: \"kubernetes.io/projected/39d57b35-653d-4314-8de9-c9bbfbfefb16-kube-api-access-thln2\") pod \"authorino-57f7f8659f-d4wgr\" (UID: \"39d57b35-653d-4314-8de9-c9bbfbfefb16\") " pod="kuadrant-system/authorino-57f7f8659f-d4wgr"
Nov 03 19:37:44.595405 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:44.595357    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-fzncl"]
Nov 03 19:37:44.598106 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:44.598066    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thln2\" (UniqueName: \"kubernetes.io/projected/39d57b35-653d-4314-8de9-c9bbfbfefb16-kube-api-access-thln2\") pod \"authorino-57f7f8659f-d4wgr\" (UID: \"39d57b35-653d-4314-8de9-c9bbfbfefb16\") " pod="kuadrant-system/authorino-57f7f8659f-d4wgr"
Nov 03 19:37:44.598834 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:44.598818    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-fzncl"
Nov 03 19:37:44.605466 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:44.605439    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-fzncl"]
Nov 03 19:37:44.690842 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:44.690808    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lwmv\" (UniqueName: \"kubernetes.io/projected/3d8cf63c-9dfa-45f9-86b8-13aa9e14e726-kube-api-access-4lwmv\") pod \"authorino-56c6c9ddd5-fzncl\" (UID: \"3d8cf63c-9dfa-45f9-86b8-13aa9e14e726\") " pod="kuadrant-system/authorino-56c6c9ddd5-fzncl"
Nov 03 19:37:44.728843 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:44.728800    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-d4wgr"
Nov 03 19:37:44.792261 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:44.792230    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lwmv\" (UniqueName: \"kubernetes.io/projected/3d8cf63c-9dfa-45f9-86b8-13aa9e14e726-kube-api-access-4lwmv\") pod \"authorino-56c6c9ddd5-fzncl\" (UID: \"3d8cf63c-9dfa-45f9-86b8-13aa9e14e726\") " pod="kuadrant-system/authorino-56c6c9ddd5-fzncl"
Nov 03 19:37:44.801183 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:44.801155    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lwmv\" (UniqueName: \"kubernetes.io/projected/3d8cf63c-9dfa-45f9-86b8-13aa9e14e726-kube-api-access-4lwmv\") pod \"authorino-56c6c9ddd5-fzncl\" (UID: \"3d8cf63c-9dfa-45f9-86b8-13aa9e14e726\") " pod="kuadrant-system/authorino-56c6c9ddd5-fzncl"
Nov 03 19:37:44.917489 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:44.917405    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-fzncl"
Nov 03 19:37:44.989696 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:44.989639    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-h2bzs" event={"ID":"8f1bb47f-e0d7-410e-a09c-0f88ba6d138a","Type":"ContainerStarted","Data":"604113f6794a2fc8365fb347d07631974f42fb0ed9636423a9a6dcd7c6a17e94"}
Nov 03 19:37:45.004870 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:45.004843    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-d4wgr"]
Nov 03 19:37:45.006854 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:37:45.006823    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod39d57b35_653d_4314_8de9_c9bbfbfefb16.slice/crio-a18f3d3a0f08a6cffdded98b4f2201dc590572db60273df627b578876e40c1b2 WatchSource:0}: Error finding container a18f3d3a0f08a6cffdded98b4f2201dc590572db60273df627b578876e40c1b2: Status 404 returned error can't find the container with id a18f3d3a0f08a6cffdded98b4f2201dc590572db60273df627b578876e40c1b2
Nov 03 19:37:45.190442 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:45.190416    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-fzncl"]
Nov 03 19:37:45.192824 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:37:45.192793    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d8cf63c_9dfa_45f9_86b8_13aa9e14e726.slice/crio-dd71aef82c33fe8b8b0766091dae6335f13542ccffb61168ab1aa792d6386c82 WatchSource:0}: Error finding container dd71aef82c33fe8b8b0766091dae6335f13542ccffb61168ab1aa792d6386c82: Status 404 returned error can't find the container with id dd71aef82c33fe8b8b0766091dae6335f13542ccffb61168ab1aa792d6386c82
Nov 03 19:37:45.996299 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:45.996244    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-h2bzs" event={"ID":"8f1bb47f-e0d7-410e-a09c-0f88ba6d138a","Type":"ContainerStarted","Data":"60cff7a588d857d2092c9ab59b1b452f46aa5860d5e6e5df2ae0c05f3808089b"}
Nov 03 19:37:45.996792 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:45.996456    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-h2bzs"
Nov 03 19:37:45.997482 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:45.997456    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-d4wgr" event={"ID":"39d57b35-653d-4314-8de9-c9bbfbfefb16","Type":"ContainerStarted","Data":"a18f3d3a0f08a6cffdded98b4f2201dc590572db60273df627b578876e40c1b2"}
Nov 03 19:37:45.998559 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:45.998535    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-fzncl" event={"ID":"3d8cf63c-9dfa-45f9-86b8-13aa9e14e726","Type":"ContainerStarted","Data":"dd71aef82c33fe8b8b0766091dae6335f13542ccffb61168ab1aa792d6386c82"}
Nov 03 19:37:46.013010 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:46.012904    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-h2bzs" podStartSLOduration=1.507299106 podStartE2EDuration="3.012886759s" totalImagesPullingTime="1.505587653s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:37:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:37:46.011868647 +0000 UTC m=+379.255834113" watchObservedRunningTime="2026-11-03 19:37:46.012886759 +0000 UTC m=+379.256852228"
Nov 03 19:37:48.012979 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:48.012936    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-fzncl" event={"ID":"3d8cf63c-9dfa-45f9-86b8-13aa9e14e726","Type":"ContainerStarted","Data":"fcacf9d47864fdc0649da177d11606731bb127848a103b058a96643414170cdb"}
Nov 03 19:37:48.014509 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:48.014482    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-d4wgr" event={"ID":"39d57b35-653d-4314-8de9-c9bbfbfefb16","Type":"ContainerStarted","Data":"80e0eb2bf8a86ca9c2dc2619a63979c749c4344c43bbf0d9d9fc0fafd5bfe280"}
Nov 03 19:37:48.025371 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:48.025320    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-fzncl" podStartSLOduration=1.825585717 podStartE2EDuration="4.02530268s" totalImagesPullingTime="2.199716963s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:37:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:37:48.024225843 +0000 UTC m=+381.268191307" watchObservedRunningTime="2026-11-03 19:37:48.02530268 +0000 UTC m=+381.269268145"
Nov 03 19:37:48.040230 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:48.040176    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-d4wgr" podStartSLOduration=1.605993147 podStartE2EDuration="4.04015956s" totalImagesPullingTime="2.434166413s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:37:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:37:48.036253891 +0000 UTC m=+381.280219358" watchObservedRunningTime="2026-11-03 19:37:48.04015956 +0000 UTC m=+381.284125024"
Nov 03 19:37:48.045357 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:48.045320    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-d4wgr"]
Nov 03 19:37:50.025136 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:50.025099    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-d4wgr" podUID="39d57b35-653d-4314-8de9-c9bbfbfefb16" containerName="authorino" containerID="cri-o://80e0eb2bf8a86ca9c2dc2619a63979c749c4344c43bbf0d9d9fc0fafd5bfe280" gracePeriod=30
Nov 03 19:37:50.278896 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:50.278830    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-d4wgr"
Nov 03 19:37:50.441284 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:50.441235    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39d57b35-653d-4314-8de9-c9bbfbfefb16-kube-api-access-thln2\" (UniqueName: \"kubernetes.io/projected/39d57b35-653d-4314-8de9-c9bbfbfefb16-kube-api-access-thln2\") pod \"39d57b35-653d-4314-8de9-c9bbfbfefb16\" (UID: \"39d57b35-653d-4314-8de9-c9bbfbfefb16\") "
Nov 03 19:37:50.443282 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:50.443251    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39d57b35-653d-4314-8de9-c9bbfbfefb16-kube-api-access-thln2" pod "39d57b35-653d-4314-8de9-c9bbfbfefb16" (UID: "39d57b35-653d-4314-8de9-c9bbfbfefb16"). InnerVolumeSpecName "kube-api-access-thln2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:37:50.542107 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:50.542029    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thln2\" (UniqueName: \"kubernetes.io/projected/39d57b35-653d-4314-8de9-c9bbfbfefb16-kube-api-access-thln2\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:51.032821 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:51.032788    4921 generic.go:356] "Generic (PLEG): container finished" podID="39d57b35-653d-4314-8de9-c9bbfbfefb16" containerID="80e0eb2bf8a86ca9c2dc2619a63979c749c4344c43bbf0d9d9fc0fafd5bfe280" exitCode=0
Nov 03 19:37:51.033305 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:51.032843    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-d4wgr"
Nov 03 19:37:51.033305 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:51.032875    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-d4wgr" event={"ID":"39d57b35-653d-4314-8de9-c9bbfbfefb16","Type":"ContainerDied","Data":"80e0eb2bf8a86ca9c2dc2619a63979c749c4344c43bbf0d9d9fc0fafd5bfe280"}
Nov 03 19:37:51.033305 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:51.032931    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-d4wgr" event={"ID":"39d57b35-653d-4314-8de9-c9bbfbfefb16","Type":"ContainerDied","Data":"a18f3d3a0f08a6cffdded98b4f2201dc590572db60273df627b578876e40c1b2"}
Nov 03 19:37:51.033305 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:51.032952    4921 scope.go:122] "RemoveContainer" containerID="80e0eb2bf8a86ca9c2dc2619a63979c749c4344c43bbf0d9d9fc0fafd5bfe280"
Nov 03 19:37:51.042124 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:51.042106    4921 scope.go:122] "RemoveContainer" containerID="80e0eb2bf8a86ca9c2dc2619a63979c749c4344c43bbf0d9d9fc0fafd5bfe280"
Nov 03 19:37:51.042474 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:37:51.042452    4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80e0eb2bf8a86ca9c2dc2619a63979c749c4344c43bbf0d9d9fc0fafd5bfe280\": container with ID starting with 80e0eb2bf8a86ca9c2dc2619a63979c749c4344c43bbf0d9d9fc0fafd5bfe280 not found: ID does not exist" containerID="80e0eb2bf8a86ca9c2dc2619a63979c749c4344c43bbf0d9d9fc0fafd5bfe280"
Nov 03 19:37:51.042584 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:51.042480    4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80e0eb2bf8a86ca9c2dc2619a63979c749c4344c43bbf0d9d9fc0fafd5bfe280"} err="failed to get container status \"80e0eb2bf8a86ca9c2dc2619a63979c749c4344c43bbf0d9d9fc0fafd5bfe280\": rpc error: code = NotFound desc = could not find container \"80e0eb2bf8a86ca9c2dc2619a63979c749c4344c43bbf0d9d9fc0fafd5bfe280\": container with ID starting with 80e0eb2bf8a86ca9c2dc2619a63979c749c4344c43bbf0d9d9fc0fafd5bfe280 not found: ID does not exist"
Nov 03 19:37:51.046636 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:51.046606    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-d4wgr"]
Nov 03 19:37:51.048732 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:51.048709    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-d4wgr"]
Nov 03 19:37:52.886079 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:52.886045    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39d57b35-653d-4314-8de9-c9bbfbfefb16" path="/var/lib/kubelet/pods/39d57b35-653d-4314-8de9-c9bbfbfefb16/volumes"
Nov 03 19:37:57.007438 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:57.007409    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-h2bzs"
Nov 03 19:37:59.026529 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:59.026475    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-h2bzs"]
Nov 03 19:37:59.027097 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:59.026793    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-h2bzs" podUID="8f1bb47f-e0d7-410e-a09c-0f88ba6d138a" containerName="limitador" containerID="cri-o://60cff7a588d857d2092c9ab59b1b452f46aa5860d5e6e5df2ae0c05f3808089b" gracePeriod=30
Nov 03 19:37:59.582218 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:59.582188    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-h2bzs"
Nov 03 19:37:59.728183 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:59.728150    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f1bb47f-e0d7-410e-a09c-0f88ba6d138a-kube-api-access-rgm5p\" (UniqueName: \"kubernetes.io/projected/8f1bb47f-e0d7-410e-a09c-0f88ba6d138a-kube-api-access-rgm5p\") pod \"8f1bb47f-e0d7-410e-a09c-0f88ba6d138a\" (UID: \"8f1bb47f-e0d7-410e-a09c-0f88ba6d138a\") "
Nov 03 19:37:59.728379 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:59.728220    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f1bb47f-e0d7-410e-a09c-0f88ba6d138a-config-file\" (UniqueName: \"kubernetes.io/configmap/8f1bb47f-e0d7-410e-a09c-0f88ba6d138a-config-file\") pod \"8f1bb47f-e0d7-410e-a09c-0f88ba6d138a\" (UID: \"8f1bb47f-e0d7-410e-a09c-0f88ba6d138a\") "
Nov 03 19:37:59.728563 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:59.728539    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f1bb47f-e0d7-410e-a09c-0f88ba6d138a-config-file" pod "8f1bb47f-e0d7-410e-a09c-0f88ba6d138a" (UID: "8f1bb47f-e0d7-410e-a09c-0f88ba6d138a"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:37:59.730300 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:59.730281    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f1bb47f-e0d7-410e-a09c-0f88ba6d138a-kube-api-access-rgm5p" pod "8f1bb47f-e0d7-410e-a09c-0f88ba6d138a" (UID: "8f1bb47f-e0d7-410e-a09c-0f88ba6d138a"). InnerVolumeSpecName "kube-api-access-rgm5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:37:59.829020 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:59.828984    4921 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8f1bb47f-e0d7-410e-a09c-0f88ba6d138a-config-file\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:59.829020 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:37:59.829016    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgm5p\" (UniqueName: \"kubernetes.io/projected/8f1bb47f-e0d7-410e-a09c-0f88ba6d138a-kube-api-access-rgm5p\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:38:00.083632 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:00.083538    4921 generic.go:356] "Generic (PLEG): container finished" podID="8f1bb47f-e0d7-410e-a09c-0f88ba6d138a" containerID="60cff7a588d857d2092c9ab59b1b452f46aa5860d5e6e5df2ae0c05f3808089b" exitCode=0
Nov 03 19:38:00.083632 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:00.083583    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-h2bzs"
Nov 03 19:38:00.084150 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:00.083589    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-h2bzs" event={"ID":"8f1bb47f-e0d7-410e-a09c-0f88ba6d138a","Type":"ContainerDied","Data":"60cff7a588d857d2092c9ab59b1b452f46aa5860d5e6e5df2ae0c05f3808089b"}
Nov 03 19:38:00.084150 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:00.083704    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-h2bzs" event={"ID":"8f1bb47f-e0d7-410e-a09c-0f88ba6d138a","Type":"ContainerDied","Data":"604113f6794a2fc8365fb347d07631974f42fb0ed9636423a9a6dcd7c6a17e94"}
Nov 03 19:38:00.084150 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:00.083729    4921 scope.go:122] "RemoveContainer" containerID="60cff7a588d857d2092c9ab59b1b452f46aa5860d5e6e5df2ae0c05f3808089b"
Nov 03 19:38:00.093117 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:00.093096    4921 scope.go:122] "RemoveContainer" containerID="60cff7a588d857d2092c9ab59b1b452f46aa5860d5e6e5df2ae0c05f3808089b"
Nov 03 19:38:00.093382 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:38:00.093356    4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60cff7a588d857d2092c9ab59b1b452f46aa5860d5e6e5df2ae0c05f3808089b\": container with ID starting with 60cff7a588d857d2092c9ab59b1b452f46aa5860d5e6e5df2ae0c05f3808089b not found: ID does not exist" containerID="60cff7a588d857d2092c9ab59b1b452f46aa5860d5e6e5df2ae0c05f3808089b"
Nov 03 19:38:00.093451 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:00.093395    4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60cff7a588d857d2092c9ab59b1b452f46aa5860d5e6e5df2ae0c05f3808089b"} err="failed to get container status \"60cff7a588d857d2092c9ab59b1b452f46aa5860d5e6e5df2ae0c05f3808089b\": rpc error: code = NotFound desc = could not find container \"60cff7a588d857d2092c9ab59b1b452f46aa5860d5e6e5df2ae0c05f3808089b\": container with ID starting with 60cff7a588d857d2092c9ab59b1b452f46aa5860d5e6e5df2ae0c05f3808089b not found: ID does not exist"
Nov 03 19:38:00.107721 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:00.105842    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-h2bzs"]
Nov 03 19:38:00.112200 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:00.112171    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-h2bzs"]
Nov 03 19:38:00.887498 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:00.887464    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f1bb47f-e0d7-410e-a09c-0f88ba6d138a" path="/var/lib/kubelet/pods/8f1bb47f-e0d7-410e-a09c-0f88ba6d138a/volumes"
Nov 03 19:38:02.492263 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:02.492226    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-jhps4"]
Nov 03 19:38:02.492660 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:02.492586    4921 cpu_manager.go:414] "containerMap: removing container" podUID="8f1bb47f-e0d7-410e-a09c-0f88ba6d138a" containerName="limitador"
Nov 03 19:38:02.492660 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:02.492597    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f1bb47f-e0d7-410e-a09c-0f88ba6d138a" containerName="limitador"
Nov 03 19:38:02.492660 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:02.492621    4921 cpu_manager.go:414] "containerMap: removing container" podUID="39d57b35-653d-4314-8de9-c9bbfbfefb16" containerName="authorino"
Nov 03 19:38:02.492660 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:02.492626    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39d57b35-653d-4314-8de9-c9bbfbfefb16" containerName="authorino"
Nov 03 19:38:02.492874 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:02.492712    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f1bb47f-e0d7-410e-a09c-0f88ba6d138a" containerName="limitador"
Nov 03 19:38:02.492874 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:02.492724    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="39d57b35-653d-4314-8de9-c9bbfbfefb16" containerName="authorino"
Nov 03 19:38:02.499349 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:02.499321    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-jhps4"
Nov 03 19:38:02.504546 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:02.504517    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-jhps4"]
Nov 03 19:38:02.505227 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:02.505207    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:38:02.505533 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:02.505514    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Nov 03 19:38:02.505533 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:02.505522    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-n29tc\""
Nov 03 19:38:02.505695 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:02.505579    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:38:02.554979 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:02.554940    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-456d4\" (UniqueName: \"kubernetes.io/projected/2dc0536d-df3d-4cf9-829f-83a55b746a05-kube-api-access-456d4\") pod \"postgres-7cdbf6dc44-jhps4\" (UID: \"2dc0536d-df3d-4cf9-829f-83a55b746a05\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-jhps4"
Nov 03 19:38:02.555188 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:02.555013    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2dc0536d-df3d-4cf9-829f-83a55b746a05-data\") pod \"postgres-7cdbf6dc44-jhps4\" (UID: \"2dc0536d-df3d-4cf9-829f-83a55b746a05\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-jhps4"
Nov 03 19:38:02.656400 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:02.656352    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-456d4\" (UniqueName: \"kubernetes.io/projected/2dc0536d-df3d-4cf9-829f-83a55b746a05-kube-api-access-456d4\") pod \"postgres-7cdbf6dc44-jhps4\" (UID: \"2dc0536d-df3d-4cf9-829f-83a55b746a05\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-jhps4"
Nov 03 19:38:02.656617 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:02.656421    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2dc0536d-df3d-4cf9-829f-83a55b746a05-data\") pod \"postgres-7cdbf6dc44-jhps4\" (UID: \"2dc0536d-df3d-4cf9-829f-83a55b746a05\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-jhps4"
Nov 03 19:38:02.656787 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:02.656769    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2dc0536d-df3d-4cf9-829f-83a55b746a05-data\") pod \"postgres-7cdbf6dc44-jhps4\" (UID: \"2dc0536d-df3d-4cf9-829f-83a55b746a05\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-jhps4"
Nov 03 19:38:02.669681 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:02.669631    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-456d4\" (UniqueName: \"kubernetes.io/projected/2dc0536d-df3d-4cf9-829f-83a55b746a05-kube-api-access-456d4\") pod \"postgres-7cdbf6dc44-jhps4\" (UID: \"2dc0536d-df3d-4cf9-829f-83a55b746a05\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-jhps4"
Nov 03 19:38:02.814327 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:02.814239    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-jhps4"
Nov 03 19:38:02.920510 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:38:02.920471    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 19:38:03.105941 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:03.105915    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-jhps4"]
Nov 03 19:38:03.108445 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:38:03.108415    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dc0536d_df3d_4cf9_829f_83a55b746a05.slice/crio-dcf1c9a0f89eb1f85f1abc5d2ddf8d6e08e2b9fe9c200be28ff200f241c6e9d2 WatchSource:0}: Error finding container dcf1c9a0f89eb1f85f1abc5d2ddf8d6e08e2b9fe9c200be28ff200f241c6e9d2: Status 404 returned error can't find the container with id dcf1c9a0f89eb1f85f1abc5d2ddf8d6e08e2b9fe9c200be28ff200f241c6e9d2
Nov 03 19:38:04.107342 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:04.107305    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-jhps4" event={"ID":"2dc0536d-df3d-4cf9-829f-83a55b746a05","Type":"ContainerStarted","Data":"dcf1c9a0f89eb1f85f1abc5d2ddf8d6e08e2b9fe9c200be28ff200f241c6e9d2"}
Nov 03 19:38:05.448197 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:05.448164    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-c5ldk"]
Nov 03 19:38:05.452335 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:05.452295    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-c5ldk"
Nov 03 19:38:05.458994 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:05.458966    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Nov 03 19:38:05.461879 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:05.461837    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-dkwhv\""
Nov 03 19:38:05.466890 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:05.466863    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-c5ldk"]
Nov 03 19:38:05.469468 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:05.469445    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-9ddrt"]
Nov 03 19:38:05.472930 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:05.472907    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9ddrt"
Nov 03 19:38:05.483477 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:05.483451    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-9ddrt"]
Nov 03 19:38:05.486783 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:05.486743    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlvhz\" (UniqueName: \"kubernetes.io/projected/e3ffc1c6-b38c-4c4c-9113-13bcf50d1fa6-kube-api-access-vlvhz\") pod \"llmisvc-controller-manager-869d5dcc5d-9ddrt\" (UID: \"e3ffc1c6-b38c-4c4c-9113-13bcf50d1fa6\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9ddrt"
Nov 03 19:38:05.487048 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:05.486964    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8sgc\" (UniqueName: \"kubernetes.io/projected/b45584be-2f2e-4d38-9954-00befca7e68f-kube-api-access-l8sgc\") pod \"kserve-controller-manager-776cc66d47-c5ldk\" (UID: \"b45584be-2f2e-4d38-9954-00befca7e68f\") " pod="opendatahub/kserve-controller-manager-776cc66d47-c5ldk"
Nov 03 19:38:05.487207 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:05.487189    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e3ffc1c6-b38c-4c4c-9113-13bcf50d1fa6-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-9ddrt\" (UID: \"e3ffc1c6-b38c-4c4c-9113-13bcf50d1fa6\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9ddrt"
Nov 03 19:38:05.487359 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:05.487336    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b45584be-2f2e-4d38-9954-00befca7e68f-cert\") pod \"kserve-controller-manager-776cc66d47-c5ldk\" (UID: \"b45584be-2f2e-4d38-9954-00befca7e68f\") " pod="opendatahub/kserve-controller-manager-776cc66d47-c5ldk"
Nov 03 19:38:05.491882 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:05.491855    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-6r9xw"]
Nov 03 19:38:05.496704 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:05.496658    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-6r9xw"
Nov 03 19:38:05.505785 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:05.505756    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-6r9xw"]
Nov 03 19:38:05.559566 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:05.559536    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Nov 03 19:38:05.559566 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:05.559536    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-wtf6d\""
Nov 03 19:38:05.559863 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:05.559598    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-scnnh\""
Nov 03 19:38:05.559863 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:05.559631    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Nov 03 19:38:05.589217 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:05.589184    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21d707aa-f4eb-4b58-b122-86a668ba2b6c-tls-certs\") pod \"model-serving-api-75bd89fdfc-6r9xw\" (UID: \"21d707aa-f4eb-4b58-b122-86a668ba2b6c\") " pod="opendatahub/model-serving-api-75bd89fdfc-6r9xw"
Nov 03 19:38:05.589414 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:05.589240    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qct68\" (UniqueName: \"kubernetes.io/projected/21d707aa-f4eb-4b58-b122-86a668ba2b6c-kube-api-access-qct68\") pod \"model-serving-api-75bd89fdfc-6r9xw\" (UID: \"21d707aa-f4eb-4b58-b122-86a668ba2b6c\") " pod="opendatahub/model-serving-api-75bd89fdfc-6r9xw"
Nov 03 19:38:05.589414 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:05.589370    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlvhz\" (UniqueName: \"kubernetes.io/projected/e3ffc1c6-b38c-4c4c-9113-13bcf50d1fa6-kube-api-access-vlvhz\") pod \"llmisvc-controller-manager-869d5dcc5d-9ddrt\" (UID: \"e3ffc1c6-b38c-4c4c-9113-13bcf50d1fa6\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9ddrt"
Nov 03 19:38:05.589542 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:05.589416    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8sgc\" (UniqueName: \"kubernetes.io/projected/b45584be-2f2e-4d38-9954-00befca7e68f-kube-api-access-l8sgc\") pod \"kserve-controller-manager-776cc66d47-c5ldk\" (UID: \"b45584be-2f2e-4d38-9954-00befca7e68f\") " pod="opendatahub/kserve-controller-manager-776cc66d47-c5ldk"
Nov 03 19:38:05.589542 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:05.589449    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e3ffc1c6-b38c-4c4c-9113-13bcf50d1fa6-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-9ddrt\" (UID: \"e3ffc1c6-b38c-4c4c-9113-13bcf50d1fa6\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9ddrt"
Nov 03 19:38:05.589542 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:05.589499    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b45584be-2f2e-4d38-9954-00befca7e68f-cert\") pod \"kserve-controller-manager-776cc66d47-c5ldk\" (UID: \"b45584be-2f2e-4d38-9954-00befca7e68f\") " pod="opendatahub/kserve-controller-manager-776cc66d47-c5ldk"
Nov 03 19:38:05.589724 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:38:05.589638    4921 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Nov 03 19:38:05.589793 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:38:05.589750    4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b45584be-2f2e-4d38-9954-00befca7e68f-cert podName:b45584be-2f2e-4d38-9954-00befca7e68f nodeName:}" failed. No retries permitted until 2026-11-03 19:38:06.08972891 +0000 UTC m=+399.333694367 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/b45584be-2f2e-4d38-9954-00befca7e68f-cert") pod "kserve-controller-manager-776cc66d47-c5ldk" (UID: "b45584be-2f2e-4d38-9954-00befca7e68f") : secret "kserve-webhook-server-cert" not found
Nov 03 19:38:05.592227 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:05.592207    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e3ffc1c6-b38c-4c4c-9113-13bcf50d1fa6-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-9ddrt\" (UID: \"e3ffc1c6-b38c-4c4c-9113-13bcf50d1fa6\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9ddrt"
Nov 03 19:38:05.600034 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:05.600006    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlvhz\" (UniqueName: \"kubernetes.io/projected/e3ffc1c6-b38c-4c4c-9113-13bcf50d1fa6-kube-api-access-vlvhz\") pod \"llmisvc-controller-manager-869d5dcc5d-9ddrt\" (UID: \"e3ffc1c6-b38c-4c4c-9113-13bcf50d1fa6\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9ddrt"
Nov 03 19:38:05.600141 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:05.600003    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8sgc\" (UniqueName: \"kubernetes.io/projected/b45584be-2f2e-4d38-9954-00befca7e68f-kube-api-access-l8sgc\") pod \"kserve-controller-manager-776cc66d47-c5ldk\" (UID: \"b45584be-2f2e-4d38-9954-00befca7e68f\") " pod="opendatahub/kserve-controller-manager-776cc66d47-c5ldk"
Nov 03 19:38:05.690357 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:05.690318    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qct68\" (UniqueName: \"kubernetes.io/projected/21d707aa-f4eb-4b58-b122-86a668ba2b6c-kube-api-access-qct68\") pod \"model-serving-api-75bd89fdfc-6r9xw\" (UID: \"21d707aa-f4eb-4b58-b122-86a668ba2b6c\") " pod="opendatahub/model-serving-api-75bd89fdfc-6r9xw"
Nov 03 19:38:05.690539 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:05.690434    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21d707aa-f4eb-4b58-b122-86a668ba2b6c-tls-certs\") pod \"model-serving-api-75bd89fdfc-6r9xw\" (UID: \"21d707aa-f4eb-4b58-b122-86a668ba2b6c\") " pod="opendatahub/model-serving-api-75bd89fdfc-6r9xw"
Nov 03 19:38:05.692882 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:05.692859    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21d707aa-f4eb-4b58-b122-86a668ba2b6c-tls-certs\") pod \"model-serving-api-75bd89fdfc-6r9xw\" (UID: \"21d707aa-f4eb-4b58-b122-86a668ba2b6c\") " pod="opendatahub/model-serving-api-75bd89fdfc-6r9xw"
Nov 03 19:38:05.700617 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:05.700551    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qct68\" (UniqueName: \"kubernetes.io/projected/21d707aa-f4eb-4b58-b122-86a668ba2b6c-kube-api-access-qct68\") pod \"model-serving-api-75bd89fdfc-6r9xw\" (UID: \"21d707aa-f4eb-4b58-b122-86a668ba2b6c\") " pod="opendatahub/model-serving-api-75bd89fdfc-6r9xw"
Nov 03 19:38:05.801521 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:05.801479    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9ddrt"
Nov 03 19:38:05.815868 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:05.815836    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-6r9xw"
Nov 03 19:38:06.094312 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:06.094200    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b45584be-2f2e-4d38-9954-00befca7e68f-cert\") pod \"kserve-controller-manager-776cc66d47-c5ldk\" (UID: \"b45584be-2f2e-4d38-9954-00befca7e68f\") " pod="opendatahub/kserve-controller-manager-776cc66d47-c5ldk"
Nov 03 19:38:06.095908 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:06.095883    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-9ddrt"]
Nov 03 19:38:06.097118 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:06.097094    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b45584be-2f2e-4d38-9954-00befca7e68f-cert\") pod \"kserve-controller-manager-776cc66d47-c5ldk\" (UID: \"b45584be-2f2e-4d38-9954-00befca7e68f\") " pod="opendatahub/kserve-controller-manager-776cc66d47-c5ldk"
Nov 03 19:38:06.098187 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:38:06.098164    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode3ffc1c6_b38c_4c4c_9113_13bcf50d1fa6.slice/crio-057e2eeeee115725f454d1981dadb44f1c15bbe00e6df8c023cf2b92af630e91 WatchSource:0}: Error finding container 057e2eeeee115725f454d1981dadb44f1c15bbe00e6df8c023cf2b92af630e91: Status 404 returned error can't find the container with id 057e2eeeee115725f454d1981dadb44f1c15bbe00e6df8c023cf2b92af630e91
Nov 03 19:38:06.126506 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:06.126464    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9ddrt" event={"ID":"e3ffc1c6-b38c-4c4c-9113-13bcf50d1fa6","Type":"ContainerStarted","Data":"057e2eeeee115725f454d1981dadb44f1c15bbe00e6df8c023cf2b92af630e91"}
Nov 03 19:38:06.129328 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:06.129298    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-6r9xw"]
Nov 03 19:38:06.133847 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:38:06.133815    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21d707aa_f4eb_4b58_b122_86a668ba2b6c.slice/crio-86541b1beac503c49f717cfad646005af70821ccd86f8c5cc7a4a41d4c2bfda5 WatchSource:0}: Error finding container 86541b1beac503c49f717cfad646005af70821ccd86f8c5cc7a4a41d4c2bfda5: Status 404 returned error can't find the container with id 86541b1beac503c49f717cfad646005af70821ccd86f8c5cc7a4a41d4c2bfda5
Nov 03 19:38:06.370926 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:06.370847    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-c5ldk"
Nov 03 19:38:06.665601 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:06.665567    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-c5ldk"]
Nov 03 19:38:06.667309 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:38:06.667277    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb45584be_2f2e_4d38_9954_00befca7e68f.slice/crio-b9b314236c3924b70865471cfaa99f839faaed5c63a1d4267ad2ac8779bb6914 WatchSource:0}: Error finding container b9b314236c3924b70865471cfaa99f839faaed5c63a1d4267ad2ac8779bb6914: Status 404 returned error can't find the container with id b9b314236c3924b70865471cfaa99f839faaed5c63a1d4267ad2ac8779bb6914
Nov 03 19:38:07.140273 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:07.140225    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-6r9xw" event={"ID":"21d707aa-f4eb-4b58-b122-86a668ba2b6c","Type":"ContainerStarted","Data":"86541b1beac503c49f717cfad646005af70821ccd86f8c5cc7a4a41d4c2bfda5"}
Nov 03 19:38:07.142717 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:07.142584    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-c5ldk" event={"ID":"b45584be-2f2e-4d38-9954-00befca7e68f","Type":"ContainerStarted","Data":"b9b314236c3924b70865471cfaa99f839faaed5c63a1d4267ad2ac8779bb6914"}
Nov 03 19:38:09.160219 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:09.160172    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-6r9xw" event={"ID":"21d707aa-f4eb-4b58-b122-86a668ba2b6c","Type":"ContainerStarted","Data":"304cf324ee2914c46822638d0cc67de2280844ab76a59d5ed6055800492a0d2a"}
Nov 03 19:38:09.160719 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:09.160260    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-6r9xw"
Nov 03 19:38:09.162376 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:09.162343    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9ddrt" event={"ID":"e3ffc1c6-b38c-4c4c-9113-13bcf50d1fa6","Type":"ContainerStarted","Data":"252153c81d09aa7e7f84170e00ed4d3472d66bfcd317e23202cbf03126184b8d"}
Nov 03 19:38:09.162533 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:09.162513    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9ddrt"
Nov 03 19:38:09.178785 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:09.176634    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-6r9xw" podStartSLOduration=1.913885993 podStartE2EDuration="4.176616532s" totalImagesPullingTime="2.262730539s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:38:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:38:09.175199692 +0000 UTC m=+402.419165161" watchObservedRunningTime="2026-11-03 19:38:09.176616532 +0000 UTC m=+402.420581997"
Nov 03 19:38:09.193825 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:09.192160    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9ddrt" podStartSLOduration=1.7334300470000001 podStartE2EDuration="4.192146519s" totalImagesPullingTime="2.458716472s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:38:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:38:09.189587667 +0000 UTC m=+402.433553136" watchObservedRunningTime="2026-11-03 19:38:09.192146519 +0000 UTC m=+402.436111985"
Nov 03 19:38:10.170451 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:10.170404    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-c5ldk" event={"ID":"b45584be-2f2e-4d38-9954-00befca7e68f","Type":"ContainerStarted","Data":"5412fe2a1d57bc0a2840042efec06ea209a6c1dbcc98276243e072bd48e0e1cf"}
Nov 03 19:38:10.170922 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:10.170712    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-c5ldk"
Nov 03 19:38:10.172404 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:10.172373    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-jhps4" event={"ID":"2dc0536d-df3d-4cf9-829f-83a55b746a05","Type":"ContainerStarted","Data":"143cfc82a60f0539ed4c50a03818434f20ff9c5ea88d93f3e8aae7dc9f575ab4"}
Nov 03 19:38:10.172547 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:10.172421    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-jhps4"
Nov 03 19:38:10.187559 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:10.187506    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-c5ldk" podStartSLOduration=2.4166895999999998 podStartE2EDuration="5.18749218s" totalImagesPullingTime="2.77080258s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:38:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:38:10.184579476 +0000 UTC m=+403.428544958" watchObservedRunningTime="2026-11-03 19:38:10.18749218 +0000 UTC m=+403.431457678"
Nov 03 19:38:10.198578 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:10.198528    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-jhps4" podStartSLOduration=1.878703365 podStartE2EDuration="8.198513508s" totalImagesPullingTime="6.319810143s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:38:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:38:10.197171361 +0000 UTC m=+403.441136826" watchObservedRunningTime="2026-11-03 19:38:10.198513508 +0000 UTC m=+403.442478972"
Nov 03 19:38:16.209065 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:16.209032    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-jhps4"
Nov 03 19:38:20.177024 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:20.176991    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-6r9xw"
Nov 03 19:38:32.920843 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:38:32.920797    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 19:38:40.175195 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:40.175159    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9ddrt"
Nov 03 19:38:41.184967 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:38:41.184937    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-c5ldk"
Nov 03 19:39:02.921442 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:39:02.921399    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 19:39:11.052184 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:11.052148    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-kwqhq"]
Nov 03 19:39:11.055517 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:11.055498    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-kwqhq"
Nov 03 19:39:11.061039 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:11.061013    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-kwqhq"]
Nov 03 19:39:11.102007 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:11.101971    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-kwqhq"]
Nov 03 19:39:11.102223 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:39:11.102199    4921 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-bxgr9], unattached volumes=[], failed to process volumes=[kube-api-access-bxgr9]: context canceled" pod="kuadrant-system/authorino-6f6944bcdd-kwqhq" podUID="abe77ec5-1813-4981-82e2-0a223dd89efa"
Nov 03 19:39:11.128941 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:11.128903    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-2bmcm"]
Nov 03 19:39:11.132478 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:11.132453    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-2bmcm"
Nov 03 19:39:11.139987 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:11.139961    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Nov 03 19:39:11.142911 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:11.142875    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkbnz\" (UniqueName: \"kubernetes.io/projected/be5b592d-7646-4fd9-89c9-d749ce0665c5-kube-api-access-rkbnz\") pod \"authorino-6889cbd8c4-2bmcm\" (UID: \"be5b592d-7646-4fd9-89c9-d749ce0665c5\") " pod="kuadrant-system/authorino-6889cbd8c4-2bmcm"
Nov 03 19:39:11.143069 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:11.142949    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxgr9\" (UniqueName: \"kubernetes.io/projected/abe77ec5-1813-4981-82e2-0a223dd89efa-kube-api-access-bxgr9\") pod \"authorino-6f6944bcdd-kwqhq\" (UID: \"abe77ec5-1813-4981-82e2-0a223dd89efa\") " pod="kuadrant-system/authorino-6f6944bcdd-kwqhq"
Nov 03 19:39:11.143069 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:11.143006    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/be5b592d-7646-4fd9-89c9-d749ce0665c5-tls-cert\") pod \"authorino-6889cbd8c4-2bmcm\" (UID: \"be5b592d-7646-4fd9-89c9-d749ce0665c5\") " pod="kuadrant-system/authorino-6889cbd8c4-2bmcm"
Nov 03 19:39:11.144078 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:11.144055    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-2bmcm"]
Nov 03 19:39:11.243929 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:11.243891    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/be5b592d-7646-4fd9-89c9-d749ce0665c5-tls-cert\") pod \"authorino-6889cbd8c4-2bmcm\" (UID: \"be5b592d-7646-4fd9-89c9-d749ce0665c5\") " pod="kuadrant-system/authorino-6889cbd8c4-2bmcm"
Nov 03 19:39:11.244113 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:11.243956    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkbnz\" (UniqueName: \"kubernetes.io/projected/be5b592d-7646-4fd9-89c9-d749ce0665c5-kube-api-access-rkbnz\") pod \"authorino-6889cbd8c4-2bmcm\" (UID: \"be5b592d-7646-4fd9-89c9-d749ce0665c5\") " pod="kuadrant-system/authorino-6889cbd8c4-2bmcm"
Nov 03 19:39:11.244113 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:11.243995    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxgr9\" (UniqueName: \"kubernetes.io/projected/abe77ec5-1813-4981-82e2-0a223dd89efa-kube-api-access-bxgr9\") pod \"authorino-6f6944bcdd-kwqhq\" (UID: \"abe77ec5-1813-4981-82e2-0a223dd89efa\") " pod="kuadrant-system/authorino-6f6944bcdd-kwqhq"
Nov 03 19:39:11.246377 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:11.246350    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/be5b592d-7646-4fd9-89c9-d749ce0665c5-tls-cert\") pod \"authorino-6889cbd8c4-2bmcm\" (UID: \"be5b592d-7646-4fd9-89c9-d749ce0665c5\") " pod="kuadrant-system/authorino-6889cbd8c4-2bmcm"
Nov 03 19:39:11.250829 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:11.250804    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkbnz\" (UniqueName: \"kubernetes.io/projected/be5b592d-7646-4fd9-89c9-d749ce0665c5-kube-api-access-rkbnz\") pod \"authorino-6889cbd8c4-2bmcm\" (UID: \"be5b592d-7646-4fd9-89c9-d749ce0665c5\") " pod="kuadrant-system/authorino-6889cbd8c4-2bmcm"
Nov 03 19:39:11.250963 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:11.250930    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxgr9\" (UniqueName: \"kubernetes.io/projected/abe77ec5-1813-4981-82e2-0a223dd89efa-kube-api-access-bxgr9\") pod \"authorino-6f6944bcdd-kwqhq\" (UID: \"abe77ec5-1813-4981-82e2-0a223dd89efa\") " pod="kuadrant-system/authorino-6f6944bcdd-kwqhq"
Nov 03 19:39:11.377681 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:11.377584    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-2bmcm"]
Nov 03 19:39:11.377829 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:11.377817    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-2bmcm"
Nov 03 19:39:11.404511 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:11.404475    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-654c79df68-hl2n6"]
Nov 03 19:39:11.408508 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:11.408481    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-654c79df68-hl2n6"
Nov 03 19:39:11.418863 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:11.418806    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-654c79df68-hl2n6"]
Nov 03 19:39:11.529187 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:11.529157    4921 generic.go:356] "Generic (PLEG): container finished" podID="d0e23fd0-6cce-402f-9e00-0f7da02ed53d" containerID="bf56445ec8fad51d29e43cc5de28a72b3b7ee5ad6e6941ff946ecd62b8f32eaa" exitCode=0
Nov 03 19:39:11.529369 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:11.529234    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zl5bx" event={"ID":"d0e23fd0-6cce-402f-9e00-0f7da02ed53d","Type":"ContainerDied","Data":"bf56445ec8fad51d29e43cc5de28a72b3b7ee5ad6e6941ff946ecd62b8f32eaa"}
Nov 03 19:39:11.529418 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:11.529366    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-kwqhq"
Nov 03 19:39:11.529628 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:11.529616    4921 scope.go:122] "RemoveContainer" containerID="bf56445ec8fad51d29e43cc5de28a72b3b7ee5ad6e6941ff946ecd62b8f32eaa"
Nov 03 19:39:11.530852 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:11.530822    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:39:11.530926 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:11.530901    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:39:11.530961 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:11.530943    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:39:11.536495 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:11.536477    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-kwqhq"
Nov 03 19:39:11.549317 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:11.549294    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/44b11870-4ef3-4061-87bd-09fd3d0f47f8-tls-cert\") pod \"authorino-654c79df68-hl2n6\" (UID: \"44b11870-4ef3-4061-87bd-09fd3d0f47f8\") " pod="kuadrant-system/authorino-654c79df68-hl2n6"
Nov 03 19:39:11.549430 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:11.549404    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmppb\" (UniqueName: \"kubernetes.io/projected/44b11870-4ef3-4061-87bd-09fd3d0f47f8-kube-api-access-wmppb\") pod \"authorino-654c79df68-hl2n6\" (UID: \"44b11870-4ef3-4061-87bd-09fd3d0f47f8\") " pod="kuadrant-system/authorino-654c79df68-hl2n6"
Nov 03 19:39:11.650895 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:11.650803    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abe77ec5-1813-4981-82e2-0a223dd89efa-kube-api-access-bxgr9\" (UniqueName: \"kubernetes.io/projected/abe77ec5-1813-4981-82e2-0a223dd89efa-kube-api-access-bxgr9\") pod \"abe77ec5-1813-4981-82e2-0a223dd89efa\" (UID: \"abe77ec5-1813-4981-82e2-0a223dd89efa\") "
Nov 03 19:39:11.651051 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:11.651016    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmppb\" (UniqueName: \"kubernetes.io/projected/44b11870-4ef3-4061-87bd-09fd3d0f47f8-kube-api-access-wmppb\") pod \"authorino-654c79df68-hl2n6\" (UID: \"44b11870-4ef3-4061-87bd-09fd3d0f47f8\") " pod="kuadrant-system/authorino-654c79df68-hl2n6"
Nov 03 19:39:11.651455 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:11.651163    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/44b11870-4ef3-4061-87bd-09fd3d0f47f8-tls-cert\") pod \"authorino-654c79df68-hl2n6\" (UID: \"44b11870-4ef3-4061-87bd-09fd3d0f47f8\") " pod="kuadrant-system/authorino-654c79df68-hl2n6"
Nov 03 19:39:11.654453 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:11.654379    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/44b11870-4ef3-4061-87bd-09fd3d0f47f8-tls-cert\") pod \"authorino-654c79df68-hl2n6\" (UID: \"44b11870-4ef3-4061-87bd-09fd3d0f47f8\") " pod="kuadrant-system/authorino-654c79df68-hl2n6"
Nov 03 19:39:11.654691 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:11.654601    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abe77ec5-1813-4981-82e2-0a223dd89efa-kube-api-access-bxgr9" pod "abe77ec5-1813-4981-82e2-0a223dd89efa" (UID: "abe77ec5-1813-4981-82e2-0a223dd89efa"). InnerVolumeSpecName "kube-api-access-bxgr9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:39:11.657618 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:11.657592    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-2bmcm"]
Nov 03 19:39:11.658869 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:11.658845    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmppb\" (UniqueName: \"kubernetes.io/projected/44b11870-4ef3-4061-87bd-09fd3d0f47f8-kube-api-access-wmppb\") pod \"authorino-654c79df68-hl2n6\" (UID: \"44b11870-4ef3-4061-87bd-09fd3d0f47f8\") " pod="kuadrant-system/authorino-654c79df68-hl2n6"
Nov 03 19:39:11.659444 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:39:11.659414    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbe5b592d_7646_4fd9_89c9_d749ce0665c5.slice/crio-29162e42e77b63d71e025b150986a35a56c26bd35d10f78a312edc1720b3cb6e WatchSource:0}: Error finding container 29162e42e77b63d71e025b150986a35a56c26bd35d10f78a312edc1720b3cb6e: Status 404 returned error can't find the container with id 29162e42e77b63d71e025b150986a35a56c26bd35d10f78a312edc1720b3cb6e
Nov 03 19:39:11.726356 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:11.726313    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-654c79df68-hl2n6"
Nov 03 19:39:11.751840 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:11.751808    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxgr9\" (UniqueName: \"kubernetes.io/projected/abe77ec5-1813-4981-82e2-0a223dd89efa-kube-api-access-bxgr9\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:39:12.001048 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:12.001018    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-654c79df68-hl2n6"]
Nov 03 19:39:12.003445 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:39:12.003414    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod44b11870_4ef3_4061_87bd_09fd3d0f47f8.slice/crio-5eab2dbfd604b295b960b81d2efef4df1f866ae4d7903ecc8c280ee7863ad479 WatchSource:0}: Error finding container 5eab2dbfd604b295b960b81d2efef4df1f866ae4d7903ecc8c280ee7863ad479: Status 404 returned error can't find the container with id 5eab2dbfd604b295b960b81d2efef4df1f866ae4d7903ecc8c280ee7863ad479
Nov 03 19:39:12.536215 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:12.536180    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-654c79df68-hl2n6" event={"ID":"44b11870-4ef3-4061-87bd-09fd3d0f47f8","Type":"ContainerStarted","Data":"5eab2dbfd604b295b960b81d2efef4df1f866ae4d7903ecc8c280ee7863ad479"}
Nov 03 19:39:12.538533 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:12.538501    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zl5bx" event={"ID":"d0e23fd0-6cce-402f-9e00-0f7da02ed53d","Type":"ContainerStarted","Data":"3b6eaf217d9e63ab2d593826f9f8ad5f3231655b4455eff84ae56150300fdd48"}
Nov 03 19:39:12.540171 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:12.540140    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-2bmcm" event={"ID":"be5b592d-7646-4fd9-89c9-d749ce0665c5","Type":"ContainerStarted","Data":"6b5582f3928ae73a1604093211862bdb2c67d5de0aaad6234faaee08d814fd98"}
Nov 03 19:39:12.540530 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:12.540183    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-2bmcm" event={"ID":"be5b592d-7646-4fd9-89c9-d749ce0665c5","Type":"ContainerStarted","Data":"29162e42e77b63d71e025b150986a35a56c26bd35d10f78a312edc1720b3cb6e"}
Nov 03 19:39:12.540530 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:12.540193    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6889cbd8c4-2bmcm" podUID="be5b592d-7646-4fd9-89c9-d749ce0665c5" containerName="authorino" containerID="cri-o://6b5582f3928ae73a1604093211862bdb2c67d5de0aaad6234faaee08d814fd98" gracePeriod=30
Nov 03 19:39:12.540530 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:12.540221    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-kwqhq"
Nov 03 19:39:12.597506 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:12.597473    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-kwqhq"]
Nov 03 19:39:12.599434 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:12.599410    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-kwqhq"]
Nov 03 19:39:12.609105 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:12.609061    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6889cbd8c4-2bmcm" podStartSLOduration=0.856029758 podStartE2EDuration="1.609046492s" totalImagesPullingTime="753.016734ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:39:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:39:12.607957503 +0000 UTC m=+465.851922968" watchObservedRunningTime="2026-11-03 19:39:12.609046492 +0000 UTC m=+465.853011957"
Nov 03 19:39:12.842992 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:12.842963    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-2bmcm"
Nov 03 19:39:12.888894 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:12.888865    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abe77ec5-1813-4981-82e2-0a223dd89efa" path="/var/lib/kubelet/pods/abe77ec5-1813-4981-82e2-0a223dd89efa/volumes"
Nov 03 19:39:12.983272 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:12.983230    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be5b592d-7646-4fd9-89c9-d749ce0665c5-tls-cert\" (UniqueName: \"kubernetes.io/secret/be5b592d-7646-4fd9-89c9-d749ce0665c5-tls-cert\") pod \"be5b592d-7646-4fd9-89c9-d749ce0665c5\" (UID: \"be5b592d-7646-4fd9-89c9-d749ce0665c5\") "
Nov 03 19:39:12.983474 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:12.983300    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be5b592d-7646-4fd9-89c9-d749ce0665c5-kube-api-access-rkbnz\" (UniqueName: \"kubernetes.io/projected/be5b592d-7646-4fd9-89c9-d749ce0665c5-kube-api-access-rkbnz\") pod \"be5b592d-7646-4fd9-89c9-d749ce0665c5\" (UID: \"be5b592d-7646-4fd9-89c9-d749ce0665c5\") "
Nov 03 19:39:12.985591 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:12.985552    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be5b592d-7646-4fd9-89c9-d749ce0665c5-kube-api-access-rkbnz" pod "be5b592d-7646-4fd9-89c9-d749ce0665c5" (UID: "be5b592d-7646-4fd9-89c9-d749ce0665c5"). InnerVolumeSpecName "kube-api-access-rkbnz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:39:12.997598 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:12.997554    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be5b592d-7646-4fd9-89c9-d749ce0665c5-tls-cert" pod "be5b592d-7646-4fd9-89c9-d749ce0665c5" (UID: "be5b592d-7646-4fd9-89c9-d749ce0665c5"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:39:13.000746 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:13.000716    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zl5bx"
Nov 03 19:39:13.000940 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:13.000919    4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-zl5bx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Nov 03 19:39:13.001007 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:13.000963    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zl5bx" podUID="d0e23fd0-6cce-402f-9e00-0f7da02ed53d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Nov 03 19:39:13.084575 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:13.084540    4921 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/be5b592d-7646-4fd9-89c9-d749ce0665c5-tls-cert\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:39:13.084575 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:13.084573    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkbnz\" (UniqueName: \"kubernetes.io/projected/be5b592d-7646-4fd9-89c9-d749ce0665c5-kube-api-access-rkbnz\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:39:13.547156 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:13.547114    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-654c79df68-hl2n6" event={"ID":"44b11870-4ef3-4061-87bd-09fd3d0f47f8","Type":"ContainerStarted","Data":"a33fb68b631d2d96f303ce68c66c4218e090408afd8e8663ea57652b6d987f2f"}
Nov 03 19:39:13.548393 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:13.548366    4921 generic.go:356] "Generic (PLEG): container finished" podID="be5b592d-7646-4fd9-89c9-d749ce0665c5" containerID="6b5582f3928ae73a1604093211862bdb2c67d5de0aaad6234faaee08d814fd98" exitCode=0
Nov 03 19:39:13.548536 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:13.548402    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-2bmcm"
Nov 03 19:39:13.548536 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:13.548423    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-2bmcm" event={"ID":"be5b592d-7646-4fd9-89c9-d749ce0665c5","Type":"ContainerDied","Data":"6b5582f3928ae73a1604093211862bdb2c67d5de0aaad6234faaee08d814fd98"}
Nov 03 19:39:13.548536 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:13.548460    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-2bmcm" event={"ID":"be5b592d-7646-4fd9-89c9-d749ce0665c5","Type":"ContainerDied","Data":"29162e42e77b63d71e025b150986a35a56c26bd35d10f78a312edc1720b3cb6e"}
Nov 03 19:39:13.548536 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:13.548477    4921 scope.go:122] "RemoveContainer" containerID="6b5582f3928ae73a1604093211862bdb2c67d5de0aaad6234faaee08d814fd98"
Nov 03 19:39:13.578736 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:13.578374    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-654c79df68-hl2n6" podStartSLOduration=1.888085336 podStartE2EDuration="2.578360426s" totalImagesPullingTime="690.27509ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:39:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:39:13.578026799 +0000 UTC m=+466.821992266" watchObservedRunningTime="2026-11-03 19:39:13.578360426 +0000 UTC m=+466.822325909"
Nov 03 19:39:13.590024 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:13.590003    4921 scope.go:122] "RemoveContainer" containerID="6b5582f3928ae73a1604093211862bdb2c67d5de0aaad6234faaee08d814fd98"
Nov 03 19:39:13.590348 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:39:13.590328    4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b5582f3928ae73a1604093211862bdb2c67d5de0aaad6234faaee08d814fd98\": container with ID starting with 6b5582f3928ae73a1604093211862bdb2c67d5de0aaad6234faaee08d814fd98 not found: ID does not exist" containerID="6b5582f3928ae73a1604093211862bdb2c67d5de0aaad6234faaee08d814fd98"
Nov 03 19:39:13.590422 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:13.590359    4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b5582f3928ae73a1604093211862bdb2c67d5de0aaad6234faaee08d814fd98"} err="failed to get container status \"6b5582f3928ae73a1604093211862bdb2c67d5de0aaad6234faaee08d814fd98\": rpc error: code = NotFound desc = could not find container \"6b5582f3928ae73a1604093211862bdb2c67d5de0aaad6234faaee08d814fd98\": container with ID starting with 6b5582f3928ae73a1604093211862bdb2c67d5de0aaad6234faaee08d814fd98 not found: ID does not exist"
Nov 03 19:39:13.598151 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:13.598124    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-fzncl"]
Nov 03 19:39:13.598363 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:13.598340    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-fzncl" podUID="3d8cf63c-9dfa-45f9-86b8-13aa9e14e726" containerName="authorino" containerID="cri-o://fcacf9d47864fdc0649da177d11606731bb127848a103b058a96643414170cdb" gracePeriod=30
Nov 03 19:39:13.604799 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:13.603439    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-2bmcm"]
Nov 03 19:39:13.609476 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:13.609451    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-2bmcm"]
Nov 03 19:39:13.885419 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:13.885392    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-fzncl"
Nov 03 19:39:13.890960 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:13.890934    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d8cf63c-9dfa-45f9-86b8-13aa9e14e726-kube-api-access-4lwmv\" (UniqueName: \"kubernetes.io/projected/3d8cf63c-9dfa-45f9-86b8-13aa9e14e726-kube-api-access-4lwmv\") pod \"3d8cf63c-9dfa-45f9-86b8-13aa9e14e726\" (UID: \"3d8cf63c-9dfa-45f9-86b8-13aa9e14e726\") "
Nov 03 19:39:13.893056 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:13.893021    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d8cf63c-9dfa-45f9-86b8-13aa9e14e726-kube-api-access-4lwmv" pod "3d8cf63c-9dfa-45f9-86b8-13aa9e14e726" (UID: "3d8cf63c-9dfa-45f9-86b8-13aa9e14e726"). InnerVolumeSpecName "kube-api-access-4lwmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:39:13.992293 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:13.992249    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lwmv\" (UniqueName: \"kubernetes.io/projected/3d8cf63c-9dfa-45f9-86b8-13aa9e14e726-kube-api-access-4lwmv\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:39:14.000392 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:14.000363    4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-zl5bx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Nov 03 19:39:14.000503 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:14.000415    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zl5bx" podUID="d0e23fd0-6cce-402f-9e00-0f7da02ed53d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Nov 03 19:39:14.477693 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:14.477224    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-gp6bm"]
Nov 03 19:39:14.477901 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:14.477881    4921 cpu_manager.go:414] "containerMap: removing container" podUID="3d8cf63c-9dfa-45f9-86b8-13aa9e14e726" containerName="authorino"
Nov 03 19:39:14.477974 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:14.477901    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d8cf63c-9dfa-45f9-86b8-13aa9e14e726" containerName="authorino"
Nov 03 19:39:14.477974 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:14.477952    4921 cpu_manager.go:414] "containerMap: removing container" podUID="be5b592d-7646-4fd9-89c9-d749ce0665c5" containerName="authorino"
Nov 03 19:39:14.477974 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:14.477960    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be5b592d-7646-4fd9-89c9-d749ce0665c5" containerName="authorino"
Nov 03 19:39:14.478136 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:14.478053    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="be5b592d-7646-4fd9-89c9-d749ce0665c5" containerName="authorino"
Nov 03 19:39:14.478136 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:14.478067    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d8cf63c-9dfa-45f9-86b8-13aa9e14e726" containerName="authorino"
Nov 03 19:39:14.481348 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:14.481325    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-gp6bm"]
Nov 03 19:39:14.481477 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:14.481419    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-gp6bm"
Nov 03 19:39:14.495637 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:14.495609    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9c518498-21b1-4e86-9e9c-4f67855afa5f-cert\") pod \"llmisvc-controller-manager-6466c554d8-gp6bm\" (UID: \"9c518498-21b1-4e86-9e9c-4f67855afa5f\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-gp6bm"
Nov 03 19:39:14.495794 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:14.495664    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nfhs\" (UniqueName: \"kubernetes.io/projected/9c518498-21b1-4e86-9e9c-4f67855afa5f-kube-api-access-8nfhs\") pod \"llmisvc-controller-manager-6466c554d8-gp6bm\" (UID: \"9c518498-21b1-4e86-9e9c-4f67855afa5f\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-gp6bm"
Nov 03 19:39:14.581238 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:14.581192    4921 generic.go:356] "Generic (PLEG): container finished" podID="3d8cf63c-9dfa-45f9-86b8-13aa9e14e726" containerID="fcacf9d47864fdc0649da177d11606731bb127848a103b058a96643414170cdb" exitCode=0
Nov 03 19:39:14.581760 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:14.581369    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-fzncl" event={"ID":"3d8cf63c-9dfa-45f9-86b8-13aa9e14e726","Type":"ContainerDied","Data":"fcacf9d47864fdc0649da177d11606731bb127848a103b058a96643414170cdb"}
Nov 03 19:39:14.581760 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:14.581412    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-fzncl" event={"ID":"3d8cf63c-9dfa-45f9-86b8-13aa9e14e726","Type":"ContainerDied","Data":"dd71aef82c33fe8b8b0766091dae6335f13542ccffb61168ab1aa792d6386c82"}
Nov 03 19:39:14.581760 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:14.581442    4921 scope.go:122] "RemoveContainer" containerID="fcacf9d47864fdc0649da177d11606731bb127848a103b058a96643414170cdb"
Nov 03 19:39:14.581760 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:14.581616    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-fzncl"
Nov 03 19:39:14.594570 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:14.594547    4921 scope.go:122] "RemoveContainer" containerID="fcacf9d47864fdc0649da177d11606731bb127848a103b058a96643414170cdb"
Nov 03 19:39:14.594878 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:39:14.594855    4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcacf9d47864fdc0649da177d11606731bb127848a103b058a96643414170cdb\": container with ID starting with fcacf9d47864fdc0649da177d11606731bb127848a103b058a96643414170cdb not found: ID does not exist" containerID="fcacf9d47864fdc0649da177d11606731bb127848a103b058a96643414170cdb"
Nov 03 19:39:14.594957 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:14.594888    4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcacf9d47864fdc0649da177d11606731bb127848a103b058a96643414170cdb"} err="failed to get container status \"fcacf9d47864fdc0649da177d11606731bb127848a103b058a96643414170cdb\": rpc error: code = NotFound desc = could not find container \"fcacf9d47864fdc0649da177d11606731bb127848a103b058a96643414170cdb\": container with ID starting with fcacf9d47864fdc0649da177d11606731bb127848a103b058a96643414170cdb not found: ID does not exist"
Nov 03 19:39:14.596270 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:14.596247    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9c518498-21b1-4e86-9e9c-4f67855afa5f-cert\") pod \"llmisvc-controller-manager-6466c554d8-gp6bm\" (UID: \"9c518498-21b1-4e86-9e9c-4f67855afa5f\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-gp6bm"
Nov 03 19:39:14.596385 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:14.596306    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nfhs\" (UniqueName: \"kubernetes.io/projected/9c518498-21b1-4e86-9e9c-4f67855afa5f-kube-api-access-8nfhs\") pod \"llmisvc-controller-manager-6466c554d8-gp6bm\" (UID: \"9c518498-21b1-4e86-9e9c-4f67855afa5f\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-gp6bm"
Nov 03 19:39:14.598551 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:14.598527    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9c518498-21b1-4e86-9e9c-4f67855afa5f-cert\") pod \"llmisvc-controller-manager-6466c554d8-gp6bm\" (UID: \"9c518498-21b1-4e86-9e9c-4f67855afa5f\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-gp6bm"
Nov 03 19:39:14.602008 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:14.601985    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-fzncl"]
Nov 03 19:39:14.604714 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:14.604695    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nfhs\" (UniqueName: \"kubernetes.io/projected/9c518498-21b1-4e86-9e9c-4f67855afa5f-kube-api-access-8nfhs\") pod \"llmisvc-controller-manager-6466c554d8-gp6bm\" (UID: \"9c518498-21b1-4e86-9e9c-4f67855afa5f\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-gp6bm"
Nov 03 19:39:14.605542 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:14.605521    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-fzncl"]
Nov 03 19:39:14.794708 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:14.794598    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-gp6bm"
Nov 03 19:39:14.890306 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:14.890270    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d8cf63c-9dfa-45f9-86b8-13aa9e14e726" path="/var/lib/kubelet/pods/3d8cf63c-9dfa-45f9-86b8-13aa9e14e726/volumes"
Nov 03 19:39:14.890637 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:14.890622    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be5b592d-7646-4fd9-89c9-d749ce0665c5" path="/var/lib/kubelet/pods/be5b592d-7646-4fd9-89c9-d749ce0665c5/volumes"
Nov 03 19:39:15.000761 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:15.000720    4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-zl5bx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Nov 03 19:39:15.000934 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:15.000776    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zl5bx" podUID="d0e23fd0-6cce-402f-9e00-0f7da02ed53d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Nov 03 19:39:15.077418 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:15.077383    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-gp6bm"]
Nov 03 19:39:15.078496 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:39:15.078471    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c518498_21b1_4e86_9e9c_4f67855afa5f.slice/crio-77bceebab8bc13aa76cad00e197f528cfd8b26a7f9f4f735bbd52cc5806d1e57 WatchSource:0}: Error finding container 77bceebab8bc13aa76cad00e197f528cfd8b26a7f9f4f735bbd52cc5806d1e57: Status 404 returned error can't find the container with id 77bceebab8bc13aa76cad00e197f528cfd8b26a7f9f4f735bbd52cc5806d1e57
Nov 03 19:39:15.594633 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:15.594601    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-gp6bm" event={"ID":"9c518498-21b1-4e86-9e9c-4f67855afa5f","Type":"ContainerStarted","Data":"46c54ac74cd471a61108321d2f81b24bcd32b2b21faf10892d3d34e478d11dc1"}
Nov 03 19:39:15.595030 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:15.594655    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-gp6bm" event={"ID":"9c518498-21b1-4e86-9e9c-4f67855afa5f","Type":"ContainerStarted","Data":"77bceebab8bc13aa76cad00e197f528cfd8b26a7f9f4f735bbd52cc5806d1e57"}
Nov 03 19:39:15.595030 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:15.594816    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-gp6bm"
Nov 03 19:39:15.609100 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:15.609040    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-gp6bm" podStartSLOduration=1.213977875 podStartE2EDuration="1.609020622s" totalImagesPullingTime="395.042747ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:39:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:39:15.607578585 +0000 UTC m=+468.851544084" watchObservedRunningTime="2026-11-03 19:39:15.609020622 +0000 UTC m=+468.852986088"
Nov 03 19:39:16.001372 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:16.001338    4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-zl5bx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Nov 03 19:39:16.001555 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:16.001393    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zl5bx" podUID="d0e23fd0-6cce-402f-9e00-0f7da02ed53d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Nov 03 19:39:17.000447 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:17.000404    4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-zl5bx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Nov 03 19:39:17.001002 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:17.000457    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zl5bx" podUID="d0e23fd0-6cce-402f-9e00-0f7da02ed53d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Nov 03 19:39:18.004466 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:18.004433    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zl5bx"
Nov 03 19:39:18.004865 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:18.004718    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zl5bx"
Nov 03 19:39:18.005566 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:18.005548    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zl5bx"
Nov 03 19:39:32.920369 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:39:32.920331    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 19:39:46.603452 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:46.603416    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-gp6bm"
Nov 03 19:39:46.647126 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:46.647089    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-9ddrt"]
Nov 03 19:39:46.647334 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:46.647298    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9ddrt" podUID="e3ffc1c6-b38c-4c4c-9113-13bcf50d1fa6" containerName="manager" containerID="cri-o://252153c81d09aa7e7f84170e00ed4d3472d66bfcd317e23202cbf03126184b8d" gracePeriod=30
Nov 03 19:39:46.901172 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:46.901146    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9ddrt"
Nov 03 19:39:46.955028 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:46.954988    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3ffc1c6-b38c-4c4c-9113-13bcf50d1fa6-kube-api-access-vlvhz\" (UniqueName: \"kubernetes.io/projected/e3ffc1c6-b38c-4c4c-9113-13bcf50d1fa6-kube-api-access-vlvhz\") pod \"e3ffc1c6-b38c-4c4c-9113-13bcf50d1fa6\" (UID: \"e3ffc1c6-b38c-4c4c-9113-13bcf50d1fa6\") "
Nov 03 19:39:46.955196 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:46.955136    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3ffc1c6-b38c-4c4c-9113-13bcf50d1fa6-cert\" (UniqueName: \"kubernetes.io/secret/e3ffc1c6-b38c-4c4c-9113-13bcf50d1fa6-cert\") pod \"e3ffc1c6-b38c-4c4c-9113-13bcf50d1fa6\" (UID: \"e3ffc1c6-b38c-4c4c-9113-13bcf50d1fa6\") "
Nov 03 19:39:46.957299 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:46.957259    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3ffc1c6-b38c-4c4c-9113-13bcf50d1fa6-kube-api-access-vlvhz" pod "e3ffc1c6-b38c-4c4c-9113-13bcf50d1fa6" (UID: "e3ffc1c6-b38c-4c4c-9113-13bcf50d1fa6"). InnerVolumeSpecName "kube-api-access-vlvhz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:39:46.957438 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:46.957348    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3ffc1c6-b38c-4c4c-9113-13bcf50d1fa6-cert" pod "e3ffc1c6-b38c-4c4c-9113-13bcf50d1fa6" (UID: "e3ffc1c6-b38c-4c4c-9113-13bcf50d1fa6"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:39:47.056586 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:47.056544    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlvhz\" (UniqueName: \"kubernetes.io/projected/e3ffc1c6-b38c-4c4c-9113-13bcf50d1fa6-kube-api-access-vlvhz\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:39:47.056586 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:47.056581    4921 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e3ffc1c6-b38c-4c4c-9113-13bcf50d1fa6-cert\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:39:47.767022 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:47.766986    4921 generic.go:356] "Generic (PLEG): container finished" podID="e3ffc1c6-b38c-4c4c-9113-13bcf50d1fa6" containerID="252153c81d09aa7e7f84170e00ed4d3472d66bfcd317e23202cbf03126184b8d" exitCode=0
Nov 03 19:39:47.767505 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:47.767032    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9ddrt"
Nov 03 19:39:47.767505 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:47.767070    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9ddrt" event={"ID":"e3ffc1c6-b38c-4c4c-9113-13bcf50d1fa6","Type":"ContainerDied","Data":"252153c81d09aa7e7f84170e00ed4d3472d66bfcd317e23202cbf03126184b8d"}
Nov 03 19:39:47.767505 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:47.767134    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9ddrt" event={"ID":"e3ffc1c6-b38c-4c4c-9113-13bcf50d1fa6","Type":"ContainerDied","Data":"057e2eeeee115725f454d1981dadb44f1c15bbe00e6df8c023cf2b92af630e91"}
Nov 03 19:39:47.767505 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:47.767165    4921 scope.go:122] "RemoveContainer" containerID="252153c81d09aa7e7f84170e00ed4d3472d66bfcd317e23202cbf03126184b8d"
Nov 03 19:39:47.777199 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:47.777180    4921 scope.go:122] "RemoveContainer" containerID="252153c81d09aa7e7f84170e00ed4d3472d66bfcd317e23202cbf03126184b8d"
Nov 03 19:39:47.777521 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:39:47.777497    4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"252153c81d09aa7e7f84170e00ed4d3472d66bfcd317e23202cbf03126184b8d\": container with ID starting with 252153c81d09aa7e7f84170e00ed4d3472d66bfcd317e23202cbf03126184b8d not found: ID does not exist" containerID="252153c81d09aa7e7f84170e00ed4d3472d66bfcd317e23202cbf03126184b8d"
Nov 03 19:39:47.777591 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:47.777534    4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"252153c81d09aa7e7f84170e00ed4d3472d66bfcd317e23202cbf03126184b8d"} err="failed to get container status \"252153c81d09aa7e7f84170e00ed4d3472d66bfcd317e23202cbf03126184b8d\": rpc error: code = NotFound desc = could not find container \"252153c81d09aa7e7f84170e00ed4d3472d66bfcd317e23202cbf03126184b8d\": container with ID starting with 252153c81d09aa7e7f84170e00ed4d3472d66bfcd317e23202cbf03126184b8d not found: ID does not exist"
Nov 03 19:39:47.786830 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:47.786802    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-9ddrt"]
Nov 03 19:39:47.790612 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:47.790587    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-9ddrt"]
Nov 03 19:39:48.886825 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:39:48.886791    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3ffc1c6-b38c-4c4c-9113-13bcf50d1fa6" path="/var/lib/kubelet/pods/e3ffc1c6-b38c-4c4c-9113-13bcf50d1fa6/volumes"
Nov 03 19:40:02.921247 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:40:02.921192    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 19:40:32.921732 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:40:32.921693    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 19:40:33.943996 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:33.943958    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6ccf875c55-rz2d8"]
Nov 03 19:40:33.944379 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:33.944366    4921 cpu_manager.go:414] "containerMap: removing container" podUID="e3ffc1c6-b38c-4c4c-9113-13bcf50d1fa6" containerName="manager"
Nov 03 19:40:33.944379 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:33.944378    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3ffc1c6-b38c-4c4c-9113-13bcf50d1fa6" containerName="manager"
Nov 03 19:40:33.944473 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:33.944451    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3ffc1c6-b38c-4c4c-9113-13bcf50d1fa6" containerName="manager"
Nov 03 19:40:33.948848 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:33.948826    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6ccf875c55-rz2d8"
Nov 03 19:40:33.957320 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:33.957294    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6ccf875c55-rz2d8"]
Nov 03 19:40:33.998608 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:33.998573    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7e77f59c-fa04-4fef-80f6-6da7f2fbf046-tls-cert\") pod \"authorino-6ccf875c55-rz2d8\" (UID: \"7e77f59c-fa04-4fef-80f6-6da7f2fbf046\") " pod="kuadrant-system/authorino-6ccf875c55-rz2d8"
Nov 03 19:40:33.998794 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:33.998621    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv6hv\" (UniqueName: \"kubernetes.io/projected/7e77f59c-fa04-4fef-80f6-6da7f2fbf046-kube-api-access-rv6hv\") pod \"authorino-6ccf875c55-rz2d8\" (UID: \"7e77f59c-fa04-4fef-80f6-6da7f2fbf046\") " pod="kuadrant-system/authorino-6ccf875c55-rz2d8"
Nov 03 19:40:34.099369 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:34.099335    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7e77f59c-fa04-4fef-80f6-6da7f2fbf046-tls-cert\") pod \"authorino-6ccf875c55-rz2d8\" (UID: \"7e77f59c-fa04-4fef-80f6-6da7f2fbf046\") " pod="kuadrant-system/authorino-6ccf875c55-rz2d8"
Nov 03 19:40:34.099369 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:34.099369    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv6hv\" (UniqueName: \"kubernetes.io/projected/7e77f59c-fa04-4fef-80f6-6da7f2fbf046-kube-api-access-rv6hv\") pod \"authorino-6ccf875c55-rz2d8\" (UID: \"7e77f59c-fa04-4fef-80f6-6da7f2fbf046\") " pod="kuadrant-system/authorino-6ccf875c55-rz2d8"
Nov 03 19:40:34.101908 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:34.101884    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7e77f59c-fa04-4fef-80f6-6da7f2fbf046-tls-cert\") pod \"authorino-6ccf875c55-rz2d8\" (UID: \"7e77f59c-fa04-4fef-80f6-6da7f2fbf046\") " pod="kuadrant-system/authorino-6ccf875c55-rz2d8"
Nov 03 19:40:34.106248 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:34.106223    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv6hv\" (UniqueName: \"kubernetes.io/projected/7e77f59c-fa04-4fef-80f6-6da7f2fbf046-kube-api-access-rv6hv\") pod \"authorino-6ccf875c55-rz2d8\" (UID: \"7e77f59c-fa04-4fef-80f6-6da7f2fbf046\") " pod="kuadrant-system/authorino-6ccf875c55-rz2d8"
Nov 03 19:40:34.263421 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:34.263387    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6ccf875c55-rz2d8"
Nov 03 19:40:34.535750 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:34.535723    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6ccf875c55-rz2d8"]
Nov 03 19:40:34.538644 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:40:34.538605    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7e77f59c_fa04_4fef_80f6_6da7f2fbf046.slice/crio-306c37e64bb7505c26443e51beff3cf1c358b557958b23faa854fa62e9e6ea0e WatchSource:0}: Error finding container 306c37e64bb7505c26443e51beff3cf1c358b557958b23faa854fa62e9e6ea0e: Status 404 returned error can't find the container with id 306c37e64bb7505c26443e51beff3cf1c358b557958b23faa854fa62e9e6ea0e
Nov 03 19:40:35.025733 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:35.025699    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6ccf875c55-rz2d8" event={"ID":"7e77f59c-fa04-4fef-80f6-6da7f2fbf046","Type":"ContainerStarted","Data":"306c37e64bb7505c26443e51beff3cf1c358b557958b23faa854fa62e9e6ea0e"}
Nov 03 19:40:36.073859 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:36.073829    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6ccf875c55-rz2d8" event={"ID":"7e77f59c-fa04-4fef-80f6-6da7f2fbf046","Type":"ContainerStarted","Data":"3aa9603ba05064cebe8b3f3624bbf19eedfd28ec5fbcfc08dad252ff81d72176"}
Nov 03 19:40:36.091830 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:36.091771    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6ccf875c55-rz2d8" podStartSLOduration=2.332040847 podStartE2EDuration="3.091751373s" totalImagesPullingTime="759.710526ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:40:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:40:36.090509343 +0000 UTC m=+549.334474808" watchObservedRunningTime="2026-11-03 19:40:36.091751373 +0000 UTC m=+549.335716840"
Nov 03 19:40:36.126622 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:36.126580    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-654c79df68-hl2n6"]
Nov 03 19:40:36.126839 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:36.126814    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-654c79df68-hl2n6" podUID="44b11870-4ef3-4061-87bd-09fd3d0f47f8" containerName="authorino" containerID="cri-o://a33fb68b631d2d96f303ce68c66c4218e090408afd8e8663ea57652b6d987f2f" gracePeriod=30
Nov 03 19:40:36.399378 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:36.399348    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-654c79df68-hl2n6"
Nov 03 19:40:36.466099 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:36.466060    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44b11870-4ef3-4061-87bd-09fd3d0f47f8-kube-api-access-wmppb\" (UniqueName: \"kubernetes.io/projected/44b11870-4ef3-4061-87bd-09fd3d0f47f8-kube-api-access-wmppb\") pod \"44b11870-4ef3-4061-87bd-09fd3d0f47f8\" (UID: \"44b11870-4ef3-4061-87bd-09fd3d0f47f8\") "
Nov 03 19:40:36.466299 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:36.466151    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44b11870-4ef3-4061-87bd-09fd3d0f47f8-tls-cert\" (UniqueName: \"kubernetes.io/secret/44b11870-4ef3-4061-87bd-09fd3d0f47f8-tls-cert\") pod \"44b11870-4ef3-4061-87bd-09fd3d0f47f8\" (UID: \"44b11870-4ef3-4061-87bd-09fd3d0f47f8\") "
Nov 03 19:40:36.469512 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:36.469450    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44b11870-4ef3-4061-87bd-09fd3d0f47f8-kube-api-access-wmppb" pod "44b11870-4ef3-4061-87bd-09fd3d0f47f8" (UID: "44b11870-4ef3-4061-87bd-09fd3d0f47f8"). InnerVolumeSpecName "kube-api-access-wmppb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:40:36.480373 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:36.480336    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44b11870-4ef3-4061-87bd-09fd3d0f47f8-tls-cert" pod "44b11870-4ef3-4061-87bd-09fd3d0f47f8" (UID: "44b11870-4ef3-4061-87bd-09fd3d0f47f8"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:40:36.567361 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:36.567317    4921 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/44b11870-4ef3-4061-87bd-09fd3d0f47f8-tls-cert\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:40:36.567361 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:36.567345    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmppb\" (UniqueName: \"kubernetes.io/projected/44b11870-4ef3-4061-87bd-09fd3d0f47f8-kube-api-access-wmppb\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:40:37.081503 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:37.081462    4921 generic.go:356] "Generic (PLEG): container finished" podID="44b11870-4ef3-4061-87bd-09fd3d0f47f8" containerID="a33fb68b631d2d96f303ce68c66c4218e090408afd8e8663ea57652b6d987f2f" exitCode=0
Nov 03 19:40:37.082005 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:37.081533    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-654c79df68-hl2n6"
Nov 03 19:40:37.082005 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:37.081555    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-654c79df68-hl2n6" event={"ID":"44b11870-4ef3-4061-87bd-09fd3d0f47f8","Type":"ContainerDied","Data":"a33fb68b631d2d96f303ce68c66c4218e090408afd8e8663ea57652b6d987f2f"}
Nov 03 19:40:37.082005 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:37.081601    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-654c79df68-hl2n6" event={"ID":"44b11870-4ef3-4061-87bd-09fd3d0f47f8","Type":"ContainerDied","Data":"5eab2dbfd604b295b960b81d2efef4df1f866ae4d7903ecc8c280ee7863ad479"}
Nov 03 19:40:37.082005 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:37.081622    4921 scope.go:122] "RemoveContainer" containerID="a33fb68b631d2d96f303ce68c66c4218e090408afd8e8663ea57652b6d987f2f"
Nov 03 19:40:37.092880 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:37.092858    4921 scope.go:122] "RemoveContainer" containerID="a33fb68b631d2d96f303ce68c66c4218e090408afd8e8663ea57652b6d987f2f"
Nov 03 19:40:37.093215 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:40:37.093195    4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a33fb68b631d2d96f303ce68c66c4218e090408afd8e8663ea57652b6d987f2f\": container with ID starting with a33fb68b631d2d96f303ce68c66c4218e090408afd8e8663ea57652b6d987f2f not found: ID does not exist" containerID="a33fb68b631d2d96f303ce68c66c4218e090408afd8e8663ea57652b6d987f2f"
Nov 03 19:40:37.093292 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:37.093227    4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a33fb68b631d2d96f303ce68c66c4218e090408afd8e8663ea57652b6d987f2f"} err="failed to get container status \"a33fb68b631d2d96f303ce68c66c4218e090408afd8e8663ea57652b6d987f2f\": rpc error: code = NotFound desc = could not find container \"a33fb68b631d2d96f303ce68c66c4218e090408afd8e8663ea57652b6d987f2f\": container with ID starting with a33fb68b631d2d96f303ce68c66c4218e090408afd8e8663ea57652b6d987f2f not found: ID does not exist"
Nov 03 19:40:37.095731 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:37.095710    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-654c79df68-hl2n6"]
Nov 03 19:40:37.099077 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:37.099056    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-654c79df68-hl2n6"]
Nov 03 19:40:38.888800 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:38.888766    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44b11870-4ef3-4061-87bd-09fd3d0f47f8" path="/var/lib/kubelet/pods/44b11870-4ef3-4061-87bd-09fd3d0f47f8/volumes"
Nov 03 19:40:53.769059 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:53.769020    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rl9tr"]
Nov 03 19:40:53.769450 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:53.769428    4921 cpu_manager.go:414] "containerMap: removing container" podUID="44b11870-4ef3-4061-87bd-09fd3d0f47f8" containerName="authorino"
Nov 03 19:40:53.769450 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:53.769443    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44b11870-4ef3-4061-87bd-09fd3d0f47f8" containerName="authorino"
Nov 03 19:40:53.769550 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:53.769539    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="44b11870-4ef3-4061-87bd-09fd3d0f47f8" containerName="authorino"
Nov 03 19:40:53.772937 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:53.772915    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rl9tr"
Nov 03 19:40:53.786181 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:53.786153    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cz4nm"]
Nov 03 19:40:53.790384 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:53.790362    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rl9tr"]
Nov 03 19:40:53.790498 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:53.790449    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cz4nm"
Nov 03 19:40:53.800343 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:53.800317    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cz4nm"]
Nov 03 19:40:53.923150 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:53.923104    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-rl9tr\" (UID: \"cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rl9tr"
Nov 03 19:40:53.923150 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:53.923148    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5053431a-0bcc-4e99-8776-c226339e3bde-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-cz4nm\" (UID: \"5053431a-0bcc-4e99-8776-c226339e3bde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cz4nm"
Nov 03 19:40:53.923376 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:53.923225    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-rl9tr\" (UID: \"cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rl9tr"
Nov 03 19:40:53.923376 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:53.923277    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5053431a-0bcc-4e99-8776-c226339e3bde-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-cz4nm\" (UID: \"5053431a-0bcc-4e99-8776-c226339e3bde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cz4nm"
Nov 03 19:40:53.923376 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:53.923305    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5053431a-0bcc-4e99-8776-c226339e3bde-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-cz4nm\" (UID: \"5053431a-0bcc-4e99-8776-c226339e3bde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cz4nm"
Nov 03 19:40:53.923376 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:53.923326    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5053431a-0bcc-4e99-8776-c226339e3bde-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-cz4nm\" (UID: \"5053431a-0bcc-4e99-8776-c226339e3bde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cz4nm"
Nov 03 19:40:53.923376 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:53.923343    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-rl9tr\" (UID: \"cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rl9tr"
Nov 03 19:40:53.923376 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:53.923369    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-rl9tr\" (UID: \"cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rl9tr"
Nov 03 19:40:53.923658 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:53.923530    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-rl9tr\" (UID: \"cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rl9tr"
Nov 03 19:40:53.923658 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:53.923566    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-rl9tr\" (UID: \"cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rl9tr"
Nov 03 19:40:53.923658 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:53.923589    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-rl9tr\" (UID: \"cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rl9tr"
Nov 03 19:40:53.923658 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:53.923640    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-rl9tr\" (UID: \"cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rl9tr"
Nov 03 19:40:53.923892 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:53.923736    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5053431a-0bcc-4e99-8776-c226339e3bde-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-cz4nm\" (UID: \"5053431a-0bcc-4e99-8776-c226339e3bde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cz4nm"
Nov 03 19:40:53.923892 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:53.923840    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5053431a-0bcc-4e99-8776-c226339e3bde-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-cz4nm\" (UID: \"5053431a-0bcc-4e99-8776-c226339e3bde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cz4nm"
Nov 03 19:40:53.923892 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:53.923867    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5053431a-0bcc-4e99-8776-c226339e3bde-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-cz4nm\" (UID: \"5053431a-0bcc-4e99-8776-c226339e3bde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cz4nm"
Nov 03 19:40:53.924046 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:53.923892    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5053431a-0bcc-4e99-8776-c226339e3bde-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-cz4nm\" (UID: \"5053431a-0bcc-4e99-8776-c226339e3bde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cz4nm"
Nov 03 19:40:53.924046 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:53.923960    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkjw6\" (UniqueName: \"kubernetes.io/projected/5053431a-0bcc-4e99-8776-c226339e3bde-kube-api-access-hkjw6\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-cz4nm\" (UID: \"5053431a-0bcc-4e99-8776-c226339e3bde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cz4nm"
Nov 03 19:40:53.924046 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:53.924002    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spz2z\" (UniqueName: \"kubernetes.io/projected/cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd-kube-api-access-spz2z\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-rl9tr\" (UID: \"cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rl9tr"
Nov 03 19:40:54.024530 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:54.024439    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spz2z\" (UniqueName: \"kubernetes.io/projected/cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd-kube-api-access-spz2z\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-rl9tr\" (UID: \"cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rl9tr"
Nov 03 19:40:54.024530 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:54.024478    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-rl9tr\" (UID: \"cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rl9tr"
Nov 03 19:40:54.024530 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:54.024495    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5053431a-0bcc-4e99-8776-c226339e3bde-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-cz4nm\" (UID: \"5053431a-0bcc-4e99-8776-c226339e3bde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cz4nm"
Nov 03 19:40:54.024530 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:54.024515    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-rl9tr\" (UID: \"cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rl9tr"
Nov 03 19:40:54.024530 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:54.024532    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5053431a-0bcc-4e99-8776-c226339e3bde-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-cz4nm\" (UID: \"5053431a-0bcc-4e99-8776-c226339e3bde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cz4nm"
Nov 03 19:40:54.024987 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:54.024551    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5053431a-0bcc-4e99-8776-c226339e3bde-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-cz4nm\" (UID: \"5053431a-0bcc-4e99-8776-c226339e3bde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cz4nm"
Nov 03 19:40:54.024987 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:54.024573    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5053431a-0bcc-4e99-8776-c226339e3bde-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-cz4nm\" (UID: \"5053431a-0bcc-4e99-8776-c226339e3bde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cz4nm"
Nov 03 19:40:54.024987 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:54.024601    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-rl9tr\" (UID: \"cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rl9tr"
Nov 03 19:40:54.024987 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:54.024625    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-rl9tr\" (UID: \"cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rl9tr"
Nov 03 19:40:54.024987 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:54.024656    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-rl9tr\" (UID: \"cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rl9tr"
Nov 03 19:40:54.024987 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:54.024710    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-rl9tr\" (UID: \"cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rl9tr"
Nov 03 19:40:54.024987 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:54.024742    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-rl9tr\" (UID: \"cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rl9tr"
Nov 03 19:40:54.024987 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:54.024773    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-rl9tr\" (UID: \"cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rl9tr"
Nov 03 19:40:54.024987 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:54.024803    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5053431a-0bcc-4e99-8776-c226339e3bde-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-cz4nm\" (UID: \"5053431a-0bcc-4e99-8776-c226339e3bde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cz4nm"
Nov 03 19:40:54.024987 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:54.024894    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5053431a-0bcc-4e99-8776-c226339e3bde-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-cz4nm\" (UID: \"5053431a-0bcc-4e99-8776-c226339e3bde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cz4nm"
Nov 03 19:40:54.024987 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:54.024922    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5053431a-0bcc-4e99-8776-c226339e3bde-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-cz4nm\" (UID: \"5053431a-0bcc-4e99-8776-c226339e3bde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cz4nm"
Nov 03 19:40:54.024987 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:54.024948    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5053431a-0bcc-4e99-8776-c226339e3bde-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-cz4nm\" (UID: \"5053431a-0bcc-4e99-8776-c226339e3bde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cz4nm"
Nov 03 19:40:54.024987 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:54.024980    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkjw6\" (UniqueName: \"kubernetes.io/projected/5053431a-0bcc-4e99-8776-c226339e3bde-kube-api-access-hkjw6\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-cz4nm\" (UID: \"5053431a-0bcc-4e99-8776-c226339e3bde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cz4nm"
Nov 03 19:40:54.025696 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:54.025007    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5053431a-0bcc-4e99-8776-c226339e3bde-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-cz4nm\" (UID: \"5053431a-0bcc-4e99-8776-c226339e3bde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cz4nm"
Nov 03 19:40:54.025696 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:54.025033    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-rl9tr\" (UID: \"cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rl9tr"
Nov 03 19:40:54.025696 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:54.025281    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5053431a-0bcc-4e99-8776-c226339e3bde-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-cz4nm\" (UID: \"5053431a-0bcc-4e99-8776-c226339e3bde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cz4nm"
Nov 03 19:40:54.025696 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:54.025373    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-rl9tr\" (UID: \"cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rl9tr"
Nov 03 19:40:54.025696 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:54.025460    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5053431a-0bcc-4e99-8776-c226339e3bde-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-cz4nm\" (UID: \"5053431a-0bcc-4e99-8776-c226339e3bde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cz4nm"
Nov 03 19:40:54.025696 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:54.025564    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-rl9tr\" (UID: \"cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rl9tr"
Nov 03 19:40:54.026046 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:54.025740    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-rl9tr\" (UID: \"cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rl9tr"
Nov 03 19:40:54.026046 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:54.025758    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5053431a-0bcc-4e99-8776-c226339e3bde-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-cz4nm\" (UID: \"5053431a-0bcc-4e99-8776-c226339e3bde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cz4nm"
Nov 03 19:40:54.026046 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:54.025920    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-rl9tr\" (UID: \"cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rl9tr"
Nov 03 19:40:54.026046 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:54.025959    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5053431a-0bcc-4e99-8776-c226339e3bde-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-cz4nm\" (UID: \"5053431a-0bcc-4e99-8776-c226339e3bde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cz4nm"
Nov 03 19:40:54.027427 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:54.027400    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5053431a-0bcc-4e99-8776-c226339e3bde-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-cz4nm\" (UID: \"5053431a-0bcc-4e99-8776-c226339e3bde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cz4nm"
Nov 03 19:40:54.027532 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:54.027406    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-rl9tr\" (UID: \"cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rl9tr"
Nov 03 19:40:54.027805 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:54.027789    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5053431a-0bcc-4e99-8776-c226339e3bde-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-cz4nm\" (UID: \"5053431a-0bcc-4e99-8776-c226339e3bde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cz4nm"
Nov 03 19:40:54.027940 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:54.027924    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-rl9tr\" (UID: \"cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rl9tr"
Nov 03 19:40:54.031456 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:54.031431    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spz2z\" (UniqueName: \"kubernetes.io/projected/cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd-kube-api-access-spz2z\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-rl9tr\" (UID: \"cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rl9tr"
Nov 03 19:40:54.031550 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:54.031532    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-rl9tr\" (UID: \"cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rl9tr"
Nov 03 19:40:54.031723 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:54.031705    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5053431a-0bcc-4e99-8776-c226339e3bde-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-cz4nm\" (UID: \"5053431a-0bcc-4e99-8776-c226339e3bde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cz4nm"
Nov 03 19:40:54.032147 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:54.032131    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkjw6\" (UniqueName: \"kubernetes.io/projected/5053431a-0bcc-4e99-8776-c226339e3bde-kube-api-access-hkjw6\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-cz4nm\" (UID: \"5053431a-0bcc-4e99-8776-c226339e3bde\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cz4nm"
Nov 03 19:40:54.092257 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:54.092218    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rl9tr"
Nov 03 19:40:54.105594 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:54.105570    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cz4nm"
Nov 03 19:40:54.380806 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:54.380779    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rl9tr"]
Nov 03 19:40:54.382265 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:40:54.382237    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcce688e1_b7b2_49b6_bb88_21bbc4d1c9cd.slice/crio-b29660423e049ddb04ef59ad053fa3fc392a5a275046f2c26df3cd5c92f1ac35 WatchSource:0}: Error finding container b29660423e049ddb04ef59ad053fa3fc392a5a275046f2c26df3cd5c92f1ac35: Status 404 returned error can't find the container with id b29660423e049ddb04ef59ad053fa3fc392a5a275046f2c26df3cd5c92f1ac35
Nov 03 19:40:54.385140 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:54.385109    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:40:54.385228 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:54.385171    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:40:54.385228 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:54.385200    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:40:54.397103 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:54.397074    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cz4nm"]
Nov 03 19:40:54.400439 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:40:54.400412    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5053431a_0bcc_4e99_8776_c226339e3bde.slice/crio-12ce2ba5e434ff5cae42d553f3e41c61a8439ac637b7b2cac5fa98f751e88ba8 WatchSource:0}: Error finding container 12ce2ba5e434ff5cae42d553f3e41c61a8439ac637b7b2cac5fa98f751e88ba8: Status 404 returned error can't find the container with id 12ce2ba5e434ff5cae42d553f3e41c61a8439ac637b7b2cac5fa98f751e88ba8
Nov 03 19:40:54.402810 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:54.402778    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:40:54.402890 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:54.402841    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:40:54.402890 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:54.402868    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:40:55.203363 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:55.203303    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rl9tr" event={"ID":"cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd","Type":"ContainerStarted","Data":"76fb700ed57858ed7215c3174791b2cfba78745c09e53fedd4ea62b8e92c7075"}
Nov 03 19:40:55.203989 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:55.203948    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rl9tr" event={"ID":"cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd","Type":"ContainerStarted","Data":"b29660423e049ddb04ef59ad053fa3fc392a5a275046f2c26df3cd5c92f1ac35"}
Nov 03 19:40:55.206925 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:55.206889    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cz4nm" event={"ID":"5053431a-0bcc-4e99-8776-c226339e3bde","Type":"ContainerStarted","Data":"6235f125f8d878227dbf12bdbc7a5620eac4d88fca5f3a03211ae52d8c58c8e4"}
Nov 03 19:40:55.207081 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:55.206951    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cz4nm" event={"ID":"5053431a-0bcc-4e99-8776-c226339e3bde","Type":"ContainerStarted","Data":"12ce2ba5e434ff5cae42d553f3e41c61a8439ac637b7b2cac5fa98f751e88ba8"}
Nov 03 19:40:55.224564 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:55.224514    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rl9tr" podStartSLOduration=2.224500301 podStartE2EDuration="2.224500301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:40:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:40:55.22300085 +0000 UTC m=+568.466966315" watchObservedRunningTime="2026-11-03 19:40:55.224500301 +0000 UTC m=+568.468465765"
Nov 03 19:40:55.241131 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:55.241081    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cz4nm" podStartSLOduration=2.241068207 podStartE2EDuration="2.241068207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:40:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:40:55.239030764 +0000 UTC m=+568.482996229" watchObservedRunningTime="2026-11-03 19:40:55.241068207 +0000 UTC m=+568.485033672"
Nov 03 19:40:56.095372 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:56.095337    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rl9tr"
Nov 03 19:40:56.095732 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:56.095708    4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-rl9tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Nov 03 19:40:56.095902 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:56.095879    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rl9tr" podUID="cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Nov 03 19:40:56.106627 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:56.106602    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cz4nm"
Nov 03 19:40:56.107849 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:56.107820    4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-cz4nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Nov 03 19:40:56.107935 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:56.107857    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cz4nm" podUID="5053431a-0bcc-4e99-8776-c226339e3bde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Nov 03 19:40:57.099087 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:57.099046    4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-rl9tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Nov 03 19:40:57.099604 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:57.099116    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rl9tr" podUID="cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Nov 03 19:40:57.106508 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:57.106481    4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-cz4nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Nov 03 19:40:57.106628 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:57.106546    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cz4nm" podUID="5053431a-0bcc-4e99-8776-c226339e3bde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Nov 03 19:40:58.093166 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:58.093125    4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-rl9tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Nov 03 19:40:58.093588 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:58.093551    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rl9tr" podUID="cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Nov 03 19:40:58.105996 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:58.105959    4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-cz4nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Nov 03 19:40:58.106372 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:58.106013    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cz4nm" podUID="5053431a-0bcc-4e99-8776-c226339e3bde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Nov 03 19:40:59.096145 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:59.096104    4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-rl9tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Nov 03 19:40:59.096455 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:59.096428    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rl9tr" podUID="cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Nov 03 19:40:59.106911 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:59.106878    4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-cz4nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Nov 03 19:40:59.107288 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:40:59.106927    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cz4nm" podUID="5053431a-0bcc-4e99-8776-c226339e3bde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Nov 03 19:41:00.095644 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:00.095145    4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-rl9tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Nov 03 19:41:00.095644 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:00.095226    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rl9tr" podUID="cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Nov 03 19:41:00.106311 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:00.106279    4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-cz4nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Nov 03 19:41:00.106491 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:00.106352    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cz4nm" podUID="5053431a-0bcc-4e99-8776-c226339e3bde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Nov 03 19:41:01.093277 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:01.093222    4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-rl9tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Nov 03 19:41:01.093850 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:01.093322    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rl9tr" podUID="cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Nov 03 19:41:01.106156 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:01.106118    4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-cz4nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Nov 03 19:41:01.106365 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:01.106185    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cz4nm" podUID="5053431a-0bcc-4e99-8776-c226339e3bde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Nov 03 19:41:02.733950 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:02.733896    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cz4nm"
Nov 03 19:41:02.734746 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:02.734707    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cz4nm"
Nov 03 19:41:02.735695 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:02.735644    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cz4nm"
Nov 03 19:41:02.791239 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:02.791200    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rl9tr"
Nov 03 19:41:02.792476 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:02.792453    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rl9tr"
Nov 03 19:41:02.793152 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:02.793128    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rl9tr"
Nov 03 19:41:02.920844 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:41:02.920810    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 19:41:26.947471 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:26.947394    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gzf8k_6f215b42-0f09-46a0-bc98-392c0fa909c9/console-operator/2.log"
Nov 03 19:41:26.950021 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:26.949998    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gzf8k_6f215b42-0f09-46a0-bc98-392c0fa909c9/console-operator/2.log"
Nov 03 19:41:32.594959 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:32.594905    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9"]
Nov 03 19:41:32.601043 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:32.601017    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9"
Nov 03 19:41:32.605418 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:32.605391    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-0486e8-openshift-default-dockercfg-4ckf8\""
Nov 03 19:41:32.611889 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:32.611864    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9"]
Nov 03 19:41:32.735856 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:32.735813    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/902fa980-da03-4354-8484-ad392278c1d9-istio-podinfo\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9\" (UID: \"902fa980-da03-4354-8484-ad392278c1d9\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9"
Nov 03 19:41:32.735856 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:32.735855    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/902fa980-da03-4354-8484-ad392278c1d9-istiod-ca-cert\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9\" (UID: \"902fa980-da03-4354-8484-ad392278c1d9\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9"
Nov 03 19:41:32.736088 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:32.735935    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf7hp\" (UniqueName: \"kubernetes.io/projected/902fa980-da03-4354-8484-ad392278c1d9-kube-api-access-jf7hp\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9\" (UID: \"902fa980-da03-4354-8484-ad392278c1d9\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9"
Nov 03 19:41:32.736088 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:32.736022    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/902fa980-da03-4354-8484-ad392278c1d9-istio-envoy\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9\" (UID: \"902fa980-da03-4354-8484-ad392278c1d9\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9"
Nov 03 19:41:32.736088 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:32.736053    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/902fa980-da03-4354-8484-ad392278c1d9-workload-socket\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9\" (UID: \"902fa980-da03-4354-8484-ad392278c1d9\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9"
Nov 03 19:41:32.736088 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:32.736077    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/902fa980-da03-4354-8484-ad392278c1d9-istio-data\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9\" (UID: \"902fa980-da03-4354-8484-ad392278c1d9\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9"
Nov 03 19:41:32.736333 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:32.736129    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/902fa980-da03-4354-8484-ad392278c1d9-istio-token\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9\" (UID: \"902fa980-da03-4354-8484-ad392278c1d9\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9"
Nov 03 19:41:32.736333 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:32.736190    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/902fa980-da03-4354-8484-ad392278c1d9-workload-certs\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9\" (UID: \"902fa980-da03-4354-8484-ad392278c1d9\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9"
Nov 03 19:41:32.736333 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:32.736217    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/902fa980-da03-4354-8484-ad392278c1d9-credential-socket\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9\" (UID: \"902fa980-da03-4354-8484-ad392278c1d9\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9"
Nov 03 19:41:32.837658 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:32.837627    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/902fa980-da03-4354-8484-ad392278c1d9-istio-envoy\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9\" (UID: \"902fa980-da03-4354-8484-ad392278c1d9\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9"
Nov 03 19:41:32.837866 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:32.837679    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/902fa980-da03-4354-8484-ad392278c1d9-workload-socket\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9\" (UID: \"902fa980-da03-4354-8484-ad392278c1d9\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9"
Nov 03 19:41:32.837866 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:32.837708    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/902fa980-da03-4354-8484-ad392278c1d9-istio-data\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9\" (UID: \"902fa980-da03-4354-8484-ad392278c1d9\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9"
Nov 03 19:41:32.837866 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:32.837734    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/902fa980-da03-4354-8484-ad392278c1d9-istio-token\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9\" (UID: \"902fa980-da03-4354-8484-ad392278c1d9\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9"
Nov 03 19:41:32.837866 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:32.837771    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/902fa980-da03-4354-8484-ad392278c1d9-workload-certs\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9\" (UID: \"902fa980-da03-4354-8484-ad392278c1d9\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9"
Nov 03 19:41:32.837866 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:32.837807    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/902fa980-da03-4354-8484-ad392278c1d9-credential-socket\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9\" (UID: \"902fa980-da03-4354-8484-ad392278c1d9\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9"
Nov 03 19:41:32.838136 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:32.838001    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/902fa980-da03-4354-8484-ad392278c1d9-istio-podinfo\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9\" (UID: \"902fa980-da03-4354-8484-ad392278c1d9\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9"
Nov 03 19:41:32.838136 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:32.838041    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/902fa980-da03-4354-8484-ad392278c1d9-istiod-ca-cert\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9\" (UID: \"902fa980-da03-4354-8484-ad392278c1d9\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9"
Nov 03 19:41:32.838136 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:32.838089    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf7hp\" (UniqueName: \"kubernetes.io/projected/902fa980-da03-4354-8484-ad392278c1d9-kube-api-access-jf7hp\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9\" (UID: \"902fa980-da03-4354-8484-ad392278c1d9\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9"
Nov 03 19:41:32.838287 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:32.838179    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/902fa980-da03-4354-8484-ad392278c1d9-workload-socket\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9\" (UID: \"902fa980-da03-4354-8484-ad392278c1d9\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9"
Nov 03 19:41:32.838287 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:32.838207    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/902fa980-da03-4354-8484-ad392278c1d9-istio-data\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9\" (UID: \"902fa980-da03-4354-8484-ad392278c1d9\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9"
Nov 03 19:41:32.838287 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:32.838252    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/902fa980-da03-4354-8484-ad392278c1d9-credential-socket\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9\" (UID: \"902fa980-da03-4354-8484-ad392278c1d9\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9"
Nov 03 19:41:32.838617 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:32.838590    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/902fa980-da03-4354-8484-ad392278c1d9-workload-certs\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9\" (UID: \"902fa980-da03-4354-8484-ad392278c1d9\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9"
Nov 03 19:41:32.838898 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:32.838871    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/902fa980-da03-4354-8484-ad392278c1d9-istiod-ca-cert\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9\" (UID: \"902fa980-da03-4354-8484-ad392278c1d9\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9"
Nov 03 19:41:32.840568 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:32.840547    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/902fa980-da03-4354-8484-ad392278c1d9-istio-envoy\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9\" (UID: \"902fa980-da03-4354-8484-ad392278c1d9\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9"
Nov 03 19:41:32.841255 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:32.841235    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/902fa980-da03-4354-8484-ad392278c1d9-istio-podinfo\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9\" (UID: \"902fa980-da03-4354-8484-ad392278c1d9\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9"
Nov 03 19:41:32.845458 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:32.845406    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/902fa980-da03-4354-8484-ad392278c1d9-istio-token\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9\" (UID: \"902fa980-da03-4354-8484-ad392278c1d9\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9"
Nov 03 19:41:32.845940 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:32.845923    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf7hp\" (UniqueName: \"kubernetes.io/projected/902fa980-da03-4354-8484-ad392278c1d9-kube-api-access-jf7hp\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9\" (UID: \"902fa980-da03-4354-8484-ad392278c1d9\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9"
Nov 03 19:41:32.919243 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:32.918949    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9"
Nov 03 19:41:32.968735 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:41:32.968651    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 19:41:33.230391 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:33.230298    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9"]
Nov 03 19:41:33.232922 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:41:33.232887    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod902fa980_da03_4354_8484_ad392278c1d9.slice/crio-5177993d834aedf1282ea91a6ed07b2f6debdf5a42d57e1b262ae2640c7c963f WatchSource:0}: Error finding container 5177993d834aedf1282ea91a6ed07b2f6debdf5a42d57e1b262ae2640c7c963f: Status 404 returned error can't find the container with id 5177993d834aedf1282ea91a6ed07b2f6debdf5a42d57e1b262ae2640c7c963f
Nov 03 19:41:33.235178 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:33.235141    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:41:33.235290 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:33.235207    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:41:33.235290 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:33.235236    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:41:33.422884 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:33.422844    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9" event={"ID":"902fa980-da03-4354-8484-ad392278c1d9","Type":"ContainerStarted","Data":"00ffdf26993b52e3e6c8035e50c8b0e14c3f4ffbf8f617d1e7d2886d27a289fa"}
Nov 03 19:41:33.423094 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:33.422943    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9" event={"ID":"902fa980-da03-4354-8484-ad392278c1d9","Type":"ContainerStarted","Data":"5177993d834aedf1282ea91a6ed07b2f6debdf5a42d57e1b262ae2640c7c963f"}
Nov 03 19:41:33.441689 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:33.440933    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9" podStartSLOduration=1.4409182440000001 podStartE2EDuration="1.440918244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:41:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:41:33.44062415 +0000 UTC m=+606.684589620" watchObservedRunningTime="2026-11-03 19:41:33.440918244 +0000 UTC m=+606.684883708"
Nov 03 19:41:33.920720 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:33.920658    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9"
Nov 03 19:41:34.925431 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:34.925400    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9"
Nov 03 19:41:35.436649 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:35.436614    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9"
Nov 03 19:41:35.437385 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:35.437367    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9"
Nov 03 19:41:39.139592 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:39.139559    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-a35658-ff9b85698-4wqq2"]
Nov 03 19:41:39.143643 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:39.143616    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a35658-ff9b85698-4wqq2"
Nov 03 19:41:39.154859 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:39.154790    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-a35658-ff9b85698-4wqq2"]
Nov 03 19:41:39.172365 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:39.172332    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-a35658-serving-cert\""
Nov 03 19:41:39.172539 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:39.172487    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rj6j4\""
Nov 03 19:41:39.172539 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:39.172539    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 03 19:41:39.196822 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:39.196788    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-8d4127-6fb576bf67-qr2v6"]
Nov 03 19:41:39.200409 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:39.200393    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8d4127-6fb576bf67-qr2v6"
Nov 03 19:41:39.211998 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:39.211968    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-8d4127-dockercfg-jdf88\""
Nov 03 19:41:39.212642 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:39.212559    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-8d4127\""
Nov 03 19:41:39.218332 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:39.218203    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-8d4127-6fb576bf67-qr2v6"]
Nov 03 19:41:39.219482 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:39.219456    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bda8b9f-49ce-45d8-9c50-3e56a865b849-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-8d4127-6fb576bf67-qr2v6\" (UID: \"4bda8b9f-49ce-45d8-9c50-3e56a865b849\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8d4127-6fb576bf67-qr2v6"
Nov 03 19:41:39.219589 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:39.219512    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f48e7cf-0a2d-44ff-8949-77d277a40138-metrics-certs\") pod \"maas-api-e2e-worker-w1-a35658-ff9b85698-4wqq2\" (UID: \"0f48e7cf-0a2d-44ff-8949-77d277a40138\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a35658-ff9b85698-4wqq2"
Nov 03 19:41:39.219654 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:39.219612    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f48e7cf-0a2d-44ff-8949-77d277a40138-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-a35658-ff9b85698-4wqq2\" (UID: \"0f48e7cf-0a2d-44ff-8949-77d277a40138\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a35658-ff9b85698-4wqq2"
Nov 03 19:41:39.219783 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:39.219681    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwg9q\" (UniqueName: \"kubernetes.io/projected/4bda8b9f-49ce-45d8-9c50-3e56a865b849-kube-api-access-xwg9q\") pod \"payload-pre-processing-e2e-worker-w2-8d4127-6fb576bf67-qr2v6\" (UID: \"4bda8b9f-49ce-45d8-9c50-3e56a865b849\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8d4127-6fb576bf67-qr2v6"
Nov 03 19:41:39.219783 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:39.219730    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2pzm\" (UniqueName: \"kubernetes.io/projected/0f48e7cf-0a2d-44ff-8949-77d277a40138-kube-api-access-x2pzm\") pod \"maas-api-e2e-worker-w1-a35658-ff9b85698-4wqq2\" (UID: \"0f48e7cf-0a2d-44ff-8949-77d277a40138\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a35658-ff9b85698-4wqq2"
Nov 03 19:41:39.219783 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:39.219777    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f48e7cf-0a2d-44ff-8949-77d277a40138-maas-api-tls\") pod \"maas-api-e2e-worker-w1-a35658-ff9b85698-4wqq2\" (UID: \"0f48e7cf-0a2d-44ff-8949-77d277a40138\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a35658-ff9b85698-4wqq2"
Nov 03 19:41:39.320612 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:39.320558    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f48e7cf-0a2d-44ff-8949-77d277a40138-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-a35658-ff9b85698-4wqq2\" (UID: \"0f48e7cf-0a2d-44ff-8949-77d277a40138\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a35658-ff9b85698-4wqq2"
Nov 03 19:41:39.320784 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:39.320633    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwg9q\" (UniqueName: \"kubernetes.io/projected/4bda8b9f-49ce-45d8-9c50-3e56a865b849-kube-api-access-xwg9q\") pod \"payload-pre-processing-e2e-worker-w2-8d4127-6fb576bf67-qr2v6\" (UID: \"4bda8b9f-49ce-45d8-9c50-3e56a865b849\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8d4127-6fb576bf67-qr2v6"
Nov 03 19:41:39.320784 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:39.320679    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2pzm\" (UniqueName: \"kubernetes.io/projected/0f48e7cf-0a2d-44ff-8949-77d277a40138-kube-api-access-x2pzm\") pod \"maas-api-e2e-worker-w1-a35658-ff9b85698-4wqq2\" (UID: \"0f48e7cf-0a2d-44ff-8949-77d277a40138\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a35658-ff9b85698-4wqq2"
Nov 03 19:41:39.320784 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:39.320729    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f48e7cf-0a2d-44ff-8949-77d277a40138-maas-api-tls\") pod \"maas-api-e2e-worker-w1-a35658-ff9b85698-4wqq2\" (UID: \"0f48e7cf-0a2d-44ff-8949-77d277a40138\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a35658-ff9b85698-4wqq2"
Nov 03 19:41:39.320784 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:39.320758    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bda8b9f-49ce-45d8-9c50-3e56a865b849-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-8d4127-6fb576bf67-qr2v6\" (UID: \"4bda8b9f-49ce-45d8-9c50-3e56a865b849\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8d4127-6fb576bf67-qr2v6"
Nov 03 19:41:39.320942 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:39.320846    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f48e7cf-0a2d-44ff-8949-77d277a40138-metrics-certs\") pod \"maas-api-e2e-worker-w1-a35658-ff9b85698-4wqq2\" (UID: \"0f48e7cf-0a2d-44ff-8949-77d277a40138\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a35658-ff9b85698-4wqq2"
Nov 03 19:41:39.321381 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:39.321359    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f48e7cf-0a2d-44ff-8949-77d277a40138-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-a35658-ff9b85698-4wqq2\" (UID: \"0f48e7cf-0a2d-44ff-8949-77d277a40138\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a35658-ff9b85698-4wqq2"
Nov 03 19:41:39.321560 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:39.321539    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bda8b9f-49ce-45d8-9c50-3e56a865b849-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-8d4127-6fb576bf67-qr2v6\" (UID: \"4bda8b9f-49ce-45d8-9c50-3e56a865b849\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8d4127-6fb576bf67-qr2v6"
Nov 03 19:41:39.323226 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:39.323205    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f48e7cf-0a2d-44ff-8949-77d277a40138-metrics-certs\") pod \"maas-api-e2e-worker-w1-a35658-ff9b85698-4wqq2\" (UID: \"0f48e7cf-0a2d-44ff-8949-77d277a40138\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a35658-ff9b85698-4wqq2"
Nov 03 19:41:39.323324 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:39.323252    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f48e7cf-0a2d-44ff-8949-77d277a40138-maas-api-tls\") pod \"maas-api-e2e-worker-w1-a35658-ff9b85698-4wqq2\" (UID: \"0f48e7cf-0a2d-44ff-8949-77d277a40138\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a35658-ff9b85698-4wqq2"
Nov 03 19:41:39.328342 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:39.328315    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwg9q\" (UniqueName: \"kubernetes.io/projected/4bda8b9f-49ce-45d8-9c50-3e56a865b849-kube-api-access-xwg9q\") pod \"payload-pre-processing-e2e-worker-w2-8d4127-6fb576bf67-qr2v6\" (UID: \"4bda8b9f-49ce-45d8-9c50-3e56a865b849\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8d4127-6fb576bf67-qr2v6"
Nov 03 19:41:39.328505 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:39.328489    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2pzm\" (UniqueName: \"kubernetes.io/projected/0f48e7cf-0a2d-44ff-8949-77d277a40138-kube-api-access-x2pzm\") pod \"maas-api-e2e-worker-w1-a35658-ff9b85698-4wqq2\" (UID: \"0f48e7cf-0a2d-44ff-8949-77d277a40138\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a35658-ff9b85698-4wqq2"
Nov 03 19:41:39.462127 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:39.462047    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a35658-ff9b85698-4wqq2"
Nov 03 19:41:39.518966 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:39.518923    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8d4127-6fb576bf67-qr2v6"
Nov 03 19:41:39.748645 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:39.748620    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-a35658-ff9b85698-4wqq2"]
Nov 03 19:41:39.750803 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:41:39.750775    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f48e7cf_0a2d_44ff_8949_77d277a40138.slice/crio-f40ba4ff70c39c7c3016bbb74775e528da3688768440e79ae1d59099a831c787 WatchSource:0}: Error finding container f40ba4ff70c39c7c3016bbb74775e528da3688768440e79ae1d59099a831c787: Status 404 returned error can't find the container with id f40ba4ff70c39c7c3016bbb74775e528da3688768440e79ae1d59099a831c787
Nov 03 19:41:39.752013 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:39.751996    4921 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 19:41:39.808476 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:39.808453    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-8d4127-6fb576bf67-qr2v6"]
Nov 03 19:41:39.810160 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:41:39.810129    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bda8b9f_49ce_45d8_9c50_3e56a865b849.slice/crio-e4532155e908aef9d53551e331490ef3604e48c0ea70e2fc67d3bb8a5519f2cd WatchSource:0}: Error finding container e4532155e908aef9d53551e331490ef3604e48c0ea70e2fc67d3bb8a5519f2cd: Status 404 returned error can't find the container with id e4532155e908aef9d53551e331490ef3604e48c0ea70e2fc67d3bb8a5519f2cd
Nov 03 19:41:40.470895 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:40.470855    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8d4127-6fb576bf67-qr2v6" event={"ID":"4bda8b9f-49ce-45d8-9c50-3e56a865b849","Type":"ContainerStarted","Data":"e4532155e908aef9d53551e331490ef3604e48c0ea70e2fc67d3bb8a5519f2cd"}
Nov 03 19:41:40.472970 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:40.472938    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a35658-ff9b85698-4wqq2" event={"ID":"0f48e7cf-0a2d-44ff-8949-77d277a40138","Type":"ContainerStarted","Data":"f40ba4ff70c39c7c3016bbb74775e528da3688768440e79ae1d59099a831c787"}
Nov 03 19:41:41.520071 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:41.520031    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-102379-64bd4947ff-7qsss"]
Nov 03 19:41:41.524273 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:41.524244    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-102379-64bd4947ff-7qsss"
Nov 03 19:41:41.531235 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:41.530638    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-102379\""
Nov 03 19:41:41.531235 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:41.530744    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-102379-dockercfg-qg9q5\""
Nov 03 19:41:41.535264 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:41.535176    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-102379-64bd4947ff-7qsss"]
Nov 03 19:41:41.567198 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:41.567158    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06971289-0da8-4921-9df7-140ff477056c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-102379-64bd4947ff-7qsss\" (UID: \"06971289-0da8-4921-9df7-140ff477056c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-102379-64bd4947ff-7qsss"
Nov 03 19:41:41.567393 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:41.567286    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bdcj\" (UniqueName: \"kubernetes.io/projected/06971289-0da8-4921-9df7-140ff477056c-kube-api-access-6bdcj\") pod \"payload-pre-processing-e2e-worker-w6-102379-64bd4947ff-7qsss\" (UID: \"06971289-0da8-4921-9df7-140ff477056c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-102379-64bd4947ff-7qsss"
Nov 03 19:41:41.569000 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:41.568972    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-102379-5c958cc4ff-t94ls"]
Nov 03 19:41:41.573923 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:41.573893    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-102379-5c958cc4ff-t94ls"
Nov 03 19:41:41.583945 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:41.583907    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-102379-5c958cc4ff-t94ls"]
Nov 03 19:41:41.675976 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:41.667993    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfk9f\" (UniqueName: \"kubernetes.io/projected/2fd7fd41-37ac-4e6c-8042-2c7b4a1f0674-kube-api-access-bfk9f\") pod \"payload-processing-e2e-worker-w6-102379-5c958cc4ff-t94ls\" (UID: \"2fd7fd41-37ac-4e6c-8042-2c7b4a1f0674\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-102379-5c958cc4ff-t94ls"
Nov 03 19:41:41.675976 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:41.668100    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bdcj\" (UniqueName: \"kubernetes.io/projected/06971289-0da8-4921-9df7-140ff477056c-kube-api-access-6bdcj\") pod \"payload-pre-processing-e2e-worker-w6-102379-64bd4947ff-7qsss\" (UID: \"06971289-0da8-4921-9df7-140ff477056c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-102379-64bd4947ff-7qsss"
Nov 03 19:41:41.675976 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:41.668165    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fd7fd41-37ac-4e6c-8042-2c7b4a1f0674-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-102379-5c958cc4ff-t94ls\" (UID: \"2fd7fd41-37ac-4e6c-8042-2c7b4a1f0674\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-102379-5c958cc4ff-t94ls"
Nov 03 19:41:41.675976 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:41.668221    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06971289-0da8-4921-9df7-140ff477056c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-102379-64bd4947ff-7qsss\" (UID: \"06971289-0da8-4921-9df7-140ff477056c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-102379-64bd4947ff-7qsss"
Nov 03 19:41:41.675976 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:41.670722    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06971289-0da8-4921-9df7-140ff477056c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-102379-64bd4947ff-7qsss\" (UID: \"06971289-0da8-4921-9df7-140ff477056c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-102379-64bd4947ff-7qsss"
Nov 03 19:41:41.688760 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:41.682937    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bdcj\" (UniqueName: \"kubernetes.io/projected/06971289-0da8-4921-9df7-140ff477056c-kube-api-access-6bdcj\") pod \"payload-pre-processing-e2e-worker-w6-102379-64bd4947ff-7qsss\" (UID: \"06971289-0da8-4921-9df7-140ff477056c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-102379-64bd4947ff-7qsss"
Nov 03 19:41:41.769592 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:41.769550    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fd7fd41-37ac-4e6c-8042-2c7b4a1f0674-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-102379-5c958cc4ff-t94ls\" (UID: \"2fd7fd41-37ac-4e6c-8042-2c7b4a1f0674\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-102379-5c958cc4ff-t94ls"
Nov 03 19:41:41.769792 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:41.769695    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfk9f\" (UniqueName: \"kubernetes.io/projected/2fd7fd41-37ac-4e6c-8042-2c7b4a1f0674-kube-api-access-bfk9f\") pod \"payload-processing-e2e-worker-w6-102379-5c958cc4ff-t94ls\" (UID: \"2fd7fd41-37ac-4e6c-8042-2c7b4a1f0674\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-102379-5c958cc4ff-t94ls"
Nov 03 19:41:41.771110 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:41.771044    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fd7fd41-37ac-4e6c-8042-2c7b4a1f0674-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-102379-5c958cc4ff-t94ls\" (UID: \"2fd7fd41-37ac-4e6c-8042-2c7b4a1f0674\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-102379-5c958cc4ff-t94ls"
Nov 03 19:41:41.777654 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:41.777628    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfk9f\" (UniqueName: \"kubernetes.io/projected/2fd7fd41-37ac-4e6c-8042-2c7b4a1f0674-kube-api-access-bfk9f\") pod \"payload-processing-e2e-worker-w6-102379-5c958cc4ff-t94ls\" (UID: \"2fd7fd41-37ac-4e6c-8042-2c7b4a1f0674\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-102379-5c958cc4ff-t94ls"
Nov 03 19:41:41.857899 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:41.857863    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-102379-64bd4947ff-7qsss"
Nov 03 19:41:41.896863 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:41.896559    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-102379-5c958cc4ff-t94ls"
Nov 03 19:41:42.185647 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:42.185603    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-102379-64bd4947ff-7qsss"]
Nov 03 19:41:42.208155 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:41:42.208056    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06971289_0da8_4921_9df7_140ff477056c.slice/crio-88abdc46cfaab1695d8e2e6b89e0a202f6580dbc1aad407097c1f07be0aee1e6 WatchSource:0}: Error finding container 88abdc46cfaab1695d8e2e6b89e0a202f6580dbc1aad407097c1f07be0aee1e6: Status 404 returned error can't find the container with id 88abdc46cfaab1695d8e2e6b89e0a202f6580dbc1aad407097c1f07be0aee1e6
Nov 03 19:41:42.258122 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:42.258030    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-102379-5c958cc4ff-t94ls"]
Nov 03 19:41:42.491717 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:42.491679    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-102379-64bd4947ff-7qsss" event={"ID":"06971289-0da8-4921-9df7-140ff477056c","Type":"ContainerStarted","Data":"88abdc46cfaab1695d8e2e6b89e0a202f6580dbc1aad407097c1f07be0aee1e6"}
Nov 03 19:41:42.493215 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:42.493185    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-102379-5c958cc4ff-t94ls" event={"ID":"2fd7fd41-37ac-4e6c-8042-2c7b4a1f0674","Type":"ContainerStarted","Data":"d4180486282705bab8df29a8da4aa045c244b2e3bebd3139b2c462b1105509d4"}
Nov 03 19:41:43.513846 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:43.513803    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8d4127-6fb576bf67-qr2v6" event={"ID":"4bda8b9f-49ce-45d8-9c50-3e56a865b849","Type":"ContainerStarted","Data":"716c5ffad050785d674c0827a94fc0c872735446fa401b34607eadfa83ddb46c"}
Nov 03 19:41:43.514812 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:43.514758    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8d4127-6fb576bf67-qr2v6"
Nov 03 19:41:43.534025 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:43.533951    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8d4127-6fb576bf67-qr2v6" podStartSLOduration=0.988982562 podStartE2EDuration="4.53393676s" totalImagesPullingTime="3.544954198s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:41:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:41:43.530498038 +0000 UTC m=+616.774463526" watchObservedRunningTime="2026-11-03 19:41:43.53393676 +0000 UTC m=+616.777902247"
Nov 03 19:41:44.475843 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:44.475807    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-0486e8-5b894565b7-76jhb"]
Nov 03 19:41:44.480184 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:44.480161    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-0486e8-5b894565b7-76jhb"
Nov 03 19:41:44.489158 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:44.489131    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-0486e8-5b894565b7-76jhb"]
Nov 03 19:41:44.490997 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:44.490960    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-0486e8-serving-cert\""
Nov 03 19:41:44.520834 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:44.520792    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a35658-ff9b85698-4wqq2" event={"ID":"0f48e7cf-0a2d-44ff-8949-77d277a40138","Type":"ContainerStarted","Data":"dd806fd9ed8b5f223b5156a6144f5d95b4d89066479f7fa2ac5b353951a94457"}
Nov 03 19:41:44.521314 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:44.520853    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a35658-ff9b85698-4wqq2"
Nov 03 19:41:44.522476 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:44.522435    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-102379-64bd4947ff-7qsss" event={"ID":"06971289-0da8-4921-9df7-140ff477056c","Type":"ContainerStarted","Data":"6fbecabd718fedb595dc30d1e25f95eb5572e0ea3f98a6c73d1f0ad159af4614"}
Nov 03 19:41:44.522590 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:44.522495    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-102379-64bd4947ff-7qsss"
Nov 03 19:41:44.523977 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:44.523953    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-102379-5c958cc4ff-t94ls" event={"ID":"2fd7fd41-37ac-4e6c-8042-2c7b4a1f0674","Type":"ContainerStarted","Data":"fe8d07760a751fe0ac4ce6791008bcc8e054dd89fad7cae8d2698c419ba3d90a"}
Nov 03 19:41:44.524175 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:44.524163    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-102379-5c958cc4ff-t94ls"
Nov 03 19:41:44.533033 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:44.533012    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfb9124e-3670-46b5-aa88-702ec75bda1f-maas-api-tls\") pod \"maas-api-e2e-worker-w5-0486e8-5b894565b7-76jhb\" (UID: \"dfb9124e-3670-46b5-aa88-702ec75bda1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-0486e8-5b894565b7-76jhb"
Nov 03 19:41:44.533119 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:44.533045    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfb9124e-3670-46b5-aa88-702ec75bda1f-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-0486e8-5b894565b7-76jhb\" (UID: \"dfb9124e-3670-46b5-aa88-702ec75bda1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-0486e8-5b894565b7-76jhb"
Nov 03 19:41:44.533119 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:44.533079    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvxhh\" (UniqueName: \"kubernetes.io/projected/dfb9124e-3670-46b5-aa88-702ec75bda1f-kube-api-access-cvxhh\") pod \"maas-api-e2e-worker-w5-0486e8-5b894565b7-76jhb\" (UID: \"dfb9124e-3670-46b5-aa88-702ec75bda1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-0486e8-5b894565b7-76jhb"
Nov 03 19:41:44.533218 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:44.533207    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfb9124e-3670-46b5-aa88-702ec75bda1f-metrics-certs\") pod \"maas-api-e2e-worker-w5-0486e8-5b894565b7-76jhb\" (UID: \"dfb9124e-3670-46b5-aa88-702ec75bda1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-0486e8-5b894565b7-76jhb"
Nov 03 19:41:44.537144 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:44.537098    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a35658-ff9b85698-4wqq2" podStartSLOduration=1.2859949990000001 podStartE2EDuration="5.53708647s" totalImagesPullingTime="4.251091471s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:41:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:41:44.536473303 +0000 UTC m=+617.780438768" watchObservedRunningTime="2026-11-03 19:41:44.53708647 +0000 UTC m=+617.781051945"
Nov 03 19:41:44.550298 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:44.550247    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-102379-64bd4947ff-7qsss" podStartSLOduration=1.870698731 podStartE2EDuration="3.550234319s" totalImagesPullingTime="1.679535588s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:41:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:41:44.548895837 +0000 UTC m=+617.792861325" watchObservedRunningTime="2026-11-03 19:41:44.550234319 +0000 UTC m=+617.794199783"
Nov 03 19:41:44.565607 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:44.565540    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-102379-5c958cc4ff-t94ls" podStartSLOduration=1.876408294 podStartE2EDuration="3.565521346s" totalImagesPullingTime="1.689113052s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:41:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:41:44.563937579 +0000 UTC m=+617.807903045" watchObservedRunningTime="2026-11-03 19:41:44.565521346 +0000 UTC m=+617.809486813"
Nov 03 19:41:44.633967 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:44.633926    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfb9124e-3670-46b5-aa88-702ec75bda1f-metrics-certs\") pod \"maas-api-e2e-worker-w5-0486e8-5b894565b7-76jhb\" (UID: \"dfb9124e-3670-46b5-aa88-702ec75bda1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-0486e8-5b894565b7-76jhb"
Nov 03 19:41:44.634188 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:44.633992    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfb9124e-3670-46b5-aa88-702ec75bda1f-maas-api-tls\") pod \"maas-api-e2e-worker-w5-0486e8-5b894565b7-76jhb\" (UID: \"dfb9124e-3670-46b5-aa88-702ec75bda1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-0486e8-5b894565b7-76jhb"
Nov 03 19:41:44.634188 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:44.634148    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfb9124e-3670-46b5-aa88-702ec75bda1f-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-0486e8-5b894565b7-76jhb\" (UID: \"dfb9124e-3670-46b5-aa88-702ec75bda1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-0486e8-5b894565b7-76jhb"
Nov 03 19:41:44.634313 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:44.634242    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvxhh\" (UniqueName: \"kubernetes.io/projected/dfb9124e-3670-46b5-aa88-702ec75bda1f-kube-api-access-cvxhh\") pod \"maas-api-e2e-worker-w5-0486e8-5b894565b7-76jhb\" (UID: \"dfb9124e-3670-46b5-aa88-702ec75bda1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-0486e8-5b894565b7-76jhb"
Nov 03 19:41:44.635278 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:44.635250    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfb9124e-3670-46b5-aa88-702ec75bda1f-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-0486e8-5b894565b7-76jhb\" (UID: \"dfb9124e-3670-46b5-aa88-702ec75bda1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-0486e8-5b894565b7-76jhb"
Nov 03 19:41:44.636543 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:44.636518    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfb9124e-3670-46b5-aa88-702ec75bda1f-maas-api-tls\") pod \"maas-api-e2e-worker-w5-0486e8-5b894565b7-76jhb\" (UID: \"dfb9124e-3670-46b5-aa88-702ec75bda1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-0486e8-5b894565b7-76jhb"
Nov 03 19:41:44.636695 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:44.636652    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfb9124e-3670-46b5-aa88-702ec75bda1f-metrics-certs\") pod \"maas-api-e2e-worker-w5-0486e8-5b894565b7-76jhb\" (UID: \"dfb9124e-3670-46b5-aa88-702ec75bda1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-0486e8-5b894565b7-76jhb"
Nov 03 19:41:44.642481 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:44.642460    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvxhh\" (UniqueName: \"kubernetes.io/projected/dfb9124e-3670-46b5-aa88-702ec75bda1f-kube-api-access-cvxhh\") pod \"maas-api-e2e-worker-w5-0486e8-5b894565b7-76jhb\" (UID: \"dfb9124e-3670-46b5-aa88-702ec75bda1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-0486e8-5b894565b7-76jhb"
Nov 03 19:41:44.796469 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:44.796438    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-0486e8-5b894565b7-76jhb"
Nov 03 19:41:45.074137 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:45.074109    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-0486e8-5b894565b7-76jhb"]
Nov 03 19:41:45.075939 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:41:45.075910    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfb9124e_3670_46b5_aa88_702ec75bda1f.slice/crio-f4199046d869189ce8a67f83c86f269631db22c2ebe053c6a9025e1c7c3bd3b9 WatchSource:0}: Error finding container f4199046d869189ce8a67f83c86f269631db22c2ebe053c6a9025e1c7c3bd3b9: Status 404 returned error can't find the container with id f4199046d869189ce8a67f83c86f269631db22c2ebe053c6a9025e1c7c3bd3b9
Nov 03 19:41:45.531411 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:45.531372    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-0486e8-5b894565b7-76jhb" event={"ID":"dfb9124e-3670-46b5-aa88-702ec75bda1f","Type":"ContainerStarted","Data":"f4199046d869189ce8a67f83c86f269631db22c2ebe053c6a9025e1c7c3bd3b9"}
Nov 03 19:41:46.540048 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:46.540006    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-0486e8-5b894565b7-76jhb" event={"ID":"dfb9124e-3670-46b5-aa88-702ec75bda1f","Type":"ContainerStarted","Data":"de96b32c357cbf61dd527cdcfac6f654bb1f7cb0ee63f0a38157ec5274b72f08"}
Nov 03 19:41:46.540473 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:46.540133    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-0486e8-5b894565b7-76jhb"
Nov 03 19:41:46.557939 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:46.557890    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-0486e8-5b894565b7-76jhb" podStartSLOduration=2.076688905 podStartE2EDuration="2.557876437s" totalImagesPullingTime="481.187532ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:41:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:41:46.554918716 +0000 UTC m=+619.798884198" watchObservedRunningTime="2026-11-03 19:41:46.557876437 +0000 UTC m=+619.801841902"
Nov 03 19:41:50.537468 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:50.537436    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a35658-ff9b85698-4wqq2"
Nov 03 19:41:52.560896 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:52.560855    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-0486e8-5b894565b7-76jhb"
Nov 03 19:41:54.680977 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:54.680934    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt"]
Nov 03 19:41:54.697987 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:54.697941    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt"]
Nov 03 19:41:54.698177 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:54.698085    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt"
Nov 03 19:41:54.704950 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:54.704927    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-8d4127\"/\"simulator-w2-8d4127-kserve-self-signed-certs\""
Nov 03 19:41:54.705139 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:54.704977    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-8d4127\"/\"kube-root-ca.crt\""
Nov 03 19:41:54.705139 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:54.705017    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-8d4127\"/\"default-dockercfg-2rgsc\""
Nov 03 19:41:54.705139 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:54.705103    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-8d4127\"/\"openshift-service-ca.crt\""
Nov 03 19:41:54.743236 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:54.743204    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17dc0ed2-5164-449d-be12-a38bdfeb946d-dshm\") pod \"simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt\" (UID: \"17dc0ed2-5164-449d-be12-a38bdfeb946d\") " pod="e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt"
Nov 03 19:41:54.743405 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:54.743290    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17dc0ed2-5164-449d-be12-a38bdfeb946d-model-cache\") pod \"simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt\" (UID: \"17dc0ed2-5164-449d-be12-a38bdfeb946d\") " pod="e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt"
Nov 03 19:41:54.743405 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:54.743315    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17dc0ed2-5164-449d-be12-a38bdfeb946d-tmp-dir\") pod \"simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt\" (UID: \"17dc0ed2-5164-449d-be12-a38bdfeb946d\") " pod="e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt"
Nov 03 19:41:54.743405 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:54.743351    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17dc0ed2-5164-449d-be12-a38bdfeb946d-home\") pod \"simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt\" (UID: \"17dc0ed2-5164-449d-be12-a38bdfeb946d\") " pod="e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt"
Nov 03 19:41:54.743584 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:54.743542    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17dc0ed2-5164-449d-be12-a38bdfeb946d-tls-certs\") pod \"simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt\" (UID: \"17dc0ed2-5164-449d-be12-a38bdfeb946d\") " pod="e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt"
Nov 03 19:41:54.743638 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:54.743599    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-284jx\" (UniqueName: \"kubernetes.io/projected/17dc0ed2-5164-449d-be12-a38bdfeb946d-kube-api-access-284jx\") pod \"simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt\" (UID: \"17dc0ed2-5164-449d-be12-a38bdfeb946d\") " pod="e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt"
Nov 03 19:41:54.844366 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:54.844334    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17dc0ed2-5164-449d-be12-a38bdfeb946d-model-cache\") pod \"simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt\" (UID: \"17dc0ed2-5164-449d-be12-a38bdfeb946d\") " pod="e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt"
Nov 03 19:41:54.844366 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:54.844366    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17dc0ed2-5164-449d-be12-a38bdfeb946d-tmp-dir\") pod \"simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt\" (UID: \"17dc0ed2-5164-449d-be12-a38bdfeb946d\") " pod="e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt"
Nov 03 19:41:54.844618 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:54.844390    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17dc0ed2-5164-449d-be12-a38bdfeb946d-home\") pod \"simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt\" (UID: \"17dc0ed2-5164-449d-be12-a38bdfeb946d\") " pod="e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt"
Nov 03 19:41:54.844618 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:54.844438    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17dc0ed2-5164-449d-be12-a38bdfeb946d-tls-certs\") pod \"simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt\" (UID: \"17dc0ed2-5164-449d-be12-a38bdfeb946d\") " pod="e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt"
Nov 03 19:41:54.844618 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:54.844466    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-284jx\" (UniqueName: \"kubernetes.io/projected/17dc0ed2-5164-449d-be12-a38bdfeb946d-kube-api-access-284jx\") pod \"simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt\" (UID: \"17dc0ed2-5164-449d-be12-a38bdfeb946d\") " pod="e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt"
Nov 03 19:41:54.844618 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:54.844492    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17dc0ed2-5164-449d-be12-a38bdfeb946d-dshm\") pod \"simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt\" (UID: \"17dc0ed2-5164-449d-be12-a38bdfeb946d\") " pod="e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt"
Nov 03 19:41:54.844866 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:54.844805    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17dc0ed2-5164-449d-be12-a38bdfeb946d-tmp-dir\") pod \"simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt\" (UID: \"17dc0ed2-5164-449d-be12-a38bdfeb946d\") " pod="e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt"
Nov 03 19:41:54.844866 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:54.844853    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17dc0ed2-5164-449d-be12-a38bdfeb946d-model-cache\") pod \"simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt\" (UID: \"17dc0ed2-5164-449d-be12-a38bdfeb946d\") " pod="e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt"
Nov 03 19:41:54.844940 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:54.844883    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17dc0ed2-5164-449d-be12-a38bdfeb946d-home\") pod \"simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt\" (UID: \"17dc0ed2-5164-449d-be12-a38bdfeb946d\") " pod="e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt"
Nov 03 19:41:54.846945 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:54.846924    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17dc0ed2-5164-449d-be12-a38bdfeb946d-dshm\") pod \"simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt\" (UID: \"17dc0ed2-5164-449d-be12-a38bdfeb946d\") " pod="e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt"
Nov 03 19:41:54.847283 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:54.847262    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17dc0ed2-5164-449d-be12-a38bdfeb946d-tls-certs\") pod \"simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt\" (UID: \"17dc0ed2-5164-449d-be12-a38bdfeb946d\") " pod="e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt"
Nov 03 19:41:54.863691 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:54.859355    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-284jx\" (UniqueName: \"kubernetes.io/projected/17dc0ed2-5164-449d-be12-a38bdfeb946d-kube-api-access-284jx\") pod \"simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt\" (UID: \"17dc0ed2-5164-449d-be12-a38bdfeb946d\") " pod="e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt"
Nov 03 19:41:55.015836 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:55.015801    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt"
Nov 03 19:41:55.318490 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:55.318458    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt"]
Nov 03 19:41:55.320102 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:41:55.320077    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod17dc0ed2_5164_449d_be12_a38bdfeb946d.slice/crio-ff3b8a957d477abeca4d240d8c4bf29c3d1946300483167e463c35861442beb4 WatchSource:0}: Error finding container ff3b8a957d477abeca4d240d8c4bf29c3d1946300483167e463c35861442beb4: Status 404 returned error can't find the container with id ff3b8a957d477abeca4d240d8c4bf29c3d1946300483167e463c35861442beb4
Nov 03 19:41:55.535146 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:55.535109    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-102379-5c958cc4ff-t94ls"
Nov 03 19:41:55.535334 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:55.535187    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8d4127-6fb576bf67-qr2v6"
Nov 03 19:41:55.535334 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:55.535329    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-102379-64bd4947ff-7qsss"
Nov 03 19:41:55.617257 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:55.617172    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt" event={"ID":"17dc0ed2-5164-449d-be12-a38bdfeb946d","Type":"ContainerStarted","Data":"ff3b8a957d477abeca4d240d8c4bf29c3d1946300483167e463c35861442beb4"}
Nov 03 19:41:57.631877 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:57.631839    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt" event={"ID":"17dc0ed2-5164-449d-be12-a38bdfeb946d","Type":"ContainerStarted","Data":"09809d4956de377304508ee5361a64aa2ffb2e284311ca8303ef18bb9edab5c2"}
Nov 03 19:41:57.648711 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:57.648565    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt" podStartSLOduration=1.854752368 podStartE2EDuration="3.648548128s" totalImagesPullingTime="1.79379576s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:41:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:41:57.646911358 +0000 UTC m=+630.890876824" watchObservedRunningTime="2026-11-03 19:41:57.648548128 +0000 UTC m=+630.892513594"
Nov 03 19:41:58.016402 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:41:58.016358    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt"
Nov 03 19:42:00.655414 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:00.655334    4921 generic.go:356] "Generic (PLEG): container finished" podID="902fa980-da03-4354-8484-ad392278c1d9" containerID="00ffdf26993b52e3e6c8035e50c8b0e14c3f4ffbf8f617d1e7d2886d27a289fa" exitCode=0
Nov 03 19:42:00.655414 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:00.655377    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9" event={"ID":"902fa980-da03-4354-8484-ad392278c1d9","Type":"ContainerDied","Data":"00ffdf26993b52e3e6c8035e50c8b0e14c3f4ffbf8f617d1e7d2886d27a289fa"}
Nov 03 19:42:00.655894 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:00.655809    4921 scope.go:122] "RemoveContainer" containerID="00ffdf26993b52e3e6c8035e50c8b0e14c3f4ffbf8f617d1e7d2886d27a289fa"
Nov 03 19:42:00.657191 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:00.657160    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:42:00.657299 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:00.657241    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:42:00.657363 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:00.657296    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:42:01.674156 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:01.674117    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9" event={"ID":"902fa980-da03-4354-8484-ad392278c1d9","Type":"ContainerStarted","Data":"a51393deb579f21fb040fc87d3911e67269bcd5d8200a4ba9644e71a01bfc0dc"}
Nov 03 19:42:01.920478 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:01.920433    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9"
Nov 03 19:42:01.921963 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:01.921934    4921 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Nov 03 19:42:01.922109 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:01.921994    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9" podUID="902fa980-da03-4354-8484-ad392278c1d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Nov 03 19:42:02.920052 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:02.920020    4921 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Nov 03 19:42:02.920489 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:02.920081    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9" podUID="902fa980-da03-4354-8484-ad392278c1d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Nov 03 19:42:02.921778 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:42:02.921748    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 19:42:02.982189 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:02.982155    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rfxvm"]
Nov 03 19:42:02.985556 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:02.985535    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rfxvm"
Nov 03 19:42:02.988444 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:02.988416    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 03 19:42:02.988587 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:02.988486    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 03 19:42:02.988653 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:02.988585    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 03 19:42:02.988740 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:02.988709    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-79jcj\""
Nov 03 19:42:03.000308 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:03.000277    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rfxvm"]
Nov 03 19:42:03.077012 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:03.076959    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/effc08c9-9b29-4cd0-ac6b-3e52d748ab34-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rfxvm\" (UID: \"effc08c9-9b29-4cd0-ac6b-3e52d748ab34\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rfxvm"
Nov 03 19:42:03.077225 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:03.077055    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/effc08c9-9b29-4cd0-ac6b-3e52d748ab34-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rfxvm\" (UID: \"effc08c9-9b29-4cd0-ac6b-3e52d748ab34\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rfxvm"
Nov 03 19:42:03.077225 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:03.077100    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/effc08c9-9b29-4cd0-ac6b-3e52d748ab34-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rfxvm\" (UID: \"effc08c9-9b29-4cd0-ac6b-3e52d748ab34\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rfxvm"
Nov 03 19:42:03.077225 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:03.077132    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ldk5\" (UniqueName: \"kubernetes.io/projected/effc08c9-9b29-4cd0-ac6b-3e52d748ab34-kube-api-access-6ldk5\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rfxvm\" (UID: \"effc08c9-9b29-4cd0-ac6b-3e52d748ab34\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rfxvm"
Nov 03 19:42:03.077225 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:03.077171    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/effc08c9-9b29-4cd0-ac6b-3e52d748ab34-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rfxvm\" (UID: \"effc08c9-9b29-4cd0-ac6b-3e52d748ab34\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rfxvm"
Nov 03 19:42:03.077476 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:03.077237    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/effc08c9-9b29-4cd0-ac6b-3e52d748ab34-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rfxvm\" (UID: \"effc08c9-9b29-4cd0-ac6b-3e52d748ab34\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rfxvm"
Nov 03 19:42:03.077476 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:03.077292    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/effc08c9-9b29-4cd0-ac6b-3e52d748ab34-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rfxvm\" (UID: \"effc08c9-9b29-4cd0-ac6b-3e52d748ab34\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rfxvm"
Nov 03 19:42:03.178581 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:03.178479    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/effc08c9-9b29-4cd0-ac6b-3e52d748ab34-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rfxvm\" (UID: \"effc08c9-9b29-4cd0-ac6b-3e52d748ab34\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rfxvm"
Nov 03 19:42:03.178581 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:03.178531    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/effc08c9-9b29-4cd0-ac6b-3e52d748ab34-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rfxvm\" (UID: \"effc08c9-9b29-4cd0-ac6b-3e52d748ab34\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rfxvm"
Nov 03 19:42:03.178867 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:03.178592    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/effc08c9-9b29-4cd0-ac6b-3e52d748ab34-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rfxvm\" (UID: \"effc08c9-9b29-4cd0-ac6b-3e52d748ab34\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rfxvm"
Nov 03 19:42:03.178867 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:03.178625    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/effc08c9-9b29-4cd0-ac6b-3e52d748ab34-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rfxvm\" (UID: \"effc08c9-9b29-4cd0-ac6b-3e52d748ab34\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rfxvm"
Nov 03 19:42:03.178867 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:03.178652    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ldk5\" (UniqueName: \"kubernetes.io/projected/effc08c9-9b29-4cd0-ac6b-3e52d748ab34-kube-api-access-6ldk5\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rfxvm\" (UID: \"effc08c9-9b29-4cd0-ac6b-3e52d748ab34\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rfxvm"
Nov 03 19:42:03.178867 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:03.178714    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/effc08c9-9b29-4cd0-ac6b-3e52d748ab34-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rfxvm\" (UID: \"effc08c9-9b29-4cd0-ac6b-3e52d748ab34\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rfxvm"
Nov 03 19:42:03.178867 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:03.178774    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/effc08c9-9b29-4cd0-ac6b-3e52d748ab34-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rfxvm\" (UID: \"effc08c9-9b29-4cd0-ac6b-3e52d748ab34\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rfxvm"
Nov 03 19:42:03.179518 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:03.179488    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/effc08c9-9b29-4cd0-ac6b-3e52d748ab34-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rfxvm\" (UID: \"effc08c9-9b29-4cd0-ac6b-3e52d748ab34\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rfxvm"
Nov 03 19:42:03.181694 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:03.181656    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/effc08c9-9b29-4cd0-ac6b-3e52d748ab34-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rfxvm\" (UID: \"effc08c9-9b29-4cd0-ac6b-3e52d748ab34\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rfxvm"
Nov 03 19:42:03.181694 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:03.181686    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/effc08c9-9b29-4cd0-ac6b-3e52d748ab34-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rfxvm\" (UID: \"effc08c9-9b29-4cd0-ac6b-3e52d748ab34\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rfxvm"
Nov 03 19:42:03.181845 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:03.181659    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/effc08c9-9b29-4cd0-ac6b-3e52d748ab34-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rfxvm\" (UID: \"effc08c9-9b29-4cd0-ac6b-3e52d748ab34\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rfxvm"
Nov 03 19:42:03.181845 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:03.181785    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/effc08c9-9b29-4cd0-ac6b-3e52d748ab34-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rfxvm\" (UID: \"effc08c9-9b29-4cd0-ac6b-3e52d748ab34\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rfxvm"
Nov 03 19:42:03.185973 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:03.185954    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/effc08c9-9b29-4cd0-ac6b-3e52d748ab34-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rfxvm\" (UID: \"effc08c9-9b29-4cd0-ac6b-3e52d748ab34\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rfxvm"
Nov 03 19:42:03.186388 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:03.186365    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ldk5\" (UniqueName: \"kubernetes.io/projected/effc08c9-9b29-4cd0-ac6b-3e52d748ab34-kube-api-access-6ldk5\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rfxvm\" (UID: \"effc08c9-9b29-4cd0-ac6b-3e52d748ab34\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rfxvm"
Nov 03 19:42:03.299189 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:03.299149    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rfxvm"
Nov 03 19:42:03.620231 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:03.620202    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rfxvm"]
Nov 03 19:42:03.623997 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:42:03.623958    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeffc08c9_9b29_4cd0_ac6b_3e52d748ab34.slice/crio-1bf3b443cc9fa98e2f8c35e507241b51ac42a83cb9f84ce691670a187c3526f5 WatchSource:0}: Error finding container 1bf3b443cc9fa98e2f8c35e507241b51ac42a83cb9f84ce691670a187c3526f5: Status 404 returned error can't find the container with id 1bf3b443cc9fa98e2f8c35e507241b51ac42a83cb9f84ce691670a187c3526f5
Nov 03 19:42:03.694991 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:03.694950    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rfxvm" event={"ID":"effc08c9-9b29-4cd0-ac6b-3e52d748ab34","Type":"ContainerStarted","Data":"1bf3b443cc9fa98e2f8c35e507241b51ac42a83cb9f84ce691670a187c3526f5"}
Nov 03 19:42:03.920246 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:03.920163    4921 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Nov 03 19:42:03.920246 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:03.920220    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9" podUID="902fa980-da03-4354-8484-ad392278c1d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Nov 03 19:42:04.920852 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:04.920813    4921 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Nov 03 19:42:04.921356 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:04.920875    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9" podUID="902fa980-da03-4354-8484-ad392278c1d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Nov 03 19:42:05.017237 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:05.017199    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt"
Nov 03 19:42:05.032429 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:05.032400    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt"
Nov 03 19:42:05.436740 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:05.436705    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9"
Nov 03 19:42:05.723743 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:05.723461    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt"
Nov 03 19:42:05.919958 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:05.919918    4921 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Nov 03 19:42:05.920146 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:05.919986    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9" podUID="902fa980-da03-4354-8484-ad392278c1d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Nov 03 19:42:06.927131 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:06.927093    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9"
Nov 03 19:42:06.929437 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:06.929416    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9"
Nov 03 19:42:07.639259 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:07.639226    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:42:07.639370 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:07.639291    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:42:08.731739 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:08.731702    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rfxvm" event={"ID":"effc08c9-9b29-4cd0-ac6b-3e52d748ab34","Type":"ContainerStarted","Data":"b9bdf8fcb631521403e4c415852fdcfbc5fa7a3f37efb9dd572407e5a17cbff0"}
Nov 03 19:42:08.750692 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:08.750499    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rfxvm" podStartSLOduration=2.738902704 podStartE2EDuration="6.750480873s" totalImagesPullingTime="4.011578169s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:42:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:42:08.750244843 +0000 UTC m=+641.994210308" watchObservedRunningTime="2026-11-03 19:42:08.750480873 +0000 UTC m=+641.994446337"
Nov 03 19:42:09.737558 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:09.737518    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rfxvm"
Nov 03 19:42:09.739428 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:09.739402    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rfxvm"
Nov 03 19:42:32.927538 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:42:32.927501    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 19:42:34.799219 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:34.799171    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv"]
Nov 03 19:42:34.805952 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:34.805923    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv"
Nov 03 19:42:34.812624 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:34.812398    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-f4e123\"/\"openshift-service-ca.crt\""
Nov 03 19:42:34.812624 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:34.812475    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-f4e123\"/\"default-dockercfg-dzh8r\""
Nov 03 19:42:34.812624 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:34.812543    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-f4e123\"/\"premium-w4-f4e123-kserve-self-signed-certs\""
Nov 03 19:42:34.812624 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:34.812595    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-f4e123\"/\"kube-root-ca.crt\""
Nov 03 19:42:34.817250 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:34.817148    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv"]
Nov 03 19:42:34.938547 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:34.938503    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01c749b9-d188-4673-9a3d-99b9c4ae0793-home\") pod \"premium-w4-f4e123-kserve-67fdcf4b67-bwmkv\" (UID: \"01c749b9-d188-4673-9a3d-99b9c4ae0793\") " pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv"
Nov 03 19:42:34.938933 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:34.938905    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46s9s\" (UniqueName: \"kubernetes.io/projected/01c749b9-d188-4673-9a3d-99b9c4ae0793-kube-api-access-46s9s\") pod \"premium-w4-f4e123-kserve-67fdcf4b67-bwmkv\" (UID: \"01c749b9-d188-4673-9a3d-99b9c4ae0793\") " pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv"
Nov 03 19:42:34.939071 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:34.939055    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01c749b9-d188-4673-9a3d-99b9c4ae0793-tls-certs\") pod \"premium-w4-f4e123-kserve-67fdcf4b67-bwmkv\" (UID: \"01c749b9-d188-4673-9a3d-99b9c4ae0793\") " pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv"
Nov 03 19:42:34.939180 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:34.939166    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01c749b9-d188-4673-9a3d-99b9c4ae0793-dshm\") pod \"premium-w4-f4e123-kserve-67fdcf4b67-bwmkv\" (UID: \"01c749b9-d188-4673-9a3d-99b9c4ae0793\") " pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv"
Nov 03 19:42:34.939288 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:34.939274    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01c749b9-d188-4673-9a3d-99b9c4ae0793-model-cache\") pod \"premium-w4-f4e123-kserve-67fdcf4b67-bwmkv\" (UID: \"01c749b9-d188-4673-9a3d-99b9c4ae0793\") " pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv"
Nov 03 19:42:34.939436 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:34.939423    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01c749b9-d188-4673-9a3d-99b9c4ae0793-tmp-dir\") pod \"premium-w4-f4e123-kserve-67fdcf4b67-bwmkv\" (UID: \"01c749b9-d188-4673-9a3d-99b9c4ae0793\") " pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv"
Nov 03 19:42:35.041000 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:35.040968    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01c749b9-d188-4673-9a3d-99b9c4ae0793-home\") pod \"premium-w4-f4e123-kserve-67fdcf4b67-bwmkv\" (UID: \"01c749b9-d188-4673-9a3d-99b9c4ae0793\") " pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv"
Nov 03 19:42:35.041000 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:35.041003    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46s9s\" (UniqueName: \"kubernetes.io/projected/01c749b9-d188-4673-9a3d-99b9c4ae0793-kube-api-access-46s9s\") pod \"premium-w4-f4e123-kserve-67fdcf4b67-bwmkv\" (UID: \"01c749b9-d188-4673-9a3d-99b9c4ae0793\") " pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv"
Nov 03 19:42:35.041289 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:35.041128    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01c749b9-d188-4673-9a3d-99b9c4ae0793-tls-certs\") pod \"premium-w4-f4e123-kserve-67fdcf4b67-bwmkv\" (UID: \"01c749b9-d188-4673-9a3d-99b9c4ae0793\") " pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv"
Nov 03 19:42:35.041289 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:35.041192    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01c749b9-d188-4673-9a3d-99b9c4ae0793-dshm\") pod \"premium-w4-f4e123-kserve-67fdcf4b67-bwmkv\" (UID: \"01c749b9-d188-4673-9a3d-99b9c4ae0793\") " pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv"
Nov 03 19:42:35.041289 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:35.041246    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01c749b9-d188-4673-9a3d-99b9c4ae0793-model-cache\") pod \"premium-w4-f4e123-kserve-67fdcf4b67-bwmkv\" (UID: \"01c749b9-d188-4673-9a3d-99b9c4ae0793\") " pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv"
Nov 03 19:42:35.041461 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:35.041356    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01c749b9-d188-4673-9a3d-99b9c4ae0793-tmp-dir\") pod \"premium-w4-f4e123-kserve-67fdcf4b67-bwmkv\" (UID: \"01c749b9-d188-4673-9a3d-99b9c4ae0793\") " pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv"
Nov 03 19:42:35.041461 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:35.041406    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01c749b9-d188-4673-9a3d-99b9c4ae0793-home\") pod \"premium-w4-f4e123-kserve-67fdcf4b67-bwmkv\" (UID: \"01c749b9-d188-4673-9a3d-99b9c4ae0793\") " pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv"
Nov 03 19:42:35.041748 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:35.041700    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01c749b9-d188-4673-9a3d-99b9c4ae0793-model-cache\") pod \"premium-w4-f4e123-kserve-67fdcf4b67-bwmkv\" (UID: \"01c749b9-d188-4673-9a3d-99b9c4ae0793\") " pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv"
Nov 03 19:42:35.042028 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:35.041941    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01c749b9-d188-4673-9a3d-99b9c4ae0793-tmp-dir\") pod \"premium-w4-f4e123-kserve-67fdcf4b67-bwmkv\" (UID: \"01c749b9-d188-4673-9a3d-99b9c4ae0793\") " pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv"
Nov 03 19:42:35.044054 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:35.044033    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01c749b9-d188-4673-9a3d-99b9c4ae0793-dshm\") pod \"premium-w4-f4e123-kserve-67fdcf4b67-bwmkv\" (UID: \"01c749b9-d188-4673-9a3d-99b9c4ae0793\") " pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv"
Nov 03 19:42:35.044256 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:35.044237    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01c749b9-d188-4673-9a3d-99b9c4ae0793-tls-certs\") pod \"premium-w4-f4e123-kserve-67fdcf4b67-bwmkv\" (UID: \"01c749b9-d188-4673-9a3d-99b9c4ae0793\") " pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv"
Nov 03 19:42:35.048161 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:35.048135    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46s9s\" (UniqueName: \"kubernetes.io/projected/01c749b9-d188-4673-9a3d-99b9c4ae0793-kube-api-access-46s9s\") pod \"premium-w4-f4e123-kserve-67fdcf4b67-bwmkv\" (UID: \"01c749b9-d188-4673-9a3d-99b9c4ae0793\") " pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv"
Nov 03 19:42:35.124177 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:35.124066    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv"
Nov 03 19:42:35.407817 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:35.407785    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv"]
Nov 03 19:42:35.408451 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:42:35.408426    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod01c749b9_d188_4673_9a3d_99b9c4ae0793.slice/crio-ff6dd056f65745ffaaee00a3d549d9a7e7f2d9ea90e99bb31c73b1b96209a625 WatchSource:0}: Error finding container ff6dd056f65745ffaaee00a3d549d9a7e7f2d9ea90e99bb31c73b1b96209a625: Status 404 returned error can't find the container with id ff6dd056f65745ffaaee00a3d549d9a7e7f2d9ea90e99bb31c73b1b96209a625
Nov 03 19:42:35.910054 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:35.909975    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv" event={"ID":"01c749b9-d188-4673-9a3d-99b9c4ae0793","Type":"ContainerStarted","Data":"ab440e7a2ef9ea255636d7dffb0c5e02ea286d89a6519ae4e020dc0365c055e8"}
Nov 03 19:42:35.910054 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:35.910024    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv" event={"ID":"01c749b9-d188-4673-9a3d-99b9c4ae0793","Type":"ContainerStarted","Data":"ff6dd056f65745ffaaee00a3d549d9a7e7f2d9ea90e99bb31c73b1b96209a625"}
Nov 03 19:42:35.927396 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:35.927354    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv" podStartSLOduration=1.927341958 podStartE2EDuration="1.927341958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:42:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:42:35.923630345 +0000 UTC m=+669.167595809" watchObservedRunningTime="2026-11-03 19:42:35.927341958 +0000 UTC m=+669.171307423"
Nov 03 19:42:36.124298 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:36.124263    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv"
Nov 03 19:42:45.124737 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:45.124698    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv"
Nov 03 19:42:45.138489 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:45.138460    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv"
Nov 03 19:42:45.984800 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:45.984757    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv"
Nov 03 19:42:49.411243 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:49.411210    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv"]
Nov 03 19:42:49.419018 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:49.418991    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv"
Nov 03 19:42:49.429223 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:49.429172    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv"]
Nov 03 19:42:49.429700 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:49.429661    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-82dbbf\"/\"openshift-service-ca.crt\""
Nov 03 19:42:49.429967 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:49.429947    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-82dbbf\"/\"kube-root-ca.crt\""
Nov 03 19:42:49.430081 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:49.430065    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-82dbbf\"/\"default-dockercfg-lc8tj\""
Nov 03 19:42:49.430195 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:49.430179    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-82dbbf\"/\"premium-w0-82dbbf-kserve-self-signed-certs\""
Nov 03 19:42:49.502826 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:49.502786    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96b73342-0079-4709-8570-3dbce57e5243-tls-certs\") pod \"premium-w0-82dbbf-kserve-7cb8b9765d-s64gv\" (UID: \"96b73342-0079-4709-8570-3dbce57e5243\") " pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv"
Nov 03 19:42:49.502826 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:49.502823    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxsdr\" (UniqueName: \"kubernetes.io/projected/96b73342-0079-4709-8570-3dbce57e5243-kube-api-access-lxsdr\") pod \"premium-w0-82dbbf-kserve-7cb8b9765d-s64gv\" (UID: \"96b73342-0079-4709-8570-3dbce57e5243\") " pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv"
Nov 03 19:42:49.503083 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:49.502854    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96b73342-0079-4709-8570-3dbce57e5243-dshm\") pod \"premium-w0-82dbbf-kserve-7cb8b9765d-s64gv\" (UID: \"96b73342-0079-4709-8570-3dbce57e5243\") " pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv"
Nov 03 19:42:49.503083 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:49.502975    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96b73342-0079-4709-8570-3dbce57e5243-tmp-dir\") pod \"premium-w0-82dbbf-kserve-7cb8b9765d-s64gv\" (UID: \"96b73342-0079-4709-8570-3dbce57e5243\") " pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv"
Nov 03 19:42:49.503083 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:49.503008    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96b73342-0079-4709-8570-3dbce57e5243-home\") pod \"premium-w0-82dbbf-kserve-7cb8b9765d-s64gv\" (UID: \"96b73342-0079-4709-8570-3dbce57e5243\") " pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv"
Nov 03 19:42:49.503083 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:49.503032    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96b73342-0079-4709-8570-3dbce57e5243-model-cache\") pod \"premium-w0-82dbbf-kserve-7cb8b9765d-s64gv\" (UID: \"96b73342-0079-4709-8570-3dbce57e5243\") " pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv"
Nov 03 19:42:49.603903 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:49.603867    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96b73342-0079-4709-8570-3dbce57e5243-tls-certs\") pod \"premium-w0-82dbbf-kserve-7cb8b9765d-s64gv\" (UID: \"96b73342-0079-4709-8570-3dbce57e5243\") " pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv"
Nov 03 19:42:49.603903 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:49.603909    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxsdr\" (UniqueName: \"kubernetes.io/projected/96b73342-0079-4709-8570-3dbce57e5243-kube-api-access-lxsdr\") pod \"premium-w0-82dbbf-kserve-7cb8b9765d-s64gv\" (UID: \"96b73342-0079-4709-8570-3dbce57e5243\") " pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv"
Nov 03 19:42:49.604146 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:49.603948    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96b73342-0079-4709-8570-3dbce57e5243-dshm\") pod \"premium-w0-82dbbf-kserve-7cb8b9765d-s64gv\" (UID: \"96b73342-0079-4709-8570-3dbce57e5243\") " pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv"
Nov 03 19:42:49.604146 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:49.603988    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96b73342-0079-4709-8570-3dbce57e5243-tmp-dir\") pod \"premium-w0-82dbbf-kserve-7cb8b9765d-s64gv\" (UID: \"96b73342-0079-4709-8570-3dbce57e5243\") " pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv"
Nov 03 19:42:49.604146 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:49.604033    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96b73342-0079-4709-8570-3dbce57e5243-home\") pod \"premium-w0-82dbbf-kserve-7cb8b9765d-s64gv\" (UID: \"96b73342-0079-4709-8570-3dbce57e5243\") " pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv"
Nov 03 19:42:49.604146 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:49.604068    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96b73342-0079-4709-8570-3dbce57e5243-model-cache\") pod \"premium-w0-82dbbf-kserve-7cb8b9765d-s64gv\" (UID: \"96b73342-0079-4709-8570-3dbce57e5243\") " pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv"
Nov 03 19:42:49.604531 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:49.604496    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96b73342-0079-4709-8570-3dbce57e5243-model-cache\") pod \"premium-w0-82dbbf-kserve-7cb8b9765d-s64gv\" (UID: \"96b73342-0079-4709-8570-3dbce57e5243\") " pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv"
Nov 03 19:42:49.604632 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:49.604559    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96b73342-0079-4709-8570-3dbce57e5243-home\") pod \"premium-w0-82dbbf-kserve-7cb8b9765d-s64gv\" (UID: \"96b73342-0079-4709-8570-3dbce57e5243\") " pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv"
Nov 03 19:42:49.604632 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:49.604581    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96b73342-0079-4709-8570-3dbce57e5243-tmp-dir\") pod \"premium-w0-82dbbf-kserve-7cb8b9765d-s64gv\" (UID: \"96b73342-0079-4709-8570-3dbce57e5243\") " pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv"
Nov 03 19:42:49.606330 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:49.606307    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96b73342-0079-4709-8570-3dbce57e5243-dshm\") pod \"premium-w0-82dbbf-kserve-7cb8b9765d-s64gv\" (UID: \"96b73342-0079-4709-8570-3dbce57e5243\") " pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv"
Nov 03 19:42:49.606538 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:49.606524    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96b73342-0079-4709-8570-3dbce57e5243-tls-certs\") pod \"premium-w0-82dbbf-kserve-7cb8b9765d-s64gv\" (UID: \"96b73342-0079-4709-8570-3dbce57e5243\") " pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv"
Nov 03 19:42:49.617397 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:49.617376    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxsdr\" (UniqueName: \"kubernetes.io/projected/96b73342-0079-4709-8570-3dbce57e5243-kube-api-access-lxsdr\") pod \"premium-w0-82dbbf-kserve-7cb8b9765d-s64gv\" (UID: \"96b73342-0079-4709-8570-3dbce57e5243\") " pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv"
Nov 03 19:42:49.737124 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:49.737092    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv"
Nov 03 19:42:50.012225 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:50.012200    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv"]
Nov 03 19:42:50.014158 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:42:50.014136    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod96b73342_0079_4709_8570_3dbce57e5243.slice/crio-29c8e1be5b53076fb3e866ceebb8c136104e8ce2f4ed963b54a65d26e19a0035 WatchSource:0}: Error finding container 29c8e1be5b53076fb3e866ceebb8c136104e8ce2f4ed963b54a65d26e19a0035: Status 404 returned error can't find the container with id 29c8e1be5b53076fb3e866ceebb8c136104e8ce2f4ed963b54a65d26e19a0035
Nov 03 19:42:51.008078 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:51.008038    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv" event={"ID":"96b73342-0079-4709-8570-3dbce57e5243","Type":"ContainerStarted","Data":"61708f049c22b2df2a4dd0624f8354a4e37d3a2e7d2d44a41c051e5812287ff5"}
Nov 03 19:42:51.008078 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:51.008084    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv" event={"ID":"96b73342-0079-4709-8570-3dbce57e5243","Type":"ContainerStarted","Data":"29c8e1be5b53076fb3e866ceebb8c136104e8ce2f4ed963b54a65d26e19a0035"}
Nov 03 19:42:51.025961 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:51.025894    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv" podStartSLOduration=2.025878662 podStartE2EDuration="2.025878662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:42:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:42:51.023966868 +0000 UTC m=+684.267932351" watchObservedRunningTime="2026-11-03 19:42:51.025878662 +0000 UTC m=+684.269844127"
Nov 03 19:42:51.737228 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:51.737194    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv"
Nov 03 19:42:59.737900 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:59.737862    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv"
Nov 03 19:42:59.751812 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:42:59.751785    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv"
Nov 03 19:43:00.162079 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:00.161978    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv"
Nov 03 19:43:02.921236 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:43:02.921198    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 19:43:29.890768 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:29.890726    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c598ec5c-xf4sb"]
Nov 03 19:43:29.894899 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:29.894880    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c598ec5c-xf4sb"
Nov 03 19:43:29.899153 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:29.899114    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:43:29.899638 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:29.899613    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 19:43:29.903125 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:29.902634    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c598ec5c-xf4sb"]
Nov 03 19:43:29.965485 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:29.965451    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/709ab7af-01f4-4205-bd44-1c1d03ef8136-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c598ec5c-xf4sb\" (UID: \"709ab7af-01f4-4205-bd44-1c1d03ef8136\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c598ec5c-xf4sb"
Nov 03 19:43:29.965654 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:29.965499    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsdmv\" (UniqueName: \"kubernetes.io/projected/709ab7af-01f4-4205-bd44-1c1d03ef8136-kube-api-access-zsdmv\") pod \"maas-api-revoke-sub-c598ec5c-xf4sb\" (UID: \"709ab7af-01f4-4205-bd44-1c1d03ef8136\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c598ec5c-xf4sb"
Nov 03 19:43:29.965727 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:29.965654    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/709ab7af-01f4-4205-bd44-1c1d03ef8136-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c598ec5c-xf4sb\" (UID: \"709ab7af-01f4-4205-bd44-1c1d03ef8136\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c598ec5c-xf4sb"
Nov 03 19:43:30.066640 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:30.066610    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/709ab7af-01f4-4205-bd44-1c1d03ef8136-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c598ec5c-xf4sb\" (UID: \"709ab7af-01f4-4205-bd44-1c1d03ef8136\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c598ec5c-xf4sb"
Nov 03 19:43:30.066841 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:30.066742    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/709ab7af-01f4-4205-bd44-1c1d03ef8136-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c598ec5c-xf4sb\" (UID: \"709ab7af-01f4-4205-bd44-1c1d03ef8136\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c598ec5c-xf4sb"
Nov 03 19:43:30.066841 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:30.066791    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsdmv\" (UniqueName: \"kubernetes.io/projected/709ab7af-01f4-4205-bd44-1c1d03ef8136-kube-api-access-zsdmv\") pod \"maas-api-revoke-sub-c598ec5c-xf4sb\" (UID: \"709ab7af-01f4-4205-bd44-1c1d03ef8136\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c598ec5c-xf4sb"
Nov 03 19:43:30.067397 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:30.067369    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/709ab7af-01f4-4205-bd44-1c1d03ef8136-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c598ec5c-xf4sb\" (UID: \"709ab7af-01f4-4205-bd44-1c1d03ef8136\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c598ec5c-xf4sb"
Nov 03 19:43:30.069090 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:30.069069    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/709ab7af-01f4-4205-bd44-1c1d03ef8136-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c598ec5c-xf4sb\" (UID: \"709ab7af-01f4-4205-bd44-1c1d03ef8136\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c598ec5c-xf4sb"
Nov 03 19:43:30.074326 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:30.074301    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsdmv\" (UniqueName: \"kubernetes.io/projected/709ab7af-01f4-4205-bd44-1c1d03ef8136-kube-api-access-zsdmv\") pod \"maas-api-revoke-sub-c598ec5c-xf4sb\" (UID: \"709ab7af-01f4-4205-bd44-1c1d03ef8136\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c598ec5c-xf4sb"
Nov 03 19:43:30.211355 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:30.211276    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c598ec5c-xf4sb"
Nov 03 19:43:30.488298 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:30.488264    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c598ec5c-xf4sb"]
Nov 03 19:43:30.488812 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:43:30.488787    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod709ab7af_01f4_4205_bd44_1c1d03ef8136.slice/crio-edae8728617fb1ae3b53ccb4e6a06a143dc223557d833a7a24621a1a77d90783 WatchSource:0}: Error finding container edae8728617fb1ae3b53ccb4e6a06a143dc223557d833a7a24621a1a77d90783: Status 404 returned error can't find the container with id edae8728617fb1ae3b53ccb4e6a06a143dc223557d833a7a24621a1a77d90783
Nov 03 19:43:31.306403 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:31.306370    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c598ec5c-xf4sb" event={"ID":"709ab7af-01f4-4205-bd44-1c1d03ef8136","Type":"ContainerStarted","Data":"edae8728617fb1ae3b53ccb4e6a06a143dc223557d833a7a24621a1a77d90783"}
Nov 03 19:43:32.922955 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:43:32.922915    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 19:43:34.328778 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:34.328663    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c598ec5c-xf4sb" event={"ID":"709ab7af-01f4-4205-bd44-1c1d03ef8136","Type":"ContainerStarted","Data":"92a1f022e49fde8cd45cf7dd80b75697ffa1ba6a27907f387ac311ecc33dcd4d"}
Nov 03 19:43:34.343686 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:34.343632    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c598ec5c-xf4sb" podStartSLOduration=2.018414603 podStartE2EDuration="5.343613856s" totalImagesPullingTime="3.325199253s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:43:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:43:34.341659126 +0000 UTC m=+727.585624574" watchObservedRunningTime="2026-11-03 19:43:34.343613856 +0000 UTC m=+727.587579321"
Nov 03 19:43:35.363713 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:35.363543    4921 generic.go:356] "Generic (PLEG): container finished" podID="709ab7af-01f4-4205-bd44-1c1d03ef8136" containerID="92a1f022e49fde8cd45cf7dd80b75697ffa1ba6a27907f387ac311ecc33dcd4d" exitCode=0
Nov 03 19:43:35.363713 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:35.363653    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c598ec5c-xf4sb" event={"ID":"709ab7af-01f4-4205-bd44-1c1d03ef8136","Type":"ContainerDied","Data":"92a1f022e49fde8cd45cf7dd80b75697ffa1ba6a27907f387ac311ecc33dcd4d"}
Nov 03 19:43:36.571824 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:36.571785    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c598ec5c-xf4sb"
Nov 03 19:43:36.642019 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:36.641926    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/709ab7af-01f4-4205-bd44-1c1d03ef8136-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/709ab7af-01f4-4205-bd44-1c1d03ef8136-maas-api-openshift-ca\") pod \"709ab7af-01f4-4205-bd44-1c1d03ef8136\" (UID: \"709ab7af-01f4-4205-bd44-1c1d03ef8136\") "
Nov 03 19:43:36.642193 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:36.642047    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/709ab7af-01f4-4205-bd44-1c1d03ef8136-kube-api-access-zsdmv\" (UniqueName: \"kubernetes.io/projected/709ab7af-01f4-4205-bd44-1c1d03ef8136-kube-api-access-zsdmv\") pod \"709ab7af-01f4-4205-bd44-1c1d03ef8136\" (UID: \"709ab7af-01f4-4205-bd44-1c1d03ef8136\") "
Nov 03 19:43:36.642193 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:36.642093    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/709ab7af-01f4-4205-bd44-1c1d03ef8136-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/709ab7af-01f4-4205-bd44-1c1d03ef8136-maas-api-xks-ca\") pod \"709ab7af-01f4-4205-bd44-1c1d03ef8136\" (UID: \"709ab7af-01f4-4205-bd44-1c1d03ef8136\") "
Nov 03 19:43:36.642404 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:36.642371    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/709ab7af-01f4-4205-bd44-1c1d03ef8136-maas-api-openshift-ca" pod "709ab7af-01f4-4205-bd44-1c1d03ef8136" (UID: "709ab7af-01f4-4205-bd44-1c1d03ef8136"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:43:36.642499 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:36.642475    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/709ab7af-01f4-4205-bd44-1c1d03ef8136-maas-api-openshift-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:43:36.644534 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:36.644503    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/709ab7af-01f4-4205-bd44-1c1d03ef8136-kube-api-access-zsdmv" pod "709ab7af-01f4-4205-bd44-1c1d03ef8136" (UID: "709ab7af-01f4-4205-bd44-1c1d03ef8136"). InnerVolumeSpecName "kube-api-access-zsdmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:43:36.644655 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:36.644628    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/709ab7af-01f4-4205-bd44-1c1d03ef8136-maas-api-xks-ca" pod "709ab7af-01f4-4205-bd44-1c1d03ef8136" (UID: "709ab7af-01f4-4205-bd44-1c1d03ef8136"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:43:36.743781 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:36.743751    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsdmv\" (UniqueName: \"kubernetes.io/projected/709ab7af-01f4-4205-bd44-1c1d03ef8136-kube-api-access-zsdmv\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:43:36.743781 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:36.743781    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/709ab7af-01f4-4205-bd44-1c1d03ef8136-maas-api-xks-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:43:37.378067 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:37.378032    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c598ec5c-xf4sb" event={"ID":"709ab7af-01f4-4205-bd44-1c1d03ef8136","Type":"ContainerDied","Data":"edae8728617fb1ae3b53ccb4e6a06a143dc223557d833a7a24621a1a77d90783"}
Nov 03 19:43:37.378251 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:37.378070    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c598ec5c-xf4sb"
Nov 03 19:43:37.378251 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:37.378088    4921 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="edae8728617fb1ae3b53ccb4e6a06a143dc223557d833a7a24621a1a77d90783"
Nov 03 19:43:40.272178 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:40.272146    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6db7b021-z5g6p"]
Nov 03 19:43:40.272634 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:40.272620    4921 cpu_manager.go:414] "containerMap: removing container" podUID="709ab7af-01f4-4205-bd44-1c1d03ef8136" containerName="revoke-keys"
Nov 03 19:43:40.272723 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:40.272636    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="709ab7af-01f4-4205-bd44-1c1d03ef8136" containerName="revoke-keys"
Nov 03 19:43:40.272775 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:40.272756    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="709ab7af-01f4-4205-bd44-1c1d03ef8136" containerName="revoke-keys"
Nov 03 19:43:40.276183 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:40.276165    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6db7b021-z5g6p"
Nov 03 19:43:40.280779 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:40.280757    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 19:43:40.280946 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:40.280741    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:43:40.284574 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:40.284542    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6db7b021-z5g6p"]
Nov 03 19:43:40.382165 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:40.382133    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbade572-c672-4955-bdf8-5e47fc2099c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6db7b021-z5g6p\" (UID: \"dbade572-c672-4955-bdf8-5e47fc2099c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6db7b021-z5g6p"
Nov 03 19:43:40.382334 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:40.382197    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbade572-c672-4955-bdf8-5e47fc2099c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6db7b021-z5g6p\" (UID: \"dbade572-c672-4955-bdf8-5e47fc2099c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6db7b021-z5g6p"
Nov 03 19:43:40.382334 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:40.382297    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xsl9\" (UniqueName: \"kubernetes.io/projected/dbade572-c672-4955-bdf8-5e47fc2099c7-kube-api-access-9xsl9\") pod \"maas-api-revoke-sub-6db7b021-z5g6p\" (UID: \"dbade572-c672-4955-bdf8-5e47fc2099c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6db7b021-z5g6p"
Nov 03 19:43:40.483504 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:40.483468    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbade572-c672-4955-bdf8-5e47fc2099c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6db7b021-z5g6p\" (UID: \"dbade572-c672-4955-bdf8-5e47fc2099c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6db7b021-z5g6p"
Nov 03 19:43:40.483654 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:40.483541    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbade572-c672-4955-bdf8-5e47fc2099c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6db7b021-z5g6p\" (UID: \"dbade572-c672-4955-bdf8-5e47fc2099c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6db7b021-z5g6p"
Nov 03 19:43:40.483654 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:40.483590    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xsl9\" (UniqueName: \"kubernetes.io/projected/dbade572-c672-4955-bdf8-5e47fc2099c7-kube-api-access-9xsl9\") pod \"maas-api-revoke-sub-6db7b021-z5g6p\" (UID: \"dbade572-c672-4955-bdf8-5e47fc2099c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6db7b021-z5g6p"
Nov 03 19:43:40.484174 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:40.484148    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbade572-c672-4955-bdf8-5e47fc2099c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6db7b021-z5g6p\" (UID: \"dbade572-c672-4955-bdf8-5e47fc2099c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6db7b021-z5g6p"
Nov 03 19:43:40.485860 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:40.485836    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbade572-c672-4955-bdf8-5e47fc2099c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6db7b021-z5g6p\" (UID: \"dbade572-c672-4955-bdf8-5e47fc2099c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6db7b021-z5g6p"
Nov 03 19:43:40.490224 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:40.490207    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xsl9\" (UniqueName: \"kubernetes.io/projected/dbade572-c672-4955-bdf8-5e47fc2099c7-kube-api-access-9xsl9\") pod \"maas-api-revoke-sub-6db7b021-z5g6p\" (UID: \"dbade572-c672-4955-bdf8-5e47fc2099c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6db7b021-z5g6p"
Nov 03 19:43:40.592023 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:40.591947    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6db7b021-z5g6p"
Nov 03 19:43:40.879323 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:40.879291    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6db7b021-z5g6p"]
Nov 03 19:43:40.880710 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:43:40.880664    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbade572_c672_4955_bdf8_5e47fc2099c7.slice/crio-43e80cd2621e13d38416d419c1171f021df0538de280644b2bae337d0373cf7c WatchSource:0}: Error finding container 43e80cd2621e13d38416d419c1171f021df0538de280644b2bae337d0373cf7c: Status 404 returned error can't find the container with id 43e80cd2621e13d38416d419c1171f021df0538de280644b2bae337d0373cf7c
Nov 03 19:43:41.404260 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:41.404227    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6db7b021-z5g6p" event={"ID":"dbade572-c672-4955-bdf8-5e47fc2099c7","Type":"ContainerStarted","Data":"321a599f1e7276c617b29dc72387f7db367c0888c6a95f658f1c4f5ee91efad2"}
Nov 03 19:43:41.404658 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:41.404286    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6db7b021-z5g6p" event={"ID":"dbade572-c672-4955-bdf8-5e47fc2099c7","Type":"ContainerStarted","Data":"43e80cd2621e13d38416d419c1171f021df0538de280644b2bae337d0373cf7c"}
Nov 03 19:43:41.417928 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:41.417864    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6db7b021-z5g6p" podStartSLOduration=1.417850789 podStartE2EDuration="1.417850789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:43:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:43:41.417570184 +0000 UTC m=+734.661535651" watchObservedRunningTime="2026-11-03 19:43:41.417850789 +0000 UTC m=+734.661816254"
Nov 03 19:43:42.412315 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:42.412271    4921 generic.go:356] "Generic (PLEG): container finished" podID="dbade572-c672-4955-bdf8-5e47fc2099c7" containerID="321a599f1e7276c617b29dc72387f7db367c0888c6a95f658f1c4f5ee91efad2" exitCode=0
Nov 03 19:43:42.412742 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:42.412333    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6db7b021-z5g6p" event={"ID":"dbade572-c672-4955-bdf8-5e47fc2099c7","Type":"ContainerDied","Data":"321a599f1e7276c617b29dc72387f7db367c0888c6a95f658f1c4f5ee91efad2"}
Nov 03 19:43:43.558151 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:43.558122    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6db7b021-z5g6p"
Nov 03 19:43:43.622156 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:43.622122    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbade572-c672-4955-bdf8-5e47fc2099c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbade572-c672-4955-bdf8-5e47fc2099c7-maas-api-openshift-ca\") pod \"dbade572-c672-4955-bdf8-5e47fc2099c7\" (UID: \"dbade572-c672-4955-bdf8-5e47fc2099c7\") "
Nov 03 19:43:43.622294 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:43.622274    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbade572-c672-4955-bdf8-5e47fc2099c7-kube-api-access-9xsl9\" (UniqueName: \"kubernetes.io/projected/dbade572-c672-4955-bdf8-5e47fc2099c7-kube-api-access-9xsl9\") pod \"dbade572-c672-4955-bdf8-5e47fc2099c7\" (UID: \"dbade572-c672-4955-bdf8-5e47fc2099c7\") "
Nov 03 19:43:43.622338 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:43.622324    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbade572-c672-4955-bdf8-5e47fc2099c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbade572-c672-4955-bdf8-5e47fc2099c7-maas-api-xks-ca\") pod \"dbade572-c672-4955-bdf8-5e47fc2099c7\" (UID: \"dbade572-c672-4955-bdf8-5e47fc2099c7\") "
Nov 03 19:43:43.622531 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:43.622500    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbade572-c672-4955-bdf8-5e47fc2099c7-maas-api-openshift-ca" pod "dbade572-c672-4955-bdf8-5e47fc2099c7" (UID: "dbade572-c672-4955-bdf8-5e47fc2099c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:43:43.625009 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:43.624984    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbade572-c672-4955-bdf8-5e47fc2099c7-kube-api-access-9xsl9" pod "dbade572-c672-4955-bdf8-5e47fc2099c7" (UID: "dbade572-c672-4955-bdf8-5e47fc2099c7"). InnerVolumeSpecName "kube-api-access-9xsl9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:43:43.625106 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:43.624987    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbade572-c672-4955-bdf8-5e47fc2099c7-maas-api-xks-ca" pod "dbade572-c672-4955-bdf8-5e47fc2099c7" (UID: "dbade572-c672-4955-bdf8-5e47fc2099c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:43:43.723487 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:43.723410    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbade572-c672-4955-bdf8-5e47fc2099c7-maas-api-xks-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:43:43.723487 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:43.723437    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbade572-c672-4955-bdf8-5e47fc2099c7-maas-api-openshift-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:43:43.723487 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:43.723447    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xsl9\" (UniqueName: \"kubernetes.io/projected/dbade572-c672-4955-bdf8-5e47fc2099c7-kube-api-access-9xsl9\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:43:44.051209 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:44.051169    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b01b13a-qmqfx"]
Nov 03 19:43:44.051658 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:44.051644    4921 cpu_manager.go:414] "containerMap: removing container" podUID="dbade572-c672-4955-bdf8-5e47fc2099c7" containerName="revoke-keys"
Nov 03 19:43:44.051775 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:44.051659    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbade572-c672-4955-bdf8-5e47fc2099c7" containerName="revoke-keys"
Nov 03 19:43:44.051828 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:44.051787    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbade572-c672-4955-bdf8-5e47fc2099c7" containerName="revoke-keys"
Nov 03 19:43:44.055041 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:44.055025    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b01b13a-qmqfx"
Nov 03 19:43:44.065276 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:44.065248    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b01b13a-qmqfx"]
Nov 03 19:43:44.127758 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:44.127721    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3990af60-0428-45ec-8f19-715ea138bfdc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b01b13a-qmqfx\" (UID: \"3990af60-0428-45ec-8f19-715ea138bfdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b01b13a-qmqfx"
Nov 03 19:43:44.127935 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:44.127770    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3990af60-0428-45ec-8f19-715ea138bfdc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b01b13a-qmqfx\" (UID: \"3990af60-0428-45ec-8f19-715ea138bfdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b01b13a-qmqfx"
Nov 03 19:43:44.127935 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:44.127788    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqnws\" (UniqueName: \"kubernetes.io/projected/3990af60-0428-45ec-8f19-715ea138bfdc-kube-api-access-tqnws\") pod \"maas-api-revoke-sub-5b01b13a-qmqfx\" (UID: \"3990af60-0428-45ec-8f19-715ea138bfdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b01b13a-qmqfx"
Nov 03 19:43:44.229015 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:44.228975    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3990af60-0428-45ec-8f19-715ea138bfdc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b01b13a-qmqfx\" (UID: \"3990af60-0428-45ec-8f19-715ea138bfdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b01b13a-qmqfx"
Nov 03 19:43:44.229204 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:44.229034    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3990af60-0428-45ec-8f19-715ea138bfdc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b01b13a-qmqfx\" (UID: \"3990af60-0428-45ec-8f19-715ea138bfdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b01b13a-qmqfx"
Nov 03 19:43:44.229204 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:44.229053    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqnws\" (UniqueName: \"kubernetes.io/projected/3990af60-0428-45ec-8f19-715ea138bfdc-kube-api-access-tqnws\") pod \"maas-api-revoke-sub-5b01b13a-qmqfx\" (UID: \"3990af60-0428-45ec-8f19-715ea138bfdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b01b13a-qmqfx"
Nov 03 19:43:44.229759 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:44.229732    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3990af60-0428-45ec-8f19-715ea138bfdc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b01b13a-qmqfx\" (UID: \"3990af60-0428-45ec-8f19-715ea138bfdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b01b13a-qmqfx"
Nov 03 19:43:44.231485 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:44.231457    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3990af60-0428-45ec-8f19-715ea138bfdc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b01b13a-qmqfx\" (UID: \"3990af60-0428-45ec-8f19-715ea138bfdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b01b13a-qmqfx"
Nov 03 19:43:44.235652 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:44.235627    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqnws\" (UniqueName: \"kubernetes.io/projected/3990af60-0428-45ec-8f19-715ea138bfdc-kube-api-access-tqnws\") pod \"maas-api-revoke-sub-5b01b13a-qmqfx\" (UID: \"3990af60-0428-45ec-8f19-715ea138bfdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b01b13a-qmqfx"
Nov 03 19:43:44.371873 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:44.371783    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b01b13a-qmqfx"
Nov 03 19:43:44.426992 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:44.426963    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6db7b021-z5g6p" event={"ID":"dbade572-c672-4955-bdf8-5e47fc2099c7","Type":"ContainerDied","Data":"43e80cd2621e13d38416d419c1171f021df0538de280644b2bae337d0373cf7c"}
Nov 03 19:43:44.427117 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:44.427016    4921 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43e80cd2621e13d38416d419c1171f021df0538de280644b2bae337d0373cf7c"
Nov 03 19:43:44.427117 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:44.427079    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6db7b021-z5g6p"
Nov 03 19:43:44.647978 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:44.647948    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b01b13a-qmqfx"]
Nov 03 19:43:44.649874 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:43:44.649843    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3990af60_0428_45ec_8f19_715ea138bfdc.slice/crio-a9ff103ef67dcc07ec16cf11c7c577f963b28ed6022c0caab5d431654d344e8e WatchSource:0}: Error finding container a9ff103ef67dcc07ec16cf11c7c577f963b28ed6022c0caab5d431654d344e8e: Status 404 returned error can't find the container with id a9ff103ef67dcc07ec16cf11c7c577f963b28ed6022c0caab5d431654d344e8e
Nov 03 19:43:45.434126 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:45.434092    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b01b13a-qmqfx" event={"ID":"3990af60-0428-45ec-8f19-715ea138bfdc","Type":"ContainerStarted","Data":"886ad597e86ca39fb6a8635c7ac666a4c1aa5ff31630dda8a5093a98d89bbd31"}
Nov 03 19:43:45.434297 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:45.434148    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b01b13a-qmqfx" event={"ID":"3990af60-0428-45ec-8f19-715ea138bfdc","Type":"ContainerStarted","Data":"a9ff103ef67dcc07ec16cf11c7c577f963b28ed6022c0caab5d431654d344e8e"}
Nov 03 19:43:45.451654 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:45.451596    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b01b13a-qmqfx" podStartSLOduration=1.451577675 podStartE2EDuration="1.451577675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:43:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:43:45.448126808 +0000 UTC m=+738.692092284" watchObservedRunningTime="2026-11-03 19:43:45.451577675 +0000 UTC m=+738.695543141"
Nov 03 19:43:46.450371 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:46.450329    4921 generic.go:356] "Generic (PLEG): container finished" podID="3990af60-0428-45ec-8f19-715ea138bfdc" containerID="886ad597e86ca39fb6a8635c7ac666a4c1aa5ff31630dda8a5093a98d89bbd31" exitCode=0
Nov 03 19:43:46.450826 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:46.450514    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b01b13a-qmqfx" event={"ID":"3990af60-0428-45ec-8f19-715ea138bfdc","Type":"ContainerDied","Data":"886ad597e86ca39fb6a8635c7ac666a4c1aa5ff31630dda8a5093a98d89bbd31"}
Nov 03 19:43:47.607149 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:47.607119    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b01b13a-qmqfx"
Nov 03 19:43:47.770414 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:47.770384    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3990af60-0428-45ec-8f19-715ea138bfdc-kube-api-access-tqnws\" (UniqueName: \"kubernetes.io/projected/3990af60-0428-45ec-8f19-715ea138bfdc-kube-api-access-tqnws\") pod \"3990af60-0428-45ec-8f19-715ea138bfdc\" (UID: \"3990af60-0428-45ec-8f19-715ea138bfdc\") "
Nov 03 19:43:47.770578 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:47.770451    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3990af60-0428-45ec-8f19-715ea138bfdc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3990af60-0428-45ec-8f19-715ea138bfdc-maas-api-openshift-ca\") pod \"3990af60-0428-45ec-8f19-715ea138bfdc\" (UID: \"3990af60-0428-45ec-8f19-715ea138bfdc\") "
Nov 03 19:43:47.770578 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:47.770531    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3990af60-0428-45ec-8f19-715ea138bfdc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3990af60-0428-45ec-8f19-715ea138bfdc-maas-api-xks-ca\") pod \"3990af60-0428-45ec-8f19-715ea138bfdc\" (UID: \"3990af60-0428-45ec-8f19-715ea138bfdc\") "
Nov 03 19:43:47.770840 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:47.770817    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3990af60-0428-45ec-8f19-715ea138bfdc-maas-api-openshift-ca" pod "3990af60-0428-45ec-8f19-715ea138bfdc" (UID: "3990af60-0428-45ec-8f19-715ea138bfdc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:43:47.772572 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:47.772546    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3990af60-0428-45ec-8f19-715ea138bfdc-maas-api-xks-ca" pod "3990af60-0428-45ec-8f19-715ea138bfdc" (UID: "3990af60-0428-45ec-8f19-715ea138bfdc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:43:47.772655 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:47.772576    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3990af60-0428-45ec-8f19-715ea138bfdc-kube-api-access-tqnws" pod "3990af60-0428-45ec-8f19-715ea138bfdc" (UID: "3990af60-0428-45ec-8f19-715ea138bfdc"). InnerVolumeSpecName "kube-api-access-tqnws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:43:47.872493 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:47.872416    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3990af60-0428-45ec-8f19-715ea138bfdc-maas-api-xks-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:43:47.872493 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:47.872452    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqnws\" (UniqueName: \"kubernetes.io/projected/3990af60-0428-45ec-8f19-715ea138bfdc-kube-api-access-tqnws\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:43:47.872493 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:47.872468    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3990af60-0428-45ec-8f19-715ea138bfdc-maas-api-openshift-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:43:48.464119 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:48.464088    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b01b13a-qmqfx"
Nov 03 19:43:48.464289 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:48.464125    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b01b13a-qmqfx" event={"ID":"3990af60-0428-45ec-8f19-715ea138bfdc","Type":"ContainerDied","Data":"a9ff103ef67dcc07ec16cf11c7c577f963b28ed6022c0caab5d431654d344e8e"}
Nov 03 19:43:48.464289 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:48.464173    4921 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9ff103ef67dcc07ec16cf11c7c577f963b28ed6022c0caab5d431654d344e8e"
Nov 03 19:43:50.055121 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:50.055086    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab4ea102-fqfq9"]
Nov 03 19:43:50.055595 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:50.055582    4921 cpu_manager.go:414] "containerMap: removing container" podUID="3990af60-0428-45ec-8f19-715ea138bfdc" containerName="revoke-keys"
Nov 03 19:43:50.055643 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:50.055596    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3990af60-0428-45ec-8f19-715ea138bfdc" containerName="revoke-keys"
Nov 03 19:43:50.055725 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:50.055715    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="3990af60-0428-45ec-8f19-715ea138bfdc" containerName="revoke-keys"
Nov 03 19:43:50.059084 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:50.059065    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab4ea102-fqfq9"
Nov 03 19:43:50.063262 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:50.063243    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:43:50.068075 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:50.068049    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab4ea102-fqfq9"]
Nov 03 19:43:50.164011 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:50.163979    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 19:43:50.193163 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:50.193130    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27f755da-8289-4800-b933-6c147ff53e48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab4ea102-fqfq9\" (UID: \"27f755da-8289-4800-b933-6c147ff53e48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab4ea102-fqfq9"
Nov 03 19:43:50.193163 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:50.193166    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjvlf\" (UniqueName: \"kubernetes.io/projected/27f755da-8289-4800-b933-6c147ff53e48-kube-api-access-kjvlf\") pod \"maas-api-revoke-sub-ab4ea102-fqfq9\" (UID: \"27f755da-8289-4800-b933-6c147ff53e48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab4ea102-fqfq9"
Nov 03 19:43:50.193367 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:50.193288    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27f755da-8289-4800-b933-6c147ff53e48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab4ea102-fqfq9\" (UID: \"27f755da-8289-4800-b933-6c147ff53e48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab4ea102-fqfq9"
Nov 03 19:43:50.294863 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:50.294831    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27f755da-8289-4800-b933-6c147ff53e48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab4ea102-fqfq9\" (UID: \"27f755da-8289-4800-b933-6c147ff53e48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab4ea102-fqfq9"
Nov 03 19:43:50.295028 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:50.294909    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27f755da-8289-4800-b933-6c147ff53e48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab4ea102-fqfq9\" (UID: \"27f755da-8289-4800-b933-6c147ff53e48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab4ea102-fqfq9"
Nov 03 19:43:50.295028 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:50.294931    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjvlf\" (UniqueName: \"kubernetes.io/projected/27f755da-8289-4800-b933-6c147ff53e48-kube-api-access-kjvlf\") pod \"maas-api-revoke-sub-ab4ea102-fqfq9\" (UID: \"27f755da-8289-4800-b933-6c147ff53e48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab4ea102-fqfq9"
Nov 03 19:43:50.295521 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:50.295492    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27f755da-8289-4800-b933-6c147ff53e48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab4ea102-fqfq9\" (UID: \"27f755da-8289-4800-b933-6c147ff53e48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab4ea102-fqfq9"
Nov 03 19:43:50.297295 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:50.297260    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27f755da-8289-4800-b933-6c147ff53e48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab4ea102-fqfq9\" (UID: \"27f755da-8289-4800-b933-6c147ff53e48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab4ea102-fqfq9"
Nov 03 19:43:50.302704 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:50.302682    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjvlf\" (UniqueName: \"kubernetes.io/projected/27f755da-8289-4800-b933-6c147ff53e48-kube-api-access-kjvlf\") pod \"maas-api-revoke-sub-ab4ea102-fqfq9\" (UID: \"27f755da-8289-4800-b933-6c147ff53e48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab4ea102-fqfq9"
Nov 03 19:43:50.375128 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:50.375041    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab4ea102-fqfq9"
Nov 03 19:43:50.651803 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:50.651779    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab4ea102-fqfq9"]
Nov 03 19:43:50.653930 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:43:50.653900    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27f755da_8289_4800_b933_6c147ff53e48.slice/crio-cbc840eab17ff2a0248658573ddb25cf3a719704ec486aa8be1e43abc32f0ea8 WatchSource:0}: Error finding container cbc840eab17ff2a0248658573ddb25cf3a719704ec486aa8be1e43abc32f0ea8: Status 404 returned error can't find the container with id cbc840eab17ff2a0248658573ddb25cf3a719704ec486aa8be1e43abc32f0ea8
Nov 03 19:43:51.486394 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:51.486365    4921 generic.go:356] "Generic (PLEG): container finished" podID="27f755da-8289-4800-b933-6c147ff53e48" containerID="16c2994fc09cd5da78af94f3359c0fd049607fae290b486a18f849584d64fa72" exitCode=0
Nov 03 19:43:51.486856 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:51.486412    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab4ea102-fqfq9" event={"ID":"27f755da-8289-4800-b933-6c147ff53e48","Type":"ContainerDied","Data":"16c2994fc09cd5da78af94f3359c0fd049607fae290b486a18f849584d64fa72"}
Nov 03 19:43:51.486856 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:51.486469    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab4ea102-fqfq9" event={"ID":"27f755da-8289-4800-b933-6c147ff53e48","Type":"ContainerStarted","Data":"cbc840eab17ff2a0248658573ddb25cf3a719704ec486aa8be1e43abc32f0ea8"}
Nov 03 19:43:52.686250 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:52.686218    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab4ea102-fqfq9"
Nov 03 19:43:52.829279 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:52.829175    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27f755da-8289-4800-b933-6c147ff53e48-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27f755da-8289-4800-b933-6c147ff53e48-maas-api-xks-ca\") pod \"27f755da-8289-4800-b933-6c147ff53e48\" (UID: \"27f755da-8289-4800-b933-6c147ff53e48\") "
Nov 03 19:43:52.829454 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:52.829282    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27f755da-8289-4800-b933-6c147ff53e48-kube-api-access-kjvlf\" (UniqueName: \"kubernetes.io/projected/27f755da-8289-4800-b933-6c147ff53e48-kube-api-access-kjvlf\") pod \"27f755da-8289-4800-b933-6c147ff53e48\" (UID: \"27f755da-8289-4800-b933-6c147ff53e48\") "
Nov 03 19:43:52.829454 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:52.829346    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27f755da-8289-4800-b933-6c147ff53e48-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27f755da-8289-4800-b933-6c147ff53e48-maas-api-openshift-ca\") pod \"27f755da-8289-4800-b933-6c147ff53e48\" (UID: \"27f755da-8289-4800-b933-6c147ff53e48\") "
Nov 03 19:43:52.829824 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:52.829777    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27f755da-8289-4800-b933-6c147ff53e48-maas-api-openshift-ca" pod "27f755da-8289-4800-b933-6c147ff53e48" (UID: "27f755da-8289-4800-b933-6c147ff53e48"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:43:52.834125 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:52.831696    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27f755da-8289-4800-b933-6c147ff53e48-kube-api-access-kjvlf" pod "27f755da-8289-4800-b933-6c147ff53e48" (UID: "27f755da-8289-4800-b933-6c147ff53e48"). InnerVolumeSpecName "kube-api-access-kjvlf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:43:52.834620 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:52.834592    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27f755da-8289-4800-b933-6c147ff53e48-maas-api-xks-ca" pod "27f755da-8289-4800-b933-6c147ff53e48" (UID: "27f755da-8289-4800-b933-6c147ff53e48"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:43:52.930542 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:52.930499    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27f755da-8289-4800-b933-6c147ff53e48-maas-api-xks-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:43:52.930542 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:52.930541    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjvlf\" (UniqueName: \"kubernetes.io/projected/27f755da-8289-4800-b933-6c147ff53e48-kube-api-access-kjvlf\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:43:52.930816 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:52.930563    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27f755da-8289-4800-b933-6c147ff53e48-maas-api-openshift-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:43:53.522229 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:53.522191    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab4ea102-fqfq9" event={"ID":"27f755da-8289-4800-b933-6c147ff53e48","Type":"ContainerDied","Data":"cbc840eab17ff2a0248658573ddb25cf3a719704ec486aa8be1e43abc32f0ea8"}
Nov 03 19:43:53.522415 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:53.522274    4921 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cbc840eab17ff2a0248658573ddb25cf3a719704ec486aa8be1e43abc32f0ea8"
Nov 03 19:43:53.522415 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:53.522335    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab4ea102-fqfq9"
Nov 03 19:43:54.106709 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:54.104723    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66af4baf-jd5mj"]
Nov 03 19:43:54.106709 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:54.105496    4921 cpu_manager.go:414] "containerMap: removing container" podUID="27f755da-8289-4800-b933-6c147ff53e48" containerName="revoke-keys"
Nov 03 19:43:54.106709 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:54.105515    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27f755da-8289-4800-b933-6c147ff53e48" containerName="revoke-keys"
Nov 03 19:43:54.106709 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:54.105635    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="27f755da-8289-4800-b933-6c147ff53e48" containerName="revoke-keys"
Nov 03 19:43:54.122062 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:54.122032    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66af4baf-jd5mj"]
Nov 03 19:43:54.122326 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:54.122302    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66af4baf-jd5mj"
Nov 03 19:43:54.128949 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:54.128930    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:43:54.134023 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:54.133932    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 19:43:54.263689 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:54.261208    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecdf5c52-5f0d-41f4-ba12-9377eadbf76e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66af4baf-jd5mj\" (UID: \"ecdf5c52-5f0d-41f4-ba12-9377eadbf76e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66af4baf-jd5mj"
Nov 03 19:43:54.263689 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:54.261316    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecdf5c52-5f0d-41f4-ba12-9377eadbf76e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66af4baf-jd5mj\" (UID: \"ecdf5c52-5f0d-41f4-ba12-9377eadbf76e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66af4baf-jd5mj"
Nov 03 19:43:54.263689 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:54.261396    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g47tj\" (UniqueName: \"kubernetes.io/projected/ecdf5c52-5f0d-41f4-ba12-9377eadbf76e-kube-api-access-g47tj\") pod \"maas-api-revoke-sub-66af4baf-jd5mj\" (UID: \"ecdf5c52-5f0d-41f4-ba12-9377eadbf76e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66af4baf-jd5mj"
Nov 03 19:43:54.367311 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:54.367210    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecdf5c52-5f0d-41f4-ba12-9377eadbf76e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66af4baf-jd5mj\" (UID: \"ecdf5c52-5f0d-41f4-ba12-9377eadbf76e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66af4baf-jd5mj"
Nov 03 19:43:54.368134 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:54.368108    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecdf5c52-5f0d-41f4-ba12-9377eadbf76e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66af4baf-jd5mj\" (UID: \"ecdf5c52-5f0d-41f4-ba12-9377eadbf76e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66af4baf-jd5mj"
Nov 03 19:43:54.368358 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:54.368339    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g47tj\" (UniqueName: \"kubernetes.io/projected/ecdf5c52-5f0d-41f4-ba12-9377eadbf76e-kube-api-access-g47tj\") pod \"maas-api-revoke-sub-66af4baf-jd5mj\" (UID: \"ecdf5c52-5f0d-41f4-ba12-9377eadbf76e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66af4baf-jd5mj"
Nov 03 19:43:54.389688 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:54.369563    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecdf5c52-5f0d-41f4-ba12-9377eadbf76e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66af4baf-jd5mj\" (UID: \"ecdf5c52-5f0d-41f4-ba12-9377eadbf76e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66af4baf-jd5mj"
Nov 03 19:43:54.413689 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:54.400744    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecdf5c52-5f0d-41f4-ba12-9377eadbf76e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66af4baf-jd5mj\" (UID: \"ecdf5c52-5f0d-41f4-ba12-9377eadbf76e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66af4baf-jd5mj"
Nov 03 19:43:54.417688 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:54.414684    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g47tj\" (UniqueName: \"kubernetes.io/projected/ecdf5c52-5f0d-41f4-ba12-9377eadbf76e-kube-api-access-g47tj\") pod \"maas-api-revoke-sub-66af4baf-jd5mj\" (UID: \"ecdf5c52-5f0d-41f4-ba12-9377eadbf76e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66af4baf-jd5mj"
Nov 03 19:43:54.442518 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:54.442489    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66af4baf-jd5mj"
Nov 03 19:43:54.882694 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:54.881516    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66af4baf-jd5mj"]
Nov 03 19:43:55.541649 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:55.541611    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66af4baf-jd5mj" event={"ID":"ecdf5c52-5f0d-41f4-ba12-9377eadbf76e","Type":"ContainerStarted","Data":"3eba5649c4cf9ebf9fe6be2fcda57f733fc817660787a36c616dc812581987df"}
Nov 03 19:43:55.542146 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:55.541699    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66af4baf-jd5mj" event={"ID":"ecdf5c52-5f0d-41f4-ba12-9377eadbf76e","Type":"ContainerStarted","Data":"c6cbe214b8d81805392ffa60bbd45675d0bcda36278e342c174e0fe821989e81"}
Nov 03 19:43:55.559615 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:55.559562    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66af4baf-jd5mj" podStartSLOduration=1.5595489900000001 podStartE2EDuration="1.55954899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:43:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:43:55.556085112 +0000 UTC m=+748.800050579" watchObservedRunningTime="2026-11-03 19:43:55.55954899 +0000 UTC m=+748.803514454"
Nov 03 19:43:56.073265 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:56.073228    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6"]
Nov 03 19:43:56.077853 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:56.077831    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6"
Nov 03 19:43:56.085709 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:56.085660    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6"]
Nov 03 19:43:56.085837 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:56.085663    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-8d4127\"/\"distinct-w2-8d4127-kserve-self-signed-certs\""
Nov 03 19:43:56.130682 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:56.130636    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1aaee61d-b036-4c2b-bee7-a692b8bed612-tls-certs\") pod \"distinct-w2-8d4127-kserve-c86c68c9-b5tv6\" (UID: \"1aaee61d-b036-4c2b-bee7-a692b8bed612\") " pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6"
Nov 03 19:43:56.130832 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:56.130693    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1aaee61d-b036-4c2b-bee7-a692b8bed612-model-cache\") pod \"distinct-w2-8d4127-kserve-c86c68c9-b5tv6\" (UID: \"1aaee61d-b036-4c2b-bee7-a692b8bed612\") " pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6"
Nov 03 19:43:56.130832 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:56.130794    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1aaee61d-b036-4c2b-bee7-a692b8bed612-tmp-dir\") pod \"distinct-w2-8d4127-kserve-c86c68c9-b5tv6\" (UID: \"1aaee61d-b036-4c2b-bee7-a692b8bed612\") " pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6"
Nov 03 19:43:56.130956 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:56.130929    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1aaee61d-b036-4c2b-bee7-a692b8bed612-dshm\") pod \"distinct-w2-8d4127-kserve-c86c68c9-b5tv6\" (UID: \"1aaee61d-b036-4c2b-bee7-a692b8bed612\") " pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6"
Nov 03 19:43:56.131050 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:56.131026    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxdxg\" (UniqueName: \"kubernetes.io/projected/1aaee61d-b036-4c2b-bee7-a692b8bed612-kube-api-access-hxdxg\") pod \"distinct-w2-8d4127-kserve-c86c68c9-b5tv6\" (UID: \"1aaee61d-b036-4c2b-bee7-a692b8bed612\") " pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6"
Nov 03 19:43:56.131181 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:56.131084    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1aaee61d-b036-4c2b-bee7-a692b8bed612-home\") pod \"distinct-w2-8d4127-kserve-c86c68c9-b5tv6\" (UID: \"1aaee61d-b036-4c2b-bee7-a692b8bed612\") " pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6"
Nov 03 19:43:56.231894 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:56.231861    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxdxg\" (UniqueName: \"kubernetes.io/projected/1aaee61d-b036-4c2b-bee7-a692b8bed612-kube-api-access-hxdxg\") pod \"distinct-w2-8d4127-kserve-c86c68c9-b5tv6\" (UID: \"1aaee61d-b036-4c2b-bee7-a692b8bed612\") " pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6"
Nov 03 19:43:56.232083 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:56.231910    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1aaee61d-b036-4c2b-bee7-a692b8bed612-home\") pod \"distinct-w2-8d4127-kserve-c86c68c9-b5tv6\" (UID: \"1aaee61d-b036-4c2b-bee7-a692b8bed612\") " pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6"
Nov 03 19:43:56.232083 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:56.231964    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1aaee61d-b036-4c2b-bee7-a692b8bed612-tls-certs\") pod \"distinct-w2-8d4127-kserve-c86c68c9-b5tv6\" (UID: \"1aaee61d-b036-4c2b-bee7-a692b8bed612\") " pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6"
Nov 03 19:43:56.232083 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:56.231989    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1aaee61d-b036-4c2b-bee7-a692b8bed612-model-cache\") pod \"distinct-w2-8d4127-kserve-c86c68c9-b5tv6\" (UID: \"1aaee61d-b036-4c2b-bee7-a692b8bed612\") " pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6"
Nov 03 19:43:56.232083 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:56.232026    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1aaee61d-b036-4c2b-bee7-a692b8bed612-tmp-dir\") pod \"distinct-w2-8d4127-kserve-c86c68c9-b5tv6\" (UID: \"1aaee61d-b036-4c2b-bee7-a692b8bed612\") " pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6"
Nov 03 19:43:56.232297 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:56.232119    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1aaee61d-b036-4c2b-bee7-a692b8bed612-dshm\") pod \"distinct-w2-8d4127-kserve-c86c68c9-b5tv6\" (UID: \"1aaee61d-b036-4c2b-bee7-a692b8bed612\") " pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6"
Nov 03 19:43:56.232496 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:56.232464    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1aaee61d-b036-4c2b-bee7-a692b8bed612-model-cache\") pod \"distinct-w2-8d4127-kserve-c86c68c9-b5tv6\" (UID: \"1aaee61d-b036-4c2b-bee7-a692b8bed612\") " pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6"
Nov 03 19:43:56.233374 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:56.232813    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1aaee61d-b036-4c2b-bee7-a692b8bed612-home\") pod \"distinct-w2-8d4127-kserve-c86c68c9-b5tv6\" (UID: \"1aaee61d-b036-4c2b-bee7-a692b8bed612\") " pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6"
Nov 03 19:43:56.233374 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:56.232961    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1aaee61d-b036-4c2b-bee7-a692b8bed612-tmp-dir\") pod \"distinct-w2-8d4127-kserve-c86c68c9-b5tv6\" (UID: \"1aaee61d-b036-4c2b-bee7-a692b8bed612\") " pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6"
Nov 03 19:43:56.235831 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:56.235809    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1aaee61d-b036-4c2b-bee7-a692b8bed612-dshm\") pod \"distinct-w2-8d4127-kserve-c86c68c9-b5tv6\" (UID: \"1aaee61d-b036-4c2b-bee7-a692b8bed612\") " pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6"
Nov 03 19:43:56.240867 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:56.240841    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxdxg\" (UniqueName: \"kubernetes.io/projected/1aaee61d-b036-4c2b-bee7-a692b8bed612-kube-api-access-hxdxg\") pod \"distinct-w2-8d4127-kserve-c86c68c9-b5tv6\" (UID: \"1aaee61d-b036-4c2b-bee7-a692b8bed612\") " pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6"
Nov 03 19:43:56.248369 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:56.248348    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1aaee61d-b036-4c2b-bee7-a692b8bed612-tls-certs\") pod \"distinct-w2-8d4127-kserve-c86c68c9-b5tv6\" (UID: \"1aaee61d-b036-4c2b-bee7-a692b8bed612\") " pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6"
Nov 03 19:43:56.395808 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:56.395727    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6"
Nov 03 19:43:56.548400 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:56.548363    4921 generic.go:356] "Generic (PLEG): container finished" podID="ecdf5c52-5f0d-41f4-ba12-9377eadbf76e" containerID="3eba5649c4cf9ebf9fe6be2fcda57f733fc817660787a36c616dc812581987df" exitCode=0
Nov 03 19:43:56.548776 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:56.548432    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66af4baf-jd5mj" event={"ID":"ecdf5c52-5f0d-41f4-ba12-9377eadbf76e","Type":"ContainerDied","Data":"3eba5649c4cf9ebf9fe6be2fcda57f733fc817660787a36c616dc812581987df"}
Nov 03 19:43:56.681995 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:56.681970    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6"]
Nov 03 19:43:56.683743 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:43:56.683711    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1aaee61d_b036_4c2b_bee7_a692b8bed612.slice/crio-9a8732ca19050dee4dc67889520d6a966951d78003deefeec8c979249300e560 WatchSource:0}: Error finding container 9a8732ca19050dee4dc67889520d6a966951d78003deefeec8c979249300e560: Status 404 returned error can't find the container with id 9a8732ca19050dee4dc67889520d6a966951d78003deefeec8c979249300e560
Nov 03 19:43:57.556271 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:57.556231    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6" event={"ID":"1aaee61d-b036-4c2b-bee7-a692b8bed612","Type":"ContainerStarted","Data":"207926fd238937e676e8b1d9164180443b545cd5d94e37d48e3ddf3f1df63e77"}
Nov 03 19:43:57.556716 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:57.556283    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6" event={"ID":"1aaee61d-b036-4c2b-bee7-a692b8bed612","Type":"ContainerStarted","Data":"9a8732ca19050dee4dc67889520d6a966951d78003deefeec8c979249300e560"}
Nov 03 19:43:57.572273 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:57.572214    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6" podStartSLOduration=1.572194216 podStartE2EDuration="1.572194216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:43:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:43:57.57119769 +0000 UTC m=+750.815163158" watchObservedRunningTime="2026-11-03 19:43:57.572194216 +0000 UTC m=+750.816159682"
Nov 03 19:43:57.726504 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:57.726479    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66af4baf-jd5mj"
Nov 03 19:43:57.748153 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:57.748126    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecdf5c52-5f0d-41f4-ba12-9377eadbf76e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecdf5c52-5f0d-41f4-ba12-9377eadbf76e-maas-api-openshift-ca\") pod \"ecdf5c52-5f0d-41f4-ba12-9377eadbf76e\" (UID: \"ecdf5c52-5f0d-41f4-ba12-9377eadbf76e\") "
Nov 03 19:43:57.748327 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:57.748209    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ecdf5c52-5f0d-41f4-ba12-9377eadbf76e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecdf5c52-5f0d-41f4-ba12-9377eadbf76e-maas-api-xks-ca\") pod \"ecdf5c52-5f0d-41f4-ba12-9377eadbf76e\" (UID: \"ecdf5c52-5f0d-41f4-ba12-9377eadbf76e\") "
Nov 03 19:43:57.748327 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:57.748236    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecdf5c52-5f0d-41f4-ba12-9377eadbf76e-kube-api-access-g47tj\" (UniqueName: \"kubernetes.io/projected/ecdf5c52-5f0d-41f4-ba12-9377eadbf76e-kube-api-access-g47tj\") pod \"ecdf5c52-5f0d-41f4-ba12-9377eadbf76e\" (UID: \"ecdf5c52-5f0d-41f4-ba12-9377eadbf76e\") "
Nov 03 19:43:57.749186 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:57.749162    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecdf5c52-5f0d-41f4-ba12-9377eadbf76e-maas-api-openshift-ca" pod "ecdf5c52-5f0d-41f4-ba12-9377eadbf76e" (UID: "ecdf5c52-5f0d-41f4-ba12-9377eadbf76e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:43:57.751167 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:57.751144    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecdf5c52-5f0d-41f4-ba12-9377eadbf76e-kube-api-access-g47tj" pod "ecdf5c52-5f0d-41f4-ba12-9377eadbf76e" (UID: "ecdf5c52-5f0d-41f4-ba12-9377eadbf76e"). InnerVolumeSpecName "kube-api-access-g47tj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:43:57.751276 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:57.751194    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecdf5c52-5f0d-41f4-ba12-9377eadbf76e-maas-api-xks-ca" pod "ecdf5c52-5f0d-41f4-ba12-9377eadbf76e" (UID: "ecdf5c52-5f0d-41f4-ba12-9377eadbf76e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:43:57.849307 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:57.849213    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecdf5c52-5f0d-41f4-ba12-9377eadbf76e-maas-api-openshift-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:43:57.849307 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:57.849243    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecdf5c52-5f0d-41f4-ba12-9377eadbf76e-maas-api-xks-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:43:57.849307 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:57.849252    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g47tj\" (UniqueName: \"kubernetes.io/projected/ecdf5c52-5f0d-41f4-ba12-9377eadbf76e-kube-api-access-g47tj\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:43:58.396733 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:58.396702    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6"
Nov 03 19:43:58.564325 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:58.564294    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66af4baf-jd5mj"
Nov 03 19:43:58.564748 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:58.564332    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66af4baf-jd5mj" event={"ID":"ecdf5c52-5f0d-41f4-ba12-9377eadbf76e","Type":"ContainerDied","Data":"c6cbe214b8d81805392ffa60bbd45675d0bcda36278e342c174e0fe821989e81"}
Nov 03 19:43:58.564748 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:43:58.564378    4921 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6cbe214b8d81805392ffa60bbd45675d0bcda36278e342c174e0fe821989e81"
Nov 03 19:44:02.388607 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:02.388535    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p"]
Nov 03 19:44:02.426171 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:02.425352    4921 cpu_manager.go:414] "containerMap: removing container" podUID="ecdf5c52-5f0d-41f4-ba12-9377eadbf76e" containerName="revoke-keys"
Nov 03 19:44:02.426171 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:02.425389    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecdf5c52-5f0d-41f4-ba12-9377eadbf76e" containerName="revoke-keys"
Nov 03 19:44:02.426171 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:02.425516    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecdf5c52-5f0d-41f4-ba12-9377eadbf76e" containerName="revoke-keys"
Nov 03 19:44:02.430121 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:02.430031    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p"]
Nov 03 19:44:02.430279 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:02.430168    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p"
Nov 03 19:44:02.434596 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:02.434571    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-39043d-openshift-default-dockercfg-zwlqs\""
Nov 03 19:44:02.528518 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:02.528484    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97dc926b-00fc-4a42-9283-938cac033482-workload-certs\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p\" (UID: \"97dc926b-00fc-4a42-9283-938cac033482\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p"
Nov 03 19:44:02.528719 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:02.528540    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97dc926b-00fc-4a42-9283-938cac033482-workload-socket\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p\" (UID: \"97dc926b-00fc-4a42-9283-938cac033482\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p"
Nov 03 19:44:02.528719 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:02.528576    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97dc926b-00fc-4a42-9283-938cac033482-istiod-ca-cert\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p\" (UID: \"97dc926b-00fc-4a42-9283-938cac033482\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p"
Nov 03 19:44:02.528806 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:02.528772    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97dc926b-00fc-4a42-9283-938cac033482-istio-envoy\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p\" (UID: \"97dc926b-00fc-4a42-9283-938cac033482\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p"
Nov 03 19:44:02.528847 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:02.528818    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggxw8\" (UniqueName: \"kubernetes.io/projected/97dc926b-00fc-4a42-9283-938cac033482-kube-api-access-ggxw8\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p\" (UID: \"97dc926b-00fc-4a42-9283-938cac033482\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p"
Nov 03 19:44:02.528924 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:02.528906    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97dc926b-00fc-4a42-9283-938cac033482-istio-token\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p\" (UID: \"97dc926b-00fc-4a42-9283-938cac033482\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p"
Nov 03 19:44:02.528985 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:02.528967    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97dc926b-00fc-4a42-9283-938cac033482-credential-socket\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p\" (UID: \"97dc926b-00fc-4a42-9283-938cac033482\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p"
Nov 03 19:44:02.529041 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:02.529025    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97dc926b-00fc-4a42-9283-938cac033482-istio-podinfo\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p\" (UID: \"97dc926b-00fc-4a42-9283-938cac033482\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p"
Nov 03 19:44:02.529121 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:02.529106    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97dc926b-00fc-4a42-9283-938cac033482-istio-data\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p\" (UID: \"97dc926b-00fc-4a42-9283-938cac033482\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p"
Nov 03 19:44:02.630621 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:02.630579    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggxw8\" (UniqueName: \"kubernetes.io/projected/97dc926b-00fc-4a42-9283-938cac033482-kube-api-access-ggxw8\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p\" (UID: \"97dc926b-00fc-4a42-9283-938cac033482\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p"
Nov 03 19:44:02.630621 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:02.630624    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97dc926b-00fc-4a42-9283-938cac033482-istio-token\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p\" (UID: \"97dc926b-00fc-4a42-9283-938cac033482\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p"
Nov 03 19:44:02.630875 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:02.630683    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97dc926b-00fc-4a42-9283-938cac033482-credential-socket\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p\" (UID: \"97dc926b-00fc-4a42-9283-938cac033482\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p"
Nov 03 19:44:02.630875 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:02.630729    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97dc926b-00fc-4a42-9283-938cac033482-istio-podinfo\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p\" (UID: \"97dc926b-00fc-4a42-9283-938cac033482\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p"
Nov 03 19:44:02.630875 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:02.630758    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97dc926b-00fc-4a42-9283-938cac033482-istio-data\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p\" (UID: \"97dc926b-00fc-4a42-9283-938cac033482\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p"
Nov 03 19:44:02.630875 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:02.630833    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97dc926b-00fc-4a42-9283-938cac033482-workload-certs\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p\" (UID: \"97dc926b-00fc-4a42-9283-938cac033482\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p"
Nov 03 19:44:02.631091 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:02.630878    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97dc926b-00fc-4a42-9283-938cac033482-workload-socket\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p\" (UID: \"97dc926b-00fc-4a42-9283-938cac033482\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p"
Nov 03 19:44:02.631091 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:02.630920    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97dc926b-00fc-4a42-9283-938cac033482-istiod-ca-cert\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p\" (UID: \"97dc926b-00fc-4a42-9283-938cac033482\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p"
Nov 03 19:44:02.631091 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:02.631018    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97dc926b-00fc-4a42-9283-938cac033482-istio-envoy\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p\" (UID: \"97dc926b-00fc-4a42-9283-938cac033482\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p"
Nov 03 19:44:02.631240 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:02.631186    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97dc926b-00fc-4a42-9283-938cac033482-credential-socket\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p\" (UID: \"97dc926b-00fc-4a42-9283-938cac033482\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p"
Nov 03 19:44:02.631398 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:02.631343    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97dc926b-00fc-4a42-9283-938cac033482-workload-certs\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p\" (UID: \"97dc926b-00fc-4a42-9283-938cac033482\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p"
Nov 03 19:44:02.631536 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:02.631494    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97dc926b-00fc-4a42-9283-938cac033482-workload-socket\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p\" (UID: \"97dc926b-00fc-4a42-9283-938cac033482\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p"
Nov 03 19:44:02.632180 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:02.631833    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97dc926b-00fc-4a42-9283-938cac033482-istio-data\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p\" (UID: \"97dc926b-00fc-4a42-9283-938cac033482\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p"
Nov 03 19:44:02.632180 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:02.631861    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97dc926b-00fc-4a42-9283-938cac033482-istiod-ca-cert\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p\" (UID: \"97dc926b-00fc-4a42-9283-938cac033482\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p"
Nov 03 19:44:02.633403 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:02.633378    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97dc926b-00fc-4a42-9283-938cac033482-istio-envoy\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p\" (UID: \"97dc926b-00fc-4a42-9283-938cac033482\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p"
Nov 03 19:44:02.633641 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:02.633619    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97dc926b-00fc-4a42-9283-938cac033482-istio-podinfo\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p\" (UID: \"97dc926b-00fc-4a42-9283-938cac033482\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p"
Nov 03 19:44:02.638353 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:02.638325    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97dc926b-00fc-4a42-9283-938cac033482-istio-token\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p\" (UID: \"97dc926b-00fc-4a42-9283-938cac033482\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p"
Nov 03 19:44:02.638499 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:02.638475    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggxw8\" (UniqueName: \"kubernetes.io/projected/97dc926b-00fc-4a42-9283-938cac033482-kube-api-access-ggxw8\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p\" (UID: \"97dc926b-00fc-4a42-9283-938cac033482\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p"
Nov 03 19:44:02.746967 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:02.746934    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p"
Nov 03 19:44:02.960292 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:44:02.960208    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 19:44:03.090488 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:03.090403    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p"]
Nov 03 19:44:03.109921 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:44:03.109846    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97dc926b_00fc_4a42_9283_938cac033482.slice/crio-6aebaccd3b21b5aaf45ac44e7baa831f469b288316f4849df0ef61a4d240b64b WatchSource:0}: Error finding container 6aebaccd3b21b5aaf45ac44e7baa831f469b288316f4849df0ef61a4d240b64b: Status 404 returned error can't find the container with id 6aebaccd3b21b5aaf45ac44e7baa831f469b288316f4849df0ef61a4d240b64b
Nov 03 19:44:03.119377 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:03.119339    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:44:03.119513 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:03.119426    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:44:03.119513 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:03.119475    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:44:03.612809 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:03.612771    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p" event={"ID":"97dc926b-00fc-4a42-9283-938cac033482","Type":"ContainerStarted","Data":"635c43f53926eb55869cf6e08091a17be68c98632072693b101aa881a14c2117"}
Nov 03 19:44:03.613440 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:03.613402    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p" event={"ID":"97dc926b-00fc-4a42-9283-938cac033482","Type":"ContainerStarted","Data":"6aebaccd3b21b5aaf45ac44e7baa831f469b288316f4849df0ef61a4d240b64b"}
Nov 03 19:44:03.634218 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:03.634160    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p" podStartSLOduration=1.634142947 podStartE2EDuration="1.634142947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:44:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:44:03.631118485 +0000 UTC m=+756.875083950" watchObservedRunningTime="2026-11-03 19:44:03.634142947 +0000 UTC m=+756.878108413"
Nov 03 19:44:03.747888 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:03.747861    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p"
Nov 03 19:44:03.841834 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:03.841796    4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-rl9tr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:44:03.841996 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:03.841852    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rl9tr" podUID="cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:44:04.753867 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:04.753833    4921 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Nov 03 19:44:04.754455 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:04.754419    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p" podUID="97dc926b-00fc-4a42-9283-938cac033482" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Nov 03 19:44:05.756066 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:05.756031    4921 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Nov 03 19:44:05.756633 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:05.756600    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p" podUID="97dc926b-00fc-4a42-9283-938cac033482" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Nov 03 19:44:06.396858 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:06.396825    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6"
Nov 03 19:44:06.411502 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:06.411463    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53ab2d2d-8v9d9"]
Nov 03 19:44:06.417930 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:06.417906    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6"
Nov 03 19:44:06.418078 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:06.417988    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53ab2d2d-8v9d9"
Nov 03 19:44:06.422550 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:06.422364    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:44:06.422550 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:06.422407    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53ab2d2d-8v9d9"]
Nov 03 19:44:06.422550 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:06.422471    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 19:44:06.518609 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:06.518574    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9b67cca-6629-4c9c-a69f-b4f0dabe2da3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53ab2d2d-8v9d9\" (UID: \"c9b67cca-6629-4c9c-a69f-b4f0dabe2da3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53ab2d2d-8v9d9"
Nov 03 19:44:06.518829 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:06.518803    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxprh\" (UniqueName: \"kubernetes.io/projected/c9b67cca-6629-4c9c-a69f-b4f0dabe2da3-kube-api-access-cxprh\") pod \"maas-api-revoke-sub-53ab2d2d-8v9d9\" (UID: \"c9b67cca-6629-4c9c-a69f-b4f0dabe2da3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53ab2d2d-8v9d9"
Nov 03 19:44:06.518915 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:06.518858    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9b67cca-6629-4c9c-a69f-b4f0dabe2da3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53ab2d2d-8v9d9\" (UID: \"c9b67cca-6629-4c9c-a69f-b4f0dabe2da3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53ab2d2d-8v9d9"
Nov 03 19:44:06.620180 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:06.620138    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9b67cca-6629-4c9c-a69f-b4f0dabe2da3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53ab2d2d-8v9d9\" (UID: \"c9b67cca-6629-4c9c-a69f-b4f0dabe2da3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53ab2d2d-8v9d9"
Nov 03 19:44:06.620381 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:06.620289    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxprh\" (UniqueName: \"kubernetes.io/projected/c9b67cca-6629-4c9c-a69f-b4f0dabe2da3-kube-api-access-cxprh\") pod \"maas-api-revoke-sub-53ab2d2d-8v9d9\" (UID: \"c9b67cca-6629-4c9c-a69f-b4f0dabe2da3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53ab2d2d-8v9d9"
Nov 03 19:44:06.620381 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:06.620341    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9b67cca-6629-4c9c-a69f-b4f0dabe2da3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53ab2d2d-8v9d9\" (UID: \"c9b67cca-6629-4c9c-a69f-b4f0dabe2da3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53ab2d2d-8v9d9"
Nov 03 19:44:06.621129 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:06.621098    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9b67cca-6629-4c9c-a69f-b4f0dabe2da3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53ab2d2d-8v9d9\" (UID: \"c9b67cca-6629-4c9c-a69f-b4f0dabe2da3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53ab2d2d-8v9d9"
Nov 03 19:44:06.624073 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:06.624044    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9b67cca-6629-4c9c-a69f-b4f0dabe2da3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53ab2d2d-8v9d9\" (UID: \"c9b67cca-6629-4c9c-a69f-b4f0dabe2da3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53ab2d2d-8v9d9"
Nov 03 19:44:06.627766 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:06.627735    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxprh\" (UniqueName: \"kubernetes.io/projected/c9b67cca-6629-4c9c-a69f-b4f0dabe2da3-kube-api-access-cxprh\") pod \"maas-api-revoke-sub-53ab2d2d-8v9d9\" (UID: \"c9b67cca-6629-4c9c-a69f-b4f0dabe2da3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53ab2d2d-8v9d9"
Nov 03 19:44:06.645543 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:06.645519    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6"
Nov 03 19:44:06.735948 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:06.735921    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53ab2d2d-8v9d9"
Nov 03 19:44:06.747753 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:06.747715    4921 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Nov 03 19:44:06.747881 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:06.747770    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p" podUID="97dc926b-00fc-4a42-9283-938cac033482" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Nov 03 19:44:07.045547 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:07.045477    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53ab2d2d-8v9d9"]
Nov 03 19:44:07.060688 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:44:07.056974    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9b67cca_6629_4c9c_a69f_b4f0dabe2da3.slice/crio-db791b1c72009001ba83ba371d864a6515705509491129c6835e147eb0bc7969 WatchSource:0}: Error finding container db791b1c72009001ba83ba371d864a6515705509491129c6835e147eb0bc7969: Status 404 returned error can't find the container with id db791b1c72009001ba83ba371d864a6515705509491129c6835e147eb0bc7969
Nov 03 19:44:07.644030 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:07.643992    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53ab2d2d-8v9d9" event={"ID":"c9b67cca-6629-4c9c-a69f-b4f0dabe2da3","Type":"ContainerStarted","Data":"676d8a7aadecb467cff6c6fcc2eba021efb5f5813c53ee9438bc2964b4fd229d"}
Nov 03 19:44:07.644201 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:07.644067    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53ab2d2d-8v9d9" event={"ID":"c9b67cca-6629-4c9c-a69f-b4f0dabe2da3","Type":"ContainerStarted","Data":"db791b1c72009001ba83ba371d864a6515705509491129c6835e147eb0bc7969"}
Nov 03 19:44:07.748705 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:07.748604    4921 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Nov 03 19:44:07.748705 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:07.748662    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p" podUID="97dc926b-00fc-4a42-9283-938cac033482" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Nov 03 19:44:08.662693 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:08.662510    4921 generic.go:356] "Generic (PLEG): container finished" podID="c9b67cca-6629-4c9c-a69f-b4f0dabe2da3" containerID="676d8a7aadecb467cff6c6fcc2eba021efb5f5813c53ee9438bc2964b4fd229d" exitCode=0
Nov 03 19:44:08.663206 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:08.662766    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53ab2d2d-8v9d9" event={"ID":"c9b67cca-6629-4c9c-a69f-b4f0dabe2da3","Type":"ContainerDied","Data":"676d8a7aadecb467cff6c6fcc2eba021efb5f5813c53ee9438bc2964b4fd229d"}
Nov 03 19:44:08.747867 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:08.747832    4921 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Nov 03 19:44:08.748031 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:08.747895    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p" podUID="97dc926b-00fc-4a42-9283-938cac033482" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Nov 03 19:44:08.844904 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:08.844868    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw"]
Nov 03 19:44:08.850717 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:08.850286    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw"
Nov 03 19:44:08.893970 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:08.872794    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-a35658\"/\"distinct-w1-a35658-kserve-self-signed-certs\""
Nov 03 19:44:08.893970 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:08.872923    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-a35658\"/\"kube-root-ca.crt\""
Nov 03 19:44:08.893970 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:08.873000    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-a35658\"/\"openshift-service-ca.crt\""
Nov 03 19:44:08.893970 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:08.873086    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-a35658\"/\"default-dockercfg-s6mct\""
Nov 03 19:44:08.903893 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:08.903863    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw"]
Nov 03 19:44:08.978302 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:08.978267    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72d96abc-f3b4-4eda-b3a1-b93ba658e1d5-tls-certs\") pod \"distinct-w1-a35658-kserve-54bbdfff9d-24klw\" (UID: \"72d96abc-f3b4-4eda-b3a1-b93ba658e1d5\") " pod="e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw"
Nov 03 19:44:08.979031 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:08.978994    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnj58\" (UniqueName: \"kubernetes.io/projected/72d96abc-f3b4-4eda-b3a1-b93ba658e1d5-kube-api-access-jnj58\") pod \"distinct-w1-a35658-kserve-54bbdfff9d-24klw\" (UID: \"72d96abc-f3b4-4eda-b3a1-b93ba658e1d5\") " pod="e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw"
Nov 03 19:44:08.979797 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:08.979737    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72d96abc-f3b4-4eda-b3a1-b93ba658e1d5-home\") pod \"distinct-w1-a35658-kserve-54bbdfff9d-24klw\" (UID: \"72d96abc-f3b4-4eda-b3a1-b93ba658e1d5\") " pod="e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw"
Nov 03 19:44:08.979979 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:08.979962    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72d96abc-f3b4-4eda-b3a1-b93ba658e1d5-model-cache\") pod \"distinct-w1-a35658-kserve-54bbdfff9d-24klw\" (UID: \"72d96abc-f3b4-4eda-b3a1-b93ba658e1d5\") " pod="e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw"
Nov 03 19:44:08.980169 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:08.980152    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72d96abc-f3b4-4eda-b3a1-b93ba658e1d5-tmp-dir\") pod \"distinct-w1-a35658-kserve-54bbdfff9d-24klw\" (UID: \"72d96abc-f3b4-4eda-b3a1-b93ba658e1d5\") " pod="e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw"
Nov 03 19:44:08.980342 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:08.980326    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72d96abc-f3b4-4eda-b3a1-b93ba658e1d5-dshm\") pod \"distinct-w1-a35658-kserve-54bbdfff9d-24klw\" (UID: \"72d96abc-f3b4-4eda-b3a1-b93ba658e1d5\") " pod="e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw"
Nov 03 19:44:09.093406 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:09.092307    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72d96abc-f3b4-4eda-b3a1-b93ba658e1d5-home\") pod \"distinct-w1-a35658-kserve-54bbdfff9d-24klw\" (UID: \"72d96abc-f3b4-4eda-b3a1-b93ba658e1d5\") " pod="e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw"
Nov 03 19:44:09.093406 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:09.092410    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72d96abc-f3b4-4eda-b3a1-b93ba658e1d5-model-cache\") pod \"distinct-w1-a35658-kserve-54bbdfff9d-24klw\" (UID: \"72d96abc-f3b4-4eda-b3a1-b93ba658e1d5\") " pod="e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw"
Nov 03 19:44:09.093406 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:09.092468    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72d96abc-f3b4-4eda-b3a1-b93ba658e1d5-tmp-dir\") pod \"distinct-w1-a35658-kserve-54bbdfff9d-24klw\" (UID: \"72d96abc-f3b4-4eda-b3a1-b93ba658e1d5\") " pod="e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw"
Nov 03 19:44:09.093406 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:09.092499    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72d96abc-f3b4-4eda-b3a1-b93ba658e1d5-dshm\") pod \"distinct-w1-a35658-kserve-54bbdfff9d-24klw\" (UID: \"72d96abc-f3b4-4eda-b3a1-b93ba658e1d5\") " pod="e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw"
Nov 03 19:44:09.093406 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:09.092608    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72d96abc-f3b4-4eda-b3a1-b93ba658e1d5-tls-certs\") pod \"distinct-w1-a35658-kserve-54bbdfff9d-24klw\" (UID: \"72d96abc-f3b4-4eda-b3a1-b93ba658e1d5\") " pod="e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw"
Nov 03 19:44:09.093406 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:09.092650    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnj58\" (UniqueName: \"kubernetes.io/projected/72d96abc-f3b4-4eda-b3a1-b93ba658e1d5-kube-api-access-jnj58\") pod \"distinct-w1-a35658-kserve-54bbdfff9d-24klw\" (UID: \"72d96abc-f3b4-4eda-b3a1-b93ba658e1d5\") " pod="e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw"
Nov 03 19:44:09.094887 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:09.094031    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72d96abc-f3b4-4eda-b3a1-b93ba658e1d5-home\") pod \"distinct-w1-a35658-kserve-54bbdfff9d-24klw\" (UID: \"72d96abc-f3b4-4eda-b3a1-b93ba658e1d5\") " pod="e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw"
Nov 03 19:44:09.094887 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:09.094305    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72d96abc-f3b4-4eda-b3a1-b93ba658e1d5-model-cache\") pod \"distinct-w1-a35658-kserve-54bbdfff9d-24klw\" (UID: \"72d96abc-f3b4-4eda-b3a1-b93ba658e1d5\") " pod="e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw"
Nov 03 19:44:09.095423 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:09.095377    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72d96abc-f3b4-4eda-b3a1-b93ba658e1d5-tmp-dir\") pod \"distinct-w1-a35658-kserve-54bbdfff9d-24klw\" (UID: \"72d96abc-f3b4-4eda-b3a1-b93ba658e1d5\") " pod="e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw"
Nov 03 19:44:09.113045 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:09.113016    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72d96abc-f3b4-4eda-b3a1-b93ba658e1d5-tls-certs\") pod \"distinct-w1-a35658-kserve-54bbdfff9d-24klw\" (UID: \"72d96abc-f3b4-4eda-b3a1-b93ba658e1d5\") " pod="e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw"
Nov 03 19:44:09.114511 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:09.114443    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnj58\" (UniqueName: \"kubernetes.io/projected/72d96abc-f3b4-4eda-b3a1-b93ba658e1d5-kube-api-access-jnj58\") pod \"distinct-w1-a35658-kserve-54bbdfff9d-24klw\" (UID: \"72d96abc-f3b4-4eda-b3a1-b93ba658e1d5\") " pod="e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw"
Nov 03 19:44:09.129754 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:09.125212    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72d96abc-f3b4-4eda-b3a1-b93ba658e1d5-dshm\") pod \"distinct-w1-a35658-kserve-54bbdfff9d-24klw\" (UID: \"72d96abc-f3b4-4eda-b3a1-b93ba658e1d5\") " pod="e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw"
Nov 03 19:44:09.204259 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:09.204212    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw"
Nov 03 19:44:09.536880 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:09.536847    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw"]
Nov 03 19:44:09.538318 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:44:09.538293    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod72d96abc_f3b4_4eda_b3a1_b93ba658e1d5.slice/crio-f9107159b42d8084e76657dd9c29c6802855422da97fbf19f6f97cb214c80ba7 WatchSource:0}: Error finding container f9107159b42d8084e76657dd9c29c6802855422da97fbf19f6f97cb214c80ba7: Status 404 returned error can't find the container with id f9107159b42d8084e76657dd9c29c6802855422da97fbf19f6f97cb214c80ba7
Nov 03 19:44:09.673446 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:09.673405    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw" event={"ID":"72d96abc-f3b4-4eda-b3a1-b93ba658e1d5","Type":"ContainerStarted","Data":"bb1cdc51d5b9f34156315417880d99fe919d20ab677f86d0c9adfcab2f00e631"}
Nov 03 19:44:09.673863 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:09.673460    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw" event={"ID":"72d96abc-f3b4-4eda-b3a1-b93ba658e1d5","Type":"ContainerStarted","Data":"f9107159b42d8084e76657dd9c29c6802855422da97fbf19f6f97cb214c80ba7"}
Nov 03 19:44:09.690572 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:09.690521    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw" podStartSLOduration=1.69050714 podStartE2EDuration="1.69050714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:44:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:44:09.686975018 +0000 UTC m=+762.930940517" watchObservedRunningTime="2026-11-03 19:44:09.69050714 +0000 UTC m=+762.934472646"
Nov 03 19:44:09.749334 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:09.749177    4921 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Nov 03 19:44:09.749334 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:09.749236    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p" podUID="97dc926b-00fc-4a42-9283-938cac033482" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Nov 03 19:44:09.817918 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:09.817890    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53ab2d2d-8v9d9"
Nov 03 19:44:09.902126 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:09.902094    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9b67cca-6629-4c9c-a69f-b4f0dabe2da3-kube-api-access-cxprh\" (UniqueName: \"kubernetes.io/projected/c9b67cca-6629-4c9c-a69f-b4f0dabe2da3-kube-api-access-cxprh\") pod \"c9b67cca-6629-4c9c-a69f-b4f0dabe2da3\" (UID: \"c9b67cca-6629-4c9c-a69f-b4f0dabe2da3\") "
Nov 03 19:44:09.902320 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:09.902161    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9b67cca-6629-4c9c-a69f-b4f0dabe2da3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9b67cca-6629-4c9c-a69f-b4f0dabe2da3-maas-api-openshift-ca\") pod \"c9b67cca-6629-4c9c-a69f-b4f0dabe2da3\" (UID: \"c9b67cca-6629-4c9c-a69f-b4f0dabe2da3\") "
Nov 03 19:44:09.902320 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:09.902254    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9b67cca-6629-4c9c-a69f-b4f0dabe2da3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9b67cca-6629-4c9c-a69f-b4f0dabe2da3-maas-api-xks-ca\") pod \"c9b67cca-6629-4c9c-a69f-b4f0dabe2da3\" (UID: \"c9b67cca-6629-4c9c-a69f-b4f0dabe2da3\") "
Nov 03 19:44:09.902610 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:09.902579    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9b67cca-6629-4c9c-a69f-b4f0dabe2da3-maas-api-openshift-ca" pod "c9b67cca-6629-4c9c-a69f-b4f0dabe2da3" (UID: "c9b67cca-6629-4c9c-a69f-b4f0dabe2da3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:44:09.904359 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:09.904337    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9b67cca-6629-4c9c-a69f-b4f0dabe2da3-kube-api-access-cxprh" pod "c9b67cca-6629-4c9c-a69f-b4f0dabe2da3" (UID: "c9b67cca-6629-4c9c-a69f-b4f0dabe2da3"). InnerVolumeSpecName "kube-api-access-cxprh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:44:09.904478 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:09.904457    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9b67cca-6629-4c9c-a69f-b4f0dabe2da3-maas-api-xks-ca" pod "c9b67cca-6629-4c9c-a69f-b4f0dabe2da3" (UID: "c9b67cca-6629-4c9c-a69f-b4f0dabe2da3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:44:10.003752 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:10.003715    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxprh\" (UniqueName: \"kubernetes.io/projected/c9b67cca-6629-4c9c-a69f-b4f0dabe2da3-kube-api-access-cxprh\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:44:10.003752 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:10.003748    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9b67cca-6629-4c9c-a69f-b4f0dabe2da3-maas-api-openshift-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:44:10.003752 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:10.003758    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9b67cca-6629-4c9c-a69f-b4f0dabe2da3-maas-api-xks-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:44:10.204801 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:10.204765    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw"
Nov 03 19:44:10.687700 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:10.686707    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53ab2d2d-8v9d9"
Nov 03 19:44:10.688206 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:10.688167    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53ab2d2d-8v9d9" event={"ID":"c9b67cca-6629-4c9c-a69f-b4f0dabe2da3","Type":"ContainerDied","Data":"db791b1c72009001ba83ba371d864a6515705509491129c6835e147eb0bc7969"}
Nov 03 19:44:10.688421 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:10.688403    4921 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db791b1c72009001ba83ba371d864a6515705509491129c6835e147eb0bc7969"
Nov 03 19:44:10.752097 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:10.752072    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p"
Nov 03 19:44:10.752430 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:10.752408    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p"
Nov 03 19:44:10.753266 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:10.753244    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p"
Nov 03 19:44:17.242126 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.242095    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7"]
Nov 03 19:44:17.243192 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.243167    4921 cpu_manager.go:414] "containerMap: removing container" podUID="c9b67cca-6629-4c9c-a69f-b4f0dabe2da3" containerName="revoke-keys"
Nov 03 19:44:17.243192 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.243193    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9b67cca-6629-4c9c-a69f-b4f0dabe2da3" containerName="revoke-keys"
Nov 03 19:44:17.243424 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.243381    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9b67cca-6629-4c9c-a69f-b4f0dabe2da3" containerName="revoke-keys"
Nov 03 19:44:17.248379 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.248351    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7"
Nov 03 19:44:17.258628 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.258602    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-8d4127\"/\"distinct-2-w2-8d4127-kserve-self-signed-certs\""
Nov 03 19:44:17.272014 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.271980    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7"]
Nov 03 19:44:17.293800 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.293763    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4de0090-d775-4bd7-9ee2-058b5d380638-model-cache\") pod \"distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7\" (UID: \"d4de0090-d775-4bd7-9ee2-058b5d380638\") " pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7"
Nov 03 19:44:17.294121 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.294096    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cmbh\" (UniqueName: \"kubernetes.io/projected/d4de0090-d775-4bd7-9ee2-058b5d380638-kube-api-access-2cmbh\") pod \"distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7\" (UID: \"d4de0090-d775-4bd7-9ee2-058b5d380638\") " pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7"
Nov 03 19:44:17.294473 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.294451    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4de0090-d775-4bd7-9ee2-058b5d380638-home\") pod \"distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7\" (UID: \"d4de0090-d775-4bd7-9ee2-058b5d380638\") " pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7"
Nov 03 19:44:17.294739 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.294720    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4de0090-d775-4bd7-9ee2-058b5d380638-tls-certs\") pod \"distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7\" (UID: \"d4de0090-d775-4bd7-9ee2-058b5d380638\") " pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7"
Nov 03 19:44:17.294948 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.294882    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4de0090-d775-4bd7-9ee2-058b5d380638-tmp-dir\") pod \"distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7\" (UID: \"d4de0090-d775-4bd7-9ee2-058b5d380638\") " pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7"
Nov 03 19:44:17.295140 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.295122    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4de0090-d775-4bd7-9ee2-058b5d380638-dshm\") pod \"distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7\" (UID: \"d4de0090-d775-4bd7-9ee2-058b5d380638\") " pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7"
Nov 03 19:44:17.376765 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.376716    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs"]
Nov 03 19:44:17.381795 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.381773    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs"
Nov 03 19:44:17.388383 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.388179    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-8d4127-openshift-default-dockercfg-6m7xm\""
Nov 03 19:44:17.400697 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.399556    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f838f4d0-1e63-4b98-8cfe-bd1b96db7162-istio-podinfo\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs\" (UID: \"f838f4d0-1e63-4b98-8cfe-bd1b96db7162\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs"
Nov 03 19:44:17.400697 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.399604    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f838f4d0-1e63-4b98-8cfe-bd1b96db7162-istio-data\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs\" (UID: \"f838f4d0-1e63-4b98-8cfe-bd1b96db7162\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs"
Nov 03 19:44:17.400697 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.399638    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cmbh\" (UniqueName: \"kubernetes.io/projected/d4de0090-d775-4bd7-9ee2-058b5d380638-kube-api-access-2cmbh\") pod \"distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7\" (UID: \"d4de0090-d775-4bd7-9ee2-058b5d380638\") " pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7"
Nov 03 19:44:17.400697 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.399687    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f838f4d0-1e63-4b98-8cfe-bd1b96db7162-istio-envoy\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs\" (UID: \"f838f4d0-1e63-4b98-8cfe-bd1b96db7162\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs"
Nov 03 19:44:17.400697 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.399755    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f838f4d0-1e63-4b98-8cfe-bd1b96db7162-credential-socket\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs\" (UID: \"f838f4d0-1e63-4b98-8cfe-bd1b96db7162\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs"
Nov 03 19:44:17.400697 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.399789    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4de0090-d775-4bd7-9ee2-058b5d380638-home\") pod \"distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7\" (UID: \"d4de0090-d775-4bd7-9ee2-058b5d380638\") " pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7"
Nov 03 19:44:17.400697 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.399840    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d27st\" (UniqueName: \"kubernetes.io/projected/f838f4d0-1e63-4b98-8cfe-bd1b96db7162-kube-api-access-d27st\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs\" (UID: \"f838f4d0-1e63-4b98-8cfe-bd1b96db7162\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs"
Nov 03 19:44:17.400697 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.399895    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4de0090-d775-4bd7-9ee2-058b5d380638-tls-certs\") pod \"distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7\" (UID: \"d4de0090-d775-4bd7-9ee2-058b5d380638\") " pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7"
Nov 03 19:44:17.400697 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.399937    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f838f4d0-1e63-4b98-8cfe-bd1b96db7162-istio-token\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs\" (UID: \"f838f4d0-1e63-4b98-8cfe-bd1b96db7162\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs"
Nov 03 19:44:17.400697 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.399966    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4de0090-d775-4bd7-9ee2-058b5d380638-tmp-dir\") pod \"distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7\" (UID: \"d4de0090-d775-4bd7-9ee2-058b5d380638\") " pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7"
Nov 03 19:44:17.400697 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.399999    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4de0090-d775-4bd7-9ee2-058b5d380638-dshm\") pod \"distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7\" (UID: \"d4de0090-d775-4bd7-9ee2-058b5d380638\") " pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7"
Nov 03 19:44:17.400697 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.400026    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f838f4d0-1e63-4b98-8cfe-bd1b96db7162-istiod-ca-cert\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs\" (UID: \"f838f4d0-1e63-4b98-8cfe-bd1b96db7162\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs"
Nov 03 19:44:17.400697 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.400071    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f838f4d0-1e63-4b98-8cfe-bd1b96db7162-workload-socket\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs\" (UID: \"f838f4d0-1e63-4b98-8cfe-bd1b96db7162\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs"
Nov 03 19:44:17.400697 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.400104    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f838f4d0-1e63-4b98-8cfe-bd1b96db7162-workload-certs\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs\" (UID: \"f838f4d0-1e63-4b98-8cfe-bd1b96db7162\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs"
Nov 03 19:44:17.400697 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.400148    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4de0090-d775-4bd7-9ee2-058b5d380638-model-cache\") pod \"distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7\" (UID: \"d4de0090-d775-4bd7-9ee2-058b5d380638\") " pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7"
Nov 03 19:44:17.400697 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.400565    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4de0090-d775-4bd7-9ee2-058b5d380638-model-cache\") pod \"distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7\" (UID: \"d4de0090-d775-4bd7-9ee2-058b5d380638\") " pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7"
Nov 03 19:44:17.401612 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.401189    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4de0090-d775-4bd7-9ee2-058b5d380638-home\") pod \"distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7\" (UID: \"d4de0090-d775-4bd7-9ee2-058b5d380638\") " pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7"
Nov 03 19:44:17.405370 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.405275    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4de0090-d775-4bd7-9ee2-058b5d380638-tmp-dir\") pod \"distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7\" (UID: \"d4de0090-d775-4bd7-9ee2-058b5d380638\") " pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7"
Nov 03 19:44:17.409064 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.409008    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4de0090-d775-4bd7-9ee2-058b5d380638-dshm\") pod \"distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7\" (UID: \"d4de0090-d775-4bd7-9ee2-058b5d380638\") " pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7"
Nov 03 19:44:17.410107 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.410025    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs"]
Nov 03 19:44:17.414614 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.414562    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4de0090-d775-4bd7-9ee2-058b5d380638-tls-certs\") pod \"distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7\" (UID: \"d4de0090-d775-4bd7-9ee2-058b5d380638\") " pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7"
Nov 03 19:44:17.479718 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.479598    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cmbh\" (UniqueName: \"kubernetes.io/projected/d4de0090-d775-4bd7-9ee2-058b5d380638-kube-api-access-2cmbh\") pod \"distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7\" (UID: \"d4de0090-d775-4bd7-9ee2-058b5d380638\") " pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7"
Nov 03 19:44:17.502452 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.502336    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f838f4d0-1e63-4b98-8cfe-bd1b96db7162-workload-socket\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs\" (UID: \"f838f4d0-1e63-4b98-8cfe-bd1b96db7162\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs"
Nov 03 19:44:17.502452 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.502406    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f838f4d0-1e63-4b98-8cfe-bd1b96db7162-workload-certs\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs\" (UID: \"f838f4d0-1e63-4b98-8cfe-bd1b96db7162\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs"
Nov 03 19:44:17.502681 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.502470    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f838f4d0-1e63-4b98-8cfe-bd1b96db7162-istio-podinfo\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs\" (UID: \"f838f4d0-1e63-4b98-8cfe-bd1b96db7162\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs"
Nov 03 19:44:17.502681 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.502494    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f838f4d0-1e63-4b98-8cfe-bd1b96db7162-istio-data\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs\" (UID: \"f838f4d0-1e63-4b98-8cfe-bd1b96db7162\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs"
Nov 03 19:44:17.502681 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.502545    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f838f4d0-1e63-4b98-8cfe-bd1b96db7162-istio-envoy\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs\" (UID: \"f838f4d0-1e63-4b98-8cfe-bd1b96db7162\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs"
Nov 03 19:44:17.502681 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.502608    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f838f4d0-1e63-4b98-8cfe-bd1b96db7162-credential-socket\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs\" (UID: \"f838f4d0-1e63-4b98-8cfe-bd1b96db7162\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs"
Nov 03 19:44:17.502899 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.502708    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d27st\" (UniqueName: \"kubernetes.io/projected/f838f4d0-1e63-4b98-8cfe-bd1b96db7162-kube-api-access-d27st\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs\" (UID: \"f838f4d0-1e63-4b98-8cfe-bd1b96db7162\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs"
Nov 03 19:44:17.502899 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.502798    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f838f4d0-1e63-4b98-8cfe-bd1b96db7162-istio-token\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs\" (UID: \"f838f4d0-1e63-4b98-8cfe-bd1b96db7162\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs"
Nov 03 19:44:17.502899 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.502864    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f838f4d0-1e63-4b98-8cfe-bd1b96db7162-istiod-ca-cert\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs\" (UID: \"f838f4d0-1e63-4b98-8cfe-bd1b96db7162\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs"
Nov 03 19:44:17.503905 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.503876    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f838f4d0-1e63-4b98-8cfe-bd1b96db7162-istiod-ca-cert\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs\" (UID: \"f838f4d0-1e63-4b98-8cfe-bd1b96db7162\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs"
Nov 03 19:44:17.504261 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.504234    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f838f4d0-1e63-4b98-8cfe-bd1b96db7162-workload-socket\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs\" (UID: \"f838f4d0-1e63-4b98-8cfe-bd1b96db7162\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs"
Nov 03 19:44:17.504547 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.504519    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f838f4d0-1e63-4b98-8cfe-bd1b96db7162-workload-certs\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs\" (UID: \"f838f4d0-1e63-4b98-8cfe-bd1b96db7162\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs"
Nov 03 19:44:17.505559 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.505023    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f838f4d0-1e63-4b98-8cfe-bd1b96db7162-credential-socket\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs\" (UID: \"f838f4d0-1e63-4b98-8cfe-bd1b96db7162\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs"
Nov 03 19:44:17.505559 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.505043    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f838f4d0-1e63-4b98-8cfe-bd1b96db7162-istio-data\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs\" (UID: \"f838f4d0-1e63-4b98-8cfe-bd1b96db7162\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs"
Nov 03 19:44:17.509997 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.509936    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f838f4d0-1e63-4b98-8cfe-bd1b96db7162-istio-podinfo\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs\" (UID: \"f838f4d0-1e63-4b98-8cfe-bd1b96db7162\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs"
Nov 03 19:44:17.531716 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.518965    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f838f4d0-1e63-4b98-8cfe-bd1b96db7162-istio-envoy\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs\" (UID: \"f838f4d0-1e63-4b98-8cfe-bd1b96db7162\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs"
Nov 03 19:44:17.531716 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.530762    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d27st\" (UniqueName: \"kubernetes.io/projected/f838f4d0-1e63-4b98-8cfe-bd1b96db7162-kube-api-access-d27st\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs\" (UID: \"f838f4d0-1e63-4b98-8cfe-bd1b96db7162\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs"
Nov 03 19:44:17.532458 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.532144    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f838f4d0-1e63-4b98-8cfe-bd1b96db7162-istio-token\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs\" (UID: \"f838f4d0-1e63-4b98-8cfe-bd1b96db7162\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs"
Nov 03 19:44:17.535244 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.535225    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45"]
Nov 03 19:44:17.540322 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.540305    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45"
Nov 03 19:44:17.544155 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.544133    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-a35658-openshift-default-dockercfg-m75kt\""
Nov 03 19:44:17.547741 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.547190    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45"]
Nov 03 19:44:17.568688 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.568189    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7"
Nov 03 19:44:17.603976 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.603942    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa3f8ec3-6ad5-4433-8dbb-71f8038dffce-workload-socket\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45\" (UID: \"fa3f8ec3-6ad5-4433-8dbb-71f8038dffce\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45"
Nov 03 19:44:17.604182 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.604161    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa3f8ec3-6ad5-4433-8dbb-71f8038dffce-credential-socket\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45\" (UID: \"fa3f8ec3-6ad5-4433-8dbb-71f8038dffce\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45"
Nov 03 19:44:17.604390 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.604370    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7h9q\" (UniqueName: \"kubernetes.io/projected/fa3f8ec3-6ad5-4433-8dbb-71f8038dffce-kube-api-access-h7h9q\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45\" (UID: \"fa3f8ec3-6ad5-4433-8dbb-71f8038dffce\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45"
Nov 03 19:44:17.604531 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.604514    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa3f8ec3-6ad5-4433-8dbb-71f8038dffce-istio-token\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45\" (UID: \"fa3f8ec3-6ad5-4433-8dbb-71f8038dffce\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45"
Nov 03 19:44:17.604758 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.604739    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa3f8ec3-6ad5-4433-8dbb-71f8038dffce-istiod-ca-cert\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45\" (UID: \"fa3f8ec3-6ad5-4433-8dbb-71f8038dffce\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45"
Nov 03 19:44:17.604881 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.604864    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa3f8ec3-6ad5-4433-8dbb-71f8038dffce-workload-certs\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45\" (UID: \"fa3f8ec3-6ad5-4433-8dbb-71f8038dffce\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45"
Nov 03 19:44:17.605015 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.604999    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa3f8ec3-6ad5-4433-8dbb-71f8038dffce-istio-data\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45\" (UID: \"fa3f8ec3-6ad5-4433-8dbb-71f8038dffce\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45"
Nov 03 19:44:17.605207 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.605170    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa3f8ec3-6ad5-4433-8dbb-71f8038dffce-istio-podinfo\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45\" (UID: \"fa3f8ec3-6ad5-4433-8dbb-71f8038dffce\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45"
Nov 03 19:44:17.605386 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.605368    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa3f8ec3-6ad5-4433-8dbb-71f8038dffce-istio-envoy\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45\" (UID: \"fa3f8ec3-6ad5-4433-8dbb-71f8038dffce\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45"
Nov 03 19:44:17.707323 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.706709    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa3f8ec3-6ad5-4433-8dbb-71f8038dffce-workload-socket\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45\" (UID: \"fa3f8ec3-6ad5-4433-8dbb-71f8038dffce\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45"
Nov 03 19:44:17.707521 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.707367    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa3f8ec3-6ad5-4433-8dbb-71f8038dffce-workload-socket\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45\" (UID: \"fa3f8ec3-6ad5-4433-8dbb-71f8038dffce\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45"
Nov 03 19:44:17.707866 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.707821    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa3f8ec3-6ad5-4433-8dbb-71f8038dffce-credential-socket\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45\" (UID: \"fa3f8ec3-6ad5-4433-8dbb-71f8038dffce\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45"
Nov 03 19:44:17.708240 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.708217    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs"
Nov 03 19:44:17.708506 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.707448    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa3f8ec3-6ad5-4433-8dbb-71f8038dffce-credential-socket\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45\" (UID: \"fa3f8ec3-6ad5-4433-8dbb-71f8038dffce\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45"
Nov 03 19:44:17.708589 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.708568    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7h9q\" (UniqueName: \"kubernetes.io/projected/fa3f8ec3-6ad5-4433-8dbb-71f8038dffce-kube-api-access-h7h9q\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45\" (UID: \"fa3f8ec3-6ad5-4433-8dbb-71f8038dffce\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45"
Nov 03 19:44:17.708659 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.708606    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa3f8ec3-6ad5-4433-8dbb-71f8038dffce-istio-token\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45\" (UID: \"fa3f8ec3-6ad5-4433-8dbb-71f8038dffce\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45"
Nov 03 19:44:17.709988 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.708656    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa3f8ec3-6ad5-4433-8dbb-71f8038dffce-istiod-ca-cert\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45\" (UID: \"fa3f8ec3-6ad5-4433-8dbb-71f8038dffce\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45"
Nov 03 19:44:17.709988 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.708696    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa3f8ec3-6ad5-4433-8dbb-71f8038dffce-workload-certs\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45\" (UID: \"fa3f8ec3-6ad5-4433-8dbb-71f8038dffce\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45"
Nov 03 19:44:17.709988 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.708753    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa3f8ec3-6ad5-4433-8dbb-71f8038dffce-istio-data\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45\" (UID: \"fa3f8ec3-6ad5-4433-8dbb-71f8038dffce\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45"
Nov 03 19:44:17.709988 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.708786    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa3f8ec3-6ad5-4433-8dbb-71f8038dffce-istio-podinfo\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45\" (UID: \"fa3f8ec3-6ad5-4433-8dbb-71f8038dffce\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45"
Nov 03 19:44:17.709988 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.708825    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa3f8ec3-6ad5-4433-8dbb-71f8038dffce-istio-envoy\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45\" (UID: \"fa3f8ec3-6ad5-4433-8dbb-71f8038dffce\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45"
Nov 03 19:44:17.709988 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.709302    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa3f8ec3-6ad5-4433-8dbb-71f8038dffce-workload-certs\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45\" (UID: \"fa3f8ec3-6ad5-4433-8dbb-71f8038dffce\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45"
Nov 03 19:44:17.709988 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.709598    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa3f8ec3-6ad5-4433-8dbb-71f8038dffce-istio-data\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45\" (UID: \"fa3f8ec3-6ad5-4433-8dbb-71f8038dffce\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45"
Nov 03 19:44:17.709988 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.709937    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa3f8ec3-6ad5-4433-8dbb-71f8038dffce-istiod-ca-cert\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45\" (UID: \"fa3f8ec3-6ad5-4433-8dbb-71f8038dffce\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45"
Nov 03 19:44:17.712559 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.712508    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa3f8ec3-6ad5-4433-8dbb-71f8038dffce-istio-envoy\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45\" (UID: \"fa3f8ec3-6ad5-4433-8dbb-71f8038dffce\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45"
Nov 03 19:44:17.713131 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.713106    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa3f8ec3-6ad5-4433-8dbb-71f8038dffce-istio-podinfo\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45\" (UID: \"fa3f8ec3-6ad5-4433-8dbb-71f8038dffce\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45"
Nov 03 19:44:17.717054 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.717022    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa3f8ec3-6ad5-4433-8dbb-71f8038dffce-istio-token\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45\" (UID: \"fa3f8ec3-6ad5-4433-8dbb-71f8038dffce\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45"
Nov 03 19:44:17.719710 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.719691    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7h9q\" (UniqueName: \"kubernetes.io/projected/fa3f8ec3-6ad5-4433-8dbb-71f8038dffce-kube-api-access-h7h9q\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45\" (UID: \"fa3f8ec3-6ad5-4433-8dbb-71f8038dffce\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45"
Nov 03 19:44:17.780294 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.780244    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt"]
Nov 03 19:44:17.799237 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.791180    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt"
Nov 03 19:44:17.799237 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.795692    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-f4e123-openshift-default-dockercfg-qlw2x\""
Nov 03 19:44:17.799237 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.799198    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt"]
Nov 03 19:44:17.814113 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.813542    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffb30cac-fb9f-4a3f-b954-ecd050b8cb41-istiod-ca-cert\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt\" (UID: \"ffb30cac-fb9f-4a3f-b954-ecd050b8cb41\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt"
Nov 03 19:44:17.814113 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.813589    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffb30cac-fb9f-4a3f-b954-ecd050b8cb41-istio-podinfo\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt\" (UID: \"ffb30cac-fb9f-4a3f-b954-ecd050b8cb41\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt"
Nov 03 19:44:17.814113 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.813625    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffb30cac-fb9f-4a3f-b954-ecd050b8cb41-istio-token\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt\" (UID: \"ffb30cac-fb9f-4a3f-b954-ecd050b8cb41\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt"
Nov 03 19:44:17.814113 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.813716    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqrmk\" (UniqueName: \"kubernetes.io/projected/ffb30cac-fb9f-4a3f-b954-ecd050b8cb41-kube-api-access-jqrmk\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt\" (UID: \"ffb30cac-fb9f-4a3f-b954-ecd050b8cb41\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt"
Nov 03 19:44:17.814113 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.813760    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffb30cac-fb9f-4a3f-b954-ecd050b8cb41-istio-data\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt\" (UID: \"ffb30cac-fb9f-4a3f-b954-ecd050b8cb41\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt"
Nov 03 19:44:17.814113 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.813796    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffb30cac-fb9f-4a3f-b954-ecd050b8cb41-credential-socket\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt\" (UID: \"ffb30cac-fb9f-4a3f-b954-ecd050b8cb41\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt"
Nov 03 19:44:17.814113 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.813822    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffb30cac-fb9f-4a3f-b954-ecd050b8cb41-workload-certs\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt\" (UID: \"ffb30cac-fb9f-4a3f-b954-ecd050b8cb41\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt"
Nov 03 19:44:17.814113 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.813850    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffb30cac-fb9f-4a3f-b954-ecd050b8cb41-workload-socket\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt\" (UID: \"ffb30cac-fb9f-4a3f-b954-ecd050b8cb41\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt"
Nov 03 19:44:17.814113 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.813882    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffb30cac-fb9f-4a3f-b954-ecd050b8cb41-istio-envoy\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt\" (UID: \"ffb30cac-fb9f-4a3f-b954-ecd050b8cb41\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt"
Nov 03 19:44:17.885951 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.885885    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7"]
Nov 03 19:44:17.886863 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:44:17.886836    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd4de0090_d775_4bd7_9ee2_058b5d380638.slice/crio-5921be12b2184bd512e5ba6130609749c433dde57d133108b416ed426199968e WatchSource:0}: Error finding container 5921be12b2184bd512e5ba6130609749c433dde57d133108b416ed426199968e: Status 404 returned error can't find the container with id 5921be12b2184bd512e5ba6130609749c433dde57d133108b416ed426199968e
Nov 03 19:44:17.902861 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.902840    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45"
Nov 03 19:44:17.914883 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.914859    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqrmk\" (UniqueName: \"kubernetes.io/projected/ffb30cac-fb9f-4a3f-b954-ecd050b8cb41-kube-api-access-jqrmk\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt\" (UID: \"ffb30cac-fb9f-4a3f-b954-ecd050b8cb41\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt"
Nov 03 19:44:17.915020 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.914898    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffb30cac-fb9f-4a3f-b954-ecd050b8cb41-istio-data\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt\" (UID: \"ffb30cac-fb9f-4a3f-b954-ecd050b8cb41\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt"
Nov 03 19:44:17.915020 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.914922    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffb30cac-fb9f-4a3f-b954-ecd050b8cb41-credential-socket\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt\" (UID: \"ffb30cac-fb9f-4a3f-b954-ecd050b8cb41\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt"
Nov 03 19:44:17.915020 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.914941    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffb30cac-fb9f-4a3f-b954-ecd050b8cb41-workload-certs\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt\" (UID: \"ffb30cac-fb9f-4a3f-b954-ecd050b8cb41\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt"
Nov 03 19:44:17.915020 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.914969    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffb30cac-fb9f-4a3f-b954-ecd050b8cb41-workload-socket\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt\" (UID: \"ffb30cac-fb9f-4a3f-b954-ecd050b8cb41\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt"
Nov 03 19:44:17.915020 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.914998    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffb30cac-fb9f-4a3f-b954-ecd050b8cb41-istio-envoy\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt\" (UID: \"ffb30cac-fb9f-4a3f-b954-ecd050b8cb41\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt"
Nov 03 19:44:17.915293 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.915120    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffb30cac-fb9f-4a3f-b954-ecd050b8cb41-istiod-ca-cert\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt\" (UID: \"ffb30cac-fb9f-4a3f-b954-ecd050b8cb41\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt"
Nov 03 19:44:17.915293 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.915151    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffb30cac-fb9f-4a3f-b954-ecd050b8cb41-istio-podinfo\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt\" (UID: \"ffb30cac-fb9f-4a3f-b954-ecd050b8cb41\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt"
Nov 03 19:44:17.915293 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.915184    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffb30cac-fb9f-4a3f-b954-ecd050b8cb41-istio-token\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt\" (UID: \"ffb30cac-fb9f-4a3f-b954-ecd050b8cb41\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt"
Nov 03 19:44:17.915293 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.915279    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffb30cac-fb9f-4a3f-b954-ecd050b8cb41-istio-data\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt\" (UID: \"ffb30cac-fb9f-4a3f-b954-ecd050b8cb41\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt"
Nov 03 19:44:17.915497 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.915327    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffb30cac-fb9f-4a3f-b954-ecd050b8cb41-credential-socket\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt\" (UID: \"ffb30cac-fb9f-4a3f-b954-ecd050b8cb41\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt"
Nov 03 19:44:17.915497 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.915376    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffb30cac-fb9f-4a3f-b954-ecd050b8cb41-workload-certs\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt\" (UID: \"ffb30cac-fb9f-4a3f-b954-ecd050b8cb41\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt"
Nov 03 19:44:17.915719 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.915694    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffb30cac-fb9f-4a3f-b954-ecd050b8cb41-workload-socket\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt\" (UID: \"ffb30cac-fb9f-4a3f-b954-ecd050b8cb41\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt"
Nov 03 19:44:17.916083 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.916055    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffb30cac-fb9f-4a3f-b954-ecd050b8cb41-istiod-ca-cert\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt\" (UID: \"ffb30cac-fb9f-4a3f-b954-ecd050b8cb41\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt"
Nov 03 19:44:17.917626 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.917606    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffb30cac-fb9f-4a3f-b954-ecd050b8cb41-istio-envoy\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt\" (UID: \"ffb30cac-fb9f-4a3f-b954-ecd050b8cb41\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt"
Nov 03 19:44:17.918138 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.918118    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffb30cac-fb9f-4a3f-b954-ecd050b8cb41-istio-podinfo\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt\" (UID: \"ffb30cac-fb9f-4a3f-b954-ecd050b8cb41\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt"
Nov 03 19:44:17.922561 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.922530    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqrmk\" (UniqueName: \"kubernetes.io/projected/ffb30cac-fb9f-4a3f-b954-ecd050b8cb41-kube-api-access-jqrmk\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt\" (UID: \"ffb30cac-fb9f-4a3f-b954-ecd050b8cb41\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt"
Nov 03 19:44:17.923168 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:17.923143    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffb30cac-fb9f-4a3f-b954-ecd050b8cb41-istio-token\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt\" (UID: \"ffb30cac-fb9f-4a3f-b954-ecd050b8cb41\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt"
Nov 03 19:44:18.012442 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:18.012415    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs"]
Nov 03 19:44:18.013438 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:44:18.013415    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf838f4d0_1e63_4b98_8cfe_bd1b96db7162.slice/crio-af2739dfc89639e6d5d4812608808752f05b046b0f7b2abf07f402f095f4ea9f WatchSource:0}: Error finding container af2739dfc89639e6d5d4812608808752f05b046b0f7b2abf07f402f095f4ea9f: Status 404 returned error can't find the container with id af2739dfc89639e6d5d4812608808752f05b046b0f7b2abf07f402f095f4ea9f
Nov 03 19:44:18.017692 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:18.017640    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:44:18.017772 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:18.017745    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:44:18.017806 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:18.017792    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:44:18.118784 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:18.118749    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt"
Nov 03 19:44:18.196374 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:18.191317    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45"]
Nov 03 19:44:18.196514 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:44:18.196413    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa3f8ec3_6ad5_4433_8dbb_71f8038dffce.slice/crio-719319ec07642acfe0818f8b9d12e80fff8101bf43843033e0bebe240ad45df2 WatchSource:0}: Error finding container 719319ec07642acfe0818f8b9d12e80fff8101bf43843033e0bebe240ad45df2: Status 404 returned error can't find the container with id 719319ec07642acfe0818f8b9d12e80fff8101bf43843033e0bebe240ad45df2
Nov 03 19:44:18.209483 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:18.208903    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:44:18.209483 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:18.208999    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:44:18.209483 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:18.209030    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:44:18.406951 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:18.406918    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt"]
Nov 03 19:44:18.408280 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:44:18.408252    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffb30cac_fb9f_4a3f_b954_ecd050b8cb41.slice/crio-71b35644987cc18d4a7056f19783b214f4626681eaa4fc9fdc91f36d1b3ca599 WatchSource:0}: Error finding container 71b35644987cc18d4a7056f19783b214f4626681eaa4fc9fdc91f36d1b3ca599: Status 404 returned error can't find the container with id 71b35644987cc18d4a7056f19783b214f4626681eaa4fc9fdc91f36d1b3ca599
Nov 03 19:44:18.412129 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:18.412087    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:44:18.412252 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:18.412188    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:44:18.412252 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:18.412232    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:44:18.741969 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:18.741930    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7" event={"ID":"d4de0090-d775-4bd7-9ee2-058b5d380638","Type":"ContainerStarted","Data":"ad85a1d6af803b96c22775daff77024d9fd0d0d6badd7d4993022769516c0e08"}
Nov 03 19:44:18.742149 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:18.741978    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7" event={"ID":"d4de0090-d775-4bd7-9ee2-058b5d380638","Type":"ContainerStarted","Data":"5921be12b2184bd512e5ba6130609749c433dde57d133108b416ed426199968e"}
Nov 03 19:44:18.744317 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:18.744218    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs" event={"ID":"f838f4d0-1e63-4b98-8cfe-bd1b96db7162","Type":"ContainerStarted","Data":"ac430cd619e6eac1717ec6dd03d93a6c78e1bb5104c473fb54212cbf72062599"}
Nov 03 19:44:18.744581 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:18.744546    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs" event={"ID":"f838f4d0-1e63-4b98-8cfe-bd1b96db7162","Type":"ContainerStarted","Data":"af2739dfc89639e6d5d4812608808752f05b046b0f7b2abf07f402f095f4ea9f"}
Nov 03 19:44:18.746632 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:18.746605    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45" event={"ID":"fa3f8ec3-6ad5-4433-8dbb-71f8038dffce","Type":"ContainerStarted","Data":"f15b475c18fb67278f8a04cec8e777f2c319a4f7942bb052509323f7c2ccce1a"}
Nov 03 19:44:18.746772 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:18.746654    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45" event={"ID":"fa3f8ec3-6ad5-4433-8dbb-71f8038dffce","Type":"ContainerStarted","Data":"719319ec07642acfe0818f8b9d12e80fff8101bf43843033e0bebe240ad45df2"}
Nov 03 19:44:18.748422 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:18.748401    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt" event={"ID":"ffb30cac-fb9f-4a3f-b954-ecd050b8cb41","Type":"ContainerStarted","Data":"8197b39e92a354ea25c97f74a91165cc703aa711b45383df2ee9f05f04a8dd79"}
Nov 03 19:44:18.748513 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:18.748440    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt" event={"ID":"ffb30cac-fb9f-4a3f-b954-ecd050b8cb41","Type":"ContainerStarted","Data":"71b35644987cc18d4a7056f19783b214f4626681eaa4fc9fdc91f36d1b3ca599"}
Nov 03 19:44:18.759261 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:18.759220    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7" podStartSLOduration=1.759208851 podStartE2EDuration="1.759208851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:44:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:44:18.756411059 +0000 UTC m=+772.000376525" watchObservedRunningTime="2026-11-03 19:44:18.759208851 +0000 UTC m=+772.003174316"
Nov 03 19:44:18.777033 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:18.776980    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs" podStartSLOduration=1.776962482 podStartE2EDuration="1.776962482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:44:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:44:18.775437038 +0000 UTC m=+772.019402516" watchObservedRunningTime="2026-11-03 19:44:18.776962482 +0000 UTC m=+772.020927950"
Nov 03 19:44:18.796956 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:18.796899    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45" podStartSLOduration=1.7968819150000002 podStartE2EDuration="1.796881915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:44:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:44:18.793660106 +0000 UTC m=+772.037625572" watchObservedRunningTime="2026-11-03 19:44:18.796881915 +0000 UTC m=+772.040847381"
Nov 03 19:44:18.809822 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:18.809775    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt" podStartSLOduration=1.809763725 podStartE2EDuration="1.809763725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:44:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:44:18.809320262 +0000 UTC m=+772.053285727" watchObservedRunningTime="2026-11-03 19:44:18.809763725 +0000 UTC m=+772.053729189"
Nov 03 19:44:18.903763 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:18.903725    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45"
Nov 03 19:44:19.121002 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:19.119717    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt"
Nov 03 19:44:19.121579 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:19.121551    4921 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Nov 03 19:44:19.121795 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:19.121767    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt" podUID="ffb30cac-fb9f-4a3f-b954-ecd050b8cb41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Nov 03 19:44:19.206986 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:19.206951    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw"
Nov 03 19:44:19.243814 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:19.243789    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw"
Nov 03 19:44:19.572822 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:19.572779    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7"
Nov 03 19:44:19.711773 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:19.711735    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs"
Nov 03 19:44:19.716805 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:19.716772    4921 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Nov 03 19:44:19.717070 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:19.717043    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs" podUID="f838f4d0-1e63-4b98-8cfe-bd1b96db7162" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Nov 03 19:44:19.910757 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:19.908876    4921 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 03 19:44:19.910757 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:19.908943    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45" podUID="fa3f8ec3-6ad5-4433-8dbb-71f8038dffce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 03 19:44:19.928941 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:19.928909    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw"
Nov 03 19:44:20.153693 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:20.153000    4921 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Nov 03 19:44:20.153693 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:20.153076    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt" podUID="ffb30cac-fb9f-4a3f-b954-ecd050b8cb41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Nov 03 19:44:20.711643 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:20.711605    4921 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Nov 03 19:44:20.712252 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:20.712221    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs" podUID="f838f4d0-1e63-4b98-8cfe-bd1b96db7162" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Nov 03 19:44:20.906123 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:20.906087    4921 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 03 19:44:20.906370 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:20.906342    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45" podUID="fa3f8ec3-6ad5-4433-8dbb-71f8038dffce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 03 19:44:21.201694 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:21.189520    4921 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Nov 03 19:44:21.204519 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:21.204479    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt" podUID="ffb30cac-fb9f-4a3f-b954-ecd050b8cb41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Nov 03 19:44:21.711189 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:21.711146    4921 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Nov 03 19:44:21.711485 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:21.711453    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs" podUID="f838f4d0-1e63-4b98-8cfe-bd1b96db7162" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Nov 03 19:44:21.916694 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:21.916048    4921 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 03 19:44:21.916694 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:21.916114    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45" podUID="fa3f8ec3-6ad5-4433-8dbb-71f8038dffce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 03 19:44:22.124862 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:22.121037    4921 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Nov 03 19:44:22.124862 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:22.121102    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt" podUID="ffb30cac-fb9f-4a3f-b954-ecd050b8cb41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Nov 03 19:44:22.709126 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:22.709085    4921 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Nov 03 19:44:22.709452 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:22.709408    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs" podUID="f838f4d0-1e63-4b98-8cfe-bd1b96db7162" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Nov 03 19:44:22.916305 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:22.916268    4921 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 03 19:44:22.916595 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:22.916562    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45" podUID="fa3f8ec3-6ad5-4433-8dbb-71f8038dffce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 03 19:44:23.130174 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:23.130132    4921 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Nov 03 19:44:23.130816 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:23.130787    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt" podUID="ffb30cac-fb9f-4a3f-b954-ecd050b8cb41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Nov 03 19:44:23.714691 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:23.709771    4921 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Nov 03 19:44:23.714691 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:23.709834    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs" podUID="f838f4d0-1e63-4b98-8cfe-bd1b96db7162" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Nov 03 19:44:23.937693 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:23.913070    4921 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 03 19:44:23.937693 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:23.913134    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45" podUID="fa3f8ec3-6ad5-4433-8dbb-71f8038dffce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 03 19:44:24.122693 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:24.122067    4921 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Nov 03 19:44:24.122693 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:24.122133    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt" podUID="ffb30cac-fb9f-4a3f-b954-ecd050b8cb41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Nov 03 19:44:24.713330 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:24.713290    4921 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Nov 03 19:44:24.714432 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:24.714399    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs" podUID="f838f4d0-1e63-4b98-8cfe-bd1b96db7162" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Nov 03 19:44:24.906493 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:24.906047    4921 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 03 19:44:24.906493 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:24.906116    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45" podUID="fa3f8ec3-6ad5-4433-8dbb-71f8038dffce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 03 19:44:25.120282 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:25.120242    4921 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Nov 03 19:44:25.120473 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:25.120305    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt" podUID="ffb30cac-fb9f-4a3f-b954-ecd050b8cb41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Nov 03 19:44:25.709391 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:25.709349    4921 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Nov 03 19:44:25.709626 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:25.709427    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs" podUID="f838f4d0-1e63-4b98-8cfe-bd1b96db7162" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Nov 03 19:44:25.903959 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:25.903918    4921 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Nov 03 19:44:25.904356 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:25.903999    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45" podUID="fa3f8ec3-6ad5-4433-8dbb-71f8038dffce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Nov 03 19:44:26.125048 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:26.125016    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt"
Nov 03 19:44:26.125565 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:26.125535    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt"
Nov 03 19:44:26.126379 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:26.126354    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt"
Nov 03 19:44:26.713610 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:26.713579    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs"
Nov 03 19:44:26.713936 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:26.713915    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs"
Nov 03 19:44:26.714582 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:26.714563    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs"
Nov 03 19:44:26.908092 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:26.908064    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45"
Nov 03 19:44:26.908452 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:26.908335    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45"
Nov 03 19:44:26.909065 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:26.909046    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45"
Nov 03 19:44:27.569100 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:27.569064    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7"
Nov 03 19:44:27.581853 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:27.581828    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7"
Nov 03 19:44:28.038966 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:28.038924    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7"
Nov 03 19:44:32.636092 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:32.636059    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf"]
Nov 03 19:44:32.751050 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:32.751014    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf"]
Nov 03 19:44:32.751244 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:32.751139    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf"
Nov 03 19:44:32.774849 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:32.773932    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-82dbbf-openshift-default-dockercfg-wz5bm\""
Nov 03 19:44:32.871125 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:32.871079    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e747102-94ba-49d0-a2b5-5ff6040c1556-istiod-ca-cert\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf\" (UID: \"9e747102-94ba-49d0-a2b5-5ff6040c1556\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf"
Nov 03 19:44:32.871960 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:32.871729    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e747102-94ba-49d0-a2b5-5ff6040c1556-istio-token\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf\" (UID: \"9e747102-94ba-49d0-a2b5-5ff6040c1556\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf"
Nov 03 19:44:32.872182 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:32.872160    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98klw\" (UniqueName: \"kubernetes.io/projected/9e747102-94ba-49d0-a2b5-5ff6040c1556-kube-api-access-98klw\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf\" (UID: \"9e747102-94ba-49d0-a2b5-5ff6040c1556\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf"
Nov 03 19:44:32.872702 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:32.872495    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e747102-94ba-49d0-a2b5-5ff6040c1556-workload-certs\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf\" (UID: \"9e747102-94ba-49d0-a2b5-5ff6040c1556\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf"
Nov 03 19:44:32.872878 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:32.872850    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e747102-94ba-49d0-a2b5-5ff6040c1556-istio-data\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf\" (UID: \"9e747102-94ba-49d0-a2b5-5ff6040c1556\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf"
Nov 03 19:44:32.873206 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:32.873180    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e747102-94ba-49d0-a2b5-5ff6040c1556-credential-socket\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf\" (UID: \"9e747102-94ba-49d0-a2b5-5ff6040c1556\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf"
Nov 03 19:44:32.873757 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:32.873730    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e747102-94ba-49d0-a2b5-5ff6040c1556-workload-socket\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf\" (UID: \"9e747102-94ba-49d0-a2b5-5ff6040c1556\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf"
Nov 03 19:44:32.874300 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:32.874280    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e747102-94ba-49d0-a2b5-5ff6040c1556-istio-envoy\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf\" (UID: \"9e747102-94ba-49d0-a2b5-5ff6040c1556\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf"
Nov 03 19:44:32.874633 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:32.874611    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e747102-94ba-49d0-a2b5-5ff6040c1556-istio-podinfo\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf\" (UID: \"9e747102-94ba-49d0-a2b5-5ff6040c1556\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf"
Nov 03 19:44:33.000692 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:32.998152    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e747102-94ba-49d0-a2b5-5ff6040c1556-istiod-ca-cert\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf\" (UID: \"9e747102-94ba-49d0-a2b5-5ff6040c1556\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf"
Nov 03 19:44:33.000692 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:32.998208    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e747102-94ba-49d0-a2b5-5ff6040c1556-istio-token\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf\" (UID: \"9e747102-94ba-49d0-a2b5-5ff6040c1556\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf"
Nov 03 19:44:33.000692 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:32.998240    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98klw\" (UniqueName: \"kubernetes.io/projected/9e747102-94ba-49d0-a2b5-5ff6040c1556-kube-api-access-98klw\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf\" (UID: \"9e747102-94ba-49d0-a2b5-5ff6040c1556\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf"
Nov 03 19:44:33.000692 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:32.998288    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e747102-94ba-49d0-a2b5-5ff6040c1556-workload-certs\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf\" (UID: \"9e747102-94ba-49d0-a2b5-5ff6040c1556\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf"
Nov 03 19:44:33.000692 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:32.998326    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e747102-94ba-49d0-a2b5-5ff6040c1556-istio-data\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf\" (UID: \"9e747102-94ba-49d0-a2b5-5ff6040c1556\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf"
Nov 03 19:44:33.000692 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:32.998366    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e747102-94ba-49d0-a2b5-5ff6040c1556-credential-socket\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf\" (UID: \"9e747102-94ba-49d0-a2b5-5ff6040c1556\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf"
Nov 03 19:44:33.000692 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:32.998430    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e747102-94ba-49d0-a2b5-5ff6040c1556-workload-socket\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf\" (UID: \"9e747102-94ba-49d0-a2b5-5ff6040c1556\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf"
Nov 03 19:44:33.000692 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:32.998514    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e747102-94ba-49d0-a2b5-5ff6040c1556-istio-envoy\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf\" (UID: \"9e747102-94ba-49d0-a2b5-5ff6040c1556\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf"
Nov 03 19:44:33.000692 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:32.998549    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e747102-94ba-49d0-a2b5-5ff6040c1556-istio-podinfo\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf\" (UID: \"9e747102-94ba-49d0-a2b5-5ff6040c1556\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf"
Nov 03 19:44:33.021693 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:33.015083    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e747102-94ba-49d0-a2b5-5ff6040c1556-workload-certs\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf\" (UID: \"9e747102-94ba-49d0-a2b5-5ff6040c1556\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf"
Nov 03 19:44:33.021693 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:33.015822    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e747102-94ba-49d0-a2b5-5ff6040c1556-istiod-ca-cert\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf\" (UID: \"9e747102-94ba-49d0-a2b5-5ff6040c1556\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf"
Nov 03 19:44:33.021693 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:33.016566    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e747102-94ba-49d0-a2b5-5ff6040c1556-credential-socket\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf\" (UID: \"9e747102-94ba-49d0-a2b5-5ff6040c1556\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf"
Nov 03 19:44:33.021693 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:33.016861    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e747102-94ba-49d0-a2b5-5ff6040c1556-istio-data\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf\" (UID: \"9e747102-94ba-49d0-a2b5-5ff6040c1556\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf"
Nov 03 19:44:33.021693 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:33.017117    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e747102-94ba-49d0-a2b5-5ff6040c1556-workload-socket\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf\" (UID: \"9e747102-94ba-49d0-a2b5-5ff6040c1556\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf"
Nov 03 19:44:33.091029 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:33.090993    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e747102-94ba-49d0-a2b5-5ff6040c1556-istio-envoy\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf\" (UID: \"9e747102-94ba-49d0-a2b5-5ff6040c1556\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf"
Nov 03 19:44:33.094770 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:33.094744    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr"]
Nov 03 19:44:33.123312 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:33.123283    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr"]
Nov 03 19:44:33.123621 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:33.123608    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr"
Nov 03 19:44:33.128590 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:33.128565    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-102379-openshift-default-dockercfg-2s5bm\""
Nov 03 19:44:33.135706 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:33.133498    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98klw\" (UniqueName: \"kubernetes.io/projected/9e747102-94ba-49d0-a2b5-5ff6040c1556-kube-api-access-98klw\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf\" (UID: \"9e747102-94ba-49d0-a2b5-5ff6040c1556\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf"
Nov 03 19:44:33.144049 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:44:33.143977    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 19:44:33.216189 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:33.216156    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e747102-94ba-49d0-a2b5-5ff6040c1556-istio-podinfo\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf\" (UID: \"9e747102-94ba-49d0-a2b5-5ff6040c1556\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf"
Nov 03 19:44:33.219239 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:33.219209    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e747102-94ba-49d0-a2b5-5ff6040c1556-istio-token\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf\" (UID: \"9e747102-94ba-49d0-a2b5-5ff6040c1556\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf"
Nov 03 19:44:33.225057 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:33.225027    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c05f083e-3052-44f9-ac8f-746ea890191b-workload-certs\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr\" (UID: \"c05f083e-3052-44f9-ac8f-746ea890191b\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr"
Nov 03 19:44:33.225254 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:33.225236    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c05f083e-3052-44f9-ac8f-746ea890191b-istio-token\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr\" (UID: \"c05f083e-3052-44f9-ac8f-746ea890191b\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr"
Nov 03 19:44:33.225401 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:33.225385    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c05f083e-3052-44f9-ac8f-746ea890191b-istio-podinfo\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr\" (UID: \"c05f083e-3052-44f9-ac8f-746ea890191b\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr"
Nov 03 19:44:33.225532 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:33.225516    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c05f083e-3052-44f9-ac8f-746ea890191b-istio-envoy\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr\" (UID: \"c05f083e-3052-44f9-ac8f-746ea890191b\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr"
Nov 03 19:44:33.229855 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:33.229829    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c05f083e-3052-44f9-ac8f-746ea890191b-workload-socket\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr\" (UID: \"c05f083e-3052-44f9-ac8f-746ea890191b\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr"
Nov 03 19:44:33.230024 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:33.230006    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c05f083e-3052-44f9-ac8f-746ea890191b-istio-data\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr\" (UID: \"c05f083e-3052-44f9-ac8f-746ea890191b\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr"
Nov 03 19:44:33.230264 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:33.230245    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsqqd\" (UniqueName: \"kubernetes.io/projected/c05f083e-3052-44f9-ac8f-746ea890191b-kube-api-access-lsqqd\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr\" (UID: \"c05f083e-3052-44f9-ac8f-746ea890191b\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr"
Nov 03 19:44:33.230465 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:33.230447    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c05f083e-3052-44f9-ac8f-746ea890191b-credential-socket\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr\" (UID: \"c05f083e-3052-44f9-ac8f-746ea890191b\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr"
Nov 03 19:44:33.230633 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:33.230617    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c05f083e-3052-44f9-ac8f-746ea890191b-istiod-ca-cert\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr\" (UID: \"c05f083e-3052-44f9-ac8f-746ea890191b\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr"
Nov 03 19:44:33.392464 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:33.392368    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsqqd\" (UniqueName: \"kubernetes.io/projected/c05f083e-3052-44f9-ac8f-746ea890191b-kube-api-access-lsqqd\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr\" (UID: \"c05f083e-3052-44f9-ac8f-746ea890191b\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr"
Nov 03 19:44:33.424740 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:33.424703    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsqqd\" (UniqueName: \"kubernetes.io/projected/c05f083e-3052-44f9-ac8f-746ea890191b-kube-api-access-lsqqd\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr\" (UID: \"c05f083e-3052-44f9-ac8f-746ea890191b\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr"
Nov 03 19:44:33.436702 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:33.433574    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c05f083e-3052-44f9-ac8f-746ea890191b-credential-socket\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr\" (UID: \"c05f083e-3052-44f9-ac8f-746ea890191b\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr"
Nov 03 19:44:33.436702 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:33.433692    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c05f083e-3052-44f9-ac8f-746ea890191b-istiod-ca-cert\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr\" (UID: \"c05f083e-3052-44f9-ac8f-746ea890191b\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr"
Nov 03 19:44:33.436702 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:33.433785    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c05f083e-3052-44f9-ac8f-746ea890191b-workload-certs\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr\" (UID: \"c05f083e-3052-44f9-ac8f-746ea890191b\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr"
Nov 03 19:44:33.436702 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:33.433823    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c05f083e-3052-44f9-ac8f-746ea890191b-istio-token\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr\" (UID: \"c05f083e-3052-44f9-ac8f-746ea890191b\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr"
Nov 03 19:44:33.436702 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:33.433873    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c05f083e-3052-44f9-ac8f-746ea890191b-istio-podinfo\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr\" (UID: \"c05f083e-3052-44f9-ac8f-746ea890191b\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr"
Nov 03 19:44:33.436702 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:33.433920    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c05f083e-3052-44f9-ac8f-746ea890191b-istio-envoy\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr\" (UID: \"c05f083e-3052-44f9-ac8f-746ea890191b\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr"
Nov 03 19:44:33.436702 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:33.433975    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c05f083e-3052-44f9-ac8f-746ea890191b-workload-socket\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr\" (UID: \"c05f083e-3052-44f9-ac8f-746ea890191b\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr"
Nov 03 19:44:33.436702 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:33.434008    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c05f083e-3052-44f9-ac8f-746ea890191b-istio-data\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr\" (UID: \"c05f083e-3052-44f9-ac8f-746ea890191b\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr"
Nov 03 19:44:33.436702 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:33.434494    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c05f083e-3052-44f9-ac8f-746ea890191b-istio-data\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr\" (UID: \"c05f083e-3052-44f9-ac8f-746ea890191b\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr"
Nov 03 19:44:33.436702 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:33.434782    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c05f083e-3052-44f9-ac8f-746ea890191b-credential-socket\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr\" (UID: \"c05f083e-3052-44f9-ac8f-746ea890191b\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr"
Nov 03 19:44:33.436702 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:33.435388    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c05f083e-3052-44f9-ac8f-746ea890191b-istiod-ca-cert\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr\" (UID: \"c05f083e-3052-44f9-ac8f-746ea890191b\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr"
Nov 03 19:44:33.436702 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:33.435625    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c05f083e-3052-44f9-ac8f-746ea890191b-workload-certs\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr\" (UID: \"c05f083e-3052-44f9-ac8f-746ea890191b\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr"
Nov 03 19:44:33.441143 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:33.440009    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c05f083e-3052-44f9-ac8f-746ea890191b-workload-socket\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr\" (UID: \"c05f083e-3052-44f9-ac8f-746ea890191b\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr"
Nov 03 19:44:33.446440 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:33.446387    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c05f083e-3052-44f9-ac8f-746ea890191b-istio-token\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr\" (UID: \"c05f083e-3052-44f9-ac8f-746ea890191b\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr"
Nov 03 19:44:33.446759 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:33.446738    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c05f083e-3052-44f9-ac8f-746ea890191b-istio-podinfo\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr\" (UID: \"c05f083e-3052-44f9-ac8f-746ea890191b\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr"
Nov 03 19:44:33.455698 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:33.451989    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c05f083e-3052-44f9-ac8f-746ea890191b-istio-envoy\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr\" (UID: \"c05f083e-3052-44f9-ac8f-746ea890191b\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr"
Nov 03 19:44:33.472249 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:33.472184    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf"
Nov 03 19:44:33.535404 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:33.535173    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr"
Nov 03 19:44:33.861985 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:33.860869    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf"]
Nov 03 19:44:33.869295 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:33.868750    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:44:33.869295 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:33.868849    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:44:33.869295 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:33.868896    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:44:34.029707 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:34.027331    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr"]
Nov 03 19:44:34.039272 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:34.039233    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:44:34.039380 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:34.039308    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:44:34.039380 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:34.039337    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:44:34.095878 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:34.093254    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr" event={"ID":"c05f083e-3052-44f9-ac8f-746ea890191b","Type":"ContainerStarted","Data":"eab8e0760f1f6435aabe5434457ea4132af5fe2a9007d7705dc0c86304276ea1"}
Nov 03 19:44:34.095878 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:34.095699    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf" event={"ID":"9e747102-94ba-49d0-a2b5-5ff6040c1556","Type":"ContainerStarted","Data":"61502eec9e6f9e0bc34eb6c038be35268c145fda46530d24978534d7916ae539"}
Nov 03 19:44:34.095878 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:34.095751    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf" event={"ID":"9e747102-94ba-49d0-a2b5-5ff6040c1556","Type":"ContainerStarted","Data":"f3048fee65ff651a366a75845e2d80c3a9e8953a9f191d33d30846be315215a6"}
Nov 03 19:44:34.119277 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:34.118971    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf" podStartSLOduration=2.11895348 podStartE2EDuration="2.11895348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:44:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:44:34.112199215 +0000 UTC m=+787.356164679" watchObservedRunningTime="2026-11-03 19:44:34.11895348 +0000 UTC m=+787.362918945"
Nov 03 19:44:34.476729 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:34.476608    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf"
Nov 03 19:44:34.484515 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:34.478461    4921 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Nov 03 19:44:34.484515 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:34.478525    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf" podUID="9e747102-94ba-49d0-a2b5-5ff6040c1556" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Nov 03 19:44:35.108477 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:35.108434    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr" event={"ID":"c05f083e-3052-44f9-ac8f-746ea890191b","Type":"ContainerStarted","Data":"4c1bcfe0668dc1be5482f113148b76ff1946205adf4e00c1b2427a7eb896fa57"}
Nov 03 19:44:35.125427 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:35.125373    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr" podStartSLOduration=2.125356432 podStartE2EDuration="2.125356432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:44:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:44:35.124462561 +0000 UTC m=+788.368428029" watchObservedRunningTime="2026-11-03 19:44:35.125356432 +0000 UTC m=+788.369321897"
Nov 03 19:44:35.486730 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:35.478093    4921 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Nov 03 19:44:35.486730 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:35.478156    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf" podUID="9e747102-94ba-49d0-a2b5-5ff6040c1556" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Nov 03 19:44:35.535742 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:35.535711    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr"
Nov 03 19:44:35.537483 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:35.537459    4921 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Nov 03 19:44:35.537632 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:35.537503    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr" podUID="c05f083e-3052-44f9-ac8f-746ea890191b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Nov 03 19:44:36.485693 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:36.476083    4921 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Nov 03 19:44:36.485693 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:36.476161    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf" podUID="9e747102-94ba-49d0-a2b5-5ff6040c1556" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Nov 03 19:44:36.536772 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:36.536738    4921 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Nov 03 19:44:36.536942 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:36.536791    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr" podUID="c05f083e-3052-44f9-ac8f-746ea890191b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Nov 03 19:44:37.478691 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:37.478066    4921 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Nov 03 19:44:37.478691 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:37.478177    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf" podUID="9e747102-94ba-49d0-a2b5-5ff6040c1556" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Nov 03 19:44:37.536824 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:37.536790    4921 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Nov 03 19:44:37.537270 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:37.536845    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr" podUID="c05f083e-3052-44f9-ac8f-746ea890191b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Nov 03 19:44:38.473156 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:38.473112    4921 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Nov 03 19:44:38.473462 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:38.473428    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf" podUID="9e747102-94ba-49d0-a2b5-5ff6040c1556" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Nov 03 19:44:38.536854 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:38.536818    4921 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Nov 03 19:44:38.537222 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:38.536875    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr" podUID="c05f083e-3052-44f9-ac8f-746ea890191b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Nov 03 19:44:39.482698 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:39.477899    4921 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Nov 03 19:44:39.482698 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:39.477958    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf" podUID="9e747102-94ba-49d0-a2b5-5ff6040c1556" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Nov 03 19:44:39.536646 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:39.536615    4921 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Nov 03 19:44:39.536826 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:39.536661    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr" podUID="c05f083e-3052-44f9-ac8f-746ea890191b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Nov 03 19:44:40.473123 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:40.473083    4921 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Nov 03 19:44:40.473524 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:40.473152    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf" podUID="9e747102-94ba-49d0-a2b5-5ff6040c1556" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Nov 03 19:44:40.536558 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:40.536517    4921 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Nov 03 19:44:40.536751 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:40.536577    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr" podUID="c05f083e-3052-44f9-ac8f-746ea890191b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Nov 03 19:44:41.479703 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:41.479637    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf"
Nov 03 19:44:41.480541 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:41.480514    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf"
Nov 03 19:44:41.481489 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:41.481467    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf"
Nov 03 19:44:41.547525 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:41.547495    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr"
Nov 03 19:44:41.548951 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:41.548929    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr"
Nov 03 19:44:41.553487 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:41.549799    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr"
Nov 03 19:44:57.777059 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:57.777026    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9"]
Nov 03 19:44:57.782729 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:57.782707    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9"
Nov 03 19:44:57.789561 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:57.789537    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9"]
Nov 03 19:44:57.790977 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:57.790951    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-102379\"/\"openshift-service-ca.crt\""
Nov 03 19:44:57.791112 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:57.791035    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-102379\"/\"kube-root-ca.crt\""
Nov 03 19:44:57.791443 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:57.791424    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-102379\"/\"default-dockercfg-cv6dk\""
Nov 03 19:44:57.791567 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:57.791526    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-102379\"/\"distinct-2-w6-102379-kserve-self-signed-certs\""
Nov 03 19:44:57.920717 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:57.905003    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fkrz\" (UniqueName: \"kubernetes.io/projected/e6a71aa3-33b3-42bd-b270-64ce3260cc2a-kube-api-access-4fkrz\") pod \"distinct-2-w6-102379-kserve-5595c84fbb-qtcb9\" (UID: \"e6a71aa3-33b3-42bd-b270-64ce3260cc2a\") " pod="e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9"
Nov 03 19:44:57.920717 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:57.905112    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6a71aa3-33b3-42bd-b270-64ce3260cc2a-dshm\") pod \"distinct-2-w6-102379-kserve-5595c84fbb-qtcb9\" (UID: \"e6a71aa3-33b3-42bd-b270-64ce3260cc2a\") " pod="e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9"
Nov 03 19:44:57.920717 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:57.905205    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6a71aa3-33b3-42bd-b270-64ce3260cc2a-model-cache\") pod \"distinct-2-w6-102379-kserve-5595c84fbb-qtcb9\" (UID: \"e6a71aa3-33b3-42bd-b270-64ce3260cc2a\") " pod="e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9"
Nov 03 19:44:57.920717 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:57.905249    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6a71aa3-33b3-42bd-b270-64ce3260cc2a-tmp-dir\") pod \"distinct-2-w6-102379-kserve-5595c84fbb-qtcb9\" (UID: \"e6a71aa3-33b3-42bd-b270-64ce3260cc2a\") " pod="e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9"
Nov 03 19:44:57.920717 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:57.905307    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6a71aa3-33b3-42bd-b270-64ce3260cc2a-home\") pod \"distinct-2-w6-102379-kserve-5595c84fbb-qtcb9\" (UID: \"e6a71aa3-33b3-42bd-b270-64ce3260cc2a\") " pod="e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9"
Nov 03 19:44:57.920717 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:57.905376    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6a71aa3-33b3-42bd-b270-64ce3260cc2a-tls-certs\") pod \"distinct-2-w6-102379-kserve-5595c84fbb-qtcb9\" (UID: \"e6a71aa3-33b3-42bd-b270-64ce3260cc2a\") " pod="e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9"
Nov 03 19:44:58.005996 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:58.005959    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6a71aa3-33b3-42bd-b270-64ce3260cc2a-model-cache\") pod \"distinct-2-w6-102379-kserve-5595c84fbb-qtcb9\" (UID: \"e6a71aa3-33b3-42bd-b270-64ce3260cc2a\") " pod="e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9"
Nov 03 19:44:58.005996 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:58.006002    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6a71aa3-33b3-42bd-b270-64ce3260cc2a-tmp-dir\") pod \"distinct-2-w6-102379-kserve-5595c84fbb-qtcb9\" (UID: \"e6a71aa3-33b3-42bd-b270-64ce3260cc2a\") " pod="e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9"
Nov 03 19:44:58.006264 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:58.006035    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6a71aa3-33b3-42bd-b270-64ce3260cc2a-home\") pod \"distinct-2-w6-102379-kserve-5595c84fbb-qtcb9\" (UID: \"e6a71aa3-33b3-42bd-b270-64ce3260cc2a\") " pod="e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9"
Nov 03 19:44:58.006264 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:58.006087    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6a71aa3-33b3-42bd-b270-64ce3260cc2a-tls-certs\") pod \"distinct-2-w6-102379-kserve-5595c84fbb-qtcb9\" (UID: \"e6a71aa3-33b3-42bd-b270-64ce3260cc2a\") " pod="e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9"
Nov 03 19:44:58.006264 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:58.006169    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fkrz\" (UniqueName: \"kubernetes.io/projected/e6a71aa3-33b3-42bd-b270-64ce3260cc2a-kube-api-access-4fkrz\") pod \"distinct-2-w6-102379-kserve-5595c84fbb-qtcb9\" (UID: \"e6a71aa3-33b3-42bd-b270-64ce3260cc2a\") " pod="e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9"
Nov 03 19:44:58.006264 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:58.006243    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6a71aa3-33b3-42bd-b270-64ce3260cc2a-dshm\") pod \"distinct-2-w6-102379-kserve-5595c84fbb-qtcb9\" (UID: \"e6a71aa3-33b3-42bd-b270-64ce3260cc2a\") " pod="e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9"
Nov 03 19:44:58.006482 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:58.006377    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6a71aa3-33b3-42bd-b270-64ce3260cc2a-tmp-dir\") pod \"distinct-2-w6-102379-kserve-5595c84fbb-qtcb9\" (UID: \"e6a71aa3-33b3-42bd-b270-64ce3260cc2a\") " pod="e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9"
Nov 03 19:44:58.006482 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:58.006438    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6a71aa3-33b3-42bd-b270-64ce3260cc2a-model-cache\") pod \"distinct-2-w6-102379-kserve-5595c84fbb-qtcb9\" (UID: \"e6a71aa3-33b3-42bd-b270-64ce3260cc2a\") " pod="e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9"
Nov 03 19:44:58.006592 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:58.006491    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6a71aa3-33b3-42bd-b270-64ce3260cc2a-home\") pod \"distinct-2-w6-102379-kserve-5595c84fbb-qtcb9\" (UID: \"e6a71aa3-33b3-42bd-b270-64ce3260cc2a\") " pod="e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9"
Nov 03 19:44:58.008751 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:58.008725    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6a71aa3-33b3-42bd-b270-64ce3260cc2a-dshm\") pod \"distinct-2-w6-102379-kserve-5595c84fbb-qtcb9\" (UID: \"e6a71aa3-33b3-42bd-b270-64ce3260cc2a\") " pod="e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9"
Nov 03 19:44:58.009177 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:58.009150    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6a71aa3-33b3-42bd-b270-64ce3260cc2a-tls-certs\") pod \"distinct-2-w6-102379-kserve-5595c84fbb-qtcb9\" (UID: \"e6a71aa3-33b3-42bd-b270-64ce3260cc2a\") " pod="e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9"
Nov 03 19:44:58.014236 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:58.014213    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fkrz\" (UniqueName: \"kubernetes.io/projected/e6a71aa3-33b3-42bd-b270-64ce3260cc2a-kube-api-access-4fkrz\") pod \"distinct-2-w6-102379-kserve-5595c84fbb-qtcb9\" (UID: \"e6a71aa3-33b3-42bd-b270-64ce3260cc2a\") " pod="e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9"
Nov 03 19:44:58.107471 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:58.107382    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9"
Nov 03 19:44:58.394465 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:58.394415    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9"]
Nov 03 19:44:58.395263 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:44:58.395234    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode6a71aa3_33b3_42bd_b270_64ce3260cc2a.slice/crio-92bd9705094ebe80f92542898488a34d7082ff55ccb75dc6e8940ea794b04cf8 WatchSource:0}: Error finding container 92bd9705094ebe80f92542898488a34d7082ff55ccb75dc6e8940ea794b04cf8: Status 404 returned error can't find the container with id 92bd9705094ebe80f92542898488a34d7082ff55ccb75dc6e8940ea794b04cf8
Nov 03 19:44:59.335908 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:59.335867    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9" event={"ID":"e6a71aa3-33b3-42bd-b270-64ce3260cc2a","Type":"ContainerStarted","Data":"0c4f0308f1137c1f5452829c6718700c7aada360b1c60edb67f1d52798b368d9"}
Nov 03 19:44:59.336326 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:59.335919    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9" event={"ID":"e6a71aa3-33b3-42bd-b270-64ce3260cc2a","Type":"ContainerStarted","Data":"92bd9705094ebe80f92542898488a34d7082ff55ccb75dc6e8940ea794b04cf8"}
Nov 03 19:44:59.352093 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:44:59.352043    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9" podStartSLOduration=2.352027108 podStartE2EDuration="2.352027108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:44:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:44:59.35122287 +0000 UTC m=+812.595188337" watchObservedRunningTime="2026-11-03 19:44:59.352027108 +0000 UTC m=+812.595992574"
Nov 03 19:45:00.108285 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:00.108250    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9"
Nov 03 19:45:00.141831 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:00.141799    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895585-b8tdx"]
Nov 03 19:45:00.145998 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:00.145975    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d4127-29895585-lwd9p"]
Nov 03 19:45:00.146143 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:00.146088    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-b8tdx"
Nov 03 19:45:00.150111 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:00.150088    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d4127-29895585-lwd9p"
Nov 03 19:45:00.153825 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:00.153799    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 19:45:00.156224 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:00.156179    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895585-b8tdx"]
Nov 03 19:45:00.166092 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:00.166066    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d4127-29895585-lwd9p"]
Nov 03 19:45:00.171507 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:00.171482    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0486e8-29895585-252sx"]
Nov 03 19:45:00.176455 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:00.176435    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0486e8-29895585-252sx"
Nov 03 19:45:00.185741 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:00.184864    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0486e8-29895585-252sx"]
Nov 03 19:45:00.231255 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:00.231218    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45523a35-8108-4813-b500-d0803ce6cc9c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895585-b8tdx\" (UID: \"45523a35-8108-4813-b500-d0803ce6cc9c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-b8tdx"
Nov 03 19:45:00.231255 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:00.231258    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jv4b\" (UniqueName: \"kubernetes.io/projected/45523a35-8108-4813-b500-d0803ce6cc9c-kube-api-access-8jv4b\") pod \"maas-api-key-cleanup-29895585-b8tdx\" (UID: \"45523a35-8108-4813-b500-d0803ce6cc9c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-b8tdx"
Nov 03 19:45:00.231455 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:00.231294    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vns6\" (UniqueName: \"kubernetes.io/projected/e2aa77e1-ea3d-4db2-84b6-da15cf42d047-kube-api-access-8vns6\") pod \"maas-api-key-cleanup-e2e-worker-w2-8d4127-29895585-lwd9p\" (UID: \"e2aa77e1-ea3d-4db2-84b6-da15cf42d047\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d4127-29895585-lwd9p"
Nov 03 19:45:00.231455 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:00.231426    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2aa77e1-ea3d-4db2-84b6-da15cf42d047-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-8d4127-29895585-lwd9p\" (UID: \"e2aa77e1-ea3d-4db2-84b6-da15cf42d047\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d4127-29895585-lwd9p"
Nov 03 19:45:00.333075 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:00.333039    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45523a35-8108-4813-b500-d0803ce6cc9c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895585-b8tdx\" (UID: \"45523a35-8108-4813-b500-d0803ce6cc9c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-b8tdx"
Nov 03 19:45:00.333075 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:00.333080    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jv4b\" (UniqueName: \"kubernetes.io/projected/45523a35-8108-4813-b500-d0803ce6cc9c-kube-api-access-8jv4b\") pod \"maas-api-key-cleanup-29895585-b8tdx\" (UID: \"45523a35-8108-4813-b500-d0803ce6cc9c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-b8tdx"
Nov 03 19:45:00.333292 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:00.333206    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vns6\" (UniqueName: \"kubernetes.io/projected/e2aa77e1-ea3d-4db2-84b6-da15cf42d047-kube-api-access-8vns6\") pod \"maas-api-key-cleanup-e2e-worker-w2-8d4127-29895585-lwd9p\" (UID: \"e2aa77e1-ea3d-4db2-84b6-da15cf42d047\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d4127-29895585-lwd9p"
Nov 03 19:45:00.333359 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:00.333341    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2aa77e1-ea3d-4db2-84b6-da15cf42d047-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-8d4127-29895585-lwd9p\" (UID: \"e2aa77e1-ea3d-4db2-84b6-da15cf42d047\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d4127-29895585-lwd9p"
Nov 03 19:45:00.333404 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:00.333383    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzkvd\" (UniqueName: \"kubernetes.io/projected/980862b5-9e6d-42a6-b7f9-141205136ef8-kube-api-access-hzkvd\") pod \"maas-api-key-cleanup-e2e-worker-w5-0486e8-29895585-252sx\" (UID: \"980862b5-9e6d-42a6-b7f9-141205136ef8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0486e8-29895585-252sx"
Nov 03 19:45:00.333452 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:00.333437    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/980862b5-9e6d-42a6-b7f9-141205136ef8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-0486e8-29895585-252sx\" (UID: \"980862b5-9e6d-42a6-b7f9-141205136ef8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0486e8-29895585-252sx"
Nov 03 19:45:00.333814 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:00.333798    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45523a35-8108-4813-b500-d0803ce6cc9c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895585-b8tdx\" (UID: \"45523a35-8108-4813-b500-d0803ce6cc9c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-b8tdx"
Nov 03 19:45:00.333945 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:00.333929    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2aa77e1-ea3d-4db2-84b6-da15cf42d047-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-8d4127-29895585-lwd9p\" (UID: \"e2aa77e1-ea3d-4db2-84b6-da15cf42d047\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d4127-29895585-lwd9p"
Nov 03 19:45:00.340268 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:00.340248    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jv4b\" (UniqueName: \"kubernetes.io/projected/45523a35-8108-4813-b500-d0803ce6cc9c-kube-api-access-8jv4b\") pod \"maas-api-key-cleanup-29895585-b8tdx\" (UID: \"45523a35-8108-4813-b500-d0803ce6cc9c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-b8tdx"
Nov 03 19:45:00.340589 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:00.340398    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vns6\" (UniqueName: \"kubernetes.io/projected/e2aa77e1-ea3d-4db2-84b6-da15cf42d047-kube-api-access-8vns6\") pod \"maas-api-key-cleanup-e2e-worker-w2-8d4127-29895585-lwd9p\" (UID: \"e2aa77e1-ea3d-4db2-84b6-da15cf42d047\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d4127-29895585-lwd9p"
Nov 03 19:45:00.435123 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:00.435044    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzkvd\" (UniqueName: \"kubernetes.io/projected/980862b5-9e6d-42a6-b7f9-141205136ef8-kube-api-access-hzkvd\") pod \"maas-api-key-cleanup-e2e-worker-w5-0486e8-29895585-252sx\" (UID: \"980862b5-9e6d-42a6-b7f9-141205136ef8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0486e8-29895585-252sx"
Nov 03 19:45:00.435123 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:00.435102    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/980862b5-9e6d-42a6-b7f9-141205136ef8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-0486e8-29895585-252sx\" (UID: \"980862b5-9e6d-42a6-b7f9-141205136ef8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0486e8-29895585-252sx"
Nov 03 19:45:00.435801 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:00.435782    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/980862b5-9e6d-42a6-b7f9-141205136ef8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-0486e8-29895585-252sx\" (UID: \"980862b5-9e6d-42a6-b7f9-141205136ef8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0486e8-29895585-252sx"
Nov 03 19:45:00.442421 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:00.442384    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzkvd\" (UniqueName: \"kubernetes.io/projected/980862b5-9e6d-42a6-b7f9-141205136ef8-kube-api-access-hzkvd\") pod \"maas-api-key-cleanup-e2e-worker-w5-0486e8-29895585-252sx\" (UID: \"980862b5-9e6d-42a6-b7f9-141205136ef8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0486e8-29895585-252sx"
Nov 03 19:45:00.475382 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:00.475349    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-b8tdx"
Nov 03 19:45:00.504634 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:00.504601    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d4127-29895585-lwd9p"
Nov 03 19:45:00.510022 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:00.509997    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0486e8-29895585-252sx"
Nov 03 19:45:00.793334 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:00.793306    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895585-b8tdx"]
Nov 03 19:45:00.795978 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:45:00.795953    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45523a35_8108_4813_b500_d0803ce6cc9c.slice/crio-0127599a3c9c39c1d22b8222cb739bc3d0d7f3467b210dc86b317d37fd5f12bf WatchSource:0}: Error finding container 0127599a3c9c39c1d22b8222cb739bc3d0d7f3467b210dc86b317d37fd5f12bf: Status 404 returned error can't find the container with id 0127599a3c9c39c1d22b8222cb739bc3d0d7f3467b210dc86b317d37fd5f12bf
Nov 03 19:45:00.856030 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:00.855262    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d4127-29895585-lwd9p"]
Nov 03 19:45:00.860560 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:45:00.860514    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2aa77e1_ea3d_4db2_84b6_da15cf42d047.slice/crio-4533638d961e31842ef7804cb9b65151028521c850d9638780f545cf23386011 WatchSource:0}: Error finding container 4533638d961e31842ef7804cb9b65151028521c850d9638780f545cf23386011: Status 404 returned error can't find the container with id 4533638d961e31842ef7804cb9b65151028521c850d9638780f545cf23386011
Nov 03 19:45:00.862557 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:45:00.862528    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod980862b5_9e6d_42a6_b7f9_141205136ef8.slice/crio-4aa3308e8982077dddd3a602a18358fe2ea4af900390aeaff2774b66cf7cf5c8 WatchSource:0}: Error finding container 4aa3308e8982077dddd3a602a18358fe2ea4af900390aeaff2774b66cf7cf5c8: Status 404 returned error can't find the container with id 4aa3308e8982077dddd3a602a18358fe2ea4af900390aeaff2774b66cf7cf5c8
Nov 03 19:45:00.862660 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:00.862618    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0486e8-29895585-252sx"]
Nov 03 19:45:01.381964 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:01.381863    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-b8tdx" event={"ID":"45523a35-8108-4813-b500-d0803ce6cc9c","Type":"ContainerStarted","Data":"e868b3680040dfe6a113aace37dc282520c7b8d0891633aa1537a6c68b5d09f8"}
Nov 03 19:45:01.381964 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:01.381927    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-b8tdx" event={"ID":"45523a35-8108-4813-b500-d0803ce6cc9c","Type":"ContainerStarted","Data":"0127599a3c9c39c1d22b8222cb739bc3d0d7f3467b210dc86b317d37fd5f12bf"}
Nov 03 19:45:01.384763 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:01.384733    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0486e8-29895585-252sx" event={"ID":"980862b5-9e6d-42a6-b7f9-141205136ef8","Type":"ContainerStarted","Data":"378599bcd17f82cf0d5e57332d925c448199b57252685a33514edc120924a826"}
Nov 03 19:45:01.384920 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:01.384788    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0486e8-29895585-252sx" event={"ID":"980862b5-9e6d-42a6-b7f9-141205136ef8","Type":"ContainerStarted","Data":"4aa3308e8982077dddd3a602a18358fe2ea4af900390aeaff2774b66cf7cf5c8"}
Nov 03 19:45:01.390158 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:01.386997    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d4127-29895585-lwd9p" event={"ID":"e2aa77e1-ea3d-4db2-84b6-da15cf42d047","Type":"ContainerStarted","Data":"c2d2ed3752124906a209acef66c198596c173feb202b7f82f1c1f0ba8bbc036c"}
Nov 03 19:45:01.390158 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:01.387057    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d4127-29895585-lwd9p" event={"ID":"e2aa77e1-ea3d-4db2-84b6-da15cf42d047","Type":"ContainerStarted","Data":"4533638d961e31842ef7804cb9b65151028521c850d9638780f545cf23386011"}
Nov 03 19:45:01.399781 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:01.399692    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-b8tdx" podStartSLOduration=1.399628581 podStartE2EDuration="1.399628581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:01.395915416 +0000 UTC m=+814.639880882" watchObservedRunningTime="2026-11-03 19:45:01.399628581 +0000 UTC m=+814.643594048"
Nov 03 19:45:01.409376 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:01.409315    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0486e8-29895585-252sx" podStartSLOduration=1.409293246 podStartE2EDuration="1.409293246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:01.408358033 +0000 UTC m=+814.652323497" watchObservedRunningTime="2026-11-03 19:45:01.409293246 +0000 UTC m=+814.653258712"
Nov 03 19:45:01.422868 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:01.422814    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d4127-29895585-lwd9p" podStartSLOduration=1.422799771 podStartE2EDuration="1.422799771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:01.421758167 +0000 UTC m=+814.665723633" watchObservedRunningTime="2026-11-03 19:45:01.422799771 +0000 UTC m=+814.666765230"
Nov 03 19:45:02.396723 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:02.396593    4921 generic.go:356] "Generic (PLEG): container finished" podID="45523a35-8108-4813-b500-d0803ce6cc9c" containerID="e868b3680040dfe6a113aace37dc282520c7b8d0891633aa1537a6c68b5d09f8" exitCode=60
Nov 03 19:45:02.397172 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:02.396708    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-b8tdx" event={"ID":"45523a35-8108-4813-b500-d0803ce6cc9c","Type":"ContainerDied","Data":"e868b3680040dfe6a113aace37dc282520c7b8d0891633aa1537a6c68b5d09f8"}
Nov 03 19:45:02.397172 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:02.396982    4921 scope.go:122] "RemoveContainer" containerID="e868b3680040dfe6a113aace37dc282520c7b8d0891633aa1537a6c68b5d09f8"
Nov 03 19:45:02.398879 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:02.398858    4921 generic.go:356] "Generic (PLEG): container finished" podID="980862b5-9e6d-42a6-b7f9-141205136ef8" containerID="378599bcd17f82cf0d5e57332d925c448199b57252685a33514edc120924a826" exitCode=60
Nov 03 19:45:02.398952 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:02.398939    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0486e8-29895585-252sx" event={"ID":"980862b5-9e6d-42a6-b7f9-141205136ef8","Type":"ContainerDied","Data":"378599bcd17f82cf0d5e57332d925c448199b57252685a33514edc120924a826"}
Nov 03 19:45:02.399542 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:02.399219    4921 scope.go:122] "RemoveContainer" containerID="378599bcd17f82cf0d5e57332d925c448199b57252685a33514edc120924a826"
Nov 03 19:45:02.401142 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:02.401120    4921 generic.go:356] "Generic (PLEG): container finished" podID="e2aa77e1-ea3d-4db2-84b6-da15cf42d047" containerID="c2d2ed3752124906a209acef66c198596c173feb202b7f82f1c1f0ba8bbc036c" exitCode=60
Nov 03 19:45:02.401289 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:02.401271    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d4127-29895585-lwd9p" event={"ID":"e2aa77e1-ea3d-4db2-84b6-da15cf42d047","Type":"ContainerDied","Data":"c2d2ed3752124906a209acef66c198596c173feb202b7f82f1c1f0ba8bbc036c"}
Nov 03 19:45:02.401437 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:02.401417    4921 scope.go:122] "RemoveContainer" containerID="c2d2ed3752124906a209acef66c198596c173feb202b7f82f1c1f0ba8bbc036c"
Nov 03 19:45:02.507302 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:02.507262    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5"]
Nov 03 19:45:02.514252 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:02.514217    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5"
Nov 03 19:45:02.533035 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:02.533003    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5"]
Nov 03 19:45:02.539517 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:02.539238    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6"]
Nov 03 19:45:02.544429 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:02.544226    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6"
Nov 03 19:45:02.561731 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:02.560754    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6"]
Nov 03 19:45:02.676343 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:02.676303    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e96267a-d738-4325-90e7-d1c9ce87eeb5-istio-podinfo\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6\" (UID: \"7e96267a-d738-4325-90e7-d1c9ce87eeb5\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6"
Nov 03 19:45:02.676549 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:02.676354    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0da622a1-45f9-4311-b668-d27eb7ec2e43-istiod-ca-cert\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5\" (UID: \"0da622a1-45f9-4311-b668-d27eb7ec2e43\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5"
Nov 03 19:45:02.676549 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:02.676388    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e96267a-d738-4325-90e7-d1c9ce87eeb5-istio-envoy\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6\" (UID: \"7e96267a-d738-4325-90e7-d1c9ce87eeb5\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6"
Nov 03 19:45:02.676549 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:02.676414    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5r7m\" (UniqueName: \"kubernetes.io/projected/0da622a1-45f9-4311-b668-d27eb7ec2e43-kube-api-access-z5r7m\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5\" (UID: \"0da622a1-45f9-4311-b668-d27eb7ec2e43\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5"
Nov 03 19:45:02.676549 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:02.676463    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e96267a-d738-4325-90e7-d1c9ce87eeb5-istio-data\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6\" (UID: \"7e96267a-d738-4325-90e7-d1c9ce87eeb5\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6"
Nov 03 19:45:02.676549 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:02.676501    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e96267a-d738-4325-90e7-d1c9ce87eeb5-credential-socket\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6\" (UID: \"7e96267a-d738-4325-90e7-d1c9ce87eeb5\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6"
Nov 03 19:45:02.676840 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:02.676620    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0da622a1-45f9-4311-b668-d27eb7ec2e43-istio-podinfo\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5\" (UID: \"0da622a1-45f9-4311-b668-d27eb7ec2e43\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5"
Nov 03 19:45:02.676840 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:02.676728    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e96267a-d738-4325-90e7-d1c9ce87eeb5-istio-token\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6\" (UID: \"7e96267a-d738-4325-90e7-d1c9ce87eeb5\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6"
Nov 03 19:45:02.676840 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:02.676760    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da622a1-45f9-4311-b668-d27eb7ec2e43-credential-socket\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5\" (UID: \"0da622a1-45f9-4311-b668-d27eb7ec2e43\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5"
Nov 03 19:45:02.676840 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:02.676811    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da622a1-45f9-4311-b668-d27eb7ec2e43-workload-socket\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5\" (UID: \"0da622a1-45f9-4311-b668-d27eb7ec2e43\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5"
Nov 03 19:45:02.677041 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:02.676839    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e96267a-d738-4325-90e7-d1c9ce87eeb5-workload-socket\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6\" (UID: \"7e96267a-d738-4325-90e7-d1c9ce87eeb5\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6"
Nov 03 19:45:02.677041 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:02.676863    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0da622a1-45f9-4311-b668-d27eb7ec2e43-istio-token\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5\" (UID: \"0da622a1-45f9-4311-b668-d27eb7ec2e43\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5"
Nov 03 19:45:02.677041 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:02.676914    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0da622a1-45f9-4311-b668-d27eb7ec2e43-workload-certs\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5\" (UID: \"0da622a1-45f9-4311-b668-d27eb7ec2e43\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5"
Nov 03 19:45:02.677041 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:02.676936    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0da622a1-45f9-4311-b668-d27eb7ec2e43-istio-envoy\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5\" (UID: \"0da622a1-45f9-4311-b668-d27eb7ec2e43\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5"
Nov 03 19:45:02.677041 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:02.676951    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0da622a1-45f9-4311-b668-d27eb7ec2e43-istio-data\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5\" (UID: \"0da622a1-45f9-4311-b668-d27eb7ec2e43\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5"
Nov 03 19:45:02.677041 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:02.676987    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e96267a-d738-4325-90e7-d1c9ce87eeb5-istiod-ca-cert\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6\" (UID: \"7e96267a-d738-4325-90e7-d1c9ce87eeb5\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6"
Nov 03 19:45:02.677041 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:02.677006    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frd24\" (UniqueName: \"kubernetes.io/projected/7e96267a-d738-4325-90e7-d1c9ce87eeb5-kube-api-access-frd24\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6\" (UID: \"7e96267a-d738-4325-90e7-d1c9ce87eeb5\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6"
Nov 03 19:45:02.677041 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:02.677030    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e96267a-d738-4325-90e7-d1c9ce87eeb5-workload-certs\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6\" (UID: \"7e96267a-d738-4325-90e7-d1c9ce87eeb5\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6"
Nov 03 19:45:02.779081 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:02.778340    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e96267a-d738-4325-90e7-d1c9ce87eeb5-istiod-ca-cert\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6\" (UID: \"7e96267a-d738-4325-90e7-d1c9ce87eeb5\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6"
Nov 03 19:45:02.779081 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:02.778388    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frd24\" (UniqueName: \"kubernetes.io/projected/7e96267a-d738-4325-90e7-d1c9ce87eeb5-kube-api-access-frd24\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6\" (UID: \"7e96267a-d738-4325-90e7-d1c9ce87eeb5\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6"
Nov 03 19:45:02.779081 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:02.778786    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e96267a-d738-4325-90e7-d1c9ce87eeb5-workload-certs\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6\" (UID: \"7e96267a-d738-4325-90e7-d1c9ce87eeb5\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6"
Nov 03 19:45:02.779081 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:02.778861    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e96267a-d738-4325-90e7-d1c9ce87eeb5-istio-podinfo\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6\" (UID: \"7e96267a-d738-4325-90e7-d1c9ce87eeb5\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6"
Nov 03 19:45:02.779081 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:02.778887    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0da622a1-45f9-4311-b668-d27eb7ec2e43-istiod-ca-cert\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5\" (UID: \"0da622a1-45f9-4311-b668-d27eb7ec2e43\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5"
Nov 03 19:45:02.779081 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:02.778916    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e96267a-d738-4325-90e7-d1c9ce87eeb5-istio-envoy\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6\" (UID: \"7e96267a-d738-4325-90e7-d1c9ce87eeb5\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6"
Nov 03 19:45:02.779081 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:02.778941    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5r7m\" (UniqueName: \"kubernetes.io/projected/0da622a1-45f9-4311-b668-d27eb7ec2e43-kube-api-access-z5r7m\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5\" (UID: \"0da622a1-45f9-4311-b668-d27eb7ec2e43\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5"
Nov 03 19:45:02.779081 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:02.778991    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e96267a-d738-4325-90e7-d1c9ce87eeb5-istio-data\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6\" (UID: \"7e96267a-d738-4325-90e7-d1c9ce87eeb5\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6"
Nov 03 19:45:02.779081 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:02.779029    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e96267a-d738-4325-90e7-d1c9ce87eeb5-credential-socket\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6\" (UID: \"7e96267a-d738-4325-90e7-d1c9ce87eeb5\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6"
Nov 03 19:45:02.779731 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:02.779117    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0da622a1-45f9-4311-b668-d27eb7ec2e43-istio-podinfo\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5\" (UID: \"0da622a1-45f9-4311-b668-d27eb7ec2e43\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5"
Nov 03 19:45:02.779731 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:02.779169    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e96267a-d738-4325-90e7-d1c9ce87eeb5-istiod-ca-cert\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6\" (UID: \"7e96267a-d738-4325-90e7-d1c9ce87eeb5\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6"
Nov 03 19:45:02.779731 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:02.779189    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e96267a-d738-4325-90e7-d1c9ce87eeb5-istio-token\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6\" (UID: \"7e96267a-d738-4325-90e7-d1c9ce87eeb5\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6"
Nov 03 19:45:02.779731 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:02.779238    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da622a1-45f9-4311-b668-d27eb7ec2e43-credential-socket\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5\" (UID: \"0da622a1-45f9-4311-b668-d27eb7ec2e43\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5"
Nov 03 19:45:02.779731 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:02.779272    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da622a1-45f9-4311-b668-d27eb7ec2e43-workload-socket\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5\" (UID: \"0da622a1-45f9-4311-b668-d27eb7ec2e43\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5"
Nov 03 19:45:02.779731 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:02.779298    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e96267a-d738-4325-90e7-d1c9ce87eeb5-workload-socket\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6\" (UID: \"7e96267a-d738-4325-90e7-d1c9ce87eeb5\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6"
Nov 03 19:45:02.779731 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:02.779328    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0da622a1-45f9-4311-b668-d27eb7ec2e43-istio-token\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5\" (UID: \"0da622a1-45f9-4311-b668-d27eb7ec2e43\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5"
Nov 03 19:45:02.779731 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:02.779390    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0da622a1-45f9-4311-b668-d27eb7ec2e43-workload-certs\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5\" (UID: \"0da622a1-45f9-4311-b668-d27eb7ec2e43\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5"
Nov 03 19:45:02.779731 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:02.779419    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0da622a1-45f9-4311-b668-d27eb7ec2e43-istio-envoy\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5\" (UID: \"0da622a1-45f9-4311-b668-d27eb7ec2e43\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5"
Nov 03 19:45:02.779731 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:02.779443    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0da622a1-45f9-4311-b668-d27eb7ec2e43-istio-data\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5\" (UID: \"0da622a1-45f9-4311-b668-d27eb7ec2e43\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5"
Nov 03 19:45:02.779731 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:02.779724    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0da622a1-45f9-4311-b668-d27eb7ec2e43-istiod-ca-cert\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5\" (UID: \"0da622a1-45f9-4311-b668-d27eb7ec2e43\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5"
Nov 03 19:45:02.780305 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:02.779825    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0da622a1-45f9-4311-b668-d27eb7ec2e43-istio-data\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5\" (UID: \"0da622a1-45f9-4311-b668-d27eb7ec2e43\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5"
Nov 03 19:45:02.780305 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:02.779983    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e96267a-d738-4325-90e7-d1c9ce87eeb5-workload-certs\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6\" (UID: \"7e96267a-d738-4325-90e7-d1c9ce87eeb5\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6"
Nov 03 19:45:02.780305 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:02.780013    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da622a1-45f9-4311-b668-d27eb7ec2e43-credential-socket\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5\" (UID: \"0da622a1-45f9-4311-b668-d27eb7ec2e43\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5"
Nov 03 19:45:02.780305 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:02.780174    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da622a1-45f9-4311-b668-d27eb7ec2e43-workload-socket\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5\" (UID: \"0da622a1-45f9-4311-b668-d27eb7ec2e43\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5"
Nov 03 19:45:02.780564 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:02.780431    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e96267a-d738-4325-90e7-d1c9ce87eeb5-workload-socket\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6\" (UID: \"7e96267a-d738-4325-90e7-d1c9ce87eeb5\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6"
Nov 03 19:45:02.780857 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:02.780837    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0da622a1-45f9-4311-b668-d27eb7ec2e43-workload-certs\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5\" (UID: \"0da622a1-45f9-4311-b668-d27eb7ec2e43\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5"
Nov 03 19:45:02.782023 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:02.781998    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e96267a-d738-4325-90e7-d1c9ce87eeb5-istio-envoy\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6\" (UID: \"7e96267a-d738-4325-90e7-d1c9ce87eeb5\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6"
Nov 03 19:45:02.782174 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:02.782148    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e96267a-d738-4325-90e7-d1c9ce87eeb5-istio-data\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6\" (UID: \"7e96267a-d738-4325-90e7-d1c9ce87eeb5\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6"
Nov 03 19:45:02.782323 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:02.782302    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e96267a-d738-4325-90e7-d1c9ce87eeb5-credential-socket\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6\" (UID: \"7e96267a-d738-4325-90e7-d1c9ce87eeb5\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6"
Nov 03 19:45:02.782801 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:02.782774    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e96267a-d738-4325-90e7-d1c9ce87eeb5-istio-podinfo\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6\" (UID: \"7e96267a-d738-4325-90e7-d1c9ce87eeb5\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6"
Nov 03 19:45:02.782915 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:02.782881    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0da622a1-45f9-4311-b668-d27eb7ec2e43-istio-envoy\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5\" (UID: \"0da622a1-45f9-4311-b668-d27eb7ec2e43\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5"
Nov 03 19:45:02.787527 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:02.787185    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e96267a-d738-4325-90e7-d1c9ce87eeb5-istio-token\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6\" (UID: \"7e96267a-d738-4325-90e7-d1c9ce87eeb5\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6"
Nov 03 19:45:02.787527 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:02.787213    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0da622a1-45f9-4311-b668-d27eb7ec2e43-istio-podinfo\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5\" (UID: \"0da622a1-45f9-4311-b668-d27eb7ec2e43\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5"
Nov 03 19:45:02.787527 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:02.787332    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frd24\" (UniqueName: \"kubernetes.io/projected/7e96267a-d738-4325-90e7-d1c9ce87eeb5-kube-api-access-frd24\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6\" (UID: \"7e96267a-d738-4325-90e7-d1c9ce87eeb5\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6"
Nov 03 19:45:02.789977 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:02.789953    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0da622a1-45f9-4311-b668-d27eb7ec2e43-istio-token\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5\" (UID: \"0da622a1-45f9-4311-b668-d27eb7ec2e43\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5"
Nov 03 19:45:02.790161 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:02.790140    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5r7m\" (UniqueName: \"kubernetes.io/projected/0da622a1-45f9-4311-b668-d27eb7ec2e43-kube-api-access-z5r7m\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5\" (UID: \"0da622a1-45f9-4311-b668-d27eb7ec2e43\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5"
Nov 03 19:45:02.836574 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:02.836533    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5"
Nov 03 19:45:02.870808 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:02.870780    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6"
Nov 03 19:45:02.997995 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:45:02.997874    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 19:45:03.010279 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:03.008182    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m"]
Nov 03 19:45:03.021913 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:03.021025    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk"]
Nov 03 19:45:03.025657 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:03.025621    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m"
Nov 03 19:45:03.031545 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:03.031502    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk"
Nov 03 19:45:03.032247 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:03.032223    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m"]
Nov 03 19:45:03.032432 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:03.032420    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk"]
Nov 03 19:45:03.165036 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:03.133086    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-82dbbf\"/\"distinct-2-w0-82dbbf-kserve-self-signed-certs\""
Nov 03 19:45:03.165036 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:03.163417    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee6baa84-41ee-4bee-98c9-a9d0f078d4b9-istio-podinfo\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m\" (UID: \"ee6baa84-41ee-4bee-98c9-a9d0f078d4b9\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m"
Nov 03 19:45:03.165036 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:03.163464    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee6baa84-41ee-4bee-98c9-a9d0f078d4b9-workload-socket\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m\" (UID: \"ee6baa84-41ee-4bee-98c9-a9d0f078d4b9\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m"
Nov 03 19:45:03.165036 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:03.163495    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee6baa84-41ee-4bee-98c9-a9d0f078d4b9-workload-certs\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m\" (UID: \"ee6baa84-41ee-4bee-98c9-a9d0f078d4b9\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m"
Nov 03 19:45:03.165036 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:03.163526    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee6baa84-41ee-4bee-98c9-a9d0f078d4b9-istiod-ca-cert\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m\" (UID: \"ee6baa84-41ee-4bee-98c9-a9d0f078d4b9\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m"
Nov 03 19:45:03.165036 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:03.163574    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d0a934e-4816-44d2-9bb5-e8b4e95dd73c-tmp-dir\") pod \"distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk\" (UID: \"2d0a934e-4816-44d2-9bb5-e8b4e95dd73c\") " pod="e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk"
Nov 03 19:45:03.165036 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:03.163652    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d0a934e-4816-44d2-9bb5-e8b4e95dd73c-dshm\") pod \"distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk\" (UID: \"2d0a934e-4816-44d2-9bb5-e8b4e95dd73c\") " pod="e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk"
Nov 03 19:45:03.165036 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:03.163733    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d0a934e-4816-44d2-9bb5-e8b4e95dd73c-tls-certs\") pod \"distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk\" (UID: \"2d0a934e-4816-44d2-9bb5-e8b4e95dd73c\") " pod="e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk"
Nov 03 19:45:03.165036 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:03.163768    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee6baa84-41ee-4bee-98c9-a9d0f078d4b9-credential-socket\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m\" (UID: \"ee6baa84-41ee-4bee-98c9-a9d0f078d4b9\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m"
Nov 03 19:45:03.165036 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:03.163861    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h55np\" (UniqueName: \"kubernetes.io/projected/ee6baa84-41ee-4bee-98c9-a9d0f078d4b9-kube-api-access-h55np\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m\" (UID: \"ee6baa84-41ee-4bee-98c9-a9d0f078d4b9\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m"
Nov 03 19:45:03.165036 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:03.164136    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee6baa84-41ee-4bee-98c9-a9d0f078d4b9-istio-envoy\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m\" (UID: \"ee6baa84-41ee-4bee-98c9-a9d0f078d4b9\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m"
Nov 03 19:45:03.165036 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:03.164187    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d0a934e-4816-44d2-9bb5-e8b4e95dd73c-model-cache\") pod \"distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk\" (UID: \"2d0a934e-4816-44d2-9bb5-e8b4e95dd73c\") " pod="e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk"
Nov 03 19:45:03.165036 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:03.164215    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cwrh\" (UniqueName: \"kubernetes.io/projected/2d0a934e-4816-44d2-9bb5-e8b4e95dd73c-kube-api-access-6cwrh\") pod \"distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk\" (UID: \"2d0a934e-4816-44d2-9bb5-e8b4e95dd73c\") " pod="e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk"
Nov 03 19:45:03.165036 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:03.164252    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee6baa84-41ee-4bee-98c9-a9d0f078d4b9-istio-data\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m\" (UID: \"ee6baa84-41ee-4bee-98c9-a9d0f078d4b9\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m"
Nov 03 19:45:03.165036 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:03.164298    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d0a934e-4816-44d2-9bb5-e8b4e95dd73c-home\") pod \"distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk\" (UID: \"2d0a934e-4816-44d2-9bb5-e8b4e95dd73c\") " pod="e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk"
Nov 03 19:45:03.165036 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:03.164371    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee6baa84-41ee-4bee-98c9-a9d0f078d4b9-istio-token\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m\" (UID: \"ee6baa84-41ee-4bee-98c9-a9d0f078d4b9\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m"
Nov 03 19:45:03.219722 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:03.219116    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:45:03.219722 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:03.219182    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:45:03.219722 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:03.219213    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:45:03.231557 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:03.231525    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5"]
Nov 03 19:45:03.265515 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:03.265468    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee6baa84-41ee-4bee-98c9-a9d0f078d4b9-istio-envoy\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m\" (UID: \"ee6baa84-41ee-4bee-98c9-a9d0f078d4b9\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m"
Nov 03 19:45:03.265723 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:03.265546    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d0a934e-4816-44d2-9bb5-e8b4e95dd73c-model-cache\") pod \"distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk\" (UID: \"2d0a934e-4816-44d2-9bb5-e8b4e95dd73c\") " pod="e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk"
Nov 03 19:45:03.265723 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:03.265573    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cwrh\" (UniqueName: \"kubernetes.io/projected/2d0a934e-4816-44d2-9bb5-e8b4e95dd73c-kube-api-access-6cwrh\") pod \"distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk\" (UID: \"2d0a934e-4816-44d2-9bb5-e8b4e95dd73c\") " pod="e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk"
Nov 03 19:45:03.265723 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:03.265610    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee6baa84-41ee-4bee-98c9-a9d0f078d4b9-istio-data\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m\" (UID: \"ee6baa84-41ee-4bee-98c9-a9d0f078d4b9\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m"
Nov 03 19:45:03.265723 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:03.265634    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d0a934e-4816-44d2-9bb5-e8b4e95dd73c-home\") pod \"distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk\" (UID: \"2d0a934e-4816-44d2-9bb5-e8b4e95dd73c\") " pod="e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk"
Nov 03 19:45:03.265723 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:03.265707    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee6baa84-41ee-4bee-98c9-a9d0f078d4b9-istio-token\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m\" (UID: \"ee6baa84-41ee-4bee-98c9-a9d0f078d4b9\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m"
Nov 03 19:45:03.266022 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:03.265776    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee6baa84-41ee-4bee-98c9-a9d0f078d4b9-istio-podinfo\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m\" (UID: \"ee6baa84-41ee-4bee-98c9-a9d0f078d4b9\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m"
Nov 03 19:45:03.266022 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:03.265809    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee6baa84-41ee-4bee-98c9-a9d0f078d4b9-workload-socket\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m\" (UID: \"ee6baa84-41ee-4bee-98c9-a9d0f078d4b9\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m"
Nov 03 19:45:03.266022 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:03.265837    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee6baa84-41ee-4bee-98c9-a9d0f078d4b9-workload-certs\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m\" (UID: \"ee6baa84-41ee-4bee-98c9-a9d0f078d4b9\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m"
Nov 03 19:45:03.266022 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:03.265867    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee6baa84-41ee-4bee-98c9-a9d0f078d4b9-istiod-ca-cert\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m\" (UID: \"ee6baa84-41ee-4bee-98c9-a9d0f078d4b9\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m"
Nov 03 19:45:03.266022 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:03.265922    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d0a934e-4816-44d2-9bb5-e8b4e95dd73c-tmp-dir\") pod \"distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk\" (UID: \"2d0a934e-4816-44d2-9bb5-e8b4e95dd73c\") " pod="e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk"
Nov 03 19:45:03.266022 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:03.265999    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d0a934e-4816-44d2-9bb5-e8b4e95dd73c-dshm\") pod \"distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk\" (UID: \"2d0a934e-4816-44d2-9bb5-e8b4e95dd73c\") " pod="e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk"
Nov 03 19:45:03.266329 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:03.266036    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d0a934e-4816-44d2-9bb5-e8b4e95dd73c-tls-certs\") pod \"distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk\" (UID: \"2d0a934e-4816-44d2-9bb5-e8b4e95dd73c\") " pod="e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk"
Nov 03 19:45:03.266329 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:03.266064    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee6baa84-41ee-4bee-98c9-a9d0f078d4b9-credential-socket\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m\" (UID: \"ee6baa84-41ee-4bee-98c9-a9d0f078d4b9\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m"
Nov 03 19:45:03.266329 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:03.266147    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h55np\" (UniqueName: \"kubernetes.io/projected/ee6baa84-41ee-4bee-98c9-a9d0f078d4b9-kube-api-access-h55np\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m\" (UID: \"ee6baa84-41ee-4bee-98c9-a9d0f078d4b9\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m"
Nov 03 19:45:03.279772 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:03.279734    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee6baa84-41ee-4bee-98c9-a9d0f078d4b9-workload-socket\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m\" (UID: \"ee6baa84-41ee-4bee-98c9-a9d0f078d4b9\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m"
Nov 03 19:45:03.280152 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:03.280126    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d0a934e-4816-44d2-9bb5-e8b4e95dd73c-model-cache\") pod \"distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk\" (UID: \"2d0a934e-4816-44d2-9bb5-e8b4e95dd73c\") " pod="e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk"
Nov 03 19:45:03.280740 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:03.280709    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee6baa84-41ee-4bee-98c9-a9d0f078d4b9-istio-data\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m\" (UID: \"ee6baa84-41ee-4bee-98c9-a9d0f078d4b9\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m"
Nov 03 19:45:03.281490 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:03.281453    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d0a934e-4816-44d2-9bb5-e8b4e95dd73c-tmp-dir\") pod \"distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk\" (UID: \"2d0a934e-4816-44d2-9bb5-e8b4e95dd73c\") " pod="e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk"
Nov 03 19:45:03.282800 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:03.281738    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee6baa84-41ee-4bee-98c9-a9d0f078d4b9-workload-certs\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m\" (UID: \"ee6baa84-41ee-4bee-98c9-a9d0f078d4b9\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m"
Nov 03 19:45:03.282800 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:03.282407    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee6baa84-41ee-4bee-98c9-a9d0f078d4b9-istiod-ca-cert\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m\" (UID: \"ee6baa84-41ee-4bee-98c9-a9d0f078d4b9\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m"
Nov 03 19:45:03.282985 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:03.282933    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d0a934e-4816-44d2-9bb5-e8b4e95dd73c-home\") pod \"distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk\" (UID: \"2d0a934e-4816-44d2-9bb5-e8b4e95dd73c\") " pod="e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk"
Nov 03 19:45:03.283577 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:03.283545    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee6baa84-41ee-4bee-98c9-a9d0f078d4b9-credential-socket\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m\" (UID: \"ee6baa84-41ee-4bee-98c9-a9d0f078d4b9\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m"
Nov 03 19:45:03.285519 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:03.285451    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h55np\" (UniqueName: \"kubernetes.io/projected/ee6baa84-41ee-4bee-98c9-a9d0f078d4b9-kube-api-access-h55np\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m\" (UID: \"ee6baa84-41ee-4bee-98c9-a9d0f078d4b9\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m"
Nov 03 19:45:03.293298 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:03.293252    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cwrh\" (UniqueName: \"kubernetes.io/projected/2d0a934e-4816-44d2-9bb5-e8b4e95dd73c-kube-api-access-6cwrh\") pod \"distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk\" (UID: \"2d0a934e-4816-44d2-9bb5-e8b4e95dd73c\") " pod="e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk"
Nov 03 19:45:03.294472 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:03.294415    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee6baa84-41ee-4bee-98c9-a9d0f078d4b9-istio-token\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m\" (UID: \"ee6baa84-41ee-4bee-98c9-a9d0f078d4b9\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m"
Nov 03 19:45:03.298953 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:03.298862    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee6baa84-41ee-4bee-98c9-a9d0f078d4b9-istio-podinfo\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m\" (UID: \"ee6baa84-41ee-4bee-98c9-a9d0f078d4b9\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m"
Nov 03 19:45:03.299533 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:03.299506    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee6baa84-41ee-4bee-98c9-a9d0f078d4b9-istio-envoy\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m\" (UID: \"ee6baa84-41ee-4bee-98c9-a9d0f078d4b9\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m"
Nov 03 19:45:03.305518 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:03.305210    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d0a934e-4816-44d2-9bb5-e8b4e95dd73c-tls-certs\") pod \"distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk\" (UID: \"2d0a934e-4816-44d2-9bb5-e8b4e95dd73c\") " pod="e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk"
Nov 03 19:45:03.312476 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:03.312445    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d0a934e-4816-44d2-9bb5-e8b4e95dd73c-dshm\") pod \"distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk\" (UID: \"2d0a934e-4816-44d2-9bb5-e8b4e95dd73c\") " pod="e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk"
Nov 03 19:45:03.313373 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:03.313346    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6"]
Nov 03 19:45:03.317514 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:45:03.317413    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e96267a_d738_4325_90e7_d1c9ce87eeb5.slice/crio-8ec0123073af5a11dda5f140782a76e73a7677ff5d557943d6a8213cfd5ebd4c WatchSource:0}: Error finding container 8ec0123073af5a11dda5f140782a76e73a7677ff5d557943d6a8213cfd5ebd4c: Status 404 returned error can't find the container with id 8ec0123073af5a11dda5f140782a76e73a7677ff5d557943d6a8213cfd5ebd4c
Nov 03 19:45:03.325147 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:03.321786    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:45:03.325147 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:03.321941    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:45:03.325147 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:03.322014    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:45:03.485770 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:03.484167    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0486e8-29895585-252sx" event={"ID":"980862b5-9e6d-42a6-b7f9-141205136ef8","Type":"ContainerStarted","Data":"d0cd4699001644a81375346ae2fbc83165c92d9e47401894646629f6efda5000"}
Nov 03 19:45:03.500971 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:03.500652    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m"
Nov 03 19:45:03.502453 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:03.502352    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d4127-29895585-lwd9p" event={"ID":"e2aa77e1-ea3d-4db2-84b6-da15cf42d047","Type":"ContainerStarted","Data":"5fd067e0e3cd6d6c72e12575f642102f3ee4db813805918f249e89cb141ea7e0"}
Nov 03 19:45:03.526852 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:03.526805    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6" event={"ID":"7e96267a-d738-4325-90e7-d1c9ce87eeb5","Type":"ContainerStarted","Data":"8ec0123073af5a11dda5f140782a76e73a7677ff5d557943d6a8213cfd5ebd4c"}
Nov 03 19:45:03.530383 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:03.530344    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5" event={"ID":"0da622a1-45f9-4311-b668-d27eb7ec2e43","Type":"ContainerStarted","Data":"133dcce05548df4da2f2b1b43cf49f4bc0e88577b8ee7203c5eeafa476f93aea"}
Nov 03 19:45:03.530518 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:03.530415    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5" event={"ID":"0da622a1-45f9-4311-b668-d27eb7ec2e43","Type":"ContainerStarted","Data":"7417e896df1c434063e535487a6305e74f48a6a9ff980793855164f79558d703"}
Nov 03 19:45:03.550830 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:03.550566    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk"
Nov 03 19:45:03.550830 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:03.550695    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-b8tdx" event={"ID":"45523a35-8108-4813-b500-d0803ce6cc9c","Type":"ContainerStarted","Data":"aab4e31e9de047148007553a3bce063ac8f9b954635fd21adb5f1240a1690b12"}
Nov 03 19:45:03.583268 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:03.583203    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6" podStartSLOduration=1.583178773 podStartE2EDuration="1.583178773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:03.565467107 +0000 UTC m=+816.809432574" watchObservedRunningTime="2026-11-03 19:45:03.583178773 +0000 UTC m=+816.827144239"
Nov 03 19:45:03.618468 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:03.618383    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5" podStartSLOduration=1.618358305 podStartE2EDuration="1.618358305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:03.598351051 +0000 UTC m=+816.842316517" watchObservedRunningTime="2026-11-03 19:45:03.618358305 +0000 UTC m=+816.862323776"
Nov 03 19:45:03.839295 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:03.839252    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5"
Nov 03 19:45:03.920956 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:03.874177    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6"
Nov 03 19:45:04.343787 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:04.293029    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk"]
Nov 03 19:45:04.369862 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:04.369815    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:45:04.370078 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:04.370050    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:45:04.370226 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:04.370204    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:45:04.459731 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:04.402866    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m"]
Nov 03 19:45:04.682697 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:04.679716    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6" event={"ID":"7e96267a-d738-4325-90e7-d1c9ce87eeb5","Type":"ContainerStarted","Data":"a25a192dcf94a7c7e77f0c45f80b61c63cdd7ff4c624c61f20f6a0ce301e5368"}
Nov 03 19:45:04.684088 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:04.683653    4921 generic.go:356] "Generic (PLEG): container finished" podID="45523a35-8108-4813-b500-d0803ce6cc9c" containerID="aab4e31e9de047148007553a3bce063ac8f9b954635fd21adb5f1240a1690b12" exitCode=60
Nov 03 19:45:04.684456 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:04.684431    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-b8tdx" event={"ID":"45523a35-8108-4813-b500-d0803ce6cc9c","Type":"ContainerDied","Data":"aab4e31e9de047148007553a3bce063ac8f9b954635fd21adb5f1240a1690b12"}
Nov 03 19:45:04.684622 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:04.684607    4921 scope.go:122] "RemoveContainer" containerID="e868b3680040dfe6a113aace37dc282520c7b8d0891633aa1537a6c68b5d09f8"
Nov 03 19:45:04.685682 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:04.685650    4921 scope.go:122] "RemoveContainer" containerID="aab4e31e9de047148007553a3bce063ac8f9b954635fd21adb5f1240a1690b12"
Nov 03 19:45:04.690500 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:04.690478    4921 generic.go:356] "Generic (PLEG): container finished" podID="980862b5-9e6d-42a6-b7f9-141205136ef8" containerID="d0cd4699001644a81375346ae2fbc83165c92d9e47401894646629f6efda5000" exitCode=60
Nov 03 19:45:04.690771 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:04.690750    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0486e8-29895585-252sx" event={"ID":"980862b5-9e6d-42a6-b7f9-141205136ef8","Type":"ContainerDied","Data":"d0cd4699001644a81375346ae2fbc83165c92d9e47401894646629f6efda5000"}
Nov 03 19:45:04.691114 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:04.691089    4921 scope.go:122] "RemoveContainer" containerID="d0cd4699001644a81375346ae2fbc83165c92d9e47401894646629f6efda5000"
Nov 03 19:45:04.691574 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:45:04.691548    4921 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-0486e8-29895585-252sx_odh-ai-gateway-infra(980862b5-9e6d-42a6-b7f9-141205136ef8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0486e8-29895585-252sx" podUID="980862b5-9e6d-42a6-b7f9-141205136ef8"
Nov 03 19:45:04.692415 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:45:04.692396    4921 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895585-b8tdx_odh-ai-gateway-infra(45523a35-8108-4813-b500-d0803ce6cc9c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-b8tdx" podUID="45523a35-8108-4813-b500-d0803ce6cc9c"
Nov 03 19:45:04.697755 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:04.697729    4921 generic.go:356] "Generic (PLEG): container finished" podID="e2aa77e1-ea3d-4db2-84b6-da15cf42d047" containerID="5fd067e0e3cd6d6c72e12575f642102f3ee4db813805918f249e89cb141ea7e0" exitCode=60
Nov 03 19:45:04.697954 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:04.697934    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d4127-29895585-lwd9p" event={"ID":"e2aa77e1-ea3d-4db2-84b6-da15cf42d047","Type":"ContainerDied","Data":"5fd067e0e3cd6d6c72e12575f642102f3ee4db813805918f249e89cb141ea7e0"}
Nov 03 19:45:04.698527 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:04.698501    4921 scope.go:122] "RemoveContainer" containerID="5fd067e0e3cd6d6c72e12575f642102f3ee4db813805918f249e89cb141ea7e0"
Nov 03 19:45:04.699082 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:45:04.699059    4921 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-8d4127-29895585-lwd9p_odh-ai-gateway-infra(e2aa77e1-ea3d-4db2-84b6-da15cf42d047)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d4127-29895585-lwd9p" podUID="e2aa77e1-ea3d-4db2-84b6-da15cf42d047"
Nov 03 19:45:04.714271 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:04.714234    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk" event={"ID":"2d0a934e-4816-44d2-9bb5-e8b4e95dd73c","Type":"ContainerStarted","Data":"47bdf4de52f7e540e4b2b513cdf4b8f923cd41286422bf2db25506f36c620aef"}
Nov 03 19:45:04.717052 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:04.717020    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m" event={"ID":"ee6baa84-41ee-4bee-98c9-a9d0f078d4b9","Type":"ContainerStarted","Data":"d653f6d08b96bd9d6c233b0e1fcd2b0988059908c72214d37a3567e0122c780e"}
Nov 03 19:45:04.862695 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:04.859067    4921 scope.go:122] "RemoveContainer" containerID="378599bcd17f82cf0d5e57332d925c448199b57252685a33514edc120924a826"
Nov 03 19:45:04.871718 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:04.869989    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh"]
Nov 03 19:45:04.887521 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:04.881305    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh"]
Nov 03 19:45:04.887521 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:04.881811    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh"
Nov 03 19:45:04.890316 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:04.887991    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-f4e123\"/\"distinct-2-w4-f4e123-kserve-self-signed-certs\""
Nov 03 19:45:04.890316 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:04.888197    4921 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 03 19:45:04.890316 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:04.888240    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6" podUID="7e96267a-d738-4325-90e7-d1c9ce87eeb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 03 19:45:04.909825 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:04.909790    4921 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 03 19:45:04.910070 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:04.910041    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5" podUID="0da622a1-45f9-4311-b668-d27eb7ec2e43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 03 19:45:05.138576 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:05.138456    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eac6c47d-8ae5-4e4b-b691-98d177887767-tmp-dir\") pod \"distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh\" (UID: \"eac6c47d-8ae5-4e4b-b691-98d177887767\") " pod="e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh"
Nov 03 19:45:05.138938 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:05.138901    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eac6c47d-8ae5-4e4b-b691-98d177887767-dshm\") pod \"distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh\" (UID: \"eac6c47d-8ae5-4e4b-b691-98d177887767\") " pod="e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh"
Nov 03 19:45:05.163708 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:05.154792    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6tf8\" (UniqueName: \"kubernetes.io/projected/eac6c47d-8ae5-4e4b-b691-98d177887767-kube-api-access-l6tf8\") pod \"distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh\" (UID: \"eac6c47d-8ae5-4e4b-b691-98d177887767\") " pod="e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh"
Nov 03 19:45:05.163708 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:05.154919    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eac6c47d-8ae5-4e4b-b691-98d177887767-home\") pod \"distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh\" (UID: \"eac6c47d-8ae5-4e4b-b691-98d177887767\") " pod="e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh"
Nov 03 19:45:05.163708 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:05.154986    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eac6c47d-8ae5-4e4b-b691-98d177887767-model-cache\") pod \"distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh\" (UID: \"eac6c47d-8ae5-4e4b-b691-98d177887767\") " pod="e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh"
Nov 03 19:45:05.163708 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:05.155087    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eac6c47d-8ae5-4e4b-b691-98d177887767-tls-certs\") pod \"distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh\" (UID: \"eac6c47d-8ae5-4e4b-b691-98d177887767\") " pod="e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh"
Nov 03 19:45:05.265086 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:05.265039    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6tf8\" (UniqueName: \"kubernetes.io/projected/eac6c47d-8ae5-4e4b-b691-98d177887767-kube-api-access-l6tf8\") pod \"distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh\" (UID: \"eac6c47d-8ae5-4e4b-b691-98d177887767\") " pod="e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh"
Nov 03 19:45:05.283268 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:05.283225    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eac6c47d-8ae5-4e4b-b691-98d177887767-home\") pod \"distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh\" (UID: \"eac6c47d-8ae5-4e4b-b691-98d177887767\") " pod="e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh"
Nov 03 19:45:05.283577 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:05.283557    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eac6c47d-8ae5-4e4b-b691-98d177887767-model-cache\") pod \"distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh\" (UID: \"eac6c47d-8ae5-4e4b-b691-98d177887767\") " pod="e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh"
Nov 03 19:45:05.283799 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:05.283782    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eac6c47d-8ae5-4e4b-b691-98d177887767-tls-certs\") pod \"distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh\" (UID: \"eac6c47d-8ae5-4e4b-b691-98d177887767\") " pod="e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh"
Nov 03 19:45:05.284070 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:05.284053    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eac6c47d-8ae5-4e4b-b691-98d177887767-tmp-dir\") pod \"distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh\" (UID: \"eac6c47d-8ae5-4e4b-b691-98d177887767\") " pod="e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh"
Nov 03 19:45:05.284290 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:05.284268    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eac6c47d-8ae5-4e4b-b691-98d177887767-dshm\") pod \"distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh\" (UID: \"eac6c47d-8ae5-4e4b-b691-98d177887767\") " pod="e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh"
Nov 03 19:45:05.285276 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:05.285219    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eac6c47d-8ae5-4e4b-b691-98d177887767-model-cache\") pod \"distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh\" (UID: \"eac6c47d-8ae5-4e4b-b691-98d177887767\") " pod="e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh"
Nov 03 19:45:05.285719 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:05.285696    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eac6c47d-8ae5-4e4b-b691-98d177887767-home\") pod \"distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh\" (UID: \"eac6c47d-8ae5-4e4b-b691-98d177887767\") " pod="e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh"
Nov 03 19:45:05.294715 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:05.293909    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eac6c47d-8ae5-4e4b-b691-98d177887767-tmp-dir\") pod \"distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh\" (UID: \"eac6c47d-8ae5-4e4b-b691-98d177887767\") " pod="e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh"
Nov 03 19:45:05.403048 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:05.392368    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6tf8\" (UniqueName: \"kubernetes.io/projected/eac6c47d-8ae5-4e4b-b691-98d177887767-kube-api-access-l6tf8\") pod \"distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh\" (UID: \"eac6c47d-8ae5-4e4b-b691-98d177887767\") " pod="e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh"
Nov 03 19:45:05.407777 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:05.403982    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eac6c47d-8ae5-4e4b-b691-98d177887767-tls-certs\") pod \"distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh\" (UID: \"eac6c47d-8ae5-4e4b-b691-98d177887767\") " pod="e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh"
Nov 03 19:45:05.407777 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:05.405897    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eac6c47d-8ae5-4e4b-b691-98d177887767-dshm\") pod \"distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh\" (UID: \"eac6c47d-8ae5-4e4b-b691-98d177887767\") " pod="e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh"
Nov 03 19:45:05.432467 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:05.428829    4921 scope.go:122] "RemoveContainer" containerID="c2d2ed3752124906a209acef66c198596c173feb202b7f82f1c1f0ba8bbc036c"
Nov 03 19:45:05.584775 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:05.583603    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh"
Nov 03 19:45:05.650337 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:05.647807    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6" podUID="1aaee61d-b036-4c2b-bee7-a692b8bed612" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:05.820851 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:05.818855    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw" podUID="72d96abc-f3b4-4eda-b3a1-b93ba658e1d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Nov 03 19:45:05.829702 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:05.829154    4921 scope.go:122] "RemoveContainer" containerID="aab4e31e9de047148007553a3bce063ac8f9b954635fd21adb5f1240a1690b12"
Nov 03 19:45:05.829702 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:45:05.829490    4921 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895585-b8tdx_odh-ai-gateway-infra(45523a35-8108-4813-b500-d0803ce6cc9c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-b8tdx" podUID="45523a35-8108-4813-b500-d0803ce6cc9c"
Nov 03 19:45:05.839546 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:05.839166    4921 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 03 19:45:05.839546 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:05.839219    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5" podUID="0da622a1-45f9-4311-b668-d27eb7ec2e43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 03 19:45:05.847507 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:05.847115    4921 scope.go:122] "RemoveContainer" containerID="d0cd4699001644a81375346ae2fbc83165c92d9e47401894646629f6efda5000"
Nov 03 19:45:05.847507 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:45:05.847447    4921 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-0486e8-29895585-252sx_odh-ai-gateway-infra(980862b5-9e6d-42a6-b7f9-141205136ef8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0486e8-29895585-252sx" podUID="980862b5-9e6d-42a6-b7f9-141205136ef8"
Nov 03 19:45:05.874150 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:05.874111    4921 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 03 19:45:05.874386 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:05.874360    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6" podUID="7e96267a-d738-4325-90e7-d1c9ce87eeb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 03 19:45:05.876805 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:05.876346    4921 scope.go:122] "RemoveContainer" containerID="5fd067e0e3cd6d6c72e12575f642102f3ee4db813805918f249e89cb141ea7e0"
Nov 03 19:45:05.876805 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:45:05.876731    4921 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-8d4127-29895585-lwd9p_odh-ai-gateway-infra(e2aa77e1-ea3d-4db2-84b6-da15cf42d047)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d4127-29895585-lwd9p" podUID="e2aa77e1-ea3d-4db2-84b6-da15cf42d047"
Nov 03 19:45:05.898737 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:05.897337    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m" event={"ID":"ee6baa84-41ee-4bee-98c9-a9d0f078d4b9","Type":"ContainerStarted","Data":"a8f40df2f93cafd7ce8790e309e58a23dad94a4e563c39af9ee3fe65cc9dc7c3"}
Nov 03 19:45:05.995859 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:05.995814    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv" podUID="01c749b9-d188-4673-9a3d-99b9c4ae0793" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:06.021919 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:06.021761    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m" podStartSLOduration=4.021725373 podStartE2EDuration="4.021725373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:05.922249101 +0000 UTC m=+819.166214595" watchObservedRunningTime="2026-11-03 19:45:06.021725373 +0000 UTC m=+819.265690840"
Nov 03 19:45:06.294731 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:06.279431    4921 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv" podUID="01c749b9-d188-4673-9a3d-99b9c4ae0793" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:06.334965 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:06.295775    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7" podUID="d4de0090-d775-4bd7-9ee2-058b5d380638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded"
Nov 03 19:45:06.335277 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:06.295811    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv" podUID="96b73342-0079-4709-8570-3dbce57e5243" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Nov 03 19:45:06.518145 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:06.504756    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m"
Nov 03 19:45:06.518145 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:06.506070    4921 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 03 19:45:06.518145 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:06.506120    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m" podUID="ee6baa84-41ee-4bee-98c9-a9d0f078d4b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 03 19:45:06.535281 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:06.535242    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh"]
Nov 03 19:45:06.683694 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:06.671702    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6" podUID="1aaee61d-b036-4c2b-bee7-a692b8bed612" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:06.683694 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:06.672534    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6"
Nov 03 19:45:06.835771 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:06.822771    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw" podUID="72d96abc-f3b4-4eda-b3a1-b93ba658e1d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:06.835771 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:06.832136    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw"
Nov 03 19:45:06.893689 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:06.893282    4921 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 03 19:45:06.893689 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:06.893341    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6" podUID="7e96267a-d738-4325-90e7-d1c9ce87eeb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 03 19:45:06.921040 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:06.920733    4921 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 03 19:45:06.921040 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:06.920786    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5" podUID="0da622a1-45f9-4311-b668-d27eb7ec2e43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 03 19:45:07.006820 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:07.006776    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv" podUID="01c749b9-d188-4673-9a3d-99b9c4ae0793" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:07.053543 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:07.053499    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh" event={"ID":"eac6c47d-8ae5-4e4b-b691-98d177887767","Type":"ContainerStarted","Data":"3c4912c5d61f1b17f31a9b3d04e47185495254546a25c44c4b07e3b22053df77"}
Nov 03 19:45:07.053979 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:07.053938    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv"
Nov 03 19:45:07.145356 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:07.145192    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv"
Nov 03 19:45:07.468361 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:07.468260    4921 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6" podUID="1aaee61d-b036-4c2b-bee7-a692b8bed612" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:07.503422 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:07.503386    4921 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 03 19:45:07.503732 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:07.503697    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m" podUID="ee6baa84-41ee-4bee-98c9-a9d0f078d4b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 03 19:45:07.664695 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:07.660696    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv" podUID="96b73342-0079-4709-8570-3dbce57e5243" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:07.664695 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:07.661082    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7" podUID="d4de0090-d775-4bd7-9ee2-058b5d380638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded"
Nov 03 19:45:07.664695 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:07.661606    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv"
Nov 03 19:45:07.664695 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:07.661681    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7"
Nov 03 19:45:07.718748 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:07.714088    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6" podUID="1aaee61d-b036-4c2b-bee7-a692b8bed612" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:07.748792 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:07.748750    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt" podUID="17dc0ed2-5164-449d-be12-a38bdfeb946d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:07.832715 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:07.831740    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw" podUID="72d96abc-f3b4-4eda-b3a1-b93ba658e1d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:07.853698 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:07.838395    4921 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 03 19:45:07.853698 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:07.838445    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5" podUID="0da622a1-45f9-4311-b668-d27eb7ec2e43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 03 19:45:07.880522 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:07.880488    4921 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 03 19:45:07.880775 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:07.880748    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6" podUID="7e96267a-d738-4325-90e7-d1c9ce87eeb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 03 19:45:08.111695 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:08.111644    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9"
Nov 03 19:45:08.548605 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:08.548511    4921 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 03 19:45:08.548605 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:08.548561    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m" podUID="ee6baa84-41ee-4bee-98c9-a9d0f078d4b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 03 19:45:08.554975 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:08.549174    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9"
Nov 03 19:45:08.571815 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:08.571771    4921 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7" podUID="d4de0090-d775-4bd7-9ee2-058b5d380638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:08.651170 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:08.650629    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw"
Nov 03 19:45:08.672004 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:08.670178    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7" podUID="d4de0090-d775-4bd7-9ee2-058b5d380638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded"
Nov 03 19:45:08.690693 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:08.689867    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6"
Nov 03 19:45:08.723692 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:08.722025    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv" podUID="96b73342-0079-4709-8570-3dbce57e5243" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:08.750734 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:08.749752    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt" podUID="17dc0ed2-5164-449d-be12-a38bdfeb946d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:08.750734 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:08.750113    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt"
Nov 03 19:45:08.840032 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:08.840001    4921 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 03 19:45:08.840243 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:08.840217    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5" podUID="0da622a1-45f9-4311-b668-d27eb7ec2e43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 03 19:45:08.943699 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:08.928786    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7"
Nov 03 19:45:08.959775 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:08.945816    4921 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 03 19:45:08.960049 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:08.960015    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6" podUID="7e96267a-d738-4325-90e7-d1c9ce87eeb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 03 19:45:09.195225 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:09.195132    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv" podUID="01c749b9-d188-4673-9a3d-99b9c4ae0793" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:09.416688 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:09.416028    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk" event={"ID":"2d0a934e-4816-44d2-9bb5-e8b4e95dd73c","Type":"ContainerStarted","Data":"952987dd0924fcc54c37caca6862e3f205ef1dc433eec329287564fcf713af06"}
Nov 03 19:45:09.438377 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:09.438340    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh" event={"ID":"eac6c47d-8ae5-4e4b-b691-98d177887767","Type":"ContainerStarted","Data":"8a18fbce0741dd69abce8924aac85d552641c9371e318c16b68052cdfe928ce4"}
Nov 03 19:45:09.444938 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:09.444912    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv"
Nov 03 19:45:09.531079 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:09.531046    4921 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 03 19:45:09.532099 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:09.532062    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m" podUID="ee6baa84-41ee-4bee-98c9-a9d0f078d4b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 03 19:45:09.549701 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:09.548008    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk" podStartSLOduration=7.547990561 podStartE2EDuration="7.547990561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:09.445536931 +0000 UTC m=+822.689502397" watchObservedRunningTime="2026-11-03 19:45:09.547990561 +0000 UTC m=+822.791956055"
Nov 03 19:45:09.642101 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:09.642061    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh"
Nov 03 19:45:09.642482 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:09.642443    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk"
Nov 03 19:45:09.647715 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:09.647648    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh" podStartSLOduration=5.647629815 podStartE2EDuration="5.647629815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:09.531622418 +0000 UTC m=+822.775587884" watchObservedRunningTime="2026-11-03 19:45:09.647629815 +0000 UTC m=+822.891595282"
Nov 03 19:45:09.819997 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:09.797090    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt" podUID="17dc0ed2-5164-449d-be12-a38bdfeb946d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:09.847510 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:09.847466    4921 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 03 19:45:09.847810 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:09.847765    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5" podUID="0da622a1-45f9-4311-b668-d27eb7ec2e43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 03 19:45:09.931706 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:09.883491    4921 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 03 19:45:09.941845 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:09.941797    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6" podUID="7e96267a-d738-4325-90e7-d1c9ce87eeb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 03 19:45:10.046781 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:10.046553    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9"
Nov 03 19:45:10.501692 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:10.501414    4921 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 03 19:45:10.501692 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:10.501489    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m" podUID="ee6baa84-41ee-4bee-98c9-a9d0f078d4b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 03 19:45:10.746695 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:10.739478    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt"
Nov 03 19:45:10.854760 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:10.842413    4921 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 03 19:45:10.854760 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:10.842474    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5" podUID="0da622a1-45f9-4311-b668-d27eb7ec2e43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 03 19:45:11.034874 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:11.034837    4921 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 03 19:45:11.035154 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:11.035120    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6" podUID="7e96267a-d738-4325-90e7-d1c9ce87eeb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 03 19:45:11.046947 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:11.046908    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9" podUID="e6a71aa3-33b3-42bd-b270-64ce3260cc2a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:11.516721 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:11.516687    4921 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 03 19:45:11.517043 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:11.517019    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m" podUID="ee6baa84-41ee-4bee-98c9-a9d0f078d4b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 03 19:45:11.746969 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:11.746926    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt" podUID="17dc0ed2-5164-449d-be12-a38bdfeb946d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Nov 03 19:45:11.747627 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:11.747194    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv" podUID="96b73342-0079-4709-8570-3dbce57e5243" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:11.748773 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:11.747229    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw" podUID="72d96abc-f3b4-4eda-b3a1-b93ba658e1d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:11.770334 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:11.747257    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6" podUID="1aaee61d-b036-4c2b-bee7-a692b8bed612" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:11.842948 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:11.842915    4921 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 03 19:45:11.843248 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:11.843219    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5" podUID="0da622a1-45f9-4311-b668-d27eb7ec2e43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 03 19:45:11.876941 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:11.876904    4921 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 03 19:45:11.877214 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:11.877185    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6" podUID="7e96267a-d738-4325-90e7-d1c9ce87eeb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 03 19:45:12.502075 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:12.502040    4921 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 03 19:45:12.502646 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:12.502615    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m" podUID="ee6baa84-41ee-4bee-98c9-a9d0f078d4b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 03 19:45:12.842751 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:12.837179    4921 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 03 19:45:12.842751 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:12.837276    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5" podUID="0da622a1-45f9-4311-b668-d27eb7ec2e43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 03 19:45:12.880188 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:12.880152    4921 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 03 19:45:12.880468 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:12.880435    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6" podUID="7e96267a-d738-4325-90e7-d1c9ce87eeb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 03 19:45:13.502262 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:13.502222    4921 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Nov 03 19:45:13.502903 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:13.502872    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m" podUID="ee6baa84-41ee-4bee-98c9-a9d0f078d4b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Nov 03 19:45:13.551329 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:13.551295    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk"
Nov 03 19:45:13.564479 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:13.564455    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk"
Nov 03 19:45:13.847736 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:13.844940    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5"
Nov 03 19:45:13.847736 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:13.845960    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5"
Nov 03 19:45:13.853880 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:13.853855    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5"
Nov 03 19:45:13.876591 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:13.876562    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6"
Nov 03 19:45:13.877335 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:13.877310    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6"
Nov 03 19:45:13.878581 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:13.878560    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6"
Nov 03 19:45:14.506379 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:14.506351    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m"
Nov 03 19:45:14.506889 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:14.506871    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m"
Nov 03 19:45:14.507510 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:14.507490    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m"
Nov 03 19:45:14.558991 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:14.558962    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk"
Nov 03 19:45:14.881560 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:14.881479    4921 scope.go:122] "RemoveContainer" containerID="aab4e31e9de047148007553a3bce063ac8f9b954635fd21adb5f1240a1690b12"
Nov 03 19:45:15.554370 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:15.554330    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-b8tdx" event={"ID":"45523a35-8108-4813-b500-d0803ce6cc9c","Type":"ContainerStarted","Data":"5b300ae8973951be1665eabba96b540db6dabd47ff864b3eb3dfb038aca31f9a"}
Nov 03 19:45:15.584883 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:15.584850    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh"
Nov 03 19:45:15.600355 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:15.600261    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh"
Nov 03 19:45:16.565459 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:16.565427    4921 generic.go:356] "Generic (PLEG): container finished" podID="45523a35-8108-4813-b500-d0803ce6cc9c" containerID="5b300ae8973951be1665eabba96b540db6dabd47ff864b3eb3dfb038aca31f9a" exitCode=60
Nov 03 19:45:16.565967 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:16.565497    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-b8tdx" event={"ID":"45523a35-8108-4813-b500-d0803ce6cc9c","Type":"ContainerDied","Data":"5b300ae8973951be1665eabba96b540db6dabd47ff864b3eb3dfb038aca31f9a"}
Nov 03 19:45:16.565967 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:16.565572    4921 scope.go:122] "RemoveContainer" containerID="aab4e31e9de047148007553a3bce063ac8f9b954635fd21adb5f1240a1690b12"
Nov 03 19:45:16.566090 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:16.565971    4921 scope.go:122] "RemoveContainer" containerID="5b300ae8973951be1665eabba96b540db6dabd47ff864b3eb3dfb038aca31f9a"
Nov 03 19:45:16.566304 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:45:16.566279    4921 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895585-b8tdx_odh-ai-gateway-infra(45523a35-8108-4813-b500-d0803ce6cc9c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-b8tdx" podUID="45523a35-8108-4813-b500-d0803ce6cc9c"
Nov 03 19:45:16.578399 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:16.578374    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh"
Nov 03 19:45:16.579105 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:16.579086    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895585-b8tdx"]
Nov 03 19:45:16.895616 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:16.895534    4921 scope.go:122] "RemoveContainer" containerID="5fd067e0e3cd6d6c72e12575f642102f3ee4db813805918f249e89cb141ea7e0"
Nov 03 19:45:17.519882 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:17.519850    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz"]
Nov 03 19:45:17.524379 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:17.524359    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz"
Nov 03 19:45:17.534076 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:17.534053    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz"]
Nov 03 19:45:17.557469 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:17.557436    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52048b65-d34b-412d-a473-36c9c72ed8e9-istio-envoy\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz\" (UID: \"52048b65-d34b-412d-a473-36c9c72ed8e9\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz"
Nov 03 19:45:17.557650 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:17.557509    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52048b65-d34b-412d-a473-36c9c72ed8e9-istio-data\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz\" (UID: \"52048b65-d34b-412d-a473-36c9c72ed8e9\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz"
Nov 03 19:45:17.557786 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:17.557661    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52048b65-d34b-412d-a473-36c9c72ed8e9-workload-certs\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz\" (UID: \"52048b65-d34b-412d-a473-36c9c72ed8e9\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz"
Nov 03 19:45:17.557847 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:17.557788    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52048b65-d34b-412d-a473-36c9c72ed8e9-istio-token\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz\" (UID: \"52048b65-d34b-412d-a473-36c9c72ed8e9\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz"
Nov 03 19:45:17.557909 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:17.557842    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52048b65-d34b-412d-a473-36c9c72ed8e9-credential-socket\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz\" (UID: \"52048b65-d34b-412d-a473-36c9c72ed8e9\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz"
Nov 03 19:45:17.557965 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:17.557923    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52048b65-d34b-412d-a473-36c9c72ed8e9-istio-podinfo\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz\" (UID: \"52048b65-d34b-412d-a473-36c9c72ed8e9\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz"
Nov 03 19:45:17.558025 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:17.557968    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s28hv\" (UniqueName: \"kubernetes.io/projected/52048b65-d34b-412d-a473-36c9c72ed8e9-kube-api-access-s28hv\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz\" (UID: \"52048b65-d34b-412d-a473-36c9c72ed8e9\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz"
Nov 03 19:45:17.558025 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:17.558003    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52048b65-d34b-412d-a473-36c9c72ed8e9-workload-socket\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz\" (UID: \"52048b65-d34b-412d-a473-36c9c72ed8e9\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz"
Nov 03 19:45:17.558135 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:17.558028    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52048b65-d34b-412d-a473-36c9c72ed8e9-istiod-ca-cert\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz\" (UID: \"52048b65-d34b-412d-a473-36c9c72ed8e9\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz"
Nov 03 19:45:17.583773 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:17.581340    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d4127-29895585-lwd9p" event={"ID":"e2aa77e1-ea3d-4db2-84b6-da15cf42d047","Type":"ContainerStarted","Data":"9fba3e5e5d7aeddad7fb7fc7ab9a7563645912ca7db832d4ff53d2310c2610d2"}
Nov 03 19:45:17.663076 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:17.663013    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52048b65-d34b-412d-a473-36c9c72ed8e9-istio-podinfo\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz\" (UID: \"52048b65-d34b-412d-a473-36c9c72ed8e9\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz"
Nov 03 19:45:17.663773 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:17.663751    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s28hv\" (UniqueName: \"kubernetes.io/projected/52048b65-d34b-412d-a473-36c9c72ed8e9-kube-api-access-s28hv\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz\" (UID: \"52048b65-d34b-412d-a473-36c9c72ed8e9\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz"
Nov 03 19:45:17.664249 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:17.664228    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52048b65-d34b-412d-a473-36c9c72ed8e9-workload-socket\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz\" (UID: \"52048b65-d34b-412d-a473-36c9c72ed8e9\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz"
Nov 03 19:45:17.664754 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:17.664737    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52048b65-d34b-412d-a473-36c9c72ed8e9-istiod-ca-cert\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz\" (UID: \"52048b65-d34b-412d-a473-36c9c72ed8e9\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz"
Nov 03 19:45:17.667007 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:17.666980    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52048b65-d34b-412d-a473-36c9c72ed8e9-istio-envoy\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz\" (UID: \"52048b65-d34b-412d-a473-36c9c72ed8e9\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz"
Nov 03 19:45:17.668911 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:17.664697    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52048b65-d34b-412d-a473-36c9c72ed8e9-workload-socket\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz\" (UID: \"52048b65-d34b-412d-a473-36c9c72ed8e9\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz"
Nov 03 19:45:17.668911 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:17.665748    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52048b65-d34b-412d-a473-36c9c72ed8e9-istiod-ca-cert\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz\" (UID: \"52048b65-d34b-412d-a473-36c9c72ed8e9\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz"
Nov 03 19:45:17.668911 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:17.668897    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52048b65-d34b-412d-a473-36c9c72ed8e9-istio-podinfo\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz\" (UID: \"52048b65-d34b-412d-a473-36c9c72ed8e9\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz"
Nov 03 19:45:17.669335 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:17.669317    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52048b65-d34b-412d-a473-36c9c72ed8e9-istio-data\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz\" (UID: \"52048b65-d34b-412d-a473-36c9c72ed8e9\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz"
Nov 03 19:45:17.669789 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:17.669689    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52048b65-d34b-412d-a473-36c9c72ed8e9-istio-data\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz\" (UID: \"52048b65-d34b-412d-a473-36c9c72ed8e9\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz"
Nov 03 19:45:17.669920 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:17.669838    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52048b65-d34b-412d-a473-36c9c72ed8e9-workload-certs\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz\" (UID: \"52048b65-d34b-412d-a473-36c9c72ed8e9\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz"
Nov 03 19:45:17.670003 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:17.669952    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52048b65-d34b-412d-a473-36c9c72ed8e9-istio-token\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz\" (UID: \"52048b65-d34b-412d-a473-36c9c72ed8e9\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz"
Nov 03 19:45:17.670087 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:17.670008    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52048b65-d34b-412d-a473-36c9c72ed8e9-credential-socket\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz\" (UID: \"52048b65-d34b-412d-a473-36c9c72ed8e9\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz"
Nov 03 19:45:17.670460 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:17.670440    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52048b65-d34b-412d-a473-36c9c72ed8e9-workload-certs\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz\" (UID: \"52048b65-d34b-412d-a473-36c9c72ed8e9\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz"
Nov 03 19:45:17.670652 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:17.670627    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52048b65-d34b-412d-a473-36c9c72ed8e9-credential-socket\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz\" (UID: \"52048b65-d34b-412d-a473-36c9c72ed8e9\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz"
Nov 03 19:45:17.671244 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:17.670986    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s28hv\" (UniqueName: \"kubernetes.io/projected/52048b65-d34b-412d-a473-36c9c72ed8e9-kube-api-access-s28hv\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz\" (UID: \"52048b65-d34b-412d-a473-36c9c72ed8e9\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz"
Nov 03 19:45:17.673301 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:17.673277    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52048b65-d34b-412d-a473-36c9c72ed8e9-istio-envoy\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz\" (UID: \"52048b65-d34b-412d-a473-36c9c72ed8e9\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz"
Nov 03 19:45:17.676414 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:17.676388    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52048b65-d34b-412d-a473-36c9c72ed8e9-istio-token\") pod \"e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz\" (UID: \"52048b65-d34b-412d-a473-36c9c72ed8e9\") " pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz"
Nov 03 19:45:17.749244 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:17.749216    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-b8tdx"
Nov 03 19:45:17.840821 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:17.840748    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz"
Nov 03 19:45:17.872296 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:17.872271    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45523a35-8108-4813-b500-d0803ce6cc9c-kube-api-access-8jv4b\" (UniqueName: \"kubernetes.io/projected/45523a35-8108-4813-b500-d0803ce6cc9c-kube-api-access-8jv4b\") pod \"45523a35-8108-4813-b500-d0803ce6cc9c\" (UID: \"45523a35-8108-4813-b500-d0803ce6cc9c\") "
Nov 03 19:45:17.872386 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:17.872310    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45523a35-8108-4813-b500-d0803ce6cc9c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45523a35-8108-4813-b500-d0803ce6cc9c-maas-api-service-ca\") pod \"45523a35-8108-4813-b500-d0803ce6cc9c\" (UID: \"45523a35-8108-4813-b500-d0803ce6cc9c\") "
Nov 03 19:45:17.872741 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:17.872715    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45523a35-8108-4813-b500-d0803ce6cc9c-maas-api-service-ca" pod "45523a35-8108-4813-b500-d0803ce6cc9c" (UID: "45523a35-8108-4813-b500-d0803ce6cc9c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:45:17.872878 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:17.872864    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45523a35-8108-4813-b500-d0803ce6cc9c-maas-api-service-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:17.874468 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:17.874446    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45523a35-8108-4813-b500-d0803ce6cc9c-kube-api-access-8jv4b" pod "45523a35-8108-4813-b500-d0803ce6cc9c" (UID: "45523a35-8108-4813-b500-d0803ce6cc9c"). InnerVolumeSpecName "kube-api-access-8jv4b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:45:17.881135 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:17.881114    4921 scope.go:122] "RemoveContainer" containerID="d0cd4699001644a81375346ae2fbc83165c92d9e47401894646629f6efda5000"
Nov 03 19:45:17.974067 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:17.974002    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jv4b\" (UniqueName: \"kubernetes.io/projected/45523a35-8108-4813-b500-d0803ce6cc9c-kube-api-access-8jv4b\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:18.173680 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:18.173629    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:45:18.173874 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:18.173837    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:45:18.173936 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:18.173902    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:45:18.174141 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:18.174120    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz"]
Nov 03 19:45:18.596762 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:18.596721    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-b8tdx" event={"ID":"45523a35-8108-4813-b500-d0803ce6cc9c","Type":"ContainerDied","Data":"0127599a3c9c39c1d22b8222cb739bc3d0d7f3467b210dc86b317d37fd5f12bf"}
Nov 03 19:45:18.596762 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:18.596724    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-b8tdx"
Nov 03 19:45:18.597313 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:18.596789    4921 scope.go:122] "RemoveContainer" containerID="5b300ae8973951be1665eabba96b540db6dabd47ff864b3eb3dfb038aca31f9a"
Nov 03 19:45:18.599647 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:18.599612    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0486e8-29895585-252sx" event={"ID":"980862b5-9e6d-42a6-b7f9-141205136ef8","Type":"ContainerStarted","Data":"1b7565989fa16834c00a3a8c6e3d53f24caa4c927e472f260730586e833611bf"}
Nov 03 19:45:18.601738 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:18.601712    4921 generic.go:356] "Generic (PLEG): container finished" podID="e2aa77e1-ea3d-4db2-84b6-da15cf42d047" containerID="9fba3e5e5d7aeddad7fb7fc7ab9a7563645912ca7db832d4ff53d2310c2610d2" exitCode=60
Nov 03 19:45:18.601852 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:18.601782    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d4127-29895585-lwd9p" event={"ID":"e2aa77e1-ea3d-4db2-84b6-da15cf42d047","Type":"ContainerDied","Data":"9fba3e5e5d7aeddad7fb7fc7ab9a7563645912ca7db832d4ff53d2310c2610d2"}
Nov 03 19:45:18.602187 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:18.602169    4921 scope.go:122] "RemoveContainer" containerID="9fba3e5e5d7aeddad7fb7fc7ab9a7563645912ca7db832d4ff53d2310c2610d2"
Nov 03 19:45:18.602497 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:45:18.602476    4921 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-8d4127-29895585-lwd9p_odh-ai-gateway-infra(e2aa77e1-ea3d-4db2-84b6-da15cf42d047)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d4127-29895585-lwd9p" podUID="e2aa77e1-ea3d-4db2-84b6-da15cf42d047"
Nov 03 19:45:18.603598 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:18.603574    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz" event={"ID":"52048b65-d34b-412d-a473-36c9c72ed8e9","Type":"ContainerStarted","Data":"bca7846d26837d58bf7508e56fd1531752fd28989919c1e93698fb045526dc39"}
Nov 03 19:45:18.603737 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:18.603620    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz" event={"ID":"52048b65-d34b-412d-a473-36c9c72ed8e9","Type":"ContainerStarted","Data":"a4654e12f76e6f82399c1cbb8204a0ae2ea8992bb92389c03cef2ccf0b8dd2aa"}
Nov 03 19:45:18.609018 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:18.608996    4921 scope.go:122] "RemoveContainer" containerID="5fd067e0e3cd6d6c72e12575f642102f3ee4db813805918f249e89cb141ea7e0"
Nov 03 19:45:18.610974 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:18.610944    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d4127-29895585-lwd9p"]
Nov 03 19:45:18.654275 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:18.654244    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895585-b8tdx"]
Nov 03 19:45:18.658031 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:18.658003    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895585-b8tdx"]
Nov 03 19:45:18.669279 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:18.669226    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz" podStartSLOduration=1.669212411 podStartE2EDuration="1.669212411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:18.666719977 +0000 UTC m=+831.910685443" watchObservedRunningTime="2026-11-03 19:45:18.669212411 +0000 UTC m=+831.913177930"
Nov 03 19:45:18.841380 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:18.841335    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz"
Nov 03 19:45:18.894823 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:18.894731    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45523a35-8108-4813-b500-d0803ce6cc9c" path="/var/lib/kubelet/pods/45523a35-8108-4813-b500-d0803ce6cc9c/volumes"
Nov 03 19:45:19.612703 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:19.612643    4921 generic.go:356] "Generic (PLEG): container finished" podID="980862b5-9e6d-42a6-b7f9-141205136ef8" containerID="1b7565989fa16834c00a3a8c6e3d53f24caa4c927e472f260730586e833611bf" exitCode=60
Nov 03 19:45:19.613141 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:19.612718    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0486e8-29895585-252sx" event={"ID":"980862b5-9e6d-42a6-b7f9-141205136ef8","Type":"ContainerDied","Data":"1b7565989fa16834c00a3a8c6e3d53f24caa4c927e472f260730586e833611bf"}
Nov 03 19:45:19.613141 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:19.612776    4921 scope.go:122] "RemoveContainer" containerID="d0cd4699001644a81375346ae2fbc83165c92d9e47401894646629f6efda5000"
Nov 03 19:45:19.613281 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:19.613174    4921 scope.go:122] "RemoveContainer" containerID="1b7565989fa16834c00a3a8c6e3d53f24caa4c927e472f260730586e833611bf"
Nov 03 19:45:19.613539 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:45:19.613509    4921 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-0486e8-29895585-252sx_odh-ai-gateway-infra(980862b5-9e6d-42a6-b7f9-141205136ef8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0486e8-29895585-252sx" podUID="980862b5-9e6d-42a6-b7f9-141205136ef8"
Nov 03 19:45:19.627280 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:19.627253    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0486e8-29895585-252sx"]
Nov 03 19:45:19.843660 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:19.843622    4921 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Nov 03 19:45:19.843820 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:19.843705    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz" podUID="52048b65-d34b-412d-a473-36c9c72ed8e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Nov 03 19:45:19.887145 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:19.887123    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d4127-29895585-lwd9p"
Nov 03 19:45:19.911041 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:19.911009    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2aa77e1-ea3d-4db2-84b6-da15cf42d047-kube-api-access-8vns6\" (UniqueName: \"kubernetes.io/projected/e2aa77e1-ea3d-4db2-84b6-da15cf42d047-kube-api-access-8vns6\") pod \"e2aa77e1-ea3d-4db2-84b6-da15cf42d047\" (UID: \"e2aa77e1-ea3d-4db2-84b6-da15cf42d047\") "
Nov 03 19:45:19.911180 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:19.911163    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2aa77e1-ea3d-4db2-84b6-da15cf42d047-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2aa77e1-ea3d-4db2-84b6-da15cf42d047-maas-api-service-ca\") pod \"e2aa77e1-ea3d-4db2-84b6-da15cf42d047\" (UID: \"e2aa77e1-ea3d-4db2-84b6-da15cf42d047\") "
Nov 03 19:45:19.911592 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:19.911561    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2aa77e1-ea3d-4db2-84b6-da15cf42d047-maas-api-service-ca" pod "e2aa77e1-ea3d-4db2-84b6-da15cf42d047" (UID: "e2aa77e1-ea3d-4db2-84b6-da15cf42d047"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:45:19.911739 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:19.911709    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2aa77e1-ea3d-4db2-84b6-da15cf42d047-maas-api-service-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:19.913486 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:19.913457    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2aa77e1-ea3d-4db2-84b6-da15cf42d047-kube-api-access-8vns6" pod "e2aa77e1-ea3d-4db2-84b6-da15cf42d047" (UID: "e2aa77e1-ea3d-4db2-84b6-da15cf42d047"). InnerVolumeSpecName "kube-api-access-8vns6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:45:20.012791 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:20.012757    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vns6\" (UniqueName: \"kubernetes.io/projected/e2aa77e1-ea3d-4db2-84b6-da15cf42d047-kube-api-access-8vns6\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:20.626962 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:20.626936    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d4127-29895585-lwd9p"
Nov 03 19:45:20.627352 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:20.626963    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d4127-29895585-lwd9p" event={"ID":"e2aa77e1-ea3d-4db2-84b6-da15cf42d047","Type":"ContainerDied","Data":"4533638d961e31842ef7804cb9b65151028521c850d9638780f545cf23386011"}
Nov 03 19:45:20.627352 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:20.627030    4921 scope.go:122] "RemoveContainer" containerID="9fba3e5e5d7aeddad7fb7fc7ab9a7563645912ca7db832d4ff53d2310c2610d2"
Nov 03 19:45:20.681527 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:20.681490    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d4127-29895585-lwd9p"]
Nov 03 19:45:20.685158 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:20.685129    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d4127-29895585-lwd9p"]
Nov 03 19:45:20.838018 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:20.837992    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0486e8-29895585-252sx"
Nov 03 19:45:20.841054 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:20.841031    4921 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Nov 03 19:45:20.841123 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:20.841078    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz" podUID="52048b65-d34b-412d-a473-36c9c72ed8e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Nov 03 19:45:20.903601 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:20.903166    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2aa77e1-ea3d-4db2-84b6-da15cf42d047" path="/var/lib/kubelet/pods/e2aa77e1-ea3d-4db2-84b6-da15cf42d047/volumes"
Nov 03 19:45:20.939058 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:20.939023    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/980862b5-9e6d-42a6-b7f9-141205136ef8-kube-api-access-hzkvd\" (UniqueName: \"kubernetes.io/projected/980862b5-9e6d-42a6-b7f9-141205136ef8-kube-api-access-hzkvd\") pod \"980862b5-9e6d-42a6-b7f9-141205136ef8\" (UID: \"980862b5-9e6d-42a6-b7f9-141205136ef8\") "
Nov 03 19:45:20.939230 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:20.939175    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/980862b5-9e6d-42a6-b7f9-141205136ef8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/980862b5-9e6d-42a6-b7f9-141205136ef8-maas-api-service-ca\") pod \"980862b5-9e6d-42a6-b7f9-141205136ef8\" (UID: \"980862b5-9e6d-42a6-b7f9-141205136ef8\") "
Nov 03 19:45:20.939521 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:20.939491    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/980862b5-9e6d-42a6-b7f9-141205136ef8-maas-api-service-ca" pod "980862b5-9e6d-42a6-b7f9-141205136ef8" (UID: "980862b5-9e6d-42a6-b7f9-141205136ef8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:45:20.941134 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:20.941108    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/980862b5-9e6d-42a6-b7f9-141205136ef8-kube-api-access-hzkvd" pod "980862b5-9e6d-42a6-b7f9-141205136ef8" (UID: "980862b5-9e6d-42a6-b7f9-141205136ef8"). InnerVolumeSpecName "kube-api-access-hzkvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:45:21.041068 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:21.041034    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/980862b5-9e6d-42a6-b7f9-141205136ef8-maas-api-service-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:21.041068 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:21.041065    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzkvd\" (UniqueName: \"kubernetes.io/projected/980862b5-9e6d-42a6-b7f9-141205136ef8-kube-api-access-hzkvd\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:21.635922 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:21.635894    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0486e8-29895585-252sx"
Nov 03 19:45:21.637710 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:21.636556    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0486e8-29895585-252sx" event={"ID":"980862b5-9e6d-42a6-b7f9-141205136ef8","Type":"ContainerDied","Data":"4aa3308e8982077dddd3a602a18358fe2ea4af900390aeaff2774b66cf7cf5c8"}
Nov 03 19:45:21.637710 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:21.636626    4921 scope.go:122] "RemoveContainer" containerID="1b7565989fa16834c00a3a8c6e3d53f24caa4c927e472f260730586e833611bf"
Nov 03 19:45:21.680531 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:21.680494    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0486e8-29895585-252sx"]
Nov 03 19:45:21.684227 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:21.684202    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-0486e8-29895585-252sx"]
Nov 03 19:45:21.841774 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:21.841741    4921 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Nov 03 19:45:21.841941 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:21.841794    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz" podUID="52048b65-d34b-412d-a473-36c9c72ed8e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Nov 03 19:45:22.841332 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:22.841294    4921 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Nov 03 19:45:22.841769 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:22.841350    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz" podUID="52048b65-d34b-412d-a473-36c9c72ed8e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Nov 03 19:45:22.888871 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:22.888831    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="980862b5-9e6d-42a6-b7f9-141205136ef8" path="/var/lib/kubelet/pods/980862b5-9e6d-42a6-b7f9-141205136ef8/volumes"
Nov 03 19:45:23.842980 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:23.842947    4921 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Nov 03 19:45:23.843734 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:23.843695    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz" podUID="52048b65-d34b-412d-a473-36c9c72ed8e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Nov 03 19:45:24.841628 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:24.841163    4921 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Nov 03 19:45:24.841628 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:24.841238    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz" podUID="52048b65-d34b-412d-a473-36c9c72ed8e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Nov 03 19:45:25.889704 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:25.887614    4921 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Nov 03 19:45:25.889704 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:25.887744    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz" podUID="52048b65-d34b-412d-a473-36c9c72ed8e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Nov 03 19:45:26.676705 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:26.567255    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk" podUID="2d0a934e-4816-44d2-9bb5-e8b4e95dd73c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Nov 03 19:45:26.709891 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:26.569629    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh" podUID="eac6c47d-8ae5-4e4b-b691-98d177887767" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded"
Nov 03 19:45:26.741092 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:26.728772    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6" podUID="1aaee61d-b036-4c2b-bee7-a692b8bed612" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:26.768732 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:26.743773    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv" podUID="96b73342-0079-4709-8570-3dbce57e5243" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Nov 03 19:45:26.768953 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:26.768891    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt" podUID="17dc0ed2-5164-449d-be12-a38bdfeb946d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:26.945015 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:26.944910    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7" podUID="d4de0090-d775-4bd7-9ee2-058b5d380638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:26.946200 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:26.946124    4921 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Nov 03 19:45:26.949782 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:26.949742    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz" podUID="52048b65-d34b-412d-a473-36c9c72ed8e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Nov 03 19:45:27.087031 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:27.085539    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9" podUID="e6a71aa3-33b3-42bd-b270-64ce3260cc2a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:27.215703 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:27.189580    4921 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:45:27.215703 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:27.189639    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt" podUID="ffb30cac-fb9f-4a3f-b954-ecd050b8cb41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:27.421655 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:27.421612    4921 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6" podUID="1aaee61d-b036-4c2b-bee7-a692b8bed612" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:27.592219 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:27.592143    4921 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:45:27.592433 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:27.592226    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr" podUID="c05f083e-3052-44f9-ac8f-746ea890191b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:28.883689 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:28.882850    4921 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:45:28.883689 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:28.882926    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz" podUID="52048b65-d34b-412d-a473-36c9c72ed8e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:29.214471 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:29.214383    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz"
Nov 03 19:45:29.214936 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:29.214899    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz"
Nov 03 19:45:29.215879 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:29.215858    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz"
Nov 03 19:45:32.649081 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:32.649049    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j"]
Nov 03 19:45:32.649709 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:32.649650    4921 cpu_manager.go:414] "containerMap: removing container" podUID="e2aa77e1-ea3d-4db2-84b6-da15cf42d047" containerName="cleanup"
Nov 03 19:45:32.649709 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:32.649681    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2aa77e1-ea3d-4db2-84b6-da15cf42d047" containerName="cleanup"
Nov 03 19:45:32.649709 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:32.649706    4921 cpu_manager.go:414] "containerMap: removing container" podUID="45523a35-8108-4813-b500-d0803ce6cc9c" containerName="cleanup"
Nov 03 19:45:32.649923 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:32.649715    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45523a35-8108-4813-b500-d0803ce6cc9c" containerName="cleanup"
Nov 03 19:45:32.649923 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:32.649731    4921 cpu_manager.go:414] "containerMap: removing container" podUID="980862b5-9e6d-42a6-b7f9-141205136ef8" containerName="cleanup"
Nov 03 19:45:32.649923 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:32.649738    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="980862b5-9e6d-42a6-b7f9-141205136ef8" containerName="cleanup"
Nov 03 19:45:32.649923 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:32.649747    4921 cpu_manager.go:414] "containerMap: removing container" podUID="45523a35-8108-4813-b500-d0803ce6cc9c" containerName="cleanup"
Nov 03 19:45:32.649923 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:32.649754    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45523a35-8108-4813-b500-d0803ce6cc9c" containerName="cleanup"
Nov 03 19:45:32.649923 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:32.649770    4921 cpu_manager.go:414] "containerMap: removing container" podUID="980862b5-9e6d-42a6-b7f9-141205136ef8" containerName="cleanup"
Nov 03 19:45:32.649923 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:32.649777    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="980862b5-9e6d-42a6-b7f9-141205136ef8" containerName="cleanup"
Nov 03 19:45:32.649923 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:32.649796    4921 cpu_manager.go:414] "containerMap: removing container" podUID="e2aa77e1-ea3d-4db2-84b6-da15cf42d047" containerName="cleanup"
Nov 03 19:45:32.649923 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:32.649803    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2aa77e1-ea3d-4db2-84b6-da15cf42d047" containerName="cleanup"
Nov 03 19:45:32.649923 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:32.649814    4921 cpu_manager.go:414] "containerMap: removing container" podUID="980862b5-9e6d-42a6-b7f9-141205136ef8" containerName="cleanup"
Nov 03 19:45:32.649923 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:32.649821    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="980862b5-9e6d-42a6-b7f9-141205136ef8" containerName="cleanup"
Nov 03 19:45:32.650405 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:32.649970    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2aa77e1-ea3d-4db2-84b6-da15cf42d047" containerName="cleanup"
Nov 03 19:45:32.650405 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:32.649982    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="45523a35-8108-4813-b500-d0803ce6cc9c" containerName="cleanup"
Nov 03 19:45:32.650405 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:32.649994    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="980862b5-9e6d-42a6-b7f9-141205136ef8" containerName="cleanup"
Nov 03 19:45:32.650405 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:32.650002    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="45523a35-8108-4813-b500-d0803ce6cc9c" containerName="cleanup"
Nov 03 19:45:32.650405 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:32.650010    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="980862b5-9e6d-42a6-b7f9-141205136ef8" containerName="cleanup"
Nov 03 19:45:32.650405 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:32.650022    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2aa77e1-ea3d-4db2-84b6-da15cf42d047" containerName="cleanup"
Nov 03 19:45:32.658687 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:32.655795    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j"
Nov 03 19:45:32.677353 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:32.677323    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j"]
Nov 03 19:45:32.800751 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:32.800711    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e78a74a9-c32a-4047-a64f-b344f8ebaf22-workload-socket\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j\" (UID: \"e78a74a9-c32a-4047-a64f-b344f8ebaf22\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j"
Nov 03 19:45:32.800921 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:32.800818    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e78a74a9-c32a-4047-a64f-b344f8ebaf22-istio-envoy\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j\" (UID: \"e78a74a9-c32a-4047-a64f-b344f8ebaf22\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j"
Nov 03 19:45:32.800921 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:32.800912    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e78a74a9-c32a-4047-a64f-b344f8ebaf22-workload-certs\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j\" (UID: \"e78a74a9-c32a-4047-a64f-b344f8ebaf22\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j"
Nov 03 19:45:32.801013 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:32.800947    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fczx\" (UniqueName: \"kubernetes.io/projected/e78a74a9-c32a-4047-a64f-b344f8ebaf22-kube-api-access-8fczx\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j\" (UID: \"e78a74a9-c32a-4047-a64f-b344f8ebaf22\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j"
Nov 03 19:45:32.801013 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:32.800993    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e78a74a9-c32a-4047-a64f-b344f8ebaf22-istiod-ca-cert\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j\" (UID: \"e78a74a9-c32a-4047-a64f-b344f8ebaf22\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j"
Nov 03 19:45:32.801112 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:32.801094    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e78a74a9-c32a-4047-a64f-b344f8ebaf22-istio-token\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j\" (UID: \"e78a74a9-c32a-4047-a64f-b344f8ebaf22\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j"
Nov 03 19:45:32.801152 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:32.801135    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e78a74a9-c32a-4047-a64f-b344f8ebaf22-istio-podinfo\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j\" (UID: \"e78a74a9-c32a-4047-a64f-b344f8ebaf22\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j"
Nov 03 19:45:32.801197 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:32.801157    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e78a74a9-c32a-4047-a64f-b344f8ebaf22-credential-socket\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j\" (UID: \"e78a74a9-c32a-4047-a64f-b344f8ebaf22\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j"
Nov 03 19:45:32.801311 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:32.801290    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e78a74a9-c32a-4047-a64f-b344f8ebaf22-istio-data\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j\" (UID: \"e78a74a9-c32a-4047-a64f-b344f8ebaf22\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j"
Nov 03 19:45:32.908751 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:32.908251    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e78a74a9-c32a-4047-a64f-b344f8ebaf22-workload-socket\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j\" (UID: \"e78a74a9-c32a-4047-a64f-b344f8ebaf22\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j"
Nov 03 19:45:32.908751 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:32.908322    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e78a74a9-c32a-4047-a64f-b344f8ebaf22-istio-envoy\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j\" (UID: \"e78a74a9-c32a-4047-a64f-b344f8ebaf22\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j"
Nov 03 19:45:32.908751 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:32.908375    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e78a74a9-c32a-4047-a64f-b344f8ebaf22-workload-certs\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j\" (UID: \"e78a74a9-c32a-4047-a64f-b344f8ebaf22\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j"
Nov 03 19:45:32.908751 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:32.908417    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fczx\" (UniqueName: \"kubernetes.io/projected/e78a74a9-c32a-4047-a64f-b344f8ebaf22-kube-api-access-8fczx\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j\" (UID: \"e78a74a9-c32a-4047-a64f-b344f8ebaf22\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j"
Nov 03 19:45:32.908751 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:32.908495    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e78a74a9-c32a-4047-a64f-b344f8ebaf22-istiod-ca-cert\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j\" (UID: \"e78a74a9-c32a-4047-a64f-b344f8ebaf22\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j"
Nov 03 19:45:32.908751 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:32.908554    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e78a74a9-c32a-4047-a64f-b344f8ebaf22-istio-token\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j\" (UID: \"e78a74a9-c32a-4047-a64f-b344f8ebaf22\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j"
Nov 03 19:45:32.908751 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:32.908602    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e78a74a9-c32a-4047-a64f-b344f8ebaf22-istio-podinfo\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j\" (UID: \"e78a74a9-c32a-4047-a64f-b344f8ebaf22\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j"
Nov 03 19:45:32.908751 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:32.908634    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e78a74a9-c32a-4047-a64f-b344f8ebaf22-credential-socket\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j\" (UID: \"e78a74a9-c32a-4047-a64f-b344f8ebaf22\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j"
Nov 03 19:45:32.908751 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:32.908684    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e78a74a9-c32a-4047-a64f-b344f8ebaf22-istio-data\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j\" (UID: \"e78a74a9-c32a-4047-a64f-b344f8ebaf22\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j"
Nov 03 19:45:32.911741 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:32.909938    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e78a74a9-c32a-4047-a64f-b344f8ebaf22-istio-data\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j\" (UID: \"e78a74a9-c32a-4047-a64f-b344f8ebaf22\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j"
Nov 03 19:45:32.911741 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:32.910635    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e78a74a9-c32a-4047-a64f-b344f8ebaf22-istiod-ca-cert\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j\" (UID: \"e78a74a9-c32a-4047-a64f-b344f8ebaf22\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j"
Nov 03 19:45:32.911741 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:32.911072    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e78a74a9-c32a-4047-a64f-b344f8ebaf22-workload-certs\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j\" (UID: \"e78a74a9-c32a-4047-a64f-b344f8ebaf22\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j"
Nov 03 19:45:32.911741 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:32.911294    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e78a74a9-c32a-4047-a64f-b344f8ebaf22-workload-socket\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j\" (UID: \"e78a74a9-c32a-4047-a64f-b344f8ebaf22\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j"
Nov 03 19:45:32.911741 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:32.911318    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e78a74a9-c32a-4047-a64f-b344f8ebaf22-credential-socket\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j\" (UID: \"e78a74a9-c32a-4047-a64f-b344f8ebaf22\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j"
Nov 03 19:45:32.924840 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:32.924275    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e78a74a9-c32a-4047-a64f-b344f8ebaf22-istio-podinfo\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j\" (UID: \"e78a74a9-c32a-4047-a64f-b344f8ebaf22\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j"
Nov 03 19:45:32.924840 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:32.924478    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e78a74a9-c32a-4047-a64f-b344f8ebaf22-istio-envoy\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j\" (UID: \"e78a74a9-c32a-4047-a64f-b344f8ebaf22\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j"
Nov 03 19:45:32.927836 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:32.927806    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fczx\" (UniqueName: \"kubernetes.io/projected/e78a74a9-c32a-4047-a64f-b344f8ebaf22-kube-api-access-8fczx\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j\" (UID: \"e78a74a9-c32a-4047-a64f-b344f8ebaf22\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j"
Nov 03 19:45:32.928834 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:32.928808    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e78a74a9-c32a-4047-a64f-b344f8ebaf22-istio-token\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j\" (UID: \"e78a74a9-c32a-4047-a64f-b344f8ebaf22\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j"
Nov 03 19:45:32.954156 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:45:32.954119    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 19:45:33.018127 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:33.018094    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j"
Nov 03 19:45:33.321968 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:33.321938    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j"]
Nov 03 19:45:33.323879 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:45:33.323849    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode78a74a9_c32a_4047_a64f_b344f8ebaf22.slice/crio-ee6669113bbc031960a4954c13ee2c62c30a95cbf0a00d493db490db2e9ce332 WatchSource:0}: Error finding container ee6669113bbc031960a4954c13ee2c62c30a95cbf0a00d493db490db2e9ce332: Status 404 returned error can't find the container with id ee6669113bbc031960a4954c13ee2c62c30a95cbf0a00d493db490db2e9ce332
Nov 03 19:45:33.326777 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:33.326742    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:45:33.326866 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:33.326808    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:45:33.326866 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:33.326836    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:45:34.009657 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:34.009617    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j" event={"ID":"e78a74a9-c32a-4047-a64f-b344f8ebaf22","Type":"ContainerStarted","Data":"1012d635dfd87be601767a1d96cda123a00bfddfd9ec6d2a97dcb84ac9f8171f"}
Nov 03 19:45:34.010079 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:34.009699    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j" event={"ID":"e78a74a9-c32a-4047-a64f-b344f8ebaf22","Type":"ContainerStarted","Data":"ee6669113bbc031960a4954c13ee2c62c30a95cbf0a00d493db490db2e9ce332"}
Nov 03 19:45:34.018879 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:34.018847    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j"
Nov 03 19:45:34.020266 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:34.020243    4921 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Nov 03 19:45:34.020382 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:34.020286    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j" podUID="e78a74a9-c32a-4047-a64f-b344f8ebaf22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Nov 03 19:45:35.018458 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:35.018428    4921 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Nov 03 19:45:35.018929 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:35.018905    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j" podUID="e78a74a9-c32a-4047-a64f-b344f8ebaf22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Nov 03 19:45:36.020536 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:36.018869    4921 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Nov 03 19:45:36.020536 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:36.018925    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j" podUID="e78a74a9-c32a-4047-a64f-b344f8ebaf22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Nov 03 19:45:37.018528 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:37.018487    4921 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Nov 03 19:45:37.018724 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:37.018552    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j" podUID="e78a74a9-c32a-4047-a64f-b344f8ebaf22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Nov 03 19:45:38.019388 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:38.019345    4921 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Nov 03 19:45:38.019800 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:38.019420    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j" podUID="e78a74a9-c32a-4047-a64f-b344f8ebaf22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Nov 03 19:45:39.019437 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:39.019397    4921 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Nov 03 19:45:39.019865 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:39.019478    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j" podUID="e78a74a9-c32a-4047-a64f-b344f8ebaf22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Nov 03 19:45:40.019474 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:40.019434    4921 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Nov 03 19:45:40.020032 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:40.019514    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j" podUID="e78a74a9-c32a-4047-a64f-b344f8ebaf22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Nov 03 19:45:41.023987 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:41.023949    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j"
Nov 03 19:45:41.024474 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:41.024448    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j"
Nov 03 19:45:41.026724 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:41.026701    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j"
Nov 03 19:45:41.046711 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:41.046610    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j" podStartSLOduration=9.046591956 podStartE2EDuration="9.046591956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:34.078074791 +0000 UTC m=+847.322040260" watchObservedRunningTime="2026-11-03 19:45:41.046591956 +0000 UTC m=+854.290557422"
Nov 03 19:45:47.674540 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.674509    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n"]
Nov 03 19:45:47.675162 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.675146    4921 cpu_manager.go:414] "containerMap: removing container" podUID="e2aa77e1-ea3d-4db2-84b6-da15cf42d047" containerName="cleanup"
Nov 03 19:45:47.675206 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.675164    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2aa77e1-ea3d-4db2-84b6-da15cf42d047" containerName="cleanup"
Nov 03 19:45:47.675206 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.675183    4921 cpu_manager.go:414] "containerMap: removing container" podUID="45523a35-8108-4813-b500-d0803ce6cc9c" containerName="cleanup"
Nov 03 19:45:47.675206 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.675189    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45523a35-8108-4813-b500-d0803ce6cc9c" containerName="cleanup"
Nov 03 19:45:47.675299 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.675272    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2aa77e1-ea3d-4db2-84b6-da15cf42d047" containerName="cleanup"
Nov 03 19:45:47.675299 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.675281    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="45523a35-8108-4813-b500-d0803ce6cc9c" containerName="cleanup"
Nov 03 19:45:47.675299 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.675289    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="980862b5-9e6d-42a6-b7f9-141205136ef8" containerName="cleanup"
Nov 03 19:45:47.678934 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.678916    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n"
Nov 03 19:45:47.693368 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.692306    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n"]
Nov 03 19:45:47.716947 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.716917    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt"]
Nov 03 19:45:47.722592 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.722565    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt"
Nov 03 19:45:47.734429 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.734400    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt"]
Nov 03 19:45:47.740874 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.740850    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww"]
Nov 03 19:45:47.746385 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.746361    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww"
Nov 03 19:45:47.756936 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.756906    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww"]
Nov 03 19:45:47.782186 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.782158    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ee1b6ae-046b-4402-a36e-bbd523e73463-workload-certs\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n\" (UID: \"4ee1b6ae-046b-4402-a36e-bbd523e73463\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n"
Nov 03 19:45:47.782318 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.782225    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdpzb\" (UniqueName: \"kubernetes.io/projected/4ee1b6ae-046b-4402-a36e-bbd523e73463-kube-api-access-wdpzb\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n\" (UID: \"4ee1b6ae-046b-4402-a36e-bbd523e73463\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n"
Nov 03 19:45:47.782318 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.782312    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ee1b6ae-046b-4402-a36e-bbd523e73463-istio-data\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n\" (UID: \"4ee1b6ae-046b-4402-a36e-bbd523e73463\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n"
Nov 03 19:45:47.782410 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.782383    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ee1b6ae-046b-4402-a36e-bbd523e73463-istio-token\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n\" (UID: \"4ee1b6ae-046b-4402-a36e-bbd523e73463\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n"
Nov 03 19:45:47.782447 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.782433    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ee1b6ae-046b-4402-a36e-bbd523e73463-credential-socket\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n\" (UID: \"4ee1b6ae-046b-4402-a36e-bbd523e73463\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n"
Nov 03 19:45:47.782491 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.782456    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ee1b6ae-046b-4402-a36e-bbd523e73463-workload-socket\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n\" (UID: \"4ee1b6ae-046b-4402-a36e-bbd523e73463\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n"
Nov 03 19:45:47.782527 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.782505    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ee1b6ae-046b-4402-a36e-bbd523e73463-istio-podinfo\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n\" (UID: \"4ee1b6ae-046b-4402-a36e-bbd523e73463\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n"
Nov 03 19:45:47.782577 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.782564    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ee1b6ae-046b-4402-a36e-bbd523e73463-istio-envoy\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n\" (UID: \"4ee1b6ae-046b-4402-a36e-bbd523e73463\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n"
Nov 03 19:45:47.782617 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.782596    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ee1b6ae-046b-4402-a36e-bbd523e73463-istiod-ca-cert\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n\" (UID: \"4ee1b6ae-046b-4402-a36e-bbd523e73463\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n"
Nov 03 19:45:47.884059 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.884023    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ee1b6ae-046b-4402-a36e-bbd523e73463-istiod-ca-cert\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n\" (UID: \"4ee1b6ae-046b-4402-a36e-bbd523e73463\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n"
Nov 03 19:45:47.884228 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.884073    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a52b2e9b-9ca0-4897-a071-a79c80e40d14-istio-token\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww\" (UID: \"a52b2e9b-9ca0-4897-a071-a79c80e40d14\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww"
Nov 03 19:45:47.884228 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.884124    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ee1b6ae-046b-4402-a36e-bbd523e73463-workload-certs\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n\" (UID: \"4ee1b6ae-046b-4402-a36e-bbd523e73463\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n"
Nov 03 19:45:47.884228 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.884139    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a52b2e9b-9ca0-4897-a071-a79c80e40d14-workload-certs\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww\" (UID: \"a52b2e9b-9ca0-4897-a071-a79c80e40d14\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww"
Nov 03 19:45:47.884228 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.884156    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a52b2e9b-9ca0-4897-a071-a79c80e40d14-istio-data\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww\" (UID: \"a52b2e9b-9ca0-4897-a071-a79c80e40d14\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww"
Nov 03 19:45:47.884228 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.884182    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a52b2e9b-9ca0-4897-a071-a79c80e40d14-workload-socket\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww\" (UID: \"a52b2e9b-9ca0-4897-a071-a79c80e40d14\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww"
Nov 03 19:45:47.884407 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.884279    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a52b2e9b-9ca0-4897-a071-a79c80e40d14-istio-envoy\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww\" (UID: \"a52b2e9b-9ca0-4897-a071-a79c80e40d14\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww"
Nov 03 19:45:47.884407 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.884300    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c2fc702-9338-4af0-a044-810fafccbfad-workload-socket\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt\" (UID: \"5c2fc702-9338-4af0-a044-810fafccbfad\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt"
Nov 03 19:45:47.884407 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.884342    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdpzb\" (UniqueName: \"kubernetes.io/projected/4ee1b6ae-046b-4402-a36e-bbd523e73463-kube-api-access-wdpzb\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n\" (UID: \"4ee1b6ae-046b-4402-a36e-bbd523e73463\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n"
Nov 03 19:45:47.884504 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.884414    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c2fc702-9338-4af0-a044-810fafccbfad-istiod-ca-cert\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt\" (UID: \"5c2fc702-9338-4af0-a044-810fafccbfad\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt"
Nov 03 19:45:47.884538 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.884516    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ee1b6ae-046b-4402-a36e-bbd523e73463-workload-certs\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n\" (UID: \"4ee1b6ae-046b-4402-a36e-bbd523e73463\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n"
Nov 03 19:45:47.884538 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.884526    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c2fc702-9338-4af0-a044-810fafccbfad-credential-socket\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt\" (UID: \"5c2fc702-9338-4af0-a044-810fafccbfad\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt"
Nov 03 19:45:47.884607 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.884592    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c2fc702-9338-4af0-a044-810fafccbfad-istio-envoy\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt\" (UID: \"5c2fc702-9338-4af0-a044-810fafccbfad\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt"
Nov 03 19:45:47.884641 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.884628    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ee1b6ae-046b-4402-a36e-bbd523e73463-istio-data\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n\" (UID: \"4ee1b6ae-046b-4402-a36e-bbd523e73463\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n"
Nov 03 19:45:47.884687 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.884661    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrnhr\" (UniqueName: \"kubernetes.io/projected/a52b2e9b-9ca0-4897-a071-a79c80e40d14-kube-api-access-lrnhr\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww\" (UID: \"a52b2e9b-9ca0-4897-a071-a79c80e40d14\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww"
Nov 03 19:45:47.884729 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.884711    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a52b2e9b-9ca0-4897-a071-a79c80e40d14-istio-podinfo\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww\" (UID: \"a52b2e9b-9ca0-4897-a071-a79c80e40d14\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww"
Nov 03 19:45:47.884780 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.884767    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c2fc702-9338-4af0-a044-810fafccbfad-workload-certs\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt\" (UID: \"5c2fc702-9338-4af0-a044-810fafccbfad\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt"
Nov 03 19:45:47.884828 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.884793    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ee1b6ae-046b-4402-a36e-bbd523e73463-istio-token\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n\" (UID: \"4ee1b6ae-046b-4402-a36e-bbd523e73463\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n"
Nov 03 19:45:47.884866 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.884825    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c2fc702-9338-4af0-a044-810fafccbfad-istio-data\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt\" (UID: \"5c2fc702-9338-4af0-a044-810fafccbfad\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt"
Nov 03 19:45:47.884866 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.884823    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ee1b6ae-046b-4402-a36e-bbd523e73463-istiod-ca-cert\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n\" (UID: \"4ee1b6ae-046b-4402-a36e-bbd523e73463\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n"
Nov 03 19:45:47.884866 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.884845    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czgdq\" (UniqueName: \"kubernetes.io/projected/5c2fc702-9338-4af0-a044-810fafccbfad-kube-api-access-czgdq\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt\" (UID: \"5c2fc702-9338-4af0-a044-810fafccbfad\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt"
Nov 03 19:45:47.884959 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.884868    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ee1b6ae-046b-4402-a36e-bbd523e73463-istio-data\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n\" (UID: \"4ee1b6ae-046b-4402-a36e-bbd523e73463\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n"
Nov 03 19:45:47.884959 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.884908    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ee1b6ae-046b-4402-a36e-bbd523e73463-credential-socket\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n\" (UID: \"4ee1b6ae-046b-4402-a36e-bbd523e73463\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n"
Nov 03 19:45:47.884959 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.884926    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c2fc702-9338-4af0-a044-810fafccbfad-istio-token\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt\" (UID: \"5c2fc702-9338-4af0-a044-810fafccbfad\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt"
Nov 03 19:45:47.884959 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.884951    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ee1b6ae-046b-4402-a36e-bbd523e73463-workload-socket\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n\" (UID: \"4ee1b6ae-046b-4402-a36e-bbd523e73463\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n"
Nov 03 19:45:47.885085 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.884997    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ee1b6ae-046b-4402-a36e-bbd523e73463-istio-podinfo\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n\" (UID: \"4ee1b6ae-046b-4402-a36e-bbd523e73463\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n"
Nov 03 19:45:47.885085 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.885042    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a52b2e9b-9ca0-4897-a071-a79c80e40d14-istiod-ca-cert\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww\" (UID: \"a52b2e9b-9ca0-4897-a071-a79c80e40d14\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww"
Nov 03 19:45:47.885152 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.885086    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a52b2e9b-9ca0-4897-a071-a79c80e40d14-credential-socket\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww\" (UID: \"a52b2e9b-9ca0-4897-a071-a79c80e40d14\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww"
Nov 03 19:45:47.885152 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.885123    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ee1b6ae-046b-4402-a36e-bbd523e73463-credential-socket\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n\" (UID: \"4ee1b6ae-046b-4402-a36e-bbd523e73463\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n"
Nov 03 19:45:47.885229 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.885157    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c2fc702-9338-4af0-a044-810fafccbfad-istio-podinfo\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt\" (UID: \"5c2fc702-9338-4af0-a044-810fafccbfad\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt"
Nov 03 19:45:47.885229 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.885188    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ee1b6ae-046b-4402-a36e-bbd523e73463-istio-envoy\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n\" (UID: \"4ee1b6ae-046b-4402-a36e-bbd523e73463\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n"
Nov 03 19:45:47.885301 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.885235    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ee1b6ae-046b-4402-a36e-bbd523e73463-workload-socket\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n\" (UID: \"4ee1b6ae-046b-4402-a36e-bbd523e73463\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n"
Nov 03 19:45:47.887311 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.887283    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ee1b6ae-046b-4402-a36e-bbd523e73463-istio-envoy\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n\" (UID: \"4ee1b6ae-046b-4402-a36e-bbd523e73463\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n"
Nov 03 19:45:47.887480 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.887464    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ee1b6ae-046b-4402-a36e-bbd523e73463-istio-podinfo\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n\" (UID: \"4ee1b6ae-046b-4402-a36e-bbd523e73463\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n"
Nov 03 19:45:47.892912 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.892892    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdpzb\" (UniqueName: \"kubernetes.io/projected/4ee1b6ae-046b-4402-a36e-bbd523e73463-kube-api-access-wdpzb\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n\" (UID: \"4ee1b6ae-046b-4402-a36e-bbd523e73463\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n"
Nov 03 19:45:47.893019 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.892988    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ee1b6ae-046b-4402-a36e-bbd523e73463-istio-token\") pod \"e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n\" (UID: \"4ee1b6ae-046b-4402-a36e-bbd523e73463\") " pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n"
Nov 03 19:45:47.938151 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.938074    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2"]
Nov 03 19:45:47.944009 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.943986    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2"
Nov 03 19:45:47.962638 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.962609    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2"]
Nov 03 19:45:47.986128 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.986099    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c2fc702-9338-4af0-a044-810fafccbfad-istio-envoy\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt\" (UID: \"5c2fc702-9338-4af0-a044-810fafccbfad\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt"
Nov 03 19:45:47.986302 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.986149    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrnhr\" (UniqueName: \"kubernetes.io/projected/a52b2e9b-9ca0-4897-a071-a79c80e40d14-kube-api-access-lrnhr\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww\" (UID: \"a52b2e9b-9ca0-4897-a071-a79c80e40d14\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww"
Nov 03 19:45:47.986302 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.986179    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a52b2e9b-9ca0-4897-a071-a79c80e40d14-istio-podinfo\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww\" (UID: \"a52b2e9b-9ca0-4897-a071-a79c80e40d14\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww"
Nov 03 19:45:47.986302 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.986208    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c2fc702-9338-4af0-a044-810fafccbfad-workload-certs\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt\" (UID: \"5c2fc702-9338-4af0-a044-810fafccbfad\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt"
Nov 03 19:45:47.986302 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.986229    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c2fc702-9338-4af0-a044-810fafccbfad-istio-data\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt\" (UID: \"5c2fc702-9338-4af0-a044-810fafccbfad\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt"
Nov 03 19:45:47.986302 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.986253    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czgdq\" (UniqueName: \"kubernetes.io/projected/5c2fc702-9338-4af0-a044-810fafccbfad-kube-api-access-czgdq\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt\" (UID: \"5c2fc702-9338-4af0-a044-810fafccbfad\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt"
Nov 03 19:45:47.986302 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.986288    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c2fc702-9338-4af0-a044-810fafccbfad-istio-token\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt\" (UID: \"5c2fc702-9338-4af0-a044-810fafccbfad\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt"
Nov 03 19:45:47.986663 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.986345    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a52b2e9b-9ca0-4897-a071-a79c80e40d14-istiod-ca-cert\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww\" (UID: \"a52b2e9b-9ca0-4897-a071-a79c80e40d14\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww"
Nov 03 19:45:47.986663 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.986379    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a52b2e9b-9ca0-4897-a071-a79c80e40d14-credential-socket\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww\" (UID: \"a52b2e9b-9ca0-4897-a071-a79c80e40d14\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww"
Nov 03 19:45:47.986663 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.986397    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c2fc702-9338-4af0-a044-810fafccbfad-istio-podinfo\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt\" (UID: \"5c2fc702-9338-4af0-a044-810fafccbfad\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt"
Nov 03 19:45:47.986663 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.986451    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a52b2e9b-9ca0-4897-a071-a79c80e40d14-istio-token\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww\" (UID: \"a52b2e9b-9ca0-4897-a071-a79c80e40d14\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww"
Nov 03 19:45:47.986663 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.986521    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a52b2e9b-9ca0-4897-a071-a79c80e40d14-workload-certs\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww\" (UID: \"a52b2e9b-9ca0-4897-a071-a79c80e40d14\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww"
Nov 03 19:45:47.986663 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.986548    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a52b2e9b-9ca0-4897-a071-a79c80e40d14-istio-data\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww\" (UID: \"a52b2e9b-9ca0-4897-a071-a79c80e40d14\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww"
Nov 03 19:45:47.986663 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.986593    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a52b2e9b-9ca0-4897-a071-a79c80e40d14-workload-socket\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww\" (UID: \"a52b2e9b-9ca0-4897-a071-a79c80e40d14\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww"
Nov 03 19:45:47.986663 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.986623    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a52b2e9b-9ca0-4897-a071-a79c80e40d14-istio-envoy\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww\" (UID: \"a52b2e9b-9ca0-4897-a071-a79c80e40d14\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww"
Nov 03 19:45:47.986663 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.986649    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c2fc702-9338-4af0-a044-810fafccbfad-workload-socket\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt\" (UID: \"5c2fc702-9338-4af0-a044-810fafccbfad\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt"
Nov 03 19:45:47.987201 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.986744    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c2fc702-9338-4af0-a044-810fafccbfad-istiod-ca-cert\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt\" (UID: \"5c2fc702-9338-4af0-a044-810fafccbfad\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt"
Nov 03 19:45:47.987201 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.986789    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c2fc702-9338-4af0-a044-810fafccbfad-credential-socket\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt\" (UID: \"5c2fc702-9338-4af0-a044-810fafccbfad\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt"
Nov 03 19:45:47.987201 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.986832    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c2fc702-9338-4af0-a044-810fafccbfad-istio-data\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt\" (UID: \"5c2fc702-9338-4af0-a044-810fafccbfad\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt"
Nov 03 19:45:47.987201 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.987147    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c2fc702-9338-4af0-a044-810fafccbfad-workload-certs\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt\" (UID: \"5c2fc702-9338-4af0-a044-810fafccbfad\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt"
Nov 03 19:45:47.987740 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.987507    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c2fc702-9338-4af0-a044-810fafccbfad-istiod-ca-cert\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt\" (UID: \"5c2fc702-9338-4af0-a044-810fafccbfad\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt"
Nov 03 19:45:47.987740 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.987569    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a52b2e9b-9ca0-4897-a071-a79c80e40d14-istiod-ca-cert\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww\" (UID: \"a52b2e9b-9ca0-4897-a071-a79c80e40d14\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww"
Nov 03 19:45:47.987934 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.987738    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c2fc702-9338-4af0-a044-810fafccbfad-workload-socket\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt\" (UID: \"5c2fc702-9338-4af0-a044-810fafccbfad\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt"
Nov 03 19:45:47.987934 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.987805    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c2fc702-9338-4af0-a044-810fafccbfad-credential-socket\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt\" (UID: \"5c2fc702-9338-4af0-a044-810fafccbfad\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt"
Nov 03 19:45:47.989385 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.989358    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a52b2e9b-9ca0-4897-a071-a79c80e40d14-istio-podinfo\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww\" (UID: \"a52b2e9b-9ca0-4897-a071-a79c80e40d14\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww"
Nov 03 19:45:47.989531 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.989506    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c2fc702-9338-4af0-a044-810fafccbfad-istio-envoy\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt\" (UID: \"5c2fc702-9338-4af0-a044-810fafccbfad\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt"
Nov 03 19:45:47.990064 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.990038    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a52b2e9b-9ca0-4897-a071-a79c80e40d14-workload-socket\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww\" (UID: \"a52b2e9b-9ca0-4897-a071-a79c80e40d14\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww"
Nov 03 19:45:47.990162 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.990091    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a52b2e9b-9ca0-4897-a071-a79c80e40d14-credential-socket\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww\" (UID: \"a52b2e9b-9ca0-4897-a071-a79c80e40d14\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww"
Nov 03 19:45:47.990162 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.990135    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a52b2e9b-9ca0-4897-a071-a79c80e40d14-istio-data\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww\" (UID: \"a52b2e9b-9ca0-4897-a071-a79c80e40d14\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww"
Nov 03 19:45:47.990283 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.990204    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a52b2e9b-9ca0-4897-a071-a79c80e40d14-workload-certs\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww\" (UID: \"a52b2e9b-9ca0-4897-a071-a79c80e40d14\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww"
Nov 03 19:45:47.991651 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.991260    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c2fc702-9338-4af0-a044-810fafccbfad-istio-podinfo\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt\" (UID: \"5c2fc702-9338-4af0-a044-810fafccbfad\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt"
Nov 03 19:45:47.992594 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.992143    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a52b2e9b-9ca0-4897-a071-a79c80e40d14-istio-envoy\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww\" (UID: \"a52b2e9b-9ca0-4897-a071-a79c80e40d14\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww"
Nov 03 19:45:47.994078 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.994053    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrnhr\" (UniqueName: \"kubernetes.io/projected/a52b2e9b-9ca0-4897-a071-a79c80e40d14-kube-api-access-lrnhr\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww\" (UID: \"a52b2e9b-9ca0-4897-a071-a79c80e40d14\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww"
Nov 03 19:45:47.994474 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.994453    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a52b2e9b-9ca0-4897-a071-a79c80e40d14-istio-token\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww\" (UID: \"a52b2e9b-9ca0-4897-a071-a79c80e40d14\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww"
Nov 03 19:45:47.995612 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.995294    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czgdq\" (UniqueName: \"kubernetes.io/projected/5c2fc702-9338-4af0-a044-810fafccbfad-kube-api-access-czgdq\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt\" (UID: \"5c2fc702-9338-4af0-a044-810fafccbfad\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt"
Nov 03 19:45:47.995612 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:47.995574    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c2fc702-9338-4af0-a044-810fafccbfad-istio-token\") pod \"e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt\" (UID: \"5c2fc702-9338-4af0-a044-810fafccbfad\") " pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt"
Nov 03 19:45:48.001606 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.001581    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n"
Nov 03 19:45:48.050856 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.050790    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt"
Nov 03 19:45:48.064381 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.064355    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww"
Nov 03 19:45:48.093991 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.088693    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/476b08ee-e24d-4f0c-a5ae-5cb23d1fa105-istio-envoy\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2\" (UID: \"476b08ee-e24d-4f0c-a5ae-5cb23d1fa105\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2"
Nov 03 19:45:48.093991 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.088767    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/476b08ee-e24d-4f0c-a5ae-5cb23d1fa105-credential-socket\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2\" (UID: \"476b08ee-e24d-4f0c-a5ae-5cb23d1fa105\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2"
Nov 03 19:45:48.093991 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.088801    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/476b08ee-e24d-4f0c-a5ae-5cb23d1fa105-workload-socket\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2\" (UID: \"476b08ee-e24d-4f0c-a5ae-5cb23d1fa105\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2"
Nov 03 19:45:48.093991 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.088868    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/476b08ee-e24d-4f0c-a5ae-5cb23d1fa105-istiod-ca-cert\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2\" (UID: \"476b08ee-e24d-4f0c-a5ae-5cb23d1fa105\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2"
Nov 03 19:45:48.093991 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.088921    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl7bj\" (UniqueName: \"kubernetes.io/projected/476b08ee-e24d-4f0c-a5ae-5cb23d1fa105-kube-api-access-fl7bj\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2\" (UID: \"476b08ee-e24d-4f0c-a5ae-5cb23d1fa105\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2"
Nov 03 19:45:48.093991 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.089056    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/476b08ee-e24d-4f0c-a5ae-5cb23d1fa105-workload-certs\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2\" (UID: \"476b08ee-e24d-4f0c-a5ae-5cb23d1fa105\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2"
Nov 03 19:45:48.093991 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.089119    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/476b08ee-e24d-4f0c-a5ae-5cb23d1fa105-istio-data\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2\" (UID: \"476b08ee-e24d-4f0c-a5ae-5cb23d1fa105\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2"
Nov 03 19:45:48.093991 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.089147    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/476b08ee-e24d-4f0c-a5ae-5cb23d1fa105-istio-podinfo\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2\" (UID: \"476b08ee-e24d-4f0c-a5ae-5cb23d1fa105\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2"
Nov 03 19:45:48.093991 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.089170    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/476b08ee-e24d-4f0c-a5ae-5cb23d1fa105-istio-token\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2\" (UID: \"476b08ee-e24d-4f0c-a5ae-5cb23d1fa105\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2"
Nov 03 19:45:48.177536 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.177508    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl"]
Nov 03 19:45:48.186253 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.186182    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl"
Nov 03 19:45:48.190029 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.189971    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl"]
Nov 03 19:45:48.190971 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.190950    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/476b08ee-e24d-4f0c-a5ae-5cb23d1fa105-istio-data\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2\" (UID: \"476b08ee-e24d-4f0c-a5ae-5cb23d1fa105\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2"
Nov 03 19:45:48.191906 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.190990    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/476b08ee-e24d-4f0c-a5ae-5cb23d1fa105-istio-podinfo\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2\" (UID: \"476b08ee-e24d-4f0c-a5ae-5cb23d1fa105\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2"
Nov 03 19:45:48.191906 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.191015    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/476b08ee-e24d-4f0c-a5ae-5cb23d1fa105-istio-token\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2\" (UID: \"476b08ee-e24d-4f0c-a5ae-5cb23d1fa105\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2"
Nov 03 19:45:48.191906 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.191089    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/476b08ee-e24d-4f0c-a5ae-5cb23d1fa105-istio-envoy\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2\" (UID: \"476b08ee-e24d-4f0c-a5ae-5cb23d1fa105\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2"
Nov 03 19:45:48.191906 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.191150    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/476b08ee-e24d-4f0c-a5ae-5cb23d1fa105-credential-socket\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2\" (UID: \"476b08ee-e24d-4f0c-a5ae-5cb23d1fa105\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2"
Nov 03 19:45:48.191906 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.191179    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/476b08ee-e24d-4f0c-a5ae-5cb23d1fa105-workload-socket\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2\" (UID: \"476b08ee-e24d-4f0c-a5ae-5cb23d1fa105\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2"
Nov 03 19:45:48.191906 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.191250    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/476b08ee-e24d-4f0c-a5ae-5cb23d1fa105-istiod-ca-cert\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2\" (UID: \"476b08ee-e24d-4f0c-a5ae-5cb23d1fa105\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2"
Nov 03 19:45:48.191906 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.191310    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl7bj\" (UniqueName: \"kubernetes.io/projected/476b08ee-e24d-4f0c-a5ae-5cb23d1fa105-kube-api-access-fl7bj\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2\" (UID: \"476b08ee-e24d-4f0c-a5ae-5cb23d1fa105\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2"
Nov 03 19:45:48.191906 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.191398    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/476b08ee-e24d-4f0c-a5ae-5cb23d1fa105-istio-data\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2\" (UID: \"476b08ee-e24d-4f0c-a5ae-5cb23d1fa105\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2"
Nov 03 19:45:48.191906 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.191479    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/476b08ee-e24d-4f0c-a5ae-5cb23d1fa105-workload-certs\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2\" (UID: \"476b08ee-e24d-4f0c-a5ae-5cb23d1fa105\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2"
Nov 03 19:45:48.191906 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.191892    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/476b08ee-e24d-4f0c-a5ae-5cb23d1fa105-workload-certs\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2\" (UID: \"476b08ee-e24d-4f0c-a5ae-5cb23d1fa105\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2"
Nov 03 19:45:48.193021 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.192142    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/476b08ee-e24d-4f0c-a5ae-5cb23d1fa105-credential-socket\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2\" (UID: \"476b08ee-e24d-4f0c-a5ae-5cb23d1fa105\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2"
Nov 03 19:45:48.193705 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.193658    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/476b08ee-e24d-4f0c-a5ae-5cb23d1fa105-istiod-ca-cert\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2\" (UID: \"476b08ee-e24d-4f0c-a5ae-5cb23d1fa105\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2"
Nov 03 19:45:48.197535 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.197514    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/476b08ee-e24d-4f0c-a5ae-5cb23d1fa105-workload-socket\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2\" (UID: \"476b08ee-e24d-4f0c-a5ae-5cb23d1fa105\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2"
Nov 03 19:45:48.198922 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.198178    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/476b08ee-e24d-4f0c-a5ae-5cb23d1fa105-istio-podinfo\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2\" (UID: \"476b08ee-e24d-4f0c-a5ae-5cb23d1fa105\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2"
Nov 03 19:45:48.198922 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.198202    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/476b08ee-e24d-4f0c-a5ae-5cb23d1fa105-istio-envoy\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2\" (UID: \"476b08ee-e24d-4f0c-a5ae-5cb23d1fa105\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2"
Nov 03 19:45:48.204725 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.204700    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj"]
Nov 03 19:45:48.211025 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.210999    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl7bj\" (UniqueName: \"kubernetes.io/projected/476b08ee-e24d-4f0c-a5ae-5cb23d1fa105-kube-api-access-fl7bj\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2\" (UID: \"476b08ee-e24d-4f0c-a5ae-5cb23d1fa105\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2"
Nov 03 19:45:48.212561 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.211995    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj"
Nov 03 19:45:48.227226 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.224861    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/476b08ee-e24d-4f0c-a5ae-5cb23d1fa105-istio-token\") pod \"e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2\" (UID: \"476b08ee-e24d-4f0c-a5ae-5cb23d1fa105\") " pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2"
Nov 03 19:45:48.242758 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.242623    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj"]
Nov 03 19:45:48.273322 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.272034    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2"
Nov 03 19:45:48.299636 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.298481    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55ff258b-9816-4fff-a74a-3c417bf0f0ee-istio-token\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl\" (UID: \"55ff258b-9816-4fff-a74a-3c417bf0f0ee\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl"
Nov 03 19:45:48.299636 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.298551    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55ff258b-9816-4fff-a74a-3c417bf0f0ee-istio-data\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl\" (UID: \"55ff258b-9816-4fff-a74a-3c417bf0f0ee\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl"
Nov 03 19:45:48.299636 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.298586    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwwq2\" (UniqueName: \"kubernetes.io/projected/30ae46dd-e9ca-4cd8-b78c-1a5463c98d94-kube-api-access-vwwq2\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj\" (UID: \"30ae46dd-e9ca-4cd8-b78c-1a5463c98d94\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj"
Nov 03 19:45:48.299636 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.298645    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55ff258b-9816-4fff-a74a-3c417bf0f0ee-workload-socket\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl\" (UID: \"55ff258b-9816-4fff-a74a-3c417bf0f0ee\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl"
Nov 03 19:45:48.299636 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.298740    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30ae46dd-e9ca-4cd8-b78c-1a5463c98d94-workload-certs\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj\" (UID: \"30ae46dd-e9ca-4cd8-b78c-1a5463c98d94\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj"
Nov 03 19:45:48.299636 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.298767    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30ae46dd-e9ca-4cd8-b78c-1a5463c98d94-istio-podinfo\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj\" (UID: \"30ae46dd-e9ca-4cd8-b78c-1a5463c98d94\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj"
Nov 03 19:45:48.299636 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.298800    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30ae46dd-e9ca-4cd8-b78c-1a5463c98d94-istio-token\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj\" (UID: \"30ae46dd-e9ca-4cd8-b78c-1a5463c98d94\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj"
Nov 03 19:45:48.299636 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.298863    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30ae46dd-e9ca-4cd8-b78c-1a5463c98d94-istio-envoy\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj\" (UID: \"30ae46dd-e9ca-4cd8-b78c-1a5463c98d94\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj"
Nov 03 19:45:48.299636 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.298892    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55ff258b-9816-4fff-a74a-3c417bf0f0ee-istiod-ca-cert\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl\" (UID: \"55ff258b-9816-4fff-a74a-3c417bf0f0ee\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl"
Nov 03 19:45:48.299636 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.298928    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55ff258b-9816-4fff-a74a-3c417bf0f0ee-istio-podinfo\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl\" (UID: \"55ff258b-9816-4fff-a74a-3c417bf0f0ee\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl"
Nov 03 19:45:48.299636 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.298951    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30ae46dd-e9ca-4cd8-b78c-1a5463c98d94-istio-data\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj\" (UID: \"30ae46dd-e9ca-4cd8-b78c-1a5463c98d94\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj"
Nov 03 19:45:48.299636 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.298991    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30ae46dd-e9ca-4cd8-b78c-1a5463c98d94-workload-socket\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj\" (UID: \"30ae46dd-e9ca-4cd8-b78c-1a5463c98d94\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj"
Nov 03 19:45:48.299636 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.299046    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55ff258b-9816-4fff-a74a-3c417bf0f0ee-workload-certs\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl\" (UID: \"55ff258b-9816-4fff-a74a-3c417bf0f0ee\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl"
Nov 03 19:45:48.299636 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.299090    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55ff258b-9816-4fff-a74a-3c417bf0f0ee-istio-envoy\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl\" (UID: \"55ff258b-9816-4fff-a74a-3c417bf0f0ee\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl"
Nov 03 19:45:48.299636 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.299121    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwr7x\" (UniqueName: \"kubernetes.io/projected/55ff258b-9816-4fff-a74a-3c417bf0f0ee-kube-api-access-qwr7x\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl\" (UID: \"55ff258b-9816-4fff-a74a-3c417bf0f0ee\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl"
Nov 03 19:45:48.299636 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.299171    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30ae46dd-e9ca-4cd8-b78c-1a5463c98d94-istiod-ca-cert\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj\" (UID: \"30ae46dd-e9ca-4cd8-b78c-1a5463c98d94\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj"
Nov 03 19:45:48.299636 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.299296    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55ff258b-9816-4fff-a74a-3c417bf0f0ee-credential-socket\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl\" (UID: \"55ff258b-9816-4fff-a74a-3c417bf0f0ee\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl"
Nov 03 19:45:48.299636 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.299330    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30ae46dd-e9ca-4cd8-b78c-1a5463c98d94-credential-socket\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj\" (UID: \"30ae46dd-e9ca-4cd8-b78c-1a5463c98d94\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj"
Nov 03 19:45:48.306958 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:45:48.306930    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ee1b6ae_046b_4402_a36e_bbd523e73463.slice/crio-102015d3a30ff0467aec55158078e1232799bd1db22e4898653afd0279ca4508 WatchSource:0}: Error finding container 102015d3a30ff0467aec55158078e1232799bd1db22e4898653afd0279ca4508: Status 404 returned error can't find the container with id 102015d3a30ff0467aec55158078e1232799bd1db22e4898653afd0279ca4508
Nov 03 19:45:48.320639 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.320576    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:45:48.320830 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.320689    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:45:48.320830 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.320732    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:45:48.328131 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.328103    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n"]
Nov 03 19:45:48.372949 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.372888    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt"]
Nov 03 19:45:48.384212 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.383812    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:45:48.384212 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.383883    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:45:48.384212 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.383914    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:45:48.398897 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.398237    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww"]
Nov 03 19:45:48.405194 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.402189    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30ae46dd-e9ca-4cd8-b78c-1a5463c98d94-workload-certs\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj\" (UID: \"30ae46dd-e9ca-4cd8-b78c-1a5463c98d94\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj"
Nov 03 19:45:48.405194 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.402232    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30ae46dd-e9ca-4cd8-b78c-1a5463c98d94-istio-podinfo\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj\" (UID: \"30ae46dd-e9ca-4cd8-b78c-1a5463c98d94\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj"
Nov 03 19:45:48.405194 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.402254    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30ae46dd-e9ca-4cd8-b78c-1a5463c98d94-istio-token\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj\" (UID: \"30ae46dd-e9ca-4cd8-b78c-1a5463c98d94\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj"
Nov 03 19:45:48.405194 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.402291    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30ae46dd-e9ca-4cd8-b78c-1a5463c98d94-istio-envoy\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj\" (UID: \"30ae46dd-e9ca-4cd8-b78c-1a5463c98d94\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj"
Nov 03 19:45:48.405194 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.402311    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55ff258b-9816-4fff-a74a-3c417bf0f0ee-istiod-ca-cert\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl\" (UID: \"55ff258b-9816-4fff-a74a-3c417bf0f0ee\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl"
Nov 03 19:45:48.405194 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.402336    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55ff258b-9816-4fff-a74a-3c417bf0f0ee-istio-podinfo\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl\" (UID: \"55ff258b-9816-4fff-a74a-3c417bf0f0ee\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl"
Nov 03 19:45:48.405194 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.402358    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30ae46dd-e9ca-4cd8-b78c-1a5463c98d94-istio-data\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj\" (UID: \"30ae46dd-e9ca-4cd8-b78c-1a5463c98d94\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj"
Nov 03 19:45:48.405194 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.402399    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30ae46dd-e9ca-4cd8-b78c-1a5463c98d94-workload-socket\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj\" (UID: \"30ae46dd-e9ca-4cd8-b78c-1a5463c98d94\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj"
Nov 03 19:45:48.405194 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.402431    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55ff258b-9816-4fff-a74a-3c417bf0f0ee-workload-certs\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl\" (UID: \"55ff258b-9816-4fff-a74a-3c417bf0f0ee\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl"
Nov 03 19:45:48.405194 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.402458    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55ff258b-9816-4fff-a74a-3c417bf0f0ee-istio-envoy\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl\" (UID: \"55ff258b-9816-4fff-a74a-3c417bf0f0ee\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl"
Nov 03 19:45:48.405194 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.403136    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30ae46dd-e9ca-4cd8-b78c-1a5463c98d94-workload-certs\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj\" (UID: \"30ae46dd-e9ca-4cd8-b78c-1a5463c98d94\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj"
Nov 03 19:45:48.405194 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.403230    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55ff258b-9816-4fff-a74a-3c417bf0f0ee-istiod-ca-cert\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl\" (UID: \"55ff258b-9816-4fff-a74a-3c417bf0f0ee\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl"
Nov 03 19:45:48.405194 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.403279    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwr7x\" (UniqueName: \"kubernetes.io/projected/55ff258b-9816-4fff-a74a-3c417bf0f0ee-kube-api-access-qwr7x\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl\" (UID: \"55ff258b-9816-4fff-a74a-3c417bf0f0ee\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl"
Nov 03 19:45:48.405194 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.403335    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30ae46dd-e9ca-4cd8-b78c-1a5463c98d94-istiod-ca-cert\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj\" (UID: \"30ae46dd-e9ca-4cd8-b78c-1a5463c98d94\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj"
Nov 03 19:45:48.405194 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.403443    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55ff258b-9816-4fff-a74a-3c417bf0f0ee-credential-socket\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl\" (UID: \"55ff258b-9816-4fff-a74a-3c417bf0f0ee\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl"
Nov 03 19:45:48.405194 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.403473    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30ae46dd-e9ca-4cd8-b78c-1a5463c98d94-credential-socket\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj\" (UID: \"30ae46dd-e9ca-4cd8-b78c-1a5463c98d94\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj"
Nov 03 19:45:48.405194 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.403521    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55ff258b-9816-4fff-a74a-3c417bf0f0ee-istio-token\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl\" (UID: \"55ff258b-9816-4fff-a74a-3c417bf0f0ee\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl"
Nov 03 19:45:48.405194 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.403565    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55ff258b-9816-4fff-a74a-3c417bf0f0ee-istio-data\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl\" (UID: \"55ff258b-9816-4fff-a74a-3c417bf0f0ee\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl"
Nov 03 19:45:48.405194 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.403592    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwwq2\" (UniqueName: \"kubernetes.io/projected/30ae46dd-e9ca-4cd8-b78c-1a5463c98d94-kube-api-access-vwwq2\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj\" (UID: \"30ae46dd-e9ca-4cd8-b78c-1a5463c98d94\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj"
Nov 03 19:45:48.405194 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.403659    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55ff258b-9816-4fff-a74a-3c417bf0f0ee-workload-socket\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl\" (UID: \"55ff258b-9816-4fff-a74a-3c417bf0f0ee\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl"
Nov 03 19:45:48.405194 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.404011    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55ff258b-9816-4fff-a74a-3c417bf0f0ee-workload-socket\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl\" (UID: \"55ff258b-9816-4fff-a74a-3c417bf0f0ee\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl"
Nov 03 19:45:48.405194 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.404358    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30ae46dd-e9ca-4cd8-b78c-1a5463c98d94-istio-data\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj\" (UID: \"30ae46dd-e9ca-4cd8-b78c-1a5463c98d94\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj"
Nov 03 19:45:48.405194 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.404407    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55ff258b-9816-4fff-a74a-3c417bf0f0ee-istio-data\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl\" (UID: \"55ff258b-9816-4fff-a74a-3c417bf0f0ee\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl"
Nov 03 19:45:48.405194 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.404799    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30ae46dd-e9ca-4cd8-b78c-1a5463c98d94-workload-socket\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj\" (UID: \"30ae46dd-e9ca-4cd8-b78c-1a5463c98d94\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj"
Nov 03 19:45:48.405194 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.405004    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55ff258b-9816-4fff-a74a-3c417bf0f0ee-credential-socket\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl\" (UID: \"55ff258b-9816-4fff-a74a-3c417bf0f0ee\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl"
Nov 03 19:45:48.405194 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.405074    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55ff258b-9816-4fff-a74a-3c417bf0f0ee-workload-certs\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl\" (UID: \"55ff258b-9816-4fff-a74a-3c417bf0f0ee\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl"
Nov 03 19:45:48.406257 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.405330    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30ae46dd-e9ca-4cd8-b78c-1a5463c98d94-istiod-ca-cert\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj\" (UID: \"30ae46dd-e9ca-4cd8-b78c-1a5463c98d94\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj"
Nov 03 19:45:48.406639 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.406506    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30ae46dd-e9ca-4cd8-b78c-1a5463c98d94-credential-socket\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj\" (UID: \"30ae46dd-e9ca-4cd8-b78c-1a5463c98d94\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj"
Nov 03 19:45:48.406639 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.406589    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55ff258b-9816-4fff-a74a-3c417bf0f0ee-istio-envoy\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl\" (UID: \"55ff258b-9816-4fff-a74a-3c417bf0f0ee\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl"
Nov 03 19:45:48.406639 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.406602    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30ae46dd-e9ca-4cd8-b78c-1a5463c98d94-istio-envoy\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj\" (UID: \"30ae46dd-e9ca-4cd8-b78c-1a5463c98d94\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj"
Nov 03 19:45:48.407147 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.407109    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30ae46dd-e9ca-4cd8-b78c-1a5463c98d94-istio-podinfo\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj\" (UID: \"30ae46dd-e9ca-4cd8-b78c-1a5463c98d94\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj"
Nov 03 19:45:48.408270 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.408250    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55ff258b-9816-4fff-a74a-3c417bf0f0ee-istio-podinfo\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl\" (UID: \"55ff258b-9816-4fff-a74a-3c417bf0f0ee\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl"
Nov 03 19:45:48.409313 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.409279    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:45:48.409422 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.409348    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:45:48.409422 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.409390    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:45:48.413838 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.411896    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55ff258b-9816-4fff-a74a-3c417bf0f0ee-istio-token\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl\" (UID: \"55ff258b-9816-4fff-a74a-3c417bf0f0ee\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl"
Nov 03 19:45:48.413838 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.413217    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwr7x\" (UniqueName: \"kubernetes.io/projected/55ff258b-9816-4fff-a74a-3c417bf0f0ee-kube-api-access-qwr7x\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl\" (UID: \"55ff258b-9816-4fff-a74a-3c417bf0f0ee\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl"
Nov 03 19:45:48.414791 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.414773    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30ae46dd-e9ca-4cd8-b78c-1a5463c98d94-istio-token\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj\" (UID: \"30ae46dd-e9ca-4cd8-b78c-1a5463c98d94\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj"
Nov 03 19:45:48.417913 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.417875    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwwq2\" (UniqueName: \"kubernetes.io/projected/30ae46dd-e9ca-4cd8-b78c-1a5463c98d94-kube-api-access-vwwq2\") pod \"e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj\" (UID: \"30ae46dd-e9ca-4cd8-b78c-1a5463c98d94\") " pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj"
Nov 03 19:45:48.513597 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.513566    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl"
Nov 03 19:45:48.539156 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.538884    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj"
Nov 03 19:45:48.626769 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:45:48.626702    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod476b08ee_e24d_4f0c_a5ae_5cb23d1fa105.slice/crio-37e81b910f6fa8843353e301c913b4411f793791d5860d5f6e60b1512b77a352 WatchSource:0}: Error finding container 37e81b910f6fa8843353e301c913b4411f793791d5860d5f6e60b1512b77a352: Status 404 returned error can't find the container with id 37e81b910f6fa8843353e301c913b4411f793791d5860d5f6e60b1512b77a352
Nov 03 19:45:48.631555 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.631381    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2"]
Nov 03 19:45:48.641208 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.639845    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:45:48.641208 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.639945    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:45:48.641208 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.639994    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:45:48.923957 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.923506    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:45:48.923957 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.923597    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:45:48.923957 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.923640    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:45:48.958309 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.958277    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl"]
Nov 03 19:45:48.998908 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.997487    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj"]
Nov 03 19:45:48.999218 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.999181    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:45:48.999333 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.999248    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:45:48.999333 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:48.999280    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:45:49.342622 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:49.342582    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj" event={"ID":"30ae46dd-e9ca-4cd8-b78c-1a5463c98d94","Type":"ContainerStarted","Data":"7260a03f31c099a7e261da6eb837d29cb48f6c8aafe972f88f5557465d2ca813"}
Nov 03 19:45:49.352022 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:49.351967    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj" event={"ID":"30ae46dd-e9ca-4cd8-b78c-1a5463c98d94","Type":"ContainerStarted","Data":"262eb77e6558b08d9b4f554e1aba96e1b3b65a9d9f2f627fad81e11c7d55f468"}
Nov 03 19:45:49.382767 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:49.382718    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt" event={"ID":"5c2fc702-9338-4af0-a044-810fafccbfad","Type":"ContainerStarted","Data":"f1a832e2f6fe841a3a93619900e9716bc01f3ec890d822bcc590edf8b761deae"}
Nov 03 19:45:49.382946 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:49.382803    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt" event={"ID":"5c2fc702-9338-4af0-a044-810fafccbfad","Type":"ContainerStarted","Data":"2bb4aa349d5304dba35ddf1710fd50959efd89f9f7164dbe4f04c94f3a681ee8"}
Nov 03 19:45:49.412708 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:49.412310    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj" podStartSLOduration=1.412287459 podStartE2EDuration="1.412287459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:49.405517167 +0000 UTC m=+862.649482634" watchObservedRunningTime="2026-11-03 19:45:49.412287459 +0000 UTC m=+862.656252925"
Nov 03 19:45:49.478707 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:49.478132    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww" event={"ID":"a52b2e9b-9ca0-4897-a071-a79c80e40d14","Type":"ContainerStarted","Data":"3e157dd7b9d6cfe501547f12915c5cd99acd3f6e3b27711109c13f06e3a31875"}
Nov 03 19:45:49.478707 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:49.478214    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww" event={"ID":"a52b2e9b-9ca0-4897-a071-a79c80e40d14","Type":"ContainerStarted","Data":"6a752ad6e84dbf3274e4fb4f05284d8e57ed47fb1f57383bb5bd76c8909f4c3d"}
Nov 03 19:45:49.514693 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:49.514015    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt" podStartSLOduration=2.513979639 podStartE2EDuration="2.513979639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:49.51240965 +0000 UTC m=+862.756375117" watchObservedRunningTime="2026-11-03 19:45:49.513979639 +0000 UTC m=+862.757945105"
Nov 03 19:45:49.544724 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:49.532905    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n" event={"ID":"4ee1b6ae-046b-4402-a36e-bbd523e73463","Type":"ContainerStarted","Data":"5491e2aed27e2d6b7ee87fbabfeb281b70e3021351b666975202e6f44dccf880"}
Nov 03 19:45:49.544724 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:49.532974    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n" event={"ID":"4ee1b6ae-046b-4402-a36e-bbd523e73463","Type":"ContainerStarted","Data":"102015d3a30ff0467aec55158078e1232799bd1db22e4898653afd0279ca4508"}
Nov 03 19:45:49.544724 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:49.539478    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj"
Nov 03 19:45:49.544724 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:49.540774    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl" event={"ID":"55ff258b-9816-4fff-a74a-3c417bf0f0ee","Type":"ContainerStarted","Data":"e1d1f3573b618e8015e9ef6897a1d097c3e97de954faf99fb4f27db3bf4cdec6"}
Nov 03 19:45:49.544724 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:49.540823    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl" event={"ID":"55ff258b-9816-4fff-a74a-3c417bf0f0ee","Type":"ContainerStarted","Data":"3be859da63518ae95264a909a91e86ac85f7d614dc10619e91980518bdd87687"}
Nov 03 19:45:49.544724 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:49.543249    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2" event={"ID":"476b08ee-e24d-4f0c-a5ae-5cb23d1fa105","Type":"ContainerStarted","Data":"63ed9099e84a399a347bdf1a5a4b7f19d524546abb5eef5a2e43ab00455c0e23"}
Nov 03 19:45:49.544724 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:49.543296    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2" event={"ID":"476b08ee-e24d-4f0c-a5ae-5cb23d1fa105","Type":"ContainerStarted","Data":"37e81b910f6fa8843353e301c913b4411f793791d5860d5f6e60b1512b77a352"}
Nov 03 19:45:49.591747 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:49.568419    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww" podStartSLOduration=2.568400989 podStartE2EDuration="2.568400989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:49.564108558 +0000 UTC m=+862.808074024" watchObservedRunningTime="2026-11-03 19:45:49.568400989 +0000 UTC m=+862.812366458"
Nov 03 19:45:49.631483 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:49.631422    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl" podStartSLOduration=1.631405344 podStartE2EDuration="1.631405344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:49.611344881 +0000 UTC m=+862.855310347" watchObservedRunningTime="2026-11-03 19:45:49.631405344 +0000 UTC m=+862.875370815"
Nov 03 19:45:49.657733 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:49.657658    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2" podStartSLOduration=2.6576349820000003 podStartE2EDuration="2.657634982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:49.645207029 +0000 UTC m=+862.889172492" watchObservedRunningTime="2026-11-03 19:45:49.657634982 +0000 UTC m=+862.901600449"
Nov 03 19:45:49.737736 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:49.731876    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n" podStartSLOduration=2.731855611 podStartE2EDuration="2.731855611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:49.720049434 +0000 UTC m=+862.964014888" watchObservedRunningTime="2026-11-03 19:45:49.731855611 +0000 UTC m=+862.975821078"
Nov 03 19:45:50.023692 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:50.010023    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n"
Nov 03 19:45:50.023692 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:50.010801    4921 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Nov 03 19:45:50.023692 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:50.010873    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n" podUID="4ee1b6ae-046b-4402-a36e-bbd523e73463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Nov 03 19:45:50.061693 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:50.058039    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt"
Nov 03 19:45:50.068845 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:50.068799    4921 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 03 19:45:50.069085 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:50.069059    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt" podUID="5c2fc702-9338-4af0-a044-810fafccbfad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 03 19:45:50.069621 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:50.069597    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww"
Nov 03 19:45:50.097700 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:50.094985    4921 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 03 19:45:50.097700 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:50.095047    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww" podUID="a52b2e9b-9ca0-4897-a071-a79c80e40d14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 03 19:45:50.278751 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:50.278069    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2"
Nov 03 19:45:50.278751 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:50.278205    4921 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 03 19:45:50.278751 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:50.278249    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2" podUID="476b08ee-e24d-4f0c-a5ae-5cb23d1fa105" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 03 19:45:50.527692 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:50.527191    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl"
Nov 03 19:45:50.554051 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:50.552477    4921 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 03 19:45:50.554051 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:50.552542    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj" podUID="30ae46dd-e9ca-4cd8-b78c-1a5463c98d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 03 19:45:50.555155 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:50.555129    4921 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Nov 03 19:45:50.555354 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:50.555332    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl" podUID="55ff258b-9816-4fff-a74a-3c417bf0f0ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Nov 03 19:45:50.932780 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:50.932661    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7" podUID="d4de0090-d775-4bd7-9ee2-058b5d380638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:51.019108 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:51.019071    4921 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Nov 03 19:45:51.019446 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:51.019404    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n" podUID="4ee1b6ae-046b-4402-a36e-bbd523e73463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Nov 03 19:45:51.053192 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:51.053154    4921 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 03 19:45:51.053793 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:51.053759    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt" podUID="5c2fc702-9338-4af0-a044-810fafccbfad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 03 19:45:51.054718 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:51.054685    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9" podUID="e6a71aa3-33b3-42bd-b270-64ce3260cc2a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Nov 03 19:45:51.079693 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:51.072811    4921 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 03 19:45:51.079693 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:51.073706    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww" podUID="a52b2e9b-9ca0-4897-a071-a79c80e40d14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 03 19:45:51.150821 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:51.150767    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv" podUID="01c749b9-d188-4673-9a3d-99b9c4ae0793" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:51.278401 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:51.278360    4921 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 03 19:45:51.278699 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:51.278650    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2" podUID="476b08ee-e24d-4f0c-a5ae-5cb23d1fa105" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 03 19:45:51.523560 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:51.523100    4921 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Nov 03 19:45:51.523560 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:51.523163    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl" podUID="55ff258b-9816-4fff-a74a-3c417bf0f0ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Nov 03 19:45:51.543184 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:51.542591    4921 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 03 19:45:51.543184 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:51.542651    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj" podUID="30ae46dd-e9ca-4cd8-b78c-1a5463c98d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 03 19:45:51.558004 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:51.557294    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk" podUID="2d0a934e-4816-44d2-9bb5-e8b4e95dd73c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:51.576692 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:51.570774    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh" podUID="eac6c47d-8ae5-4e4b-b691-98d177887767" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:51.747694 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:51.717822    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw" podUID="72d96abc-f3b4-4eda-b3a1-b93ba658e1d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Nov 03 19:45:51.748060 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:51.748016    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6" podUID="1aaee61d-b036-4c2b-bee7-a692b8bed612" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:51.792723 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:51.783325    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt" podUID="17dc0ed2-5164-449d-be12-a38bdfeb946d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Nov 03 19:45:51.799266 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:51.797497    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv" podUID="96b73342-0079-4709-8570-3dbce57e5243" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Nov 03 19:45:51.956690 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:51.946602    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7" podUID="d4de0090-d775-4bd7-9ee2-058b5d380638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:51.993709 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:51.975794    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7"
Nov 03 19:45:52.075451 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:52.075366    4921 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Nov 03 19:45:52.166852 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:52.166804    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n" podUID="4ee1b6ae-046b-4402-a36e-bbd523e73463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Nov 03 19:45:52.168449 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:52.076580    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9" podUID="e6a71aa3-33b3-42bd-b270-64ce3260cc2a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:52.169024 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:52.076660    4921 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 03 19:45:52.169201 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:52.169179    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt" podUID="5c2fc702-9338-4af0-a044-810fafccbfad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 03 19:45:52.169309 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:52.106822    4921 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 03 19:45:52.232036 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:52.231987    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww" podUID="a52b2e9b-9ca0-4897-a071-a79c80e40d14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 03 19:45:52.287795 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:52.169821    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9"
Nov 03 19:45:52.437753 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:52.168049    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv" podUID="01c749b9-d188-4673-9a3d-99b9c4ae0793" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Nov 03 19:45:52.437753 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:52.352934    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv"
Nov 03 19:45:52.437753 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:52.416368    4921 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 03 19:45:52.437753 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:52.416438    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2" podUID="476b08ee-e24d-4f0c-a5ae-5cb23d1fa105" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 03 19:45:52.585693 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:52.584864    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk" podUID="2d0a934e-4816-44d2-9bb5-e8b4e95dd73c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:52.602695 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:52.586491    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk"
Nov 03 19:45:52.639698 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:52.619838    4921 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Nov 03 19:45:52.639698 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:52.619896    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl" podUID="55ff258b-9816-4fff-a74a-3c417bf0f0ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Nov 03 19:45:52.639698 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:52.620006    4921 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 03 19:45:52.639698 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:52.620031    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj" podUID="30ae46dd-e9ca-4cd8-b78c-1a5463c98d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 03 19:45:52.735745 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:52.732999    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw" podUID="72d96abc-f3b4-4eda-b3a1-b93ba658e1d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:52.735745 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:52.734516    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw"
Nov 03 19:45:52.761904 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:52.752797    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh" podUID="eac6c47d-8ae5-4e4b-b691-98d177887767" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded"
Nov 03 19:45:52.761904 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:52.753952    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh"
Nov 03 19:45:52.799694 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:52.786800    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6" podUID="1aaee61d-b036-4c2b-bee7-a692b8bed612" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 03 19:45:52.799694 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:52.795314    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6"
Nov 03 19:45:52.861694 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:52.861635    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt" podUID="17dc0ed2-5164-449d-be12-a38bdfeb946d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:52.875702 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:52.862977    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv" podUID="96b73342-0079-4709-8570-3dbce57e5243" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Nov 03 19:45:52.875702 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:52.864136    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt"
Nov 03 19:45:52.875702 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:52.864211    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv"
Nov 03 19:45:53.079093 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:53.078993    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7" podUID="d4de0090-d775-4bd7-9ee2-058b5d380638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:53.249697 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:53.240933    4921 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 03 19:45:53.249697 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:53.241000    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww" podUID="a52b2e9b-9ca0-4897-a071-a79c80e40d14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 03 19:45:53.249697 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:53.241075    4921 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Nov 03 19:45:53.249697 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:53.241097    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n" podUID="4ee1b6ae-046b-4402-a36e-bbd523e73463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Nov 03 19:45:53.268705 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:53.263758    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9" podUID="e6a71aa3-33b3-42bd-b270-64ce3260cc2a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Nov 03 19:45:53.337744 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:53.312054    4921 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 03 19:45:53.368352 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:53.368308    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2" podUID="476b08ee-e24d-4f0c-a5ae-5cb23d1fa105" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 03 19:45:53.368601 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:53.352258    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv" podUID="01c749b9-d188-4673-9a3d-99b9c4ae0793" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:53.405693 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:53.381122    4921 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 03 19:45:53.438450 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:53.438392    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt" podUID="5c2fc702-9338-4af0-a044-810fafccbfad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 03 19:45:53.537698 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:53.523948    4921 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Nov 03 19:45:53.537698 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:53.524005    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl" podUID="55ff258b-9816-4fff-a74a-3c417bf0f0ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Nov 03 19:45:53.537698 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:53.534282    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv"
Nov 03 19:45:53.542025 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:53.541990    4921 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 03 19:45:53.542278 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:53.542247    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj" podUID="30ae46dd-e9ca-4cd8-b78c-1a5463c98d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 03 19:45:53.593869 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:53.593770    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk" podUID="2d0a934e-4816-44d2-9bb5-e8b4e95dd73c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:53.738704 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:53.735788    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw" podUID="72d96abc-f3b4-4eda-b3a1-b93ba658e1d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:53.767216 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:53.767157    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh" podUID="eac6c47d-8ae5-4e4b-b691-98d177887767" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:53.820730 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:53.788019    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6" podUID="1aaee61d-b036-4c2b-bee7-a692b8bed612" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 03 19:45:53.891961 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:53.891851    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt" podUID="17dc0ed2-5164-449d-be12-a38bdfeb946d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Nov 03 19:45:54.009318 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:54.009281    4921 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Nov 03 19:45:54.010050 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:54.010018    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n" podUID="4ee1b6ae-046b-4402-a36e-bbd523e73463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Nov 03 19:45:54.089320 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:54.089277    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7" podUID="d4de0090-d775-4bd7-9ee2-058b5d380638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded"
Nov 03 19:45:54.099691 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:54.092017    4921 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 03 19:45:54.171179 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:54.100557    4921 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 03 19:45:54.171487 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:54.171417    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt" podUID="5c2fc702-9338-4af0-a044-810fafccbfad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 03 19:45:54.246695 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:54.221797    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww" podUID="a52b2e9b-9ca0-4897-a071-a79c80e40d14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 03 19:45:54.329704 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:54.276253    4921 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 03 19:45:54.330024 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:54.329983    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2" podUID="476b08ee-e24d-4f0c-a5ae-5cb23d1fa105" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 03 19:45:54.339789 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:54.318787    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9" podUID="e6a71aa3-33b3-42bd-b270-64ce3260cc2a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:54.443250 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:54.443150    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv" podUID="96b73342-0079-4709-8570-3dbce57e5243" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:54.443845 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:54.443812    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv" podUID="01c749b9-d188-4673-9a3d-99b9c4ae0793" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Nov 03 19:45:54.568695 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:54.544211    4921 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Nov 03 19:45:54.569971 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:54.569935    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl" podUID="55ff258b-9816-4fff-a74a-3c417bf0f0ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Nov 03 19:45:54.581809 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:54.569753    4921 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk" podUID="2d0a934e-4816-44d2-9bb5-e8b4e95dd73c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": context deadline exceeded"
Nov 03 19:45:54.582015 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:54.544804    4921 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 03 19:45:54.582153 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:54.582126    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj" podUID="30ae46dd-e9ca-4cd8-b78c-1a5463c98d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 03 19:45:54.634726 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:54.596853    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk" podUID="2d0a934e-4816-44d2-9bb5-e8b4e95dd73c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:54.749575 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:54.749524    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw" podUID="72d96abc-f3b4-4eda-b3a1-b93ba658e1d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:54.784704 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:54.779502    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh" podUID="eac6c47d-8ae5-4e4b-b691-98d177887767" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:54.788232 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:54.788183    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6" podUID="1aaee61d-b036-4c2b-bee7-a692b8bed612" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:54.912799 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:54.912751    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt" podUID="17dc0ed2-5164-449d-be12-a38bdfeb946d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:55.025929 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:55.025837    4921 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Nov 03 19:45:55.044691 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:55.026143    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n" podUID="4ee1b6ae-046b-4402-a36e-bbd523e73463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Nov 03 19:45:55.061695 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:55.058923    4921 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 03 19:45:55.061695 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:55.058980    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt" podUID="5c2fc702-9338-4af0-a044-810fafccbfad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 03 19:45:55.094694 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:55.093875    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7" podUID="d4de0090-d775-4bd7-9ee2-058b5d380638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded"
Nov 03 19:45:55.126693 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:55.096269    4921 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 03 19:45:55.126693 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:55.096321    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww" podUID="a52b2e9b-9ca0-4897-a071-a79c80e40d14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 03 19:45:55.279145 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:55.279048    4921 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 03 19:45:55.280103 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:55.280071    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2" podUID="476b08ee-e24d-4f0c-a5ae-5cb23d1fa105" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 03 19:45:55.342808 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:55.342763    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9" podUID="e6a71aa3-33b3-42bd-b270-64ce3260cc2a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:55.482433 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:55.482385    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv" podUID="01c749b9-d188-4673-9a3d-99b9c4ae0793" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:55.483050 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:55.483008    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv" podUID="96b73342-0079-4709-8570-3dbce57e5243" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Nov 03 19:45:55.572980 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:55.514968    4921 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Nov 03 19:45:55.572980 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:55.540191    4921 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 03 19:45:55.572980 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:55.540241    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj" podUID="30ae46dd-e9ca-4cd8-b78c-1a5463c98d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 03 19:45:55.572980 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:55.565788    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl" podUID="55ff258b-9816-4fff-a74a-3c417bf0f0ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Nov 03 19:45:55.631729 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:55.613535    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk" podUID="2d0a934e-4816-44d2-9bb5-e8b4e95dd73c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Nov 03 19:45:55.641711 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:55.632956    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv"
Nov 03 19:45:55.905030 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:55.904920    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw" podUID="72d96abc-f3b4-4eda-b3a1-b93ba658e1d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Nov 03 19:45:56.036448 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:55.918088    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh" podUID="eac6c47d-8ae5-4e4b-b691-98d177887767" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded"
Nov 03 19:45:56.037396 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:55.918155    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6" podUID="1aaee61d-b036-4c2b-bee7-a692b8bed612" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 03 19:45:56.037896 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:55.978138    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt" podUID="17dc0ed2-5164-449d-be12-a38bdfeb946d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Nov 03 19:45:56.038292 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:56.011418    4921 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Nov 03 19:45:56.063825 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:56.063773    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n" podUID="4ee1b6ae-046b-4402-a36e-bbd523e73463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Nov 03 19:45:56.064093 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:56.062547    4921 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt" podUID="17dc0ed2-5164-449d-be12-a38bdfeb946d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": context deadline exceeded"
Nov 03 19:45:56.064310 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:56.062612    4921 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 03 19:45:56.153063 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:56.153022    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt" podUID="5c2fc702-9338-4af0-a044-810fafccbfad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 03 19:45:56.168313 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:56.162229    4921 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv" podUID="01c749b9-d188-4673-9a3d-99b9c4ae0793" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:56.168556 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:56.095960    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7" podUID="d4de0090-d775-4bd7-9ee2-058b5d380638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:56.198695 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:56.065649    4921 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 03 19:45:56.218877 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:56.218832    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww" podUID="a52b2e9b-9ca0-4897-a071-a79c80e40d14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 03 19:45:56.320627 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:56.320591    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt"
Nov 03 19:45:56.348698 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:56.329645    4921 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 03 19:45:56.352816 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:56.346386    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9" podUID="e6a71aa3-33b3-42bd-b270-64ce3260cc2a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:56.430243 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:56.430146    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2" podUID="476b08ee-e24d-4f0c-a5ae-5cb23d1fa105" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 03 19:45:56.551700 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:56.492230    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv" podUID="01c749b9-d188-4673-9a3d-99b9c4ae0793" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Nov 03 19:45:56.739754 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:56.736149    4921 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Nov 03 19:45:56.767695 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:56.672940    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk" podUID="2d0a934e-4816-44d2-9bb5-e8b4e95dd73c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:56.768950 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:56.768898    4921 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 03 19:45:56.804693 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:56.769182    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl" podUID="55ff258b-9816-4fff-a74a-3c417bf0f0ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Nov 03 19:45:56.935694 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:56.769216    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj" podUID="30ae46dd-e9ca-4cd8-b78c-1a5463c98d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 03 19:45:56.996876 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:56.672896    4921 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh" podUID="eac6c47d-8ae5-4e4b-b691-98d177887767" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": context deadline exceeded"
Nov 03 19:45:57.001746 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:56.977781    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv" podUID="96b73342-0079-4709-8570-3dbce57e5243" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:57.005156 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:57.005128    4921 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Nov 03 19:45:57.005371 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:57.005342    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n" podUID="4ee1b6ae-046b-4402-a36e-bbd523e73463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Nov 03 19:45:57.053691 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:57.050381    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh" podUID="eac6c47d-8ae5-4e4b-b691-98d177887767" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:57.053691 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:57.050818    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw" podUID="72d96abc-f3b4-4eda-b3a1-b93ba658e1d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:57.053691 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:57.052958    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6" podUID="1aaee61d-b036-4c2b-bee7-a692b8bed612" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:57.083702 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:57.082857    4921 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 03 19:45:57.083702 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:57.082919    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt" podUID="5c2fc702-9338-4af0-a044-810fafccbfad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 03 19:45:57.083702 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:57.082997    4921 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 03 19:45:57.083702 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:57.083021    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww" podUID="a52b2e9b-9ca0-4897-a071-a79c80e40d14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 03 19:45:57.171046 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:57.171003    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7" podUID="d4de0090-d775-4bd7-9ee2-058b5d380638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:57.318875 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:57.318775    4921 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 03 19:45:57.319161 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:57.319118    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2" podUID="476b08ee-e24d-4f0c-a5ae-5cb23d1fa105" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 03 19:45:57.356801 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:57.356759    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9" podUID="e6a71aa3-33b3-42bd-b270-64ce3260cc2a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:57.408855 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:57.408794    4921 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6" podUID="1aaee61d-b036-4c2b-bee7-a692b8bed612" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:57.515234 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:57.515197    4921 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Nov 03 19:45:57.515506 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:57.515478    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl" podUID="55ff258b-9816-4fff-a74a-3c417bf0f0ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Nov 03 19:45:57.561239 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:57.561204    4921 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 03 19:45:57.561484 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:57.561455    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj" podUID="30ae46dd-e9ca-4cd8-b78c-1a5463c98d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 03 19:45:57.674908 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:57.674813    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv" podUID="01c749b9-d188-4673-9a3d-99b9c4ae0793" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:57.799722 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:57.799630    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk" podUID="2d0a934e-4816-44d2-9bb5-e8b4e95dd73c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Nov 03 19:45:57.923737 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:57.921450    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt" podUID="17dc0ed2-5164-449d-be12-a38bdfeb946d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Nov 03 19:45:58.022692 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:58.006045    4921 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Nov 03 19:45:58.022692 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:58.006108    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n" podUID="4ee1b6ae-046b-4402-a36e-bbd523e73463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Nov 03 19:45:58.022692 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:58.006505    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv" podUID="96b73342-0079-4709-8570-3dbce57e5243" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Nov 03 19:45:58.054217 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:58.054176    4921 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 03 19:45:58.054498 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:58.054466    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt" podUID="5c2fc702-9338-4af0-a044-810fafccbfad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 03 19:45:58.055028 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:58.055003    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6" podUID="1aaee61d-b036-4c2b-bee7-a692b8bed612" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 03 19:45:58.055448 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:58.055426    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw" podUID="72d96abc-f3b4-4eda-b3a1-b93ba658e1d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Nov 03 19:45:58.056310 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:58.056043    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh" podUID="eac6c47d-8ae5-4e4b-b691-98d177887767" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:58.068290 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:58.068258    4921 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 03 19:45:58.068558 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:58.068522    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww" podUID="a52b2e9b-9ca0-4897-a071-a79c80e40d14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 03 19:45:58.177853 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:58.177804    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7" podUID="d4de0090-d775-4bd7-9ee2-058b5d380638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:58.278155 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:58.278064    4921 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 03 19:45:58.278446 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:58.278416    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2" podUID="476b08ee-e24d-4f0c-a5ae-5cb23d1fa105" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 03 19:45:58.368827 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:58.368776    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9" podUID="e6a71aa3-33b3-42bd-b270-64ce3260cc2a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:58.517108 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:58.517068    4921 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Nov 03 19:45:58.517368 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:58.517340    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl" podUID="55ff258b-9816-4fff-a74a-3c417bf0f0ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Nov 03 19:45:58.539087 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:58.538999    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7"
Nov 03 19:45:58.542233 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:58.542205    4921 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 03 19:45:58.562885 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:58.562842    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj" podUID="30ae46dd-e9ca-4cd8-b78c-1a5463c98d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 03 19:45:58.637287 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:58.637223    4921 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7" podUID="d4de0090-d775-4bd7-9ee2-058b5d380638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:58.741700 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:58.694772    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv" podUID="01c749b9-d188-4673-9a3d-99b9c4ae0793" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Nov 03 19:45:58.804628 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:58.804519    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk" podUID="2d0a934e-4816-44d2-9bb5-e8b4e95dd73c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:58.930945 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:58.930898    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt" podUID="17dc0ed2-5164-449d-be12-a38bdfeb946d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:59.064755 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:59.025731    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv" podUID="96b73342-0079-4709-8570-3dbce57e5243" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:59.070693 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:59.028314    4921 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Nov 03 19:45:59.074825 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:59.074777    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n" podUID="4ee1b6ae-046b-4402-a36e-bbd523e73463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Nov 03 19:45:59.091779 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:59.056034    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6" podUID="1aaee61d-b036-4c2b-bee7-a692b8bed612" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:59.092320 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:59.077450    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh" podUID="eac6c47d-8ae5-4e4b-b691-98d177887767" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded"
Nov 03 19:45:59.150386 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:59.150340    4921 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 03 19:45:59.150605 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:59.150409    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww" podUID="a52b2e9b-9ca0-4897-a071-a79c80e40d14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 03 19:45:59.151620 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:59.151588    4921 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 03 19:45:59.151764 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:59.151636    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt" podUID="5c2fc702-9338-4af0-a044-810fafccbfad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 03 19:45:59.153473 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:59.153442    4921 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9" podUID="e6a71aa3-33b3-42bd-b270-64ce3260cc2a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:59.239695 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:59.199622    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt"
Nov 03 19:45:59.331772 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:59.207808    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw" podUID="72d96abc-f3b4-4eda-b3a1-b93ba658e1d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:59.331772 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:59.294780    4921 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 03 19:45:59.331772 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:59.299538    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2" podUID="476b08ee-e24d-4f0c-a5ae-5cb23d1fa105" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 03 19:45:59.379286 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:59.379240    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9" podUID="e6a71aa3-33b3-42bd-b270-64ce3260cc2a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:59.519180 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:59.519140    4921 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Nov 03 19:45:59.529546 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:59.529501    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl" podUID="55ff258b-9816-4fff-a74a-3c417bf0f0ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Nov 03 19:45:59.544490 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:59.544451    4921 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 03 19:45:59.544739 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:59.544714    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj" podUID="30ae46dd-e9ca-4cd8-b78c-1a5463c98d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 03 19:45:59.735761 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:59.723791    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv" podUID="01c749b9-d188-4673-9a3d-99b9c4ae0793" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Nov 03 19:45:59.805829 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:59.805780    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk" podUID="2d0a934e-4816-44d2-9bb5-e8b4e95dd73c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:45:59.972693 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:45:59.958869    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt" podUID="17dc0ed2-5164-449d-be12-a38bdfeb946d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Nov 03 19:46:00.016749 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:00.011892    4921 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Nov 03 19:46:00.016749 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:00.011986    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n" podUID="4ee1b6ae-046b-4402-a36e-bbd523e73463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Nov 03 19:46:00.058576 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:00.058529    4921 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 03 19:46:00.065834 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:00.064773    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt" podUID="5c2fc702-9338-4af0-a044-810fafccbfad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 03 19:46:00.087697 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:00.086396    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv" podUID="96b73342-0079-4709-8570-3dbce57e5243" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:00.087697 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:00.087563    4921 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 03 19:46:00.087697 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:00.087626    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww" podUID="a52b2e9b-9ca0-4897-a071-a79c80e40d14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 03 19:46:00.096699 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:00.092328    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6" podUID="1aaee61d-b036-4c2b-bee7-a692b8bed612" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:00.096699 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:00.092701    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh" podUID="eac6c47d-8ae5-4e4b-b691-98d177887767" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:00.117659 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:00.117612    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7" podUID="d4de0090-d775-4bd7-9ee2-058b5d380638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:00.214844 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:00.214790    4921 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw" podUID="72d96abc-f3b4-4eda-b3a1-b93ba658e1d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:00.301853 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:00.301735    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw" podUID="72d96abc-f3b4-4eda-b3a1-b93ba658e1d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:00.339704 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:00.335950    4921 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 03 19:46:00.360788 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:00.360741    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2" podUID="476b08ee-e24d-4f0c-a5ae-5cb23d1fa105" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 03 19:46:00.444918 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:00.444858    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9" podUID="e6a71aa3-33b3-42bd-b270-64ce3260cc2a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Nov 03 19:46:00.529202 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:00.529163    4921 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Nov 03 19:46:00.529463 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:00.529433    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl" podUID="55ff258b-9816-4fff-a74a-3c417bf0f0ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Nov 03 19:46:00.574752 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:00.571471    4921 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 03 19:46:00.581693 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:00.574995    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj" podUID="30ae46dd-e9ca-4cd8-b78c-1a5463c98d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 03 19:46:00.737694 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:00.734441    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv" podUID="01c749b9-d188-4673-9a3d-99b9c4ae0793" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:00.745271 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:00.745209    4921 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv" podUID="96b73342-0079-4709-8570-3dbce57e5243" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:00.807891 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:00.807832    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk" podUID="2d0a934e-4816-44d2-9bb5-e8b4e95dd73c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:00.963900 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:00.963785    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt" podUID="17dc0ed2-5164-449d-be12-a38bdfeb946d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:01.003131 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:01.003092    4921 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Nov 03 19:46:01.003468 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:01.003435    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n" podUID="4ee1b6ae-046b-4402-a36e-bbd523e73463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Nov 03 19:46:01.054995 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:01.054959    4921 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 03 19:46:01.055339 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:01.055290    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt" podUID="5c2fc702-9338-4af0-a044-810fafccbfad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 03 19:46:01.068238 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:01.068201    4921 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 03 19:46:01.069378 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:01.069346    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww" podUID="a52b2e9b-9ca0-4897-a071-a79c80e40d14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 03 19:46:01.089815 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:01.089768    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv" podUID="96b73342-0079-4709-8570-3dbce57e5243" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Nov 03 19:46:01.112017 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:01.111971    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh" podUID="eac6c47d-8ae5-4e4b-b691-98d177887767" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:01.112639 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:01.112610    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6" podUID="1aaee61d-b036-4c2b-bee7-a692b8bed612" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 03 19:46:01.147838 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:01.147790    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7" podUID="d4de0090-d775-4bd7-9ee2-058b5d380638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:01.149493 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:01.149470    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7"
Nov 03 19:46:01.416694 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:01.407968    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw" podUID="72d96abc-f3b4-4eda-b3a1-b93ba658e1d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:01.416694 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:01.411556    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9"
Nov 03 19:46:01.450700 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:01.430811    4921 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 03 19:46:01.450700 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:01.430879    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2" podUID="476b08ee-e24d-4f0c-a5ae-5cb23d1fa105" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 03 19:46:01.762695 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:01.750787    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv" podUID="01c749b9-d188-4673-9a3d-99b9c4ae0793" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:01.825388 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:01.825341    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk" podUID="2d0a934e-4816-44d2-9bb5-e8b4e95dd73c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:01.865009 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:01.864976    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6"
Nov 03 19:46:01.896693 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:01.892824    4921 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 03 19:46:01.896693 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:01.892888    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj" podUID="30ae46dd-e9ca-4cd8-b78c-1a5463c98d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 03 19:46:01.896693 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:01.893364    4921 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Nov 03 19:46:01.896693 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:01.893398    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl" podUID="55ff258b-9816-4fff-a74a-3c417bf0f0ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Nov 03 19:46:01.989695 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:01.985936    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt" podUID="17dc0ed2-5164-449d-be12-a38bdfeb946d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:02.055230 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:02.055132    4921 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Nov 03 19:46:02.055476 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:02.055445    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n" podUID="4ee1b6ae-046b-4402-a36e-bbd523e73463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Nov 03 19:46:02.123690 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:02.116004    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh" podUID="eac6c47d-8ae5-4e4b-b691-98d177887767" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:02.123690 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:02.125618    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv" podUID="96b73342-0079-4709-8570-3dbce57e5243" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Nov 03 19:46:02.161636 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:02.161600    4921 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 03 19:46:02.161941 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:02.161911    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt" podUID="5c2fc702-9338-4af0-a044-810fafccbfad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 03 19:46:02.179613 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:02.173864    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv"
Nov 03 19:46:02.198829 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:02.198565    4921 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 03 19:46:02.199263 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:02.199230    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww" podUID="a52b2e9b-9ca0-4897-a071-a79c80e40d14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 03 19:46:02.261838 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:02.261795    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7" podUID="d4de0090-d775-4bd7-9ee2-058b5d380638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:02.301711 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:02.298804    4921 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 03 19:46:02.301711 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:02.298873    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2" podUID="476b08ee-e24d-4f0c-a5ae-5cb23d1fa105" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 03 19:46:02.301711 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:02.300934    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh"
Nov 03 19:46:02.472128 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:02.472024    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw" podUID="72d96abc-f3b4-4eda-b3a1-b93ba658e1d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:02.593723 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:02.577034    4921 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 03 19:46:02.593723 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:02.577099    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj" podUID="30ae46dd-e9ca-4cd8-b78c-1a5463c98d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 03 19:46:02.593723 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:02.577533    4921 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Nov 03 19:46:02.593723 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:02.577581    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl" podUID="55ff258b-9816-4fff-a74a-3c417bf0f0ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Nov 03 19:46:02.647426 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:02.647364    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa409566-5lzcd"]
Nov 03 19:46:02.670211 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:02.670172    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa409566-5lzcd"
Nov 03 19:46:02.693959 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:02.692487    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 19:46:02.693959 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:02.693169    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:46:02.746253 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:02.740596    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7790312d-8886-4f05-a13f-01ad360672da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa409566-5lzcd\" (UID: \"7790312d-8886-4f05-a13f-01ad360672da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa409566-5lzcd"
Nov 03 19:46:02.746253 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:02.740681    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7790312d-8886-4f05-a13f-01ad360672da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa409566-5lzcd\" (UID: \"7790312d-8886-4f05-a13f-01ad360672da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa409566-5lzcd"
Nov 03 19:46:02.746253 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:02.740833    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgsxd\" (UniqueName: \"kubernetes.io/projected/7790312d-8886-4f05-a13f-01ad360672da-kube-api-access-mgsxd\") pod \"maas-api-revoke-sub-fa409566-5lzcd\" (UID: \"7790312d-8886-4f05-a13f-01ad360672da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa409566-5lzcd"
Nov 03 19:46:02.757859 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:02.754026    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa409566-5lzcd"]
Nov 03 19:46:02.827579 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:02.826769    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk" podUID="2d0a934e-4816-44d2-9bb5-e8b4e95dd73c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Nov 03 19:46:02.851693 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:02.845846    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgsxd\" (UniqueName: \"kubernetes.io/projected/7790312d-8886-4f05-a13f-01ad360672da-kube-api-access-mgsxd\") pod \"maas-api-revoke-sub-fa409566-5lzcd\" (UID: \"7790312d-8886-4f05-a13f-01ad360672da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa409566-5lzcd"
Nov 03 19:46:02.851693 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:02.846086    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7790312d-8886-4f05-a13f-01ad360672da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa409566-5lzcd\" (UID: \"7790312d-8886-4f05-a13f-01ad360672da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa409566-5lzcd"
Nov 03 19:46:02.851693 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:02.846139    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7790312d-8886-4f05-a13f-01ad360672da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa409566-5lzcd\" (UID: \"7790312d-8886-4f05-a13f-01ad360672da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa409566-5lzcd"
Nov 03 19:46:02.851693 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:02.847145    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7790312d-8886-4f05-a13f-01ad360672da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa409566-5lzcd\" (UID: \"7790312d-8886-4f05-a13f-01ad360672da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa409566-5lzcd"
Nov 03 19:46:02.876692 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:02.876144    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7790312d-8886-4f05-a13f-01ad360672da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa409566-5lzcd\" (UID: \"7790312d-8886-4f05-a13f-01ad360672da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa409566-5lzcd"
Nov 03 19:46:02.948706 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:02.945646    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgsxd\" (UniqueName: \"kubernetes.io/projected/7790312d-8886-4f05-a13f-01ad360672da-kube-api-access-mgsxd\") pod \"maas-api-revoke-sub-fa409566-5lzcd\" (UID: \"7790312d-8886-4f05-a13f-01ad360672da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa409566-5lzcd"
Nov 03 19:46:03.061508 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:03.061465    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt"
Nov 03 19:46:03.108717 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:03.064050    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt"
Nov 03 19:46:03.108717 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:03.093838    4921 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Nov 03 19:46:03.108717 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:03.093924    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n" podUID="4ee1b6ae-046b-4402-a36e-bbd523e73463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Nov 03 19:46:03.113769 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:03.109851    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n"
Nov 03 19:46:03.142694 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:03.114189    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww"
Nov 03 19:46:03.142694 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:03.121033    4921 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 03 19:46:03.142694 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:03.121090    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww" podUID="a52b2e9b-9ca0-4897-a071-a79c80e40d14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 03 19:46:03.180695 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:03.143069    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv" podUID="96b73342-0079-4709-8570-3dbce57e5243" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:03.293693 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:03.292900    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2"
Nov 03 19:46:03.516692 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:03.506728    4921 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 03 19:46:03.516692 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:03.506807    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2" podUID="476b08ee-e24d-4f0c-a5ae-5cb23d1fa105" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 03 19:46:03.574371 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:03.559814    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw" podUID="72d96abc-f3b4-4eda-b3a1-b93ba658e1d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:03.574371 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:03.560839    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7" podUID="d4de0090-d775-4bd7-9ee2-058b5d380638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:03.668695 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:03.666496    4921 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Nov 03 19:46:03.668695 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:03.666585    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl" podUID="55ff258b-9816-4fff-a74a-3c417bf0f0ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Nov 03 19:46:03.668695 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:03.668216    4921 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 03 19:46:03.668695 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:03.668264    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj" podUID="30ae46dd-e9ca-4cd8-b78c-1a5463c98d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 03 19:46:03.693627 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:03.687702    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj"
Nov 03 19:46:03.729166 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:03.728049    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa409566-5lzcd"
Nov 03 19:46:03.786860 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:03.779360    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl"
Nov 03 19:46:04.051752 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:04.026948    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv"
Nov 03 19:46:04.067301 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:04.067253    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk" podUID="2d0a934e-4816-44d2-9bb5-e8b4e95dd73c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:04.085539 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:04.085499    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6" podUID="1aaee61d-b036-4c2b-bee7-a692b8bed612" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:04.099710 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:04.097953    4921 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:46:04.099710 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:04.098038    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt" podUID="5c2fc702-9338-4af0-a044-810fafccbfad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:04.385753 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:04.362225    4921 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 03 19:46:04.385753 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:04.362306    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2" podUID="476b08ee-e24d-4f0c-a5ae-5cb23d1fa105" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 03 19:46:04.385753 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:04.364765    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw"
Nov 03 19:46:04.457791 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:46:04.428722    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 19:46:04.476861 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:04.468973    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9" podUID="e6a71aa3-33b3-42bd-b270-64ce3260cc2a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:04.557721 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:04.541065    4921 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 03 19:46:04.557721 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:04.541137    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj" podUID="30ae46dd-e9ca-4cd8-b78c-1a5463c98d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 03 19:46:04.561024 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:04.560983    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7" podUID="d4de0090-d775-4bd7-9ee2-058b5d380638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:04.643072 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:04.642967    4921 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk" podUID="2d0a934e-4816-44d2-9bb5-e8b4e95dd73c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:04.656956 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:04.656915    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv" podUID="01c749b9-d188-4673-9a3d-99b9c4ae0793" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Nov 03 19:46:04.671206 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:04.671176    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk"
Nov 03 19:46:04.901937 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:04.901832    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh" podUID="eac6c47d-8ae5-4e4b-b691-98d177887767" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:04.990866 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:04.990790    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt" podUID="17dc0ed2-5164-449d-be12-a38bdfeb946d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:05.086694 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:05.083830    4921 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:46:05.086694 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:05.083902    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n" podUID="4ee1b6ae-046b-4402-a36e-bbd523e73463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:05.118877 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:05.113245    4921 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:46:05.118877 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:05.113315    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww" podUID="a52b2e9b-9ca0-4897-a071-a79c80e40d14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:05.147863 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:05.147822    4921 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:46:05.148110 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:05.148082    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt" podUID="5c2fc702-9338-4af0-a044-810fafccbfad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:05.305881 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:05.305848    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl"
Nov 03 19:46:05.376917 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:05.372715    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa409566-5lzcd"]
Nov 03 19:46:05.394735 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:05.392154    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl"
Nov 03 19:46:05.568490 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:05.568399    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7" podUID="d4de0090-d775-4bd7-9ee2-058b5d380638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:05.605873 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:05.597236    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n"
Nov 03 19:46:05.650118 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:05.650091    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n"
Nov 03 19:46:05.718050 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:05.718023    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7"
Nov 03 19:46:05.857182 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:05.857051    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj"
Nov 03 19:46:05.878073 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:05.867305    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj"
Nov 03 19:46:05.905049 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:05.903277    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2"
Nov 03 19:46:05.908104 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:05.907841    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2"
Nov 03 19:46:06.126313 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:06.121907    4921 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:46:06.126313 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:06.121979    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww" podUID="a52b2e9b-9ca0-4897-a071-a79c80e40d14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:06.152031 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:06.149049    4921 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:46:06.152031 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:06.151260    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt" podUID="5c2fc702-9338-4af0-a044-810fafccbfad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:06.164223 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:06.162152    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt"
Nov 03 19:46:06.170958 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:06.168221    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt"
Nov 03 19:46:06.171890 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:06.171855    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa409566-5lzcd" event={"ID":"7790312d-8886-4f05-a13f-01ad360672da","Type":"ContainerStarted","Data":"b409e19fbc80fe6e64d42c6b9cd549b247b076f2fb7b78d37dcfdf46b35afaa6"}
Nov 03 19:46:06.913124 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:06.913082    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww"
Nov 03 19:46:06.915326 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:06.915300    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww"
Nov 03 19:46:07.181058 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:07.180974    4921 generic.go:356] "Generic (PLEG): container finished" podID="7790312d-8886-4f05-a13f-01ad360672da" containerID="a2729f3f40ebc255abd98c542ee106d02ff07c2549f52976ff62421db1c72510" exitCode=0
Nov 03 19:46:07.181220 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:07.181057    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa409566-5lzcd" event={"ID":"7790312d-8886-4f05-a13f-01ad360672da","Type":"ContainerDied","Data":"a2729f3f40ebc255abd98c542ee106d02ff07c2549f52976ff62421db1c72510"}
Nov 03 19:46:08.373816 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:08.373787    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa409566-5lzcd"
Nov 03 19:46:08.501752 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:08.501716    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7790312d-8886-4f05-a13f-01ad360672da-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7790312d-8886-4f05-a13f-01ad360672da-maas-api-xks-ca\") pod \"7790312d-8886-4f05-a13f-01ad360672da\" (UID: \"7790312d-8886-4f05-a13f-01ad360672da\") "
Nov 03 19:46:08.501924 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:08.501845    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7790312d-8886-4f05-a13f-01ad360672da-kube-api-access-mgsxd\" (UniqueName: \"kubernetes.io/projected/7790312d-8886-4f05-a13f-01ad360672da-kube-api-access-mgsxd\") pod \"7790312d-8886-4f05-a13f-01ad360672da\" (UID: \"7790312d-8886-4f05-a13f-01ad360672da\") "
Nov 03 19:46:08.502609 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:08.502265    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7790312d-8886-4f05-a13f-01ad360672da-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7790312d-8886-4f05-a13f-01ad360672da-maas-api-openshift-ca\") pod \"7790312d-8886-4f05-a13f-01ad360672da\" (UID: \"7790312d-8886-4f05-a13f-01ad360672da\") "
Nov 03 19:46:08.502725 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:08.502640    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7790312d-8886-4f05-a13f-01ad360672da-maas-api-openshift-ca" pod "7790312d-8886-4f05-a13f-01ad360672da" (UID: "7790312d-8886-4f05-a13f-01ad360672da"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:46:08.503206 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:08.503182    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7790312d-8886-4f05-a13f-01ad360672da-maas-api-openshift-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:46:08.506900 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:08.506876    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7790312d-8886-4f05-a13f-01ad360672da-kube-api-access-mgsxd" pod "7790312d-8886-4f05-a13f-01ad360672da" (UID: "7790312d-8886-4f05-a13f-01ad360672da"). InnerVolumeSpecName "kube-api-access-mgsxd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:46:08.507163 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:08.507018    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7790312d-8886-4f05-a13f-01ad360672da-maas-api-xks-ca" pod "7790312d-8886-4f05-a13f-01ad360672da" (UID: "7790312d-8886-4f05-a13f-01ad360672da"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:46:08.608452 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:08.608397    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7790312d-8886-4f05-a13f-01ad360672da-maas-api-xks-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:46:08.610602 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:08.610537    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgsxd\" (UniqueName: \"kubernetes.io/projected/7790312d-8886-4f05-a13f-01ad360672da-kube-api-access-mgsxd\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:46:09.198401 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:09.198328    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa409566-5lzcd"
Nov 03 19:46:09.198401 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:09.198339    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa409566-5lzcd" event={"ID":"7790312d-8886-4f05-a13f-01ad360672da","Type":"ContainerDied","Data":"b409e19fbc80fe6e64d42c6b9cd549b247b076f2fb7b78d37dcfdf46b35afaa6"}
Nov 03 19:46:09.198582 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:09.198401    4921 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b409e19fbc80fe6e64d42c6b9cd549b247b076f2fb7b78d37dcfdf46b35afaa6"
Nov 03 19:46:11.506542 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:11.506510    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63a98a9b-bkrfb"]
Nov 03 19:46:11.507204 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:11.507189    4921 cpu_manager.go:414] "containerMap: removing container" podUID="7790312d-8886-4f05-a13f-01ad360672da" containerName="revoke-keys"
Nov 03 19:46:11.507256 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:11.507206    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7790312d-8886-4f05-a13f-01ad360672da" containerName="revoke-keys"
Nov 03 19:46:11.507314 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:11.507304    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="7790312d-8886-4f05-a13f-01ad360672da" containerName="revoke-keys"
Nov 03 19:46:11.511057 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:11.511032    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63a98a9b-bkrfb"
Nov 03 19:46:11.518204 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:11.518165    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:46:11.518364 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:11.518271    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 19:46:11.521201 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:11.521173    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63a98a9b-bkrfb"]
Nov 03 19:46:11.577886 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:11.577854    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a5b26fb-965e-4220-b649-8968cb6fb371-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63a98a9b-bkrfb\" (UID: \"0a5b26fb-965e-4220-b649-8968cb6fb371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63a98a9b-bkrfb"
Nov 03 19:46:11.578059 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:11.577912    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a5b26fb-965e-4220-b649-8968cb6fb371-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63a98a9b-bkrfb\" (UID: \"0a5b26fb-965e-4220-b649-8968cb6fb371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63a98a9b-bkrfb"
Nov 03 19:46:11.578059 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:11.578040    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpvb4\" (UniqueName: \"kubernetes.io/projected/0a5b26fb-965e-4220-b649-8968cb6fb371-kube-api-access-bpvb4\") pod \"maas-api-revoke-sub-63a98a9b-bkrfb\" (UID: \"0a5b26fb-965e-4220-b649-8968cb6fb371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63a98a9b-bkrfb"
Nov 03 19:46:11.679071 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:11.679035    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a5b26fb-965e-4220-b649-8968cb6fb371-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63a98a9b-bkrfb\" (UID: \"0a5b26fb-965e-4220-b649-8968cb6fb371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63a98a9b-bkrfb"
Nov 03 19:46:11.679254 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:11.679106    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a5b26fb-965e-4220-b649-8968cb6fb371-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63a98a9b-bkrfb\" (UID: \"0a5b26fb-965e-4220-b649-8968cb6fb371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63a98a9b-bkrfb"
Nov 03 19:46:11.679254 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:11.679161    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpvb4\" (UniqueName: \"kubernetes.io/projected/0a5b26fb-965e-4220-b649-8968cb6fb371-kube-api-access-bpvb4\") pod \"maas-api-revoke-sub-63a98a9b-bkrfb\" (UID: \"0a5b26fb-965e-4220-b649-8968cb6fb371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63a98a9b-bkrfb"
Nov 03 19:46:11.679798 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:11.679778    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a5b26fb-965e-4220-b649-8968cb6fb371-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63a98a9b-bkrfb\" (UID: \"0a5b26fb-965e-4220-b649-8968cb6fb371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63a98a9b-bkrfb"
Nov 03 19:46:11.681499 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:11.681475    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a5b26fb-965e-4220-b649-8968cb6fb371-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63a98a9b-bkrfb\" (UID: \"0a5b26fb-965e-4220-b649-8968cb6fb371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63a98a9b-bkrfb"
Nov 03 19:46:11.688291 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:11.688264    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpvb4\" (UniqueName: \"kubernetes.io/projected/0a5b26fb-965e-4220-b649-8968cb6fb371-kube-api-access-bpvb4\") pod \"maas-api-revoke-sub-63a98a9b-bkrfb\" (UID: \"0a5b26fb-965e-4220-b649-8968cb6fb371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63a98a9b-bkrfb"
Nov 03 19:46:11.832491 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:11.832409    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63a98a9b-bkrfb"
Nov 03 19:46:12.114494 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:12.114454    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63a98a9b-bkrfb"]
Nov 03 19:46:12.114880 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:46:12.114854    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a5b26fb_965e_4220_b649_8968cb6fb371.slice/crio-bad56ff84dbb6cb202a50fd1bd7dc886bc216a9d7b94f87a386bd4b986a26b68 WatchSource:0}: Error finding container bad56ff84dbb6cb202a50fd1bd7dc886bc216a9d7b94f87a386bd4b986a26b68: Status 404 returned error can't find the container with id bad56ff84dbb6cb202a50fd1bd7dc886bc216a9d7b94f87a386bd4b986a26b68
Nov 03 19:46:12.251719 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:12.251660    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63a98a9b-bkrfb" event={"ID":"0a5b26fb-965e-4220-b649-8968cb6fb371","Type":"ContainerStarted","Data":"bad56ff84dbb6cb202a50fd1bd7dc886bc216a9d7b94f87a386bd4b986a26b68"}
Nov 03 19:46:13.261389 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:13.261355    4921 generic.go:356] "Generic (PLEG): container finished" podID="0a5b26fb-965e-4220-b649-8968cb6fb371" containerID="5042d5effeb93fb51af9bcb7402e687fa6725b846b9cd947c1e14519e3086c84" exitCode=0
Nov 03 19:46:13.261853 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:13.261603    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63a98a9b-bkrfb" event={"ID":"0a5b26fb-965e-4220-b649-8968cb6fb371","Type":"ContainerDied","Data":"5042d5effeb93fb51af9bcb7402e687fa6725b846b9cd947c1e14519e3086c84"}
Nov 03 19:46:14.466551 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:14.466526    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63a98a9b-bkrfb"
Nov 03 19:46:14.555236 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:14.555148    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a5b26fb-965e-4220-b649-8968cb6fb371-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a5b26fb-965e-4220-b649-8968cb6fb371-maas-api-openshift-ca\") pod \"0a5b26fb-965e-4220-b649-8968cb6fb371\" (UID: \"0a5b26fb-965e-4220-b649-8968cb6fb371\") "
Nov 03 19:46:14.555236 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:14.555223    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a5b26fb-965e-4220-b649-8968cb6fb371-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a5b26fb-965e-4220-b649-8968cb6fb371-maas-api-xks-ca\") pod \"0a5b26fb-965e-4220-b649-8968cb6fb371\" (UID: \"0a5b26fb-965e-4220-b649-8968cb6fb371\") "
Nov 03 19:46:14.555467 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:14.555310    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a5b26fb-965e-4220-b649-8968cb6fb371-kube-api-access-bpvb4\" (UniqueName: \"kubernetes.io/projected/0a5b26fb-965e-4220-b649-8968cb6fb371-kube-api-access-bpvb4\") pod \"0a5b26fb-965e-4220-b649-8968cb6fb371\" (UID: \"0a5b26fb-965e-4220-b649-8968cb6fb371\") "
Nov 03 19:46:14.555593 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:14.555566    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a5b26fb-965e-4220-b649-8968cb6fb371-maas-api-openshift-ca" pod "0a5b26fb-965e-4220-b649-8968cb6fb371" (UID: "0a5b26fb-965e-4220-b649-8968cb6fb371"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:46:14.555791 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:14.555772    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a5b26fb-965e-4220-b649-8968cb6fb371-maas-api-openshift-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:46:14.557510 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:14.557480    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a5b26fb-965e-4220-b649-8968cb6fb371-maas-api-xks-ca" pod "0a5b26fb-965e-4220-b649-8968cb6fb371" (UID: "0a5b26fb-965e-4220-b649-8968cb6fb371"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:46:14.557610 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:14.557529    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a5b26fb-965e-4220-b649-8968cb6fb371-kube-api-access-bpvb4" pod "0a5b26fb-965e-4220-b649-8968cb6fb371" (UID: "0a5b26fb-965e-4220-b649-8968cb6fb371"). InnerVolumeSpecName "kube-api-access-bpvb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:46:14.656545 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:14.656512    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a5b26fb-965e-4220-b649-8968cb6fb371-maas-api-xks-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:46:14.656545 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:14.656542    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpvb4\" (UniqueName: \"kubernetes.io/projected/0a5b26fb-965e-4220-b649-8968cb6fb371-kube-api-access-bpvb4\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:46:15.279448 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:15.279418    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63a98a9b-bkrfb"
Nov 03 19:46:15.279619 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:15.279443    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63a98a9b-bkrfb" event={"ID":"0a5b26fb-965e-4220-b649-8968cb6fb371","Type":"ContainerDied","Data":"bad56ff84dbb6cb202a50fd1bd7dc886bc216a9d7b94f87a386bd4b986a26b68"}
Nov 03 19:46:15.279619 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:15.279488    4921 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bad56ff84dbb6cb202a50fd1bd7dc886bc216a9d7b94f87a386bd4b986a26b68"
Nov 03 19:46:17.657896 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:17.657861    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf"]
Nov 03 19:46:17.658879 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:17.658858    4921 cpu_manager.go:414] "containerMap: removing container" podUID="0a5b26fb-965e-4220-b649-8968cb6fb371" containerName="revoke-keys"
Nov 03 19:46:17.659000 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:17.658880    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a5b26fb-965e-4220-b649-8968cb6fb371" containerName="revoke-keys"
Nov 03 19:46:17.659069 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:17.659055    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a5b26fb-965e-4220-b649-8968cb6fb371" containerName="revoke-keys"
Nov 03 19:46:17.666621 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:17.666599    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf"
Nov 03 19:46:17.671601 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:17.671508    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf"]
Nov 03 19:46:17.693118 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:17.693090    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49c66530-de54-4ec9-83cb-61cafce44e7b-istiod-ca-cert\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf\" (UID: \"49c66530-de54-4ec9-83cb-61cafce44e7b\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf"
Nov 03 19:46:17.693248 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:17.693157    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49c66530-de54-4ec9-83cb-61cafce44e7b-istio-token\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf\" (UID: \"49c66530-de54-4ec9-83cb-61cafce44e7b\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf"
Nov 03 19:46:17.693316 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:17.693241    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49c66530-de54-4ec9-83cb-61cafce44e7b-istio-data\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf\" (UID: \"49c66530-de54-4ec9-83cb-61cafce44e7b\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf"
Nov 03 19:46:17.693316 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:17.693284    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gftmd\" (UniqueName: \"kubernetes.io/projected/49c66530-de54-4ec9-83cb-61cafce44e7b-kube-api-access-gftmd\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf\" (UID: \"49c66530-de54-4ec9-83cb-61cafce44e7b\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf"
Nov 03 19:46:17.693487 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:17.693467    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49c66530-de54-4ec9-83cb-61cafce44e7b-istio-podinfo\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf\" (UID: \"49c66530-de54-4ec9-83cb-61cafce44e7b\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf"
Nov 03 19:46:17.693551 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:17.693536    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49c66530-de54-4ec9-83cb-61cafce44e7b-workload-certs\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf\" (UID: \"49c66530-de54-4ec9-83cb-61cafce44e7b\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf"
Nov 03 19:46:17.693679 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:17.693648    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49c66530-de54-4ec9-83cb-61cafce44e7b-credential-socket\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf\" (UID: \"49c66530-de54-4ec9-83cb-61cafce44e7b\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf"
Nov 03 19:46:17.693746 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:17.693729    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49c66530-de54-4ec9-83cb-61cafce44e7b-workload-socket\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf\" (UID: \"49c66530-de54-4ec9-83cb-61cafce44e7b\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf"
Nov 03 19:46:17.693837 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:17.693817    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49c66530-de54-4ec9-83cb-61cafce44e7b-istio-envoy\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf\" (UID: \"49c66530-de54-4ec9-83cb-61cafce44e7b\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf"
Nov 03 19:46:17.795115 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:17.795072    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49c66530-de54-4ec9-83cb-61cafce44e7b-credential-socket\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf\" (UID: \"49c66530-de54-4ec9-83cb-61cafce44e7b\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf"
Nov 03 19:46:17.795293 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:17.795132    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49c66530-de54-4ec9-83cb-61cafce44e7b-workload-socket\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf\" (UID: \"49c66530-de54-4ec9-83cb-61cafce44e7b\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf"
Nov 03 19:46:17.795293 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:17.795176    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49c66530-de54-4ec9-83cb-61cafce44e7b-istio-envoy\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf\" (UID: \"49c66530-de54-4ec9-83cb-61cafce44e7b\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf"
Nov 03 19:46:17.795293 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:17.795231    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49c66530-de54-4ec9-83cb-61cafce44e7b-istiod-ca-cert\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf\" (UID: \"49c66530-de54-4ec9-83cb-61cafce44e7b\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf"
Nov 03 19:46:17.795293 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:17.795292    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49c66530-de54-4ec9-83cb-61cafce44e7b-istio-token\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf\" (UID: \"49c66530-de54-4ec9-83cb-61cafce44e7b\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf"
Nov 03 19:46:17.795523 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:17.795311    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49c66530-de54-4ec9-83cb-61cafce44e7b-istio-data\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf\" (UID: \"49c66530-de54-4ec9-83cb-61cafce44e7b\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf"
Nov 03 19:46:17.795523 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:17.795344    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gftmd\" (UniqueName: \"kubernetes.io/projected/49c66530-de54-4ec9-83cb-61cafce44e7b-kube-api-access-gftmd\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf\" (UID: \"49c66530-de54-4ec9-83cb-61cafce44e7b\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf"
Nov 03 19:46:17.795523 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:17.795470    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49c66530-de54-4ec9-83cb-61cafce44e7b-istio-podinfo\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf\" (UID: \"49c66530-de54-4ec9-83cb-61cafce44e7b\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf"
Nov 03 19:46:17.795523 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:17.795499    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49c66530-de54-4ec9-83cb-61cafce44e7b-credential-socket\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf\" (UID: \"49c66530-de54-4ec9-83cb-61cafce44e7b\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf"
Nov 03 19:46:17.795759 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:17.795542    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49c66530-de54-4ec9-83cb-61cafce44e7b-workload-socket\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf\" (UID: \"49c66530-de54-4ec9-83cb-61cafce44e7b\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf"
Nov 03 19:46:17.795759 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:17.795555    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49c66530-de54-4ec9-83cb-61cafce44e7b-workload-certs\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf\" (UID: \"49c66530-de54-4ec9-83cb-61cafce44e7b\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf"
Nov 03 19:46:17.795951 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:17.795929    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49c66530-de54-4ec9-83cb-61cafce44e7b-istio-data\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf\" (UID: \"49c66530-de54-4ec9-83cb-61cafce44e7b\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf"
Nov 03 19:46:17.796009 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:17.795934    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49c66530-de54-4ec9-83cb-61cafce44e7b-workload-certs\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf\" (UID: \"49c66530-de54-4ec9-83cb-61cafce44e7b\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf"
Nov 03 19:46:17.796057 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:17.796017    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49c66530-de54-4ec9-83cb-61cafce44e7b-istiod-ca-cert\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf\" (UID: \"49c66530-de54-4ec9-83cb-61cafce44e7b\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf"
Nov 03 19:46:17.798127 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:17.798099    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49c66530-de54-4ec9-83cb-61cafce44e7b-istio-podinfo\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf\" (UID: \"49c66530-de54-4ec9-83cb-61cafce44e7b\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf"
Nov 03 19:46:17.800903 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:17.800881    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49c66530-de54-4ec9-83cb-61cafce44e7b-istio-envoy\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf\" (UID: \"49c66530-de54-4ec9-83cb-61cafce44e7b\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf"
Nov 03 19:46:17.802208 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:17.802190    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49c66530-de54-4ec9-83cb-61cafce44e7b-istio-token\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf\" (UID: \"49c66530-de54-4ec9-83cb-61cafce44e7b\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf"
Nov 03 19:46:17.802726 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:17.802655    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gftmd\" (UniqueName: \"kubernetes.io/projected/49c66530-de54-4ec9-83cb-61cafce44e7b-kube-api-access-gftmd\") pod \"e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf\" (UID: \"49c66530-de54-4ec9-83cb-61cafce44e7b\") " pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf"
Nov 03 19:46:17.983951 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:17.983922    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf"
Nov 03 19:46:18.149012 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:18.148978    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766"]
Nov 03 19:46:18.154769 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:18.154733    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766"
Nov 03 19:46:18.165011 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:18.164815    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766"]
Nov 03 19:46:18.184405 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:18.184373    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6"]
Nov 03 19:46:18.195325 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:18.195291    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6"
Nov 03 19:46:18.207112 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:18.207076    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdkz7\" (UniqueName: \"kubernetes.io/projected/7ad21605-a58c-494c-89e0-6aca910b76b1-kube-api-access-fdkz7\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766\" (UID: \"7ad21605-a58c-494c-89e0-6aca910b76b1\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766"
Nov 03 19:46:18.207274 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:18.207140    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ad21605-a58c-494c-89e0-6aca910b76b1-workload-certs\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766\" (UID: \"7ad21605-a58c-494c-89e0-6aca910b76b1\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766"
Nov 03 19:46:18.207274 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:18.207184    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/052a7a9f-ed4b-4cb9-911a-413079445a1c-istio-token\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6\" (UID: \"052a7a9f-ed4b-4cb9-911a-413079445a1c\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6"
Nov 03 19:46:18.207274 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:18.207259    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rndt\" (UniqueName: \"kubernetes.io/projected/052a7a9f-ed4b-4cb9-911a-413079445a1c-kube-api-access-9rndt\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6\" (UID: \"052a7a9f-ed4b-4cb9-911a-413079445a1c\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6"
Nov 03 19:46:18.207443 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:18.207300    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ad21605-a58c-494c-89e0-6aca910b76b1-istio-token\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766\" (UID: \"7ad21605-a58c-494c-89e0-6aca910b76b1\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766"
Nov 03 19:46:18.207443 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:18.207333    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/052a7a9f-ed4b-4cb9-911a-413079445a1c-istiod-ca-cert\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6\" (UID: \"052a7a9f-ed4b-4cb9-911a-413079445a1c\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6"
Nov 03 19:46:18.207443 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:18.207387    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ad21605-a58c-494c-89e0-6aca910b76b1-istio-envoy\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766\" (UID: \"7ad21605-a58c-494c-89e0-6aca910b76b1\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766"
Nov 03 19:46:18.207443 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:18.207437    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/052a7a9f-ed4b-4cb9-911a-413079445a1c-istio-podinfo\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6\" (UID: \"052a7a9f-ed4b-4cb9-911a-413079445a1c\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6"
Nov 03 19:46:18.207651 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:18.207490    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ad21605-a58c-494c-89e0-6aca910b76b1-istiod-ca-cert\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766\" (UID: \"7ad21605-a58c-494c-89e0-6aca910b76b1\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766"
Nov 03 19:46:18.207651 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:18.207540    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ad21605-a58c-494c-89e0-6aca910b76b1-istio-data\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766\" (UID: \"7ad21605-a58c-494c-89e0-6aca910b76b1\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766"
Nov 03 19:46:18.207651 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:18.207594    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ad21605-a58c-494c-89e0-6aca910b76b1-workload-socket\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766\" (UID: \"7ad21605-a58c-494c-89e0-6aca910b76b1\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766"
Nov 03 19:46:18.207651 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:18.207630    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ad21605-a58c-494c-89e0-6aca910b76b1-istio-podinfo\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766\" (UID: \"7ad21605-a58c-494c-89e0-6aca910b76b1\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766"
Nov 03 19:46:18.207908 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:18.207686    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ad21605-a58c-494c-89e0-6aca910b76b1-credential-socket\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766\" (UID: \"7ad21605-a58c-494c-89e0-6aca910b76b1\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766"
Nov 03 19:46:18.207908 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:18.207899    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/052a7a9f-ed4b-4cb9-911a-413079445a1c-istio-data\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6\" (UID: \"052a7a9f-ed4b-4cb9-911a-413079445a1c\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6"
Nov 03 19:46:18.208012 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:18.207972    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/052a7a9f-ed4b-4cb9-911a-413079445a1c-credential-socket\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6\" (UID: \"052a7a9f-ed4b-4cb9-911a-413079445a1c\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6"
Nov 03 19:46:18.208236 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:18.208213    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/052a7a9f-ed4b-4cb9-911a-413079445a1c-workload-certs\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6\" (UID: \"052a7a9f-ed4b-4cb9-911a-413079445a1c\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6"
Nov 03 19:46:18.208307 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:18.208290    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/052a7a9f-ed4b-4cb9-911a-413079445a1c-workload-socket\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6\" (UID: \"052a7a9f-ed4b-4cb9-911a-413079445a1c\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6"
Nov 03 19:46:18.217976 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:18.217943    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/052a7a9f-ed4b-4cb9-911a-413079445a1c-istio-envoy\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6\" (UID: \"052a7a9f-ed4b-4cb9-911a-413079445a1c\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6"
Nov 03 19:46:18.220529 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:18.220500    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6"]
Nov 03 19:46:18.319825 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:18.319748    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ad21605-a58c-494c-89e0-6aca910b76b1-credential-socket\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766\" (UID: \"7ad21605-a58c-494c-89e0-6aca910b76b1\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766"
Nov 03 19:46:18.320004 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:18.319829    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/052a7a9f-ed4b-4cb9-911a-413079445a1c-istio-data\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6\" (UID: \"052a7a9f-ed4b-4cb9-911a-413079445a1c\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6"
Nov 03 19:46:18.320004 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:18.319864    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/052a7a9f-ed4b-4cb9-911a-413079445a1c-credential-socket\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6\" (UID: \"052a7a9f-ed4b-4cb9-911a-413079445a1c\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6"
Nov 03 19:46:18.320004 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:18.319880    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/052a7a9f-ed4b-4cb9-911a-413079445a1c-workload-certs\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6\" (UID: \"052a7a9f-ed4b-4cb9-911a-413079445a1c\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6"
Nov 03 19:46:18.320004 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:18.319916    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/052a7a9f-ed4b-4cb9-911a-413079445a1c-workload-socket\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6\" (UID: \"052a7a9f-ed4b-4cb9-911a-413079445a1c\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6"
Nov 03 19:46:18.320004 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:18.319957    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/052a7a9f-ed4b-4cb9-911a-413079445a1c-istio-envoy\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6\" (UID: \"052a7a9f-ed4b-4cb9-911a-413079445a1c\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6"
Nov 03 19:46:18.320004 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:18.319996    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdkz7\" (UniqueName: \"kubernetes.io/projected/7ad21605-a58c-494c-89e0-6aca910b76b1-kube-api-access-fdkz7\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766\" (UID: \"7ad21605-a58c-494c-89e0-6aca910b76b1\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766"
Nov 03 19:46:18.320336 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:18.320026    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ad21605-a58c-494c-89e0-6aca910b76b1-workload-certs\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766\" (UID: \"7ad21605-a58c-494c-89e0-6aca910b76b1\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766"
Nov 03 19:46:18.320336 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:18.320055    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/052a7a9f-ed4b-4cb9-911a-413079445a1c-istio-token\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6\" (UID: \"052a7a9f-ed4b-4cb9-911a-413079445a1c\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6"
Nov 03 19:46:18.320336 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:18.320123    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rndt\" (UniqueName: \"kubernetes.io/projected/052a7a9f-ed4b-4cb9-911a-413079445a1c-kube-api-access-9rndt\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6\" (UID: \"052a7a9f-ed4b-4cb9-911a-413079445a1c\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6"
Nov 03 19:46:18.320336 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:18.320154    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ad21605-a58c-494c-89e0-6aca910b76b1-istio-token\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766\" (UID: \"7ad21605-a58c-494c-89e0-6aca910b76b1\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766"
Nov 03 19:46:18.320336 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:18.320184    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/052a7a9f-ed4b-4cb9-911a-413079445a1c-istiod-ca-cert\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6\" (UID: \"052a7a9f-ed4b-4cb9-911a-413079445a1c\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6"
Nov 03 19:46:18.320336 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:18.320211    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ad21605-a58c-494c-89e0-6aca910b76b1-credential-socket\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766\" (UID: \"7ad21605-a58c-494c-89e0-6aca910b76b1\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766"
Nov 03 19:46:18.320336 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:18.320256    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ad21605-a58c-494c-89e0-6aca910b76b1-istio-envoy\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766\" (UID: \"7ad21605-a58c-494c-89e0-6aca910b76b1\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766"
Nov 03 19:46:18.320336 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:18.320301    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/052a7a9f-ed4b-4cb9-911a-413079445a1c-istio-podinfo\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6\" (UID: \"052a7a9f-ed4b-4cb9-911a-413079445a1c\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6"
Nov 03 19:46:18.320336 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:18.320313    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/052a7a9f-ed4b-4cb9-911a-413079445a1c-credential-socket\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6\" (UID: \"052a7a9f-ed4b-4cb9-911a-413079445a1c\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6"
Nov 03 19:46:18.320336 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:18.320341    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/052a7a9f-ed4b-4cb9-911a-413079445a1c-istio-data\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6\" (UID: \"052a7a9f-ed4b-4cb9-911a-413079445a1c\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6"
Nov 03 19:46:18.320336 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:18.320354    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ad21605-a58c-494c-89e0-6aca910b76b1-istiod-ca-cert\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766\" (UID: \"7ad21605-a58c-494c-89e0-6aca910b76b1\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766"
Nov 03 19:46:18.321003 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:18.320396    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/052a7a9f-ed4b-4cb9-911a-413079445a1c-workload-certs\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6\" (UID: \"052a7a9f-ed4b-4cb9-911a-413079445a1c\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6"
Nov 03 19:46:18.321003 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:18.320410    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ad21605-a58c-494c-89e0-6aca910b76b1-istio-data\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766\" (UID: \"7ad21605-a58c-494c-89e0-6aca910b76b1\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766"
Nov 03 19:46:18.321003 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:18.320462    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ad21605-a58c-494c-89e0-6aca910b76b1-workload-socket\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766\" (UID: \"7ad21605-a58c-494c-89e0-6aca910b76b1\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766"
Nov 03 19:46:18.321003 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:18.320493    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ad21605-a58c-494c-89e0-6aca910b76b1-istio-podinfo\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766\" (UID: \"7ad21605-a58c-494c-89e0-6aca910b76b1\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766"
Nov 03 19:46:18.321003 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:18.320569    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ad21605-a58c-494c-89e0-6aca910b76b1-workload-certs\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766\" (UID: \"7ad21605-a58c-494c-89e0-6aca910b76b1\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766"
Nov 03 19:46:18.321003 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:18.320850    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ad21605-a58c-494c-89e0-6aca910b76b1-istio-data\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766\" (UID: \"7ad21605-a58c-494c-89e0-6aca910b76b1\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766"
Nov 03 19:46:18.321003 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:18.320938    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/052a7a9f-ed4b-4cb9-911a-413079445a1c-workload-socket\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6\" (UID: \"052a7a9f-ed4b-4cb9-911a-413079445a1c\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6"
Nov 03 19:46:18.321003 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:18.321002    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/052a7a9f-ed4b-4cb9-911a-413079445a1c-istiod-ca-cert\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6\" (UID: \"052a7a9f-ed4b-4cb9-911a-413079445a1c\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6"
Nov 03 19:46:18.321490 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:18.321090    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ad21605-a58c-494c-89e0-6aca910b76b1-workload-socket\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766\" (UID: \"7ad21605-a58c-494c-89e0-6aca910b76b1\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766"
Nov 03 19:46:18.321490 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:18.321333    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ad21605-a58c-494c-89e0-6aca910b76b1-istiod-ca-cert\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766\" (UID: \"7ad21605-a58c-494c-89e0-6aca910b76b1\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766"
Nov 03 19:46:18.323039 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:18.323015    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/052a7a9f-ed4b-4cb9-911a-413079445a1c-istio-podinfo\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6\" (UID: \"052a7a9f-ed4b-4cb9-911a-413079445a1c\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6"
Nov 03 19:46:18.323146 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:18.323071    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ad21605-a58c-494c-89e0-6aca910b76b1-istio-envoy\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766\" (UID: \"7ad21605-a58c-494c-89e0-6aca910b76b1\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766"
Nov 03 19:46:18.323146 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:18.323071    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/052a7a9f-ed4b-4cb9-911a-413079445a1c-istio-envoy\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6\" (UID: \"052a7a9f-ed4b-4cb9-911a-413079445a1c\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6"
Nov 03 19:46:18.323146 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:18.323134    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ad21605-a58c-494c-89e0-6aca910b76b1-istio-podinfo\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766\" (UID: \"7ad21605-a58c-494c-89e0-6aca910b76b1\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766"
Nov 03 19:46:18.329609 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:18.329583    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/052a7a9f-ed4b-4cb9-911a-413079445a1c-istio-token\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6\" (UID: \"052a7a9f-ed4b-4cb9-911a-413079445a1c\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6"
Nov 03 19:46:18.329860 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:18.329842    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rndt\" (UniqueName: \"kubernetes.io/projected/052a7a9f-ed4b-4cb9-911a-413079445a1c-kube-api-access-9rndt\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6\" (UID: \"052a7a9f-ed4b-4cb9-911a-413079445a1c\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6"
Nov 03 19:46:18.330298 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:18.330272    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdkz7\" (UniqueName: \"kubernetes.io/projected/7ad21605-a58c-494c-89e0-6aca910b76b1-kube-api-access-fdkz7\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766\" (UID: \"7ad21605-a58c-494c-89e0-6aca910b76b1\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766"
Nov 03 19:46:18.330398 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:18.330340    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ad21605-a58c-494c-89e0-6aca910b76b1-istio-token\") pod \"e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766\" (UID: \"7ad21605-a58c-494c-89e0-6aca910b76b1\") " pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766"
Nov 03 19:46:18.330558 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:46:18.330537    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49c66530_de54_4ec9_83cb_61cafce44e7b.slice/crio-95dfb0f9de24a473dae99f88dd2775aca114f8eddbf26df8f7ac41474ae22476 WatchSource:0}: Error finding container 95dfb0f9de24a473dae99f88dd2775aca114f8eddbf26df8f7ac41474ae22476: Status 404 returned error can't find the container with id 95dfb0f9de24a473dae99f88dd2775aca114f8eddbf26df8f7ac41474ae22476
Nov 03 19:46:18.331308 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:18.331288    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf"]
Nov 03 19:46:18.333095 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:18.333062    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:46:18.333172 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:18.333126    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:46:18.333172 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:18.333154    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:46:18.485786 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:18.485418    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766"
Nov 03 19:46:18.525168 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:18.525024    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6"
Nov 03 19:46:18.798960 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:18.798928    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766"]
Nov 03 19:46:18.799324 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:46:18.799297    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ad21605_a58c_494c_89e0_6aca910b76b1.slice/crio-d912cf58fb150f6c9c67b773adf8e8d4f97477658a1f9e45a6a1fae87cd8cd55 WatchSource:0}: Error finding container d912cf58fb150f6c9c67b773adf8e8d4f97477658a1f9e45a6a1fae87cd8cd55: Status 404 returned error can't find the container with id d912cf58fb150f6c9c67b773adf8e8d4f97477658a1f9e45a6a1fae87cd8cd55
Nov 03 19:46:18.802296 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:18.802246    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:46:18.802416 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:18.802343    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:46:18.802416 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:18.802383    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:46:18.839906 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:18.839864    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6"]
Nov 03 19:46:18.844752 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:46:18.844724    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod052a7a9f_ed4b_4cb9_911a_413079445a1c.slice/crio-6ba7d9163a33a22610bd4b5bbc2bc1abe348b35b383e852f7db1f507b6e95bc1 WatchSource:0}: Error finding container 6ba7d9163a33a22610bd4b5bbc2bc1abe348b35b383e852f7db1f507b6e95bc1: Status 404 returned error can't find the container with id 6ba7d9163a33a22610bd4b5bbc2bc1abe348b35b383e852f7db1f507b6e95bc1
Nov 03 19:46:18.847360 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:18.847322    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:46:18.847467 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:18.847399    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:46:18.847467 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:18.847443    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:46:19.313634 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:19.313593    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6" event={"ID":"052a7a9f-ed4b-4cb9-911a-413079445a1c","Type":"ContainerStarted","Data":"fa3fdaac1525b4a72fe74d752a4613a5148615674024627d6ecf2e38ae7084e3"}
Nov 03 19:46:19.313823 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:19.313683    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6" event={"ID":"052a7a9f-ed4b-4cb9-911a-413079445a1c","Type":"ContainerStarted","Data":"6ba7d9163a33a22610bd4b5bbc2bc1abe348b35b383e852f7db1f507b6e95bc1"}
Nov 03 19:46:19.315932 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:19.315893    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766" event={"ID":"7ad21605-a58c-494c-89e0-6aca910b76b1","Type":"ContainerStarted","Data":"2ce740bd67ada8c9acf3c3b091c62171649d15c1f554157cdc8dd661c65a3926"}
Nov 03 19:46:19.316064 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:19.315970    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766" event={"ID":"7ad21605-a58c-494c-89e0-6aca910b76b1","Type":"ContainerStarted","Data":"d912cf58fb150f6c9c67b773adf8e8d4f97477658a1f9e45a6a1fae87cd8cd55"}
Nov 03 19:46:19.318059 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:19.318006    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf" event={"ID":"49c66530-de54-4ec9-83cb-61cafce44e7b","Type":"ContainerStarted","Data":"b1392ce25e2c9ca2cae30c03bd2245b22ef79386e97066c48a8160da89465b41"}
Nov 03 19:46:19.318183 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:19.318087    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf" event={"ID":"49c66530-de54-4ec9-83cb-61cafce44e7b","Type":"ContainerStarted","Data":"95dfb0f9de24a473dae99f88dd2775aca114f8eddbf26df8f7ac41474ae22476"}
Nov 03 19:46:19.334969 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:19.334915    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6" podStartSLOduration=1.334899518 podStartE2EDuration="1.334899518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:46:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:46:19.332536199 +0000 UTC m=+892.576501665" watchObservedRunningTime="2026-11-03 19:46:19.334899518 +0000 UTC m=+892.578864987"
Nov 03 19:46:19.359695 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:19.357548    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766" podStartSLOduration=1.357528425 podStartE2EDuration="1.357528425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:46:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:46:19.348151729 +0000 UTC m=+892.592117198" watchObservedRunningTime="2026-11-03 19:46:19.357528425 +0000 UTC m=+892.601493899"
Nov 03 19:46:19.394803 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:19.394747    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf" podStartSLOduration=2.394732597 podStartE2EDuration="2.394732597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:46:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:46:19.389884143 +0000 UTC m=+892.633849622" watchObservedRunningTime="2026-11-03 19:46:19.394732597 +0000 UTC m=+892.638698062"
Nov 03 19:46:19.486572 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:19.486538    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766"
Nov 03 19:46:19.487944 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:19.487921    4921 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Nov 03 19:46:19.488016 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:19.487968    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766" podUID="7ad21605-a58c-494c-89e0-6aca910b76b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Nov 03 19:46:19.525219 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:19.525184    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6"
Nov 03 19:46:19.526608 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:19.526572    4921 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 03 19:46:19.526688 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:19.526631    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6" podUID="052a7a9f-ed4b-4cb9-911a-413079445a1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 03 19:46:19.990692 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:19.986992    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf"
Nov 03 19:46:19.990692 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:19.987646    4921 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 03 19:46:19.990692 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:19.987731    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf" podUID="49c66530-de54-4ec9-83cb-61cafce44e7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 03 19:46:20.492702 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:20.492107    4921 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Nov 03 19:46:20.492702 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:20.492172    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766" podUID="7ad21605-a58c-494c-89e0-6aca910b76b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Nov 03 19:46:20.526244 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:20.526204    4921 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 03 19:46:20.526527 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:20.526496    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6" podUID="052a7a9f-ed4b-4cb9-911a-413079445a1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 03 19:46:20.999689 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:20.986798    4921 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 03 19:46:20.999689 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:20.986863    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf" podUID="49c66530-de54-4ec9-83cb-61cafce44e7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 03 19:46:21.506695 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:21.490380    4921 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Nov 03 19:46:21.506695 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:21.490507    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766" podUID="7ad21605-a58c-494c-89e0-6aca910b76b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Nov 03 19:46:21.536692 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:21.526608    4921 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 03 19:46:21.536692 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:21.526703    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6" podUID="052a7a9f-ed4b-4cb9-911a-413079445a1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 03 19:46:21.999692 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:21.986098    4921 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 03 19:46:21.999692 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:21.986179    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf" podUID="49c66530-de54-4ec9-83cb-61cafce44e7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 03 19:46:22.525692 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:22.524834    4921 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Nov 03 19:46:22.525692 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:22.524901    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766" podUID="7ad21605-a58c-494c-89e0-6aca910b76b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Nov 03 19:46:22.525692 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:22.525633    4921 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 03 19:46:22.525692 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:22.525692    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6" podUID="052a7a9f-ed4b-4cb9-911a-413079445a1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 03 19:46:22.999753 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:22.988910    4921 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 03 19:46:22.999753 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:22.988971    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf" podUID="49c66530-de54-4ec9-83cb-61cafce44e7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 03 19:46:23.697276 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:23.693995    4921 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 03 19:46:23.697276 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:23.694086    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6" podUID="052a7a9f-ed4b-4cb9-911a-413079445a1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 03 19:46:23.697276 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:23.694589    4921 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Nov 03 19:46:23.697276 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:23.694630    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766" podUID="7ad21605-a58c-494c-89e0-6aca910b76b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Nov 03 19:46:24.064161 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:24.064126    4921 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 03 19:46:24.064838 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:24.064807    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf" podUID="49c66530-de54-4ec9-83cb-61cafce44e7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 03 19:46:24.378812 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:24.374871    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv" podUID="01c749b9-d188-4673-9a3d-99b9c4ae0793" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Nov 03 19:46:24.494459 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:24.494420    4921 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Nov 03 19:46:24.494767 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:24.494737    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766" podUID="7ad21605-a58c-494c-89e0-6aca910b76b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Nov 03 19:46:24.527825 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:24.527787    4921 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 03 19:46:24.528101 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:24.528071    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6" podUID="052a7a9f-ed4b-4cb9-911a-413079445a1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 03 19:46:24.802136 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:24.798779    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh" podUID="eac6c47d-8ae5-4e4b-b691-98d177887767" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:24.985345 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:24.985308    4921 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 03 19:46:24.986086 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:24.986048    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf" podUID="49c66530-de54-4ec9-83cb-61cafce44e7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 03 19:46:25.084222 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:25.077129    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt" podUID="17dc0ed2-5164-449d-be12-a38bdfeb946d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:25.084222 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:25.079527    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6" podUID="1aaee61d-b036-4c2b-bee7-a692b8bed612" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:25.105807 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:25.105749    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv" podUID="96b73342-0079-4709-8570-3dbce57e5243" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Nov 03 19:46:25.442751 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:25.426085    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv" podUID="01c749b9-d188-4673-9a3d-99b9c4ae0793" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Nov 03 19:46:25.612724 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:25.572774    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv"
Nov 03 19:46:25.612724 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:25.453069    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7" podUID="d4de0090-d775-4bd7-9ee2-058b5d380638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:25.612724 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:25.538008    4921 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Nov 03 19:46:25.612724 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:25.573471    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766" podUID="7ad21605-a58c-494c-89e0-6aca910b76b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Nov 03 19:46:25.612724 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:25.538048    4921 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 03 19:46:25.612724 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:25.573521    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6" podUID="052a7a9f-ed4b-4cb9-911a-413079445a1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 03 19:46:25.612724 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:25.453603    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9" podUID="e6a71aa3-33b3-42bd-b270-64ce3260cc2a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:25.676693 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:25.632561    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk" podUID="2d0a934e-4816-44d2-9bb5-e8b4e95dd73c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:25.772824 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:25.772772    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw" podUID="72d96abc-f3b4-4eda-b3a1-b93ba658e1d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Nov 03 19:46:25.828706 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:25.827878    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh" podUID="eac6c47d-8ae5-4e4b-b691-98d177887767" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:25.832703 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:25.829791    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh"
Nov 03 19:46:26.000696 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:25.995220    4921 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 03 19:46:26.000696 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:25.995304    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf" podUID="49c66530-de54-4ec9-83cb-61cafce44e7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 03 19:46:26.084753 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:26.080891    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt" podUID="17dc0ed2-5164-449d-be12-a38bdfeb946d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:26.127696 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:26.086764    4921 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt" podUID="17dc0ed2-5164-449d-be12-a38bdfeb946d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:26.127696 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:26.086860    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6" podUID="1aaee61d-b036-4c2b-bee7-a692b8bed612" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 03 19:46:26.127696 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:26.097744    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt"
Nov 03 19:46:26.127696 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:26.098613    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6"
Nov 03 19:46:26.127696 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:26.125771    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv" podUID="96b73342-0079-4709-8570-3dbce57e5243" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:26.127696 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:26.126210    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv"
Nov 03 19:46:26.148695 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:26.141088    4921 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv" podUID="01c749b9-d188-4673-9a3d-99b9c4ae0793" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:26.596693 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:26.568293    4921 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Nov 03 19:46:26.596693 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:26.568355    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766" podUID="7ad21605-a58c-494c-89e0-6aca910b76b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Nov 03 19:46:26.596693 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:26.579715    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9" podUID="e6a71aa3-33b3-42bd-b270-64ce3260cc2a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Nov 03 19:46:26.596693 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:26.580125    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9"
Nov 03 19:46:26.596693 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:26.581696    4921 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 03 19:46:26.596693 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:26.581766    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6" podUID="052a7a9f-ed4b-4cb9-911a-413079445a1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 03 19:46:26.596693 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:26.586913    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7" podUID="d4de0090-d775-4bd7-9ee2-058b5d380638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:26.596693 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:26.587214    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv" podUID="01c749b9-d188-4673-9a3d-99b9c4ae0793" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Nov 03 19:46:26.596693 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:26.587489    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7"
Nov 03 19:46:26.706696 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:26.672960    4921 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh" podUID="eac6c47d-8ae5-4e4b-b691-98d177887767" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:26.754693 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:26.687211    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk" podUID="2d0a934e-4816-44d2-9bb5-e8b4e95dd73c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:26.754693 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:26.742526    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk"
Nov 03 19:46:26.782747 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:26.773228    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw" podUID="72d96abc-f3b4-4eda-b3a1-b93ba658e1d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:26.782747 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:26.773443    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw"
Nov 03 19:46:26.851961 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:26.851813    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh" podUID="eac6c47d-8ae5-4e4b-b691-98d177887767" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded"
Nov 03 19:46:27.111030 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:27.110933    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6" podUID="1aaee61d-b036-4c2b-bee7-a692b8bed612" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:27.114152 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:27.114118    4921 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 03 19:46:27.114371 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:27.114344    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf" podUID="49c66530-de54-4ec9-83cb-61cafce44e7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 03 19:46:27.132441 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:27.132393    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt" podUID="17dc0ed2-5164-449d-be12-a38bdfeb946d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Nov 03 19:46:27.156124 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:27.156072    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv" podUID="96b73342-0079-4709-8570-3dbce57e5243" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:27.223694 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:27.219767    4921 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:46:27.223694 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:27.219854    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt" podUID="ffb30cac-fb9f-4a3f-b954-ecd050b8cb41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:27.423818 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:27.412848    4921 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6" podUID="1aaee61d-b036-4c2b-bee7-a692b8bed612" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": context deadline exceeded"
Nov 03 19:46:27.562580 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:27.562528    4921 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 03 19:46:27.562907 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:27.562862    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6" podUID="052a7a9f-ed4b-4cb9-911a-413079445a1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 03 19:46:27.596599 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:27.539856    4921 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Nov 03 19:46:27.596599 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:27.566847    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766" podUID="7ad21605-a58c-494c-89e0-6aca910b76b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Nov 03 19:46:27.597054 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:27.597010    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9" podUID="e6a71aa3-33b3-42bd-b270-64ce3260cc2a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Nov 03 19:46:27.597780 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:27.597748    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv" podUID="01c749b9-d188-4673-9a3d-99b9c4ae0793" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:27.598243 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:27.598215    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7" podUID="d4de0090-d775-4bd7-9ee2-058b5d380638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded"
Nov 03 19:46:27.645024 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:27.644987    4921 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:46:27.673887 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:27.648756    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr" podUID="c05f083e-3052-44f9-ac8f-746ea890191b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:28.019694 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:27.750300    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk" podUID="2d0a934e-4816-44d2-9bb5-e8b4e95dd73c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:28.027695 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:27.793837    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw" podUID="72d96abc-f3b4-4eda-b3a1-b93ba658e1d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:28.027695 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:27.886875    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh" podUID="eac6c47d-8ae5-4e4b-b691-98d177887767" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:28.027695 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:28.010250    4921 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 03 19:46:28.027695 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:28.025784    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf" podUID="49c66530-de54-4ec9-83cb-61cafce44e7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 03 19:46:28.461886 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:28.461769    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv" podUID="96b73342-0079-4709-8570-3dbce57e5243" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:28.462796 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:28.462761    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6" podUID="1aaee61d-b036-4c2b-bee7-a692b8bed612" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 03 19:46:28.475699 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:28.471019    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt" podUID="17dc0ed2-5164-449d-be12-a38bdfeb946d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Nov 03 19:46:28.516295 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:28.499916    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7"
Nov 03 19:46:28.516295 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:28.514415    4921 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Nov 03 19:46:28.516295 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:28.514486    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766" podUID="7ad21605-a58c-494c-89e0-6aca910b76b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Nov 03 19:46:28.552694 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:28.539711    4921 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 03 19:46:28.552694 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:28.539794    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6" podUID="052a7a9f-ed4b-4cb9-911a-413079445a1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 03 19:46:28.652135 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:28.650488    4921 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7" podUID="d4de0090-d775-4bd7-9ee2-058b5d380638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:28.652135 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:28.651103    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9" podUID="e6a71aa3-33b3-42bd-b270-64ce3260cc2a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:28.807833 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:28.807783    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv" podUID="01c749b9-d188-4673-9a3d-99b9c4ae0793" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:28.984448 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:28.984412    4921 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 03 19:46:28.984788 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:28.984753    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf" podUID="49c66530-de54-4ec9-83cb-61cafce44e7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 03 19:46:29.025610 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:29.025543    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk" podUID="2d0a934e-4816-44d2-9bb5-e8b4e95dd73c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Nov 03 19:46:29.026217 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:29.025481    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw" podUID="72d96abc-f3b4-4eda-b3a1-b93ba658e1d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:29.032965 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:29.032915    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh" podUID="eac6c47d-8ae5-4e4b-b691-98d177887767" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:29.227761 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:29.169165    4921 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9" podUID="e6a71aa3-33b3-42bd-b270-64ce3260cc2a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:29.483464 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:29.480396    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt" podUID="17dc0ed2-5164-449d-be12-a38bdfeb946d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Nov 03 19:46:29.483464 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:29.480812    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv" podUID="96b73342-0079-4709-8570-3dbce57e5243" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Nov 03 19:46:29.483464 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:29.481092    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6" podUID="1aaee61d-b036-4c2b-bee7-a692b8bed612" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 03 19:46:29.502815 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:29.502773    4921 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Nov 03 19:46:29.503073 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:29.503042    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766" podUID="7ad21605-a58c-494c-89e0-6aca910b76b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Nov 03 19:46:29.531703 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:29.526222    4921 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 03 19:46:29.532095 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:29.532057    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6" podUID="052a7a9f-ed4b-4cb9-911a-413079445a1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 03 19:46:29.608008 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:29.607973    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw"
Nov 03 19:46:29.658700 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:29.655832    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7" podUID="d4de0090-d775-4bd7-9ee2-058b5d380638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:29.658700 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:29.656602    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9" podUID="e6a71aa3-33b3-42bd-b270-64ce3260cc2a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:29.763330 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:29.756230    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6mrqr" podUID="73136ab5-fad4-49ac-a39c-26abd533c8a1" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.132.0.38:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:29.818856 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:29.818811    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv" podUID="01c749b9-d188-4673-9a3d-99b9c4ae0793" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Nov 03 19:46:29.987712 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:29.967771    4921 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:46:29.987712 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:29.967836    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6" podUID="7e96267a-d738-4325-90e7-d1c9ce87eeb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:30.114828 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:30.095819    4921 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 03 19:46:30.114828 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:30.095884    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf" podUID="49c66530-de54-4ec9-83cb-61cafce44e7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 03 19:46:30.114828 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:30.105222    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk" podUID="2d0a934e-4816-44d2-9bb5-e8b4e95dd73c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Nov 03 19:46:30.114828 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:30.107253    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh" podUID="eac6c47d-8ae5-4e4b-b691-98d177887767" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded"
Nov 03 19:46:30.456754 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:30.453562    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gzf8k_6f215b42-0f09-46a0-bc98-392c0fa909c9/console-operator/2.log"
Nov 03 19:46:30.489837 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:30.489796    4921 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Nov 03 19:46:30.583575 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:30.490776    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6" podUID="1aaee61d-b036-4c2b-bee7-a692b8bed612" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:30.615918 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:30.490807    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt" podUID="17dc0ed2-5164-449d-be12-a38bdfeb946d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Nov 03 19:46:30.615918 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:30.490827    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv" podUID="96b73342-0079-4709-8570-3dbce57e5243" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Nov 03 19:46:30.615918 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:30.491038    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gzf8k_6f215b42-0f09-46a0-bc98-392c0fa909c9/console-operator/2.log"
Nov 03 19:46:30.615918 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:30.496976    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9"
Nov 03 19:46:30.616475 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:30.557740    4921 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 03 19:46:30.616475 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:30.616405    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6" podUID="052a7a9f-ed4b-4cb9-911a-413079445a1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 03 19:46:30.627054 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:30.627008    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766" podUID="7ad21605-a58c-494c-89e0-6aca910b76b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Nov 03 19:46:30.703771 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:30.697775    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7" podUID="d4de0090-d775-4bd7-9ee2-058b5d380638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:30.703771 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:30.701284    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7"
Nov 03 19:46:30.786983 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:30.764663    4921 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv" podUID="96b73342-0079-4709-8570-3dbce57e5243" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": context deadline exceeded"
Nov 03 19:46:30.860887 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:30.859975    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv" podUID="01c749b9-d188-4673-9a3d-99b9c4ae0793" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:31.210776 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:31.195365    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh" podUID="eac6c47d-8ae5-4e4b-b691-98d177887767" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded"
Nov 03 19:46:31.302524 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:31.302451    4921 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 03 19:46:31.319395 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:31.319354    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf" podUID="49c66530-de54-4ec9-83cb-61cafce44e7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 03 19:46:31.339284 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:31.339249    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk"
Nov 03 19:46:31.385688 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:31.366059    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7"
Nov 03 19:46:31.385688 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:31.368465    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw" podUID="72d96abc-f3b4-4eda-b3a1-b93ba658e1d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Nov 03 19:46:31.412693 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:31.400525    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt"
Nov 03 19:46:31.416078 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:31.416050    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv"
Nov 03 19:46:31.463173 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:31.463090    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6"
Nov 03 19:46:31.467360 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:31.467330    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv"
Nov 03 19:46:31.505608 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:31.505576    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh"
Nov 03 19:46:31.507858 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:31.507748    4921 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Nov 03 19:46:31.507858 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:31.507799    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766" podUID="7ad21605-a58c-494c-89e0-6aca910b76b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Nov 03 19:46:31.537599 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:31.537567    4921 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 03 19:46:31.537872 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:31.537843    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6" podUID="052a7a9f-ed4b-4cb9-911a-413079445a1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 03 19:46:31.986241 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:31.985097    4921 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 03 19:46:31.986241 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:31.985854    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf" podUID="49c66530-de54-4ec9-83cb-61cafce44e7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 03 19:46:32.486153 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:32.486112    4921 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Nov 03 19:46:32.486351 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:32.486177    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766" podUID="7ad21605-a58c-494c-89e0-6aca910b76b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Nov 03 19:46:32.973939 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:46:32.973816    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 19:46:32.984908 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:32.984156    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf"
Nov 03 19:46:33.188474 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:33.187476    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4006e3d-wdjqb"]
Nov 03 19:46:33.266254 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:33.266218    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4006e3d-wdjqb"]
Nov 03 19:46:33.266588 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:33.266571    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4006e3d-wdjqb"
Nov 03 19:46:33.271825 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:33.271801    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 19:46:33.272065 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:33.272028    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:46:33.370408 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:33.366959    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6"
Nov 03 19:46:33.370408 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:33.368608    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6"
Nov 03 19:46:33.385866 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:33.385787    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6"
Nov 03 19:46:33.409817 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:33.407909    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e41d2ecd-31d0-4110-af26-8ca6c3f9a593-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4006e3d-wdjqb\" (UID: \"e41d2ecd-31d0-4110-af26-8ca6c3f9a593\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4006e3d-wdjqb"
Nov 03 19:46:33.409817 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:33.407970    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk7ld\" (UniqueName: \"kubernetes.io/projected/e41d2ecd-31d0-4110-af26-8ca6c3f9a593-kube-api-access-hk7ld\") pod \"maas-api-revoke-sub-f4006e3d-wdjqb\" (UID: \"e41d2ecd-31d0-4110-af26-8ca6c3f9a593\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4006e3d-wdjqb"
Nov 03 19:46:33.409817 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:33.408580    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e41d2ecd-31d0-4110-af26-8ca6c3f9a593-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4006e3d-wdjqb\" (UID: \"e41d2ecd-31d0-4110-af26-8ca6c3f9a593\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4006e3d-wdjqb"
Nov 03 19:46:33.487832 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:33.486547    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766"
Nov 03 19:46:33.510769 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:33.510735    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e41d2ecd-31d0-4110-af26-8ca6c3f9a593-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4006e3d-wdjqb\" (UID: \"e41d2ecd-31d0-4110-af26-8ca6c3f9a593\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4006e3d-wdjqb"
Nov 03 19:46:33.510938 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:33.510778    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk7ld\" (UniqueName: \"kubernetes.io/projected/e41d2ecd-31d0-4110-af26-8ca6c3f9a593-kube-api-access-hk7ld\") pod \"maas-api-revoke-sub-f4006e3d-wdjqb\" (UID: \"e41d2ecd-31d0-4110-af26-8ca6c3f9a593\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4006e3d-wdjqb"
Nov 03 19:46:33.510938 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:33.510812    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e41d2ecd-31d0-4110-af26-8ca6c3f9a593-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4006e3d-wdjqb\" (UID: \"e41d2ecd-31d0-4110-af26-8ca6c3f9a593\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4006e3d-wdjqb"
Nov 03 19:46:33.513438 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:33.513406    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e41d2ecd-31d0-4110-af26-8ca6c3f9a593-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4006e3d-wdjqb\" (UID: \"e41d2ecd-31d0-4110-af26-8ca6c3f9a593\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4006e3d-wdjqb"
Nov 03 19:46:33.516056 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:33.516011    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e41d2ecd-31d0-4110-af26-8ca6c3f9a593-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4006e3d-wdjqb\" (UID: \"e41d2ecd-31d0-4110-af26-8ca6c3f9a593\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4006e3d-wdjqb"
Nov 03 19:46:33.518638 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:33.518579    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk7ld\" (UniqueName: \"kubernetes.io/projected/e41d2ecd-31d0-4110-af26-8ca6c3f9a593-kube-api-access-hk7ld\") pod \"maas-api-revoke-sub-f4006e3d-wdjqb\" (UID: \"e41d2ecd-31d0-4110-af26-8ca6c3f9a593\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4006e3d-wdjqb"
Nov 03 19:46:33.597560 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:33.597249    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4006e3d-wdjqb"
Nov 03 19:46:33.801947 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:33.801863    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766"
Nov 03 19:46:33.805098 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:33.804893    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766"
Nov 03 19:46:33.983088 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:46:33.983034    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode41d2ecd_31d0_4110_af26_8ca6c3f9a593.slice/crio-d2818166117dd23212bc74b9099784731f99dffdf4c9fd6e882530b4427924d2 WatchSource:0}: Error finding container d2818166117dd23212bc74b9099784731f99dffdf4c9fd6e882530b4427924d2: Status 404 returned error can't find the container with id d2818166117dd23212bc74b9099784731f99dffdf4c9fd6e882530b4427924d2
Nov 03 19:46:33.986385 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:33.985503    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4006e3d-wdjqb"]
Nov 03 19:46:34.032745 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:34.030712    4921 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:46:34.032745 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:34.030779    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf" podUID="49c66530-de54-4ec9-83cb-61cafce44e7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:34.650710 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:34.650640    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf"
Nov 03 19:46:34.655120 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:34.655088    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf"
Nov 03 19:46:34.666659 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:34.666615    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4006e3d-wdjqb" event={"ID":"e41d2ecd-31d0-4110-af26-8ca6c3f9a593","Type":"ContainerStarted","Data":"9ff0c21a509c7a3081189b7e3595123b1cd0074aa4bb2232ccc9172ee7a688de"}
Nov 03 19:46:34.666817 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:34.666713    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4006e3d-wdjqb" event={"ID":"e41d2ecd-31d0-4110-af26-8ca6c3f9a593","Type":"ContainerStarted","Data":"d2818166117dd23212bc74b9099784731f99dffdf4c9fd6e882530b4427924d2"}
Nov 03 19:46:34.686493 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:34.686428    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4006e3d-wdjqb" podStartSLOduration=1.686411796 podStartE2EDuration="1.686411796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:46:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:46:34.682850816 +0000 UTC m=+907.926816281" watchObservedRunningTime="2026-11-03 19:46:34.686411796 +0000 UTC m=+907.930377262"
Nov 03 19:46:35.682461 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:35.682423    4921 generic.go:356] "Generic (PLEG): container finished" podID="e41d2ecd-31d0-4110-af26-8ca6c3f9a593" containerID="9ff0c21a509c7a3081189b7e3595123b1cd0074aa4bb2232ccc9172ee7a688de" exitCode=0
Nov 03 19:46:35.682952 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:35.682521    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4006e3d-wdjqb" event={"ID":"e41d2ecd-31d0-4110-af26-8ca6c3f9a593","Type":"ContainerDied","Data":"9ff0c21a509c7a3081189b7e3595123b1cd0074aa4bb2232ccc9172ee7a688de"}
Nov 03 19:46:36.887237 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:36.887207    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4006e3d-wdjqb"
Nov 03 19:46:37.045467 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:37.045430    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e41d2ecd-31d0-4110-af26-8ca6c3f9a593-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e41d2ecd-31d0-4110-af26-8ca6c3f9a593-maas-api-openshift-ca\") pod \"e41d2ecd-31d0-4110-af26-8ca6c3f9a593\" (UID: \"e41d2ecd-31d0-4110-af26-8ca6c3f9a593\") "
Nov 03 19:46:37.045815 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:37.045791    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e41d2ecd-31d0-4110-af26-8ca6c3f9a593-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e41d2ecd-31d0-4110-af26-8ca6c3f9a593-maas-api-xks-ca\") pod \"e41d2ecd-31d0-4110-af26-8ca6c3f9a593\" (UID: \"e41d2ecd-31d0-4110-af26-8ca6c3f9a593\") "
Nov 03 19:46:37.045934 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:37.045912    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e41d2ecd-31d0-4110-af26-8ca6c3f9a593-maas-api-openshift-ca" pod "e41d2ecd-31d0-4110-af26-8ca6c3f9a593" (UID: "e41d2ecd-31d0-4110-af26-8ca6c3f9a593"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:46:37.046063 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:37.045996    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e41d2ecd-31d0-4110-af26-8ca6c3f9a593-kube-api-access-hk7ld\" (UniqueName: \"kubernetes.io/projected/e41d2ecd-31d0-4110-af26-8ca6c3f9a593-kube-api-access-hk7ld\") pod \"e41d2ecd-31d0-4110-af26-8ca6c3f9a593\" (UID: \"e41d2ecd-31d0-4110-af26-8ca6c3f9a593\") "
Nov 03 19:46:37.046943 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:37.046929    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e41d2ecd-31d0-4110-af26-8ca6c3f9a593-maas-api-openshift-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:46:37.048856 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:37.048834    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e41d2ecd-31d0-4110-af26-8ca6c3f9a593-kube-api-access-hk7ld" pod "e41d2ecd-31d0-4110-af26-8ca6c3f9a593" (UID: "e41d2ecd-31d0-4110-af26-8ca6c3f9a593"). InnerVolumeSpecName "kube-api-access-hk7ld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:46:37.051199 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:37.051172    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e41d2ecd-31d0-4110-af26-8ca6c3f9a593-maas-api-xks-ca" pod "e41d2ecd-31d0-4110-af26-8ca6c3f9a593" (UID: "e41d2ecd-31d0-4110-af26-8ca6c3f9a593"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:46:37.148837 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:37.148806    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hk7ld\" (UniqueName: \"kubernetes.io/projected/e41d2ecd-31d0-4110-af26-8ca6c3f9a593-kube-api-access-hk7ld\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:46:37.148837 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:37.148842    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e41d2ecd-31d0-4110-af26-8ca6c3f9a593-maas-api-xks-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:46:37.699621 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:37.699590    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4006e3d-wdjqb"
Nov 03 19:46:37.699621 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:37.699614    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4006e3d-wdjqb" event={"ID":"e41d2ecd-31d0-4110-af26-8ca6c3f9a593","Type":"ContainerDied","Data":"d2818166117dd23212bc74b9099784731f99dffdf4c9fd6e882530b4427924d2"}
Nov 03 19:46:37.699846 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:37.699662    4921 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2818166117dd23212bc74b9099784731f99dffdf4c9fd6e882530b4427924d2"
Nov 03 19:46:45.289940 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:45.289898    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efb8207d-87fjr"]
Nov 03 19:46:45.290588 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:45.290574    4921 cpu_manager.go:414] "containerMap: removing container" podUID="e41d2ecd-31d0-4110-af26-8ca6c3f9a593" containerName="revoke-keys"
Nov 03 19:46:45.290633 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:45.290589    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e41d2ecd-31d0-4110-af26-8ca6c3f9a593" containerName="revoke-keys"
Nov 03 19:46:45.290738 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:45.290716    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="e41d2ecd-31d0-4110-af26-8ca6c3f9a593" containerName="revoke-keys"
Nov 03 19:46:45.294405 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:45.294389    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efb8207d-87fjr"
Nov 03 19:46:45.298047 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:45.297865    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:46:45.298047 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:45.298016    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 19:46:45.303413 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:45.303293    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efb8207d-87fjr"]
Nov 03 19:46:45.324787 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:45.324752    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3d6e5c0-7246-4aea-9419-f0f5199c5474-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efb8207d-87fjr\" (UID: \"e3d6e5c0-7246-4aea-9419-f0f5199c5474\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efb8207d-87fjr"
Nov 03 19:46:45.324986 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:45.324942    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2grv\" (UniqueName: \"kubernetes.io/projected/e3d6e5c0-7246-4aea-9419-f0f5199c5474-kube-api-access-w2grv\") pod \"maas-api-revoke-sub-efb8207d-87fjr\" (UID: \"e3d6e5c0-7246-4aea-9419-f0f5199c5474\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efb8207d-87fjr"
Nov 03 19:46:45.325116 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:45.325086    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3d6e5c0-7246-4aea-9419-f0f5199c5474-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efb8207d-87fjr\" (UID: \"e3d6e5c0-7246-4aea-9419-f0f5199c5474\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efb8207d-87fjr"
Nov 03 19:46:45.426743 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:45.426662    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2grv\" (UniqueName: \"kubernetes.io/projected/e3d6e5c0-7246-4aea-9419-f0f5199c5474-kube-api-access-w2grv\") pod \"maas-api-revoke-sub-efb8207d-87fjr\" (UID: \"e3d6e5c0-7246-4aea-9419-f0f5199c5474\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efb8207d-87fjr"
Nov 03 19:46:45.426949 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:45.426860    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3d6e5c0-7246-4aea-9419-f0f5199c5474-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efb8207d-87fjr\" (UID: \"e3d6e5c0-7246-4aea-9419-f0f5199c5474\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efb8207d-87fjr"
Nov 03 19:46:45.426949 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:45.426929    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3d6e5c0-7246-4aea-9419-f0f5199c5474-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efb8207d-87fjr\" (UID: \"e3d6e5c0-7246-4aea-9419-f0f5199c5474\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efb8207d-87fjr"
Nov 03 19:46:45.427568 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:45.427544    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3d6e5c0-7246-4aea-9419-f0f5199c5474-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efb8207d-87fjr\" (UID: \"e3d6e5c0-7246-4aea-9419-f0f5199c5474\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efb8207d-87fjr"
Nov 03 19:46:45.429619 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:45.429597    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3d6e5c0-7246-4aea-9419-f0f5199c5474-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efb8207d-87fjr\" (UID: \"e3d6e5c0-7246-4aea-9419-f0f5199c5474\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efb8207d-87fjr"
Nov 03 19:46:45.433915 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:45.433894    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2grv\" (UniqueName: \"kubernetes.io/projected/e3d6e5c0-7246-4aea-9419-f0f5199c5474-kube-api-access-w2grv\") pod \"maas-api-revoke-sub-efb8207d-87fjr\" (UID: \"e3d6e5c0-7246-4aea-9419-f0f5199c5474\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efb8207d-87fjr"
Nov 03 19:46:45.611406 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:45.611315    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efb8207d-87fjr"
Nov 03 19:46:45.899239 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:45.899197    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efb8207d-87fjr"]
Nov 03 19:46:45.899411 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:46:45.899389    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3d6e5c0_7246_4aea_9419_f0f5199c5474.slice/crio-5ea2327f1accd184d2a0214dad5a8c2d0fab2341ac27f60e0347b39c70640f71 WatchSource:0}: Error finding container 5ea2327f1accd184d2a0214dad5a8c2d0fab2341ac27f60e0347b39c70640f71: Status 404 returned error can't find the container with id 5ea2327f1accd184d2a0214dad5a8c2d0fab2341ac27f60e0347b39c70640f71
Nov 03 19:46:46.789734 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:46.789239    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efb8207d-87fjr" event={"ID":"e3d6e5c0-7246-4aea-9419-f0f5199c5474","Type":"ContainerStarted","Data":"c5a18174b1497959dd9d9a2ab3fb437a63b880e479be0bd215b93e9e7320211c"}
Nov 03 19:46:46.789734 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:46.789320    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efb8207d-87fjr" event={"ID":"e3d6e5c0-7246-4aea-9419-f0f5199c5474","Type":"ContainerStarted","Data":"5ea2327f1accd184d2a0214dad5a8c2d0fab2341ac27f60e0347b39c70640f71"}
Nov 03 19:46:46.817695 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:46.817594    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efb8207d-87fjr" podStartSLOduration=1.817570426 podStartE2EDuration="1.817570426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:46:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:46:46.805653807 +0000 UTC m=+920.049619282" watchObservedRunningTime="2026-11-03 19:46:46.817570426 +0000 UTC m=+920.061535893"
Nov 03 19:46:47.800895 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:47.800856    4921 generic.go:356] "Generic (PLEG): container finished" podID="e3d6e5c0-7246-4aea-9419-f0f5199c5474" containerID="c5a18174b1497959dd9d9a2ab3fb437a63b880e479be0bd215b93e9e7320211c" exitCode=0
Nov 03 19:46:47.801312 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:47.800922    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efb8207d-87fjr" event={"ID":"e3d6e5c0-7246-4aea-9419-f0f5199c5474","Type":"ContainerDied","Data":"c5a18174b1497959dd9d9a2ab3fb437a63b880e479be0bd215b93e9e7320211c"}
Nov 03 19:46:48.977515 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:48.977486    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efb8207d-87fjr"
Nov 03 19:46:49.095417 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:49.095333    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3d6e5c0-7246-4aea-9419-f0f5199c5474-kube-api-access-w2grv\" (UniqueName: \"kubernetes.io/projected/e3d6e5c0-7246-4aea-9419-f0f5199c5474-kube-api-access-w2grv\") pod \"e3d6e5c0-7246-4aea-9419-f0f5199c5474\" (UID: \"e3d6e5c0-7246-4aea-9419-f0f5199c5474\") "
Nov 03 19:46:49.095417 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:49.095389    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3d6e5c0-7246-4aea-9419-f0f5199c5474-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3d6e5c0-7246-4aea-9419-f0f5199c5474-maas-api-openshift-ca\") pod \"e3d6e5c0-7246-4aea-9419-f0f5199c5474\" (UID: \"e3d6e5c0-7246-4aea-9419-f0f5199c5474\") "
Nov 03 19:46:49.095605 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:49.095453    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3d6e5c0-7246-4aea-9419-f0f5199c5474-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3d6e5c0-7246-4aea-9419-f0f5199c5474-maas-api-xks-ca\") pod \"e3d6e5c0-7246-4aea-9419-f0f5199c5474\" (UID: \"e3d6e5c0-7246-4aea-9419-f0f5199c5474\") "
Nov 03 19:46:49.095821 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:49.095800    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3d6e5c0-7246-4aea-9419-f0f5199c5474-maas-api-openshift-ca" pod "e3d6e5c0-7246-4aea-9419-f0f5199c5474" (UID: "e3d6e5c0-7246-4aea-9419-f0f5199c5474"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:46:49.096161 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:49.096136    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3d6e5c0-7246-4aea-9419-f0f5199c5474-maas-api-openshift-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:46:49.097545 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:49.097523    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3d6e5c0-7246-4aea-9419-f0f5199c5474-maas-api-xks-ca" pod "e3d6e5c0-7246-4aea-9419-f0f5199c5474" (UID: "e3d6e5c0-7246-4aea-9419-f0f5199c5474"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:46:49.097698 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:49.097681    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3d6e5c0-7246-4aea-9419-f0f5199c5474-kube-api-access-w2grv" pod "e3d6e5c0-7246-4aea-9419-f0f5199c5474" (UID: "e3d6e5c0-7246-4aea-9419-f0f5199c5474"). InnerVolumeSpecName "kube-api-access-w2grv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:46:49.198006 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:49.197974    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2grv\" (UniqueName: \"kubernetes.io/projected/e3d6e5c0-7246-4aea-9419-f0f5199c5474-kube-api-access-w2grv\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:46:49.198006 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:49.198003    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3d6e5c0-7246-4aea-9419-f0f5199c5474-maas-api-xks-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:46:49.892214 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:49.891987    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efb8207d-87fjr"
Nov 03 19:46:49.892533 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:49.892365    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efb8207d-87fjr" event={"ID":"e3d6e5c0-7246-4aea-9419-f0f5199c5474","Type":"ContainerDied","Data":"5ea2327f1accd184d2a0214dad5a8c2d0fab2341ac27f60e0347b39c70640f71"}
Nov 03 19:46:49.892533 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:49.892416    4921 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ea2327f1accd184d2a0214dad5a8c2d0fab2341ac27f60e0347b39c70640f71"
Nov 03 19:46:54.820687 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:54.813772    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv" podUID="01c749b9-d188-4673-9a3d-99b9c4ae0793" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:54.903950 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:54.903895    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh" podUID="eac6c47d-8ae5-4e4b-b691-98d177887767" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:55.036706 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:55.035178    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw" podUID="72d96abc-f3b4-4eda-b3a1-b93ba658e1d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:55.369692 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:55.369439    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7" podUID="d4de0090-d775-4bd7-9ee2-058b5d380638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:55.476692 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:55.475810    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6" podUID="1aaee61d-b036-4c2b-bee7-a692b8bed612" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:55.662087 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:55.661956    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9" podUID="e6a71aa3-33b3-42bd-b270-64ce3260cc2a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:55.990752 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:55.970420    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv" podUID="01c749b9-d188-4673-9a3d-99b9c4ae0793" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Nov 03 19:46:55.990752 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:55.970981    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv"
Nov 03 19:46:56.007737 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:55.991928    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh" podUID="eac6c47d-8ae5-4e4b-b691-98d177887767" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:56.026704 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:56.020513    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh"
Nov 03 19:46:56.038299 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:56.037317    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk" podUID="2d0a934e-4816-44d2-9bb5-e8b4e95dd73c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:56.325751 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:56.276782    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv" podUID="96b73342-0079-4709-8570-3dbce57e5243" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:56.325751 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:56.277085    4921 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv" podUID="01c749b9-d188-4673-9a3d-99b9c4ae0793" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:56.325751 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:56.277139    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt" podUID="17dc0ed2-5164-449d-be12-a38bdfeb946d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Nov 03 19:46:56.325751 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:56.279702    4921 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt" podUID="17dc0ed2-5164-449d-be12-a38bdfeb946d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": context deadline exceeded"
Nov 03 19:46:56.406230 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:56.406009    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7" podUID="d4de0090-d775-4bd7-9ee2-058b5d380638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded"
Nov 03 19:46:56.444140 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:56.418270    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7"
Nov 03 19:46:56.505967 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:56.505924    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6" podUID="1aaee61d-b036-4c2b-bee7-a692b8bed612" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 03 19:46:56.615747 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:56.589998    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6"
Nov 03 19:46:56.615747 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:56.598013    4921 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh" podUID="eac6c47d-8ae5-4e4b-b691-98d177887767" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:56.772297 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:56.679850    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9" podUID="e6a71aa3-33b3-42bd-b270-64ce3260cc2a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Nov 03 19:46:56.780349 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:56.773106    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9"
Nov 03 19:46:57.109727 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:57.096773    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh" podUID="eac6c47d-8ae5-4e4b-b691-98d177887767" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded"
Nov 03 19:46:57.133837 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:57.111508    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk" podUID="2d0a934e-4816-44d2-9bb5-e8b4e95dd73c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Nov 03 19:46:57.136699 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:57.136644    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv" podUID="01c749b9-d188-4673-9a3d-99b9c4ae0793" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Nov 03 19:46:57.191892 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:57.191830    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw" podUID="72d96abc-f3b4-4eda-b3a1-b93ba658e1d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:57.265790 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:57.265742    4921 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:46:57.266109 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:57.266061    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j" podUID="e78a74a9-c32a-4047-a64f-b344f8ebaf22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:57.291027 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:57.290975    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt" podUID="17dc0ed2-5164-449d-be12-a38bdfeb946d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:57.291783 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:57.291751    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv" podUID="96b73342-0079-4709-8570-3dbce57e5243" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Nov 03 19:46:57.417753 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:57.408779    4921 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6" podUID="1aaee61d-b036-4c2b-bee7-a692b8bed612" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": context deadline exceeded"
Nov 03 19:46:57.417753 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:57.414890    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7" podUID="d4de0090-d775-4bd7-9ee2-058b5d380638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded"
Nov 03 19:46:57.557234 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:57.557183    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6" podUID="1aaee61d-b036-4c2b-bee7-a692b8bed612" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 03 19:46:57.609470 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:57.609433    4921 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:46:57.609760 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:57.609726    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr" podUID="c05f083e-3052-44f9-ac8f-746ea890191b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:57.775852 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:57.775808    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9" podUID="e6a71aa3-33b3-42bd-b270-64ce3260cc2a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Nov 03 19:46:57.823939 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:57.823891    4921 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:46:57.824292 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:57.824250    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs" podUID="f838f4d0-1e63-4b98-8cfe-bd1b96db7162" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:57.843150 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:57.843112    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk"
Nov 03 19:46:57.843439 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:57.843417    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt"
Nov 03 19:46:57.843598 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:57.843585    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv"
Nov 03 19:46:57.970959 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:57.970919    4921 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:46:58.015695 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:58.014762    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45" podUID="fa3f8ec3-6ad5-4433-8dbb-71f8038dffce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:58.124217 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:58.124112    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh" podUID="eac6c47d-8ae5-4e4b-b691-98d177887767" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:58.140415 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:58.139191    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv" podUID="01c749b9-d188-4673-9a3d-99b9c4ae0793" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:58.140415 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:58.139578    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk" podUID="2d0a934e-4816-44d2-9bb5-e8b4e95dd73c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Nov 03 19:46:58.200040 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:58.199987    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw" podUID="72d96abc-f3b4-4eda-b3a1-b93ba658e1d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:58.414747 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:58.397812    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt" podUID="17dc0ed2-5164-449d-be12-a38bdfeb946d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Nov 03 19:46:58.473220 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:58.398416    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv" podUID="96b73342-0079-4709-8570-3dbce57e5243" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:58.474626 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:58.421411    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw"
Nov 03 19:46:58.601951 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:58.601907    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7" podUID="d4de0090-d775-4bd7-9ee2-058b5d380638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:58.617593 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:58.603698    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6" podUID="1aaee61d-b036-4c2b-bee7-a692b8bed612" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:58.617593 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:58.613639    4921 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7" podUID="d4de0090-d775-4bd7-9ee2-058b5d380638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:58.796851 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:58.796806    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9" podUID="e6a71aa3-33b3-42bd-b270-64ce3260cc2a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Nov 03 19:46:59.159755 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:59.122746    4921 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9" podUID="e6a71aa3-33b3-42bd-b270-64ce3260cc2a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": context deadline exceeded"
Nov 03 19:46:59.159755 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:59.130573    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh" podUID="eac6c47d-8ae5-4e4b-b691-98d177887767" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:59.159755 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:59.141122    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk" podUID="2d0a934e-4816-44d2-9bb5-e8b4e95dd73c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:59.171150 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:59.164772    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv" podUID="01c749b9-d188-4673-9a3d-99b9c4ae0793" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Nov 03 19:46:59.425921 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:59.425817    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw" podUID="72d96abc-f3b4-4eda-b3a1-b93ba658e1d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:59.512912 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:59.512867    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt" podUID="17dc0ed2-5164-449d-be12-a38bdfeb946d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Nov 03 19:46:59.564391 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:59.564338    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv" podUID="96b73342-0079-4709-8570-3dbce57e5243" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:59.636171 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:59.627840    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7" podUID="d4de0090-d775-4bd7-9ee2-058b5d380638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded"
Nov 03 19:46:59.703748 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:59.685758    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6" podUID="1aaee61d-b036-4c2b-bee7-a692b8bed612" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 03 19:46:59.804696 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:59.797559    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9" podUID="e6a71aa3-33b3-42bd-b270-64ce3260cc2a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:59.893009 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:59.892791    4921 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:46:59.893009 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:59.892897    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5" podUID="0da622a1-45f9-4311-b668-d27eb7ec2e43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:46:59.931874 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:59.931834    4921 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:46:59.932178 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:46:59.932131    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6" podUID="7e96267a-d738-4325-90e7-d1c9ce87eeb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:00.155695 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:00.135833    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh" podUID="eac6c47d-8ae5-4e4b-b691-98d177887767" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded"
Nov 03 19:47:00.174738 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:00.174689    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv" podUID="01c749b9-d188-4673-9a3d-99b9c4ae0793" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:00.175818 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:00.175787    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk" podUID="2d0a934e-4816-44d2-9bb5-e8b4e95dd73c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:00.233473 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:00.210755    4921 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw" podUID="72d96abc-f3b4-4eda-b3a1-b93ba658e1d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": context deadline exceeded"
Nov 03 19:47:00.289693 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:00.288550    4921 patch_prober.go:29] interesting pod/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:47:00.289693 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:00.288619    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz" podUID="52048b65-d34b-412d-a473-36c9c72ed8e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:00.441815 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:00.432498    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw" podUID="72d96abc-f3b4-4eda-b3a1-b93ba658e1d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:00.519695 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:00.518761    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt" podUID="17dc0ed2-5164-449d-be12-a38bdfeb946d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Nov 03 19:47:00.565678 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:00.565615    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv" podUID="96b73342-0079-4709-8570-3dbce57e5243" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:00.629806 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:00.629761    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7" podUID="d4de0090-d775-4bd7-9ee2-058b5d380638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:00.686876 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:00.686826    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6" podUID="1aaee61d-b036-4c2b-bee7-a692b8bed612" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 03 19:47:00.754154 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:00.754117    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6"
Nov 03 19:47:00.791698 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:00.788521    4921 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv" podUID="96b73342-0079-4709-8570-3dbce57e5243" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:00.826611 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:00.826565    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9" podUID="e6a71aa3-33b3-42bd-b270-64ce3260cc2a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:01.158773 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:01.143771    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh" podUID="eac6c47d-8ae5-4e4b-b691-98d177887767" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:01.188709 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:01.179054    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk" podUID="2d0a934e-4816-44d2-9bb5-e8b4e95dd73c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:01.188709 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:01.179480    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv" podUID="01c749b9-d188-4673-9a3d-99b9c4ae0793" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Nov 03 19:47:01.434398 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:01.434284    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw" podUID="72d96abc-f3b4-4eda-b3a1-b93ba658e1d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:01.530992 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:01.530945    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt" podUID="17dc0ed2-5164-449d-be12-a38bdfeb946d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:01.581870 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:01.581821    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv" podUID="96b73342-0079-4709-8570-3dbce57e5243" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:01.627640 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:01.626514    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh"
Nov 03 19:47:01.645645 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:01.644735    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7" podUID="d4de0090-d775-4bd7-9ee2-058b5d380638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:01.834816 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:01.834777    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9" podUID="e6a71aa3-33b3-42bd-b270-64ce3260cc2a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Nov 03 19:47:02.059692 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:02.058941    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7"
Nov 03 19:47:02.155001 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:02.154914    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9"
Nov 03 19:47:02.184889 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:02.184845    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv" podUID="01c749b9-d188-4673-9a3d-99b9c4ae0793" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:02.185593 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:02.185562    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk" podUID="2d0a934e-4816-44d2-9bb5-e8b4e95dd73c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Nov 03 19:47:02.348112 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:02.348083    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk"
Nov 03 19:47:02.452749 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:02.438052    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw" podUID="72d96abc-f3b4-4eda-b3a1-b93ba658e1d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:02.548843 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:02.548788    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt" podUID="17dc0ed2-5164-449d-be12-a38bdfeb946d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Nov 03 19:47:02.600494 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:02.600460    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv"
Nov 03 19:47:02.649455 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:02.649042    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv" podUID="96b73342-0079-4709-8570-3dbce57e5243" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:03.448801 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:03.448752    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw" podUID="72d96abc-f3b4-4eda-b3a1-b93ba658e1d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:03.568602 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:03.568556    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt" podUID="17dc0ed2-5164-449d-be12-a38bdfeb946d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:03.760704 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:03.755471    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6" podUID="1aaee61d-b036-4c2b-bee7-a692b8bed612" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:03.760704 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:03.755911    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7" podUID="d4de0090-d775-4bd7-9ee2-058b5d380638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded"
Nov 03 19:47:03.760704 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:03.756242    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv" podUID="96b73342-0079-4709-8570-3dbce57e5243" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Nov 03 19:47:03.760704 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:47:03.757236    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 19:47:03.791814 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:03.785397    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9" podUID="e6a71aa3-33b3-42bd-b270-64ce3260cc2a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:04.034660 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:04.018296    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw"
Nov 03 19:47:04.183855 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:04.183812    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv" podUID="01c749b9-d188-4673-9a3d-99b9c4ae0793" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Nov 03 19:47:04.219363 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:04.184287    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk" podUID="2d0a934e-4816-44d2-9bb5-e8b4e95dd73c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:04.281623 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:04.281535    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv"
Nov 03 19:47:04.285567 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:04.285491    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt"
Nov 03 19:47:14.728290 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:14.728201    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2e17271b-6fd654c77b-g5mtg"]
Nov 03 19:47:14.729060 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:14.729042    4921 cpu_manager.go:414] "containerMap: removing container" podUID="e3d6e5c0-7246-4aea-9419-f0f5199c5474" containerName="revoke-keys"
Nov 03 19:47:14.729113 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:14.729062    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3d6e5c0-7246-4aea-9419-f0f5199c5474" containerName="revoke-keys"
Nov 03 19:47:14.729183 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:14.729174    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3d6e5c0-7246-4aea-9419-f0f5199c5474" containerName="revoke-keys"
Nov 03 19:47:14.732499 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:14.732483    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2e17271b-6fd654c77b-g5mtg"
Nov 03 19:47:14.739198 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:14.739173    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-2e17271b-serving-cert\""
Nov 03 19:47:14.747096 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:14.747000    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2e17271b-6fd654c77b-g5mtg"]
Nov 03 19:47:14.819621 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:14.819589    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h78lf\" (UniqueName: \"kubernetes.io/projected/9edd35a2-d2c5-4910-b7fe-aaa3d8b52909-kube-api-access-h78lf\") pod \"maas-api-e2e-ait-2e17271b-6fd654c77b-g5mtg\" (UID: \"9edd35a2-d2c5-4910-b7fe-aaa3d8b52909\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2e17271b-6fd654c77b-g5mtg"
Nov 03 19:47:14.819817 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:14.819636    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9edd35a2-d2c5-4910-b7fe-aaa3d8b52909-metrics-certs\") pod \"maas-api-e2e-ait-2e17271b-6fd654c77b-g5mtg\" (UID: \"9edd35a2-d2c5-4910-b7fe-aaa3d8b52909\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2e17271b-6fd654c77b-g5mtg"
Nov 03 19:47:14.819889 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:14.819842    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9edd35a2-d2c5-4910-b7fe-aaa3d8b52909-openshift-service-ca\") pod \"maas-api-e2e-ait-2e17271b-6fd654c77b-g5mtg\" (UID: \"9edd35a2-d2c5-4910-b7fe-aaa3d8b52909\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2e17271b-6fd654c77b-g5mtg"
Nov 03 19:47:14.820068 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:14.820051    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9edd35a2-d2c5-4910-b7fe-aaa3d8b52909-maas-api-tls\") pod \"maas-api-e2e-ait-2e17271b-6fd654c77b-g5mtg\" (UID: \"9edd35a2-d2c5-4910-b7fe-aaa3d8b52909\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2e17271b-6fd654c77b-g5mtg"
Nov 03 19:47:14.935185 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:14.935150    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h78lf\" (UniqueName: \"kubernetes.io/projected/9edd35a2-d2c5-4910-b7fe-aaa3d8b52909-kube-api-access-h78lf\") pod \"maas-api-e2e-ait-2e17271b-6fd654c77b-g5mtg\" (UID: \"9edd35a2-d2c5-4910-b7fe-aaa3d8b52909\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2e17271b-6fd654c77b-g5mtg"
Nov 03 19:47:14.935897 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:14.935867    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9edd35a2-d2c5-4910-b7fe-aaa3d8b52909-metrics-certs\") pod \"maas-api-e2e-ait-2e17271b-6fd654c77b-g5mtg\" (UID: \"9edd35a2-d2c5-4910-b7fe-aaa3d8b52909\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2e17271b-6fd654c77b-g5mtg"
Nov 03 19:47:14.936746 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:14.936722    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9edd35a2-d2c5-4910-b7fe-aaa3d8b52909-openshift-service-ca\") pod \"maas-api-e2e-ait-2e17271b-6fd654c77b-g5mtg\" (UID: \"9edd35a2-d2c5-4910-b7fe-aaa3d8b52909\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2e17271b-6fd654c77b-g5mtg"
Nov 03 19:47:14.949753 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:14.949720    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9edd35a2-d2c5-4910-b7fe-aaa3d8b52909-maas-api-tls\") pod \"maas-api-e2e-ait-2e17271b-6fd654c77b-g5mtg\" (UID: \"9edd35a2-d2c5-4910-b7fe-aaa3d8b52909\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2e17271b-6fd654c77b-g5mtg"
Nov 03 19:47:14.966706 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:14.938134    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9edd35a2-d2c5-4910-b7fe-aaa3d8b52909-openshift-service-ca\") pod \"maas-api-e2e-ait-2e17271b-6fd654c77b-g5mtg\" (UID: \"9edd35a2-d2c5-4910-b7fe-aaa3d8b52909\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2e17271b-6fd654c77b-g5mtg"
Nov 03 19:47:14.966706 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:14.961848    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9edd35a2-d2c5-4910-b7fe-aaa3d8b52909-metrics-certs\") pod \"maas-api-e2e-ait-2e17271b-6fd654c77b-g5mtg\" (UID: \"9edd35a2-d2c5-4910-b7fe-aaa3d8b52909\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2e17271b-6fd654c77b-g5mtg"
Nov 03 19:47:14.972697 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:14.967616    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9edd35a2-d2c5-4910-b7fe-aaa3d8b52909-maas-api-tls\") pod \"maas-api-e2e-ait-2e17271b-6fd654c77b-g5mtg\" (UID: \"9edd35a2-d2c5-4910-b7fe-aaa3d8b52909\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2e17271b-6fd654c77b-g5mtg"
Nov 03 19:47:14.982878 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:14.977062    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h78lf\" (UniqueName: \"kubernetes.io/projected/9edd35a2-d2c5-4910-b7fe-aaa3d8b52909-kube-api-access-h78lf\") pod \"maas-api-e2e-ait-2e17271b-6fd654c77b-g5mtg\" (UID: \"9edd35a2-d2c5-4910-b7fe-aaa3d8b52909\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2e17271b-6fd654c77b-g5mtg"
Nov 03 19:47:15.050698 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:15.048475    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2e17271b-6fd654c77b-g5mtg"
Nov 03 19:47:15.338366 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:15.338333    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2e17271b-6fd654c77b-g5mtg"]
Nov 03 19:47:15.342137 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:15.342108    4921 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 19:47:15.552944 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:15.527402    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2e17271b-6fd654c77b-g5mtg" event={"ID":"9edd35a2-d2c5-4910-b7fe-aaa3d8b52909","Type":"ContainerStarted","Data":"d9b23c728f71e075d968c9a8b796eb19c609251e52a48139326166f17a590bdd"}
Nov 03 19:47:16.536509 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:16.536467    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2e17271b-6fd654c77b-g5mtg" event={"ID":"9edd35a2-d2c5-4910-b7fe-aaa3d8b52909","Type":"ContainerStarted","Data":"940f29bd21e6f518d945ef4164fd0edad3f8f23703f5e49544f63e5bedb3f5b1"}
Nov 03 19:47:16.537015 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:16.536535    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2e17271b-6fd654c77b-g5mtg"
Nov 03 19:47:16.552885 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:16.552828    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2e17271b-6fd654c77b-g5mtg" podStartSLOduration=2.093726329 podStartE2EDuration="2.552813266s" totalImagesPullingTime="459.086937ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:47:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:47:16.550300667 +0000 UTC m=+949.794266132" watchObservedRunningTime="2026-11-03 19:47:16.552813266 +0000 UTC m=+949.796778731"
Nov 03 19:47:19.026963 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:19.026931    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-26c00c22-7b7b5fdccd-jsfbk"]
Nov 03 19:47:19.032963 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:19.032936    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-26c00c22-7b7b5fdccd-jsfbk"
Nov 03 19:47:19.040327 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:19.040288    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-26c00c22-serving-cert\""
Nov 03 19:47:19.049487 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:19.049457    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-26c00c22-7b7b5fdccd-jsfbk"]
Nov 03 19:47:19.110392 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:19.110361    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c957cfe7-2d58-4f98-a5b2-f6b6d45b8a6b-metrics-certs\") pod \"maas-api-e2e-derive-26c00c22-7b7b5fdccd-jsfbk\" (UID: \"c957cfe7-2d58-4f98-a5b2-f6b6d45b8a6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-26c00c22-7b7b5fdccd-jsfbk"
Nov 03 19:47:19.110585 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:19.110508    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-965cc\" (UniqueName: \"kubernetes.io/projected/c957cfe7-2d58-4f98-a5b2-f6b6d45b8a6b-kube-api-access-965cc\") pod \"maas-api-e2e-derive-26c00c22-7b7b5fdccd-jsfbk\" (UID: \"c957cfe7-2d58-4f98-a5b2-f6b6d45b8a6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-26c00c22-7b7b5fdccd-jsfbk"
Nov 03 19:47:19.110585 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:19.110570    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c957cfe7-2d58-4f98-a5b2-f6b6d45b8a6b-openshift-service-ca\") pod \"maas-api-e2e-derive-26c00c22-7b7b5fdccd-jsfbk\" (UID: \"c957cfe7-2d58-4f98-a5b2-f6b6d45b8a6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-26c00c22-7b7b5fdccd-jsfbk"
Nov 03 19:47:19.110747 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:19.110726    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c957cfe7-2d58-4f98-a5b2-f6b6d45b8a6b-maas-api-tls\") pod \"maas-api-e2e-derive-26c00c22-7b7b5fdccd-jsfbk\" (UID: \"c957cfe7-2d58-4f98-a5b2-f6b6d45b8a6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-26c00c22-7b7b5fdccd-jsfbk"
Nov 03 19:47:19.212576 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:19.212534    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c957cfe7-2d58-4f98-a5b2-f6b6d45b8a6b-openshift-service-ca\") pod \"maas-api-e2e-derive-26c00c22-7b7b5fdccd-jsfbk\" (UID: \"c957cfe7-2d58-4f98-a5b2-f6b6d45b8a6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-26c00c22-7b7b5fdccd-jsfbk"
Nov 03 19:47:19.212802 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:19.212623    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c957cfe7-2d58-4f98-a5b2-f6b6d45b8a6b-maas-api-tls\") pod \"maas-api-e2e-derive-26c00c22-7b7b5fdccd-jsfbk\" (UID: \"c957cfe7-2d58-4f98-a5b2-f6b6d45b8a6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-26c00c22-7b7b5fdccd-jsfbk"
Nov 03 19:47:19.212880 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:19.212820    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c957cfe7-2d58-4f98-a5b2-f6b6d45b8a6b-metrics-certs\") pod \"maas-api-e2e-derive-26c00c22-7b7b5fdccd-jsfbk\" (UID: \"c957cfe7-2d58-4f98-a5b2-f6b6d45b8a6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-26c00c22-7b7b5fdccd-jsfbk"
Nov 03 19:47:19.212999 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:19.212946    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-965cc\" (UniqueName: \"kubernetes.io/projected/c957cfe7-2d58-4f98-a5b2-f6b6d45b8a6b-kube-api-access-965cc\") pod \"maas-api-e2e-derive-26c00c22-7b7b5fdccd-jsfbk\" (UID: \"c957cfe7-2d58-4f98-a5b2-f6b6d45b8a6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-26c00c22-7b7b5fdccd-jsfbk"
Nov 03 19:47:19.213460 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:19.213428    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c957cfe7-2d58-4f98-a5b2-f6b6d45b8a6b-openshift-service-ca\") pod \"maas-api-e2e-derive-26c00c22-7b7b5fdccd-jsfbk\" (UID: \"c957cfe7-2d58-4f98-a5b2-f6b6d45b8a6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-26c00c22-7b7b5fdccd-jsfbk"
Nov 03 19:47:19.215525 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:19.215494    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c957cfe7-2d58-4f98-a5b2-f6b6d45b8a6b-maas-api-tls\") pod \"maas-api-e2e-derive-26c00c22-7b7b5fdccd-jsfbk\" (UID: \"c957cfe7-2d58-4f98-a5b2-f6b6d45b8a6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-26c00c22-7b7b5fdccd-jsfbk"
Nov 03 19:47:19.266689 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:19.261325    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-965cc\" (UniqueName: \"kubernetes.io/projected/c957cfe7-2d58-4f98-a5b2-f6b6d45b8a6b-kube-api-access-965cc\") pod \"maas-api-e2e-derive-26c00c22-7b7b5fdccd-jsfbk\" (UID: \"c957cfe7-2d58-4f98-a5b2-f6b6d45b8a6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-26c00c22-7b7b5fdccd-jsfbk"
Nov 03 19:47:19.276298 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:19.276263    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c957cfe7-2d58-4f98-a5b2-f6b6d45b8a6b-metrics-certs\") pod \"maas-api-e2e-derive-26c00c22-7b7b5fdccd-jsfbk\" (UID: \"c957cfe7-2d58-4f98-a5b2-f6b6d45b8a6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-26c00c22-7b7b5fdccd-jsfbk"
Nov 03 19:47:19.359511 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:19.359431    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-26c00c22-7b7b5fdccd-jsfbk"
Nov 03 19:47:19.863929 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:19.862631    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-26c00c22-7b7b5fdccd-jsfbk"]
Nov 03 19:47:20.653841 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:20.652793    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-26c00c22-7b7b5fdccd-jsfbk" event={"ID":"c957cfe7-2d58-4f98-a5b2-f6b6d45b8a6b","Type":"ContainerStarted","Data":"d80b4aa4d4b40682cf2b51d1ab657f4c3ae71349666faf81d9055dc5ef76f0eb"}
Nov 03 19:47:20.653841 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:20.652868    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-26c00c22-7b7b5fdccd-jsfbk" event={"ID":"c957cfe7-2d58-4f98-a5b2-f6b6d45b8a6b","Type":"ContainerStarted","Data":"22e6fc1f70015361ad80fba00da5bc16e88353c4e5da172093d09811e05392ea"}
Nov 03 19:47:20.679704 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:20.654593    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-26c00c22-7b7b5fdccd-jsfbk"
Nov 03 19:47:20.689700 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:20.687421    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-26c00c22-7b7b5fdccd-jsfbk" podStartSLOduration=1.194924416 podStartE2EDuration="1.687400988s" totalImagesPullingTime="492.476572ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:47:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:47:20.674082437 +0000 UTC m=+953.918047902" watchObservedRunningTime="2026-11-03 19:47:20.687400988 +0000 UTC m=+953.931366454"
Nov 03 19:47:22.594692 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:22.593985    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2e17271b-6fd654c77b-g5mtg"
Nov 03 19:47:27.697922 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:27.697888    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-26c00c22-7b7b5fdccd-jsfbk"
Nov 03 19:47:30.807694 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:30.786861    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9" podUID="e6a71aa3-33b3-42bd-b270-64ce3260cc2a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:31.434807 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:31.428764    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw" podUID="72d96abc-f3b4-4eda-b3a1-b93ba658e1d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:31.482693 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:31.481755    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv" podUID="96b73342-0079-4709-8570-3dbce57e5243" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:31.547696 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:31.530824    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt" podUID="17dc0ed2-5164-449d-be12-a38bdfeb946d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Nov 03 19:47:31.616695 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:31.615968    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7" podUID="d4de0090-d775-4bd7-9ee2-058b5d380638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:31.620692 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:31.617211    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6" podUID="1aaee61d-b036-4c2b-bee7-a692b8bed612" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 03 19:47:31.787389 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:31.787340    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9" podUID="e6a71aa3-33b3-42bd-b270-64ce3260cc2a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:31.788248 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:31.788222    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9"
Nov 03 19:47:32.144850 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:32.143784    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh" podUID="eac6c47d-8ae5-4e4b-b691-98d177887767" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded"
Nov 03 19:47:32.171641 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:32.169865    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk" podUID="2d0a934e-4816-44d2-9bb5-e8b4e95dd73c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:32.171641 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:32.170264    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv" podUID="01c749b9-d188-4673-9a3d-99b9c4ae0793" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Nov 03 19:47:32.431414 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:32.431315    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw" podUID="72d96abc-f3b4-4eda-b3a1-b93ba658e1d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:32.432186 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:32.432118    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw"
Nov 03 19:47:32.517295 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:32.517227    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv" podUID="96b73342-0079-4709-8570-3dbce57e5243" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:32.531275 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:32.531244    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv"
Nov 03 19:47:32.534939 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:32.534758    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt" podUID="17dc0ed2-5164-449d-be12-a38bdfeb946d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:32.584720 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:32.555102    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt"
Nov 03 19:47:32.834694 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:32.826280    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7" podUID="d4de0090-d775-4bd7-9ee2-058b5d380638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded"
Nov 03 19:47:32.858824 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:32.858779    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9" podUID="e6a71aa3-33b3-42bd-b270-64ce3260cc2a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Nov 03 19:47:32.888887 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:32.887755    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6" podUID="1aaee61d-b036-4c2b-bee7-a692b8bed612" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 03 19:47:32.925849 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:32.900786    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7"
Nov 03 19:47:33.101194 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:33.101107    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6"
Nov 03 19:47:33.331371 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:33.157486    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh" podUID="eac6c47d-8ae5-4e4b-b691-98d177887767" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:33.342593 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:33.342563    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh"
Nov 03 19:47:33.347692 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:33.171927    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv" podUID="01c749b9-d188-4673-9a3d-99b9c4ae0793" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:33.386982 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:33.171967    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk" podUID="2d0a934e-4816-44d2-9bb5-e8b4e95dd73c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Nov 03 19:47:33.386982 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:33.386518    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv"
Nov 03 19:47:33.386982 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:33.386579    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk"
Nov 03 19:47:33.439812 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:33.439761    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw" podUID="72d96abc-f3b4-4eda-b3a1-b93ba658e1d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:33.539701 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:33.522625    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv" podUID="96b73342-0079-4709-8570-3dbce57e5243" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:33.596694 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:33.578808    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt" podUID="17dc0ed2-5164-449d-be12-a38bdfeb946d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Nov 03 19:47:33.848813 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:33.842942    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7" podUID="d4de0090-d775-4bd7-9ee2-058b5d380638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded"
Nov 03 19:47:33.897133 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:47:33.857919    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 19:47:34.001863 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:34.001819    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9" podUID="e6a71aa3-33b3-42bd-b270-64ce3260cc2a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Nov 03 19:47:34.004711 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:34.004651    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6" podUID="1aaee61d-b036-4c2b-bee7-a692b8bed612" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 03 19:47:34.375136 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:34.375089    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh" podUID="eac6c47d-8ae5-4e4b-b691-98d177887767" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded"
Nov 03 19:47:34.680748 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:34.659180    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv" podUID="01c749b9-d188-4673-9a3d-99b9c4ae0793" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:34.680748 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:34.659520    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk" podUID="2d0a934e-4816-44d2-9bb5-e8b4e95dd73c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Nov 03 19:47:34.694571 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:34.694527    4921 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk" podUID="2d0a934e-4816-44d2-9bb5-e8b4e95dd73c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": context deadline exceeded"
Nov 03 19:47:34.720865 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:34.709433    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt" podUID="17dc0ed2-5164-449d-be12-a38bdfeb946d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Nov 03 19:47:34.857343 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:34.857301    4921 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:47:34.859629 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:34.859587    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6" podUID="052a7a9f-ed4b-4cb9-911a-413079445a1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:34.894504 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:34.894458    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7" podUID="d4de0090-d775-4bd7-9ee2-058b5d380638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded"
Nov 03 19:47:34.937080 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:34.936985    4921 patch_prober.go:29] interesting pod/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:47:35.016206 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:35.016172    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766" podUID="7ad21605-a58c-494c-89e0-6aca910b76b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:35.077626 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:35.016035    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6" podUID="1aaee61d-b036-4c2b-bee7-a692b8bed612" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:35.077927 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:34.982772    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw" podUID="72d96abc-f3b4-4eda-b3a1-b93ba658e1d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:35.078081 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:35.049838    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9" podUID="e6a71aa3-33b3-42bd-b270-64ce3260cc2a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:35.295554 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:35.293841    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv" podUID="96b73342-0079-4709-8570-3dbce57e5243" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Nov 03 19:47:35.378695 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:35.376387    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh" podUID="eac6c47d-8ae5-4e4b-b691-98d177887767" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:35.698130 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:35.662758    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk" podUID="2d0a934e-4816-44d2-9bb5-e8b4e95dd73c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:35.698130 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:35.663141    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv" podUID="01c749b9-d188-4673-9a3d-99b9c4ae0793" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:35.698130 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:35.694752    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-crz4q"]
Nov 03 19:47:35.719693 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:35.701902    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-crz4q"
Nov 03 19:47:35.761388 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:35.705763    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-crz4q\" (UID: \"4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-crz4q"
Nov 03 19:47:35.761388 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:35.747213    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc4pf\" (UniqueName: \"kubernetes.io/projected/4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc-kube-api-access-tc4pf\") pod \"istiod-openshift-gateway-7ddd9cfdc8-crz4q\" (UID: \"4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-crz4q"
Nov 03 19:47:35.761388 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:35.747256    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-crz4q\" (UID: \"4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-crz4q"
Nov 03 19:47:35.761388 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:35.747291    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-crz4q\" (UID: \"4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-crz4q"
Nov 03 19:47:35.761388 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:35.747428    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-crz4q\" (UID: \"4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-crz4q"
Nov 03 19:47:35.761388 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:35.747473    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-crz4q\" (UID: \"4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-crz4q"
Nov 03 19:47:35.761388 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:35.747767    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt" podUID="17dc0ed2-5164-449d-be12-a38bdfeb946d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:35.761388 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:35.747850    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-crz4q\" (UID: \"4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-crz4q"
Nov 03 19:47:35.780491 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:35.764810    4921 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:47:35.780491 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:35.764886    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf" podUID="49c66530-de54-4ec9-83cb-61cafce44e7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:35.874691 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:35.849244    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-crz4q\" (UID: \"4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-crz4q"
Nov 03 19:47:35.882060 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:35.878854    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-crz4q\" (UID: \"4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-crz4q"
Nov 03 19:47:35.882060 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:35.878938    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc4pf\" (UniqueName: \"kubernetes.io/projected/4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc-kube-api-access-tc4pf\") pod \"istiod-openshift-gateway-7ddd9cfdc8-crz4q\" (UID: \"4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-crz4q"
Nov 03 19:47:35.882060 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:35.878967    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-crz4q\" (UID: \"4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-crz4q"
Nov 03 19:47:35.882060 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:35.879003    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-crz4q\" (UID: \"4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-crz4q"
Nov 03 19:47:35.882060 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:35.879134    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-crz4q\" (UID: \"4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-crz4q"
Nov 03 19:47:35.882060 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:35.879175    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-crz4q\" (UID: \"4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-crz4q"
Nov 03 19:47:35.882060 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:35.880458    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-crz4q\" (UID: \"4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-crz4q"
Nov 03 19:47:35.907732 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:35.899101    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7" podUID="d4de0090-d775-4bd7-9ee2-058b5d380638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:36.399529 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:36.222491    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-crz4q"]
Nov 03 19:47:36.399529 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:36.222711    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt"
Nov 03 19:47:36.399529 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:36.316561    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv" podUID="96b73342-0079-4709-8570-3dbce57e5243" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:36.399529 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:36.317435    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9" podUID="e6a71aa3-33b3-42bd-b270-64ce3260cc2a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:36.399529 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:36.317712    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw" podUID="72d96abc-f3b4-4eda-b3a1-b93ba658e1d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:36.399529 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:36.326980    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-crz4q\" (UID: \"4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-crz4q"
Nov 03 19:47:36.474756 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:36.474127    4921 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:47:36.474756 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:36.474187    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl" podUID="55ff258b-9816-4fff-a74a-3c417bf0f0ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:36.475143 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:36.475106    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh" podUID="eac6c47d-8ae5-4e4b-b691-98d177887767" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:36.574524 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:36.574472    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6" podUID="1aaee61d-b036-4c2b-bee7-a692b8bed612" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:36.708244 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:36.610757    4921 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh" podUID="eac6c47d-8ae5-4e4b-b691-98d177887767" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:36.708458 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:36.675178    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk" podUID="2d0a934e-4816-44d2-9bb5-e8b4e95dd73c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Nov 03 19:47:36.715617 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:36.675220    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv" podUID="01c749b9-d188-4673-9a3d-99b9c4ae0793" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:36.726700 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:36.707810    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-crz4q\" (UID: \"4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-crz4q"
Nov 03 19:47:36.726963 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:36.709116    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-crz4q\" (UID: \"4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-crz4q"
Nov 03 19:47:36.727121 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:36.711159    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-crz4q\" (UID: \"4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-crz4q"
Nov 03 19:47:36.759700 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:36.757387    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-crz4q\" (UID: \"4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-crz4q"
Nov 03 19:47:36.953744 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:36.941108    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7" podUID="d4de0090-d775-4bd7-9ee2-058b5d380638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:37.080506 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:37.080458    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7"
Nov 03 19:47:37.093490 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:37.093458    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc4pf\" (UniqueName: \"kubernetes.io/projected/4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc-kube-api-access-tc4pf\") pod \"istiod-openshift-gateway-7ddd9cfdc8-crz4q\" (UID: \"4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-crz4q"
Nov 03 19:47:37.412750 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:37.382867    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv" podUID="96b73342-0079-4709-8570-3dbce57e5243" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Nov 03 19:47:37.412750 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:37.399074    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9" podUID="e6a71aa3-33b3-42bd-b270-64ce3260cc2a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Nov 03 19:47:37.412750 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:37.401232    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw" podUID="72d96abc-f3b4-4eda-b3a1-b93ba658e1d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded"
Nov 03 19:47:37.436691 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:37.427028    4921 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6" podUID="1aaee61d-b036-4c2b-bee7-a692b8bed612" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:37.436691 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:37.427939    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-crz4q"
Nov 03 19:47:37.475596 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:37.467358    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6"
Nov 03 19:47:37.574696 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:37.573857    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh" podUID="eac6c47d-8ae5-4e4b-b691-98d177887767" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:37.615694 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:37.615456    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt" podUID="17dc0ed2-5164-449d-be12-a38bdfeb946d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:37.726103 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:37.715728    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk" podUID="2d0a934e-4816-44d2-9bb5-e8b4e95dd73c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:37.883658 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:37.883605    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv" podUID="01c749b9-d188-4673-9a3d-99b9c4ae0793" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:38.443775 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:38.429736    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw" podUID="72d96abc-f3b4-4eda-b3a1-b93ba658e1d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:38.446469 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:38.446431    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9" podUID="e6a71aa3-33b3-42bd-b270-64ce3260cc2a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:38.474950 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:38.473434    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv" podUID="96b73342-0079-4709-8570-3dbce57e5243" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:38.586954 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:38.586913    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh" podUID="eac6c47d-8ae5-4e4b-b691-98d177887767" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:38.652012 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:38.645071    4921 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7" podUID="d4de0090-d775-4bd7-9ee2-058b5d380638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:38.749694 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:38.719112    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk" podUID="2d0a934e-4816-44d2-9bb5-e8b4e95dd73c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:39.131015 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:39.125331    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:47:39.131015 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:39.125424    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:47:39.175934 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:39.174147    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9"
Nov 03 19:47:39.176249 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:39.176229    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv"
Nov 03 19:47:39.176429 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:39.176415    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk"
Nov 03 19:47:39.176588 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:39.176574    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw"
Nov 03 19:47:39.176771 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:39.176757    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv"
Nov 03 19:47:39.195752 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:39.195720    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh"
Nov 03 19:47:39.195968 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:39.195953    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-crz4q"]
Nov 03 19:47:39.257181 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:39.257097    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-crz4q" event={"ID":"4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc","Type":"ContainerStarted","Data":"9db66fe81c82c8f74c4401805c45e7d9b1952f9bf45ac5a82983b49f2863f08d"}
Nov 03 19:47:40.318564 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:40.316328    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-crz4q" event={"ID":"4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc","Type":"ContainerStarted","Data":"16f51738c532765a9301416e1c41576c198001166cc12ebb3e00745e487c96a4"}
Nov 03 19:47:40.319304 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:40.319221    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-crz4q"
Nov 03 19:47:40.322547 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:40.322270    4921 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-crz4q container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 19:47:40.322547 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:40.322333    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-crz4q" podUID="4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 19:47:40.362376 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:40.351713    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-crz4q" podStartSLOduration=7.351694213 podStartE2EDuration="7.351694213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:47:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:47:40.343338125 +0000 UTC m=+973.587303589" watchObservedRunningTime="2026-11-03 19:47:40.351694213 +0000 UTC m=+973.595659681"
Nov 03 19:47:41.400747 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:41.389908    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-crz4q"
Nov 03 19:47:44.412145 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:44.412107    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2e17271b-6fd654c77b-g5mtg"]
Nov 03 19:47:44.413043 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:44.412996    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2e17271b-6fd654c77b-g5mtg" podUID="9edd35a2-d2c5-4910-b7fe-aaa3d8b52909" containerName="maas-api" containerID="cri-o://940f29bd21e6f518d945ef4164fd0edad3f8f23703f5e49544f63e5bedb3f5b1" gracePeriod=30
Nov 03 19:47:45.364375 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:45.364338    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2e17271b-6fd654c77b-g5mtg"
Nov 03 19:47:45.386581 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:45.386550    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9edd35a2-d2c5-4910-b7fe-aaa3d8b52909-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9edd35a2-d2c5-4910-b7fe-aaa3d8b52909-openshift-service-ca\") pod \"9edd35a2-d2c5-4910-b7fe-aaa3d8b52909\" (UID: \"9edd35a2-d2c5-4910-b7fe-aaa3d8b52909\") "
Nov 03 19:47:45.386740 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:45.386609    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9edd35a2-d2c5-4910-b7fe-aaa3d8b52909-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9edd35a2-d2c5-4910-b7fe-aaa3d8b52909-maas-api-tls\") pod \"9edd35a2-d2c5-4910-b7fe-aaa3d8b52909\" (UID: \"9edd35a2-d2c5-4910-b7fe-aaa3d8b52909\") "
Nov 03 19:47:45.386740 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:45.386714    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9edd35a2-d2c5-4910-b7fe-aaa3d8b52909-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9edd35a2-d2c5-4910-b7fe-aaa3d8b52909-metrics-certs\") pod \"9edd35a2-d2c5-4910-b7fe-aaa3d8b52909\" (UID: \"9edd35a2-d2c5-4910-b7fe-aaa3d8b52909\") "
Nov 03 19:47:45.386923 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:45.386748    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9edd35a2-d2c5-4910-b7fe-aaa3d8b52909-kube-api-access-h78lf\" (UniqueName: \"kubernetes.io/projected/9edd35a2-d2c5-4910-b7fe-aaa3d8b52909-kube-api-access-h78lf\") pod \"9edd35a2-d2c5-4910-b7fe-aaa3d8b52909\" (UID: \"9edd35a2-d2c5-4910-b7fe-aaa3d8b52909\") "
Nov 03 19:47:45.387536 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:45.387509    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9edd35a2-d2c5-4910-b7fe-aaa3d8b52909-openshift-service-ca" pod "9edd35a2-d2c5-4910-b7fe-aaa3d8b52909" (UID: "9edd35a2-d2c5-4910-b7fe-aaa3d8b52909"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:47:45.388012 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:45.387990    4921 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9edd35a2-d2c5-4910-b7fe-aaa3d8b52909-openshift-service-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:45.389458 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:45.389426    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9edd35a2-d2c5-4910-b7fe-aaa3d8b52909-metrics-certs" pod "9edd35a2-d2c5-4910-b7fe-aaa3d8b52909" (UID: "9edd35a2-d2c5-4910-b7fe-aaa3d8b52909"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:47:45.389553 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:45.389453    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9edd35a2-d2c5-4910-b7fe-aaa3d8b52909-maas-api-tls" pod "9edd35a2-d2c5-4910-b7fe-aaa3d8b52909" (UID: "9edd35a2-d2c5-4910-b7fe-aaa3d8b52909"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:47:45.389901 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:45.389870    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9edd35a2-d2c5-4910-b7fe-aaa3d8b52909-kube-api-access-h78lf" pod "9edd35a2-d2c5-4910-b7fe-aaa3d8b52909" (UID: "9edd35a2-d2c5-4910-b7fe-aaa3d8b52909"). InnerVolumeSpecName "kube-api-access-h78lf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:47:45.495326 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:45.495246    4921 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9edd35a2-d2c5-4910-b7fe-aaa3d8b52909-metrics-certs\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:45.495326 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:45.495286    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h78lf\" (UniqueName: \"kubernetes.io/projected/9edd35a2-d2c5-4910-b7fe-aaa3d8b52909-kube-api-access-h78lf\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:45.495326 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:45.495300    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9edd35a2-d2c5-4910-b7fe-aaa3d8b52909-maas-api-tls\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:45.600941 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:45.600903    4921 generic.go:356] "Generic (PLEG): container finished" podID="9edd35a2-d2c5-4910-b7fe-aaa3d8b52909" containerID="940f29bd21e6f518d945ef4164fd0edad3f8f23703f5e49544f63e5bedb3f5b1" exitCode=0
Nov 03 19:47:45.601144 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:45.600952    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2e17271b-6fd654c77b-g5mtg"
Nov 03 19:47:45.601144 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:45.601006    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2e17271b-6fd654c77b-g5mtg" event={"ID":"9edd35a2-d2c5-4910-b7fe-aaa3d8b52909","Type":"ContainerDied","Data":"940f29bd21e6f518d945ef4164fd0edad3f8f23703f5e49544f63e5bedb3f5b1"}
Nov 03 19:47:45.601144 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:45.601057    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2e17271b-6fd654c77b-g5mtg" event={"ID":"9edd35a2-d2c5-4910-b7fe-aaa3d8b52909","Type":"ContainerDied","Data":"d9b23c728f71e075d968c9a8b796eb19c609251e52a48139326166f17a590bdd"}
Nov 03 19:47:45.601144 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:45.601079    4921 scope.go:122] "RemoveContainer" containerID="940f29bd21e6f518d945ef4164fd0edad3f8f23703f5e49544f63e5bedb3f5b1"
Nov 03 19:47:45.616716 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:45.616685    4921 scope.go:122] "RemoveContainer" containerID="940f29bd21e6f518d945ef4164fd0edad3f8f23703f5e49544f63e5bedb3f5b1"
Nov 03 19:47:45.617085 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:47:45.617064    4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"940f29bd21e6f518d945ef4164fd0edad3f8f23703f5e49544f63e5bedb3f5b1\": container with ID starting with 940f29bd21e6f518d945ef4164fd0edad3f8f23703f5e49544f63e5bedb3f5b1 not found: ID does not exist" containerID="940f29bd21e6f518d945ef4164fd0edad3f8f23703f5e49544f63e5bedb3f5b1"
Nov 03 19:47:45.617182 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:45.617090    4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"940f29bd21e6f518d945ef4164fd0edad3f8f23703f5e49544f63e5bedb3f5b1"} err="failed to get container status \"940f29bd21e6f518d945ef4164fd0edad3f8f23703f5e49544f63e5bedb3f5b1\": rpc error: code = NotFound desc = could not find container \"940f29bd21e6f518d945ef4164fd0edad3f8f23703f5e49544f63e5bedb3f5b1\": container with ID starting with 940f29bd21e6f518d945ef4164fd0edad3f8f23703f5e49544f63e5bedb3f5b1 not found: ID does not exist"
Nov 03 19:47:45.631053 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:45.631020    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2e17271b-6fd654c77b-g5mtg"]
Nov 03 19:47:45.635397 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:45.635369    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2e17271b-6fd654c77b-g5mtg"]
Nov 03 19:47:46.927695 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:46.920853    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9edd35a2-d2c5-4910-b7fe-aaa3d8b52909" path="/var/lib/kubelet/pods/9edd35a2-d2c5-4910-b7fe-aaa3d8b52909/volumes"
Nov 03 19:47:59.986449 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:59.986414    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-26c00c22-7b7b5fdccd-jsfbk"]
Nov 03 19:47:59.987003 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:47:59.986706    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-26c00c22-7b7b5fdccd-jsfbk" podUID="c957cfe7-2d58-4f98-a5b2-f6b6d45b8a6b" containerName="maas-api" containerID="cri-o://d80b4aa4d4b40682cf2b51d1ab657f4c3ae71349666faf81d9055dc5ef76f0eb" gracePeriod=30
Nov 03 19:48:00.468696 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:00.466989    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-26c00c22-7b7b5fdccd-jsfbk"
Nov 03 19:48:00.484703 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:00.477335    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c957cfe7-2d58-4f98-a5b2-f6b6d45b8a6b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c957cfe7-2d58-4f98-a5b2-f6b6d45b8a6b-maas-api-tls\") pod \"c957cfe7-2d58-4f98-a5b2-f6b6d45b8a6b\" (UID: \"c957cfe7-2d58-4f98-a5b2-f6b6d45b8a6b\") "
Nov 03 19:48:00.484703 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:00.477392    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c957cfe7-2d58-4f98-a5b2-f6b6d45b8a6b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c957cfe7-2d58-4f98-a5b2-f6b6d45b8a6b-metrics-certs\") pod \"c957cfe7-2d58-4f98-a5b2-f6b6d45b8a6b\" (UID: \"c957cfe7-2d58-4f98-a5b2-f6b6d45b8a6b\") "
Nov 03 19:48:00.484703 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:00.477538    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c957cfe7-2d58-4f98-a5b2-f6b6d45b8a6b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c957cfe7-2d58-4f98-a5b2-f6b6d45b8a6b-openshift-service-ca\") pod \"c957cfe7-2d58-4f98-a5b2-f6b6d45b8a6b\" (UID: \"c957cfe7-2d58-4f98-a5b2-f6b6d45b8a6b\") "
Nov 03 19:48:00.484703 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:00.477592    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c957cfe7-2d58-4f98-a5b2-f6b6d45b8a6b-kube-api-access-965cc\" (UniqueName: \"kubernetes.io/projected/c957cfe7-2d58-4f98-a5b2-f6b6d45b8a6b-kube-api-access-965cc\") pod \"c957cfe7-2d58-4f98-a5b2-f6b6d45b8a6b\" (UID: \"c957cfe7-2d58-4f98-a5b2-f6b6d45b8a6b\") "
Nov 03 19:48:00.493686 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:00.487472    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c957cfe7-2d58-4f98-a5b2-f6b6d45b8a6b-openshift-service-ca" pod "c957cfe7-2d58-4f98-a5b2-f6b6d45b8a6b" (UID: "c957cfe7-2d58-4f98-a5b2-f6b6d45b8a6b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:48:00.503356 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:00.499206    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c957cfe7-2d58-4f98-a5b2-f6b6d45b8a6b-kube-api-access-965cc" pod "c957cfe7-2d58-4f98-a5b2-f6b6d45b8a6b" (UID: "c957cfe7-2d58-4f98-a5b2-f6b6d45b8a6b"). InnerVolumeSpecName "kube-api-access-965cc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:48:00.705085 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:00.704320    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c957cfe7-2d58-4f98-a5b2-f6b6d45b8a6b-metrics-certs" pod "c957cfe7-2d58-4f98-a5b2-f6b6d45b8a6b" (UID: "c957cfe7-2d58-4f98-a5b2-f6b6d45b8a6b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:48:00.705085 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:00.704881    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c957cfe7-2d58-4f98-a5b2-f6b6d45b8a6b-maas-api-tls" pod "c957cfe7-2d58-4f98-a5b2-f6b6d45b8a6b" (UID: "c957cfe7-2d58-4f98-a5b2-f6b6d45b8a6b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:48:00.713694 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:00.710102    4921 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c957cfe7-2d58-4f98-a5b2-f6b6d45b8a6b-openshift-service-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:48:00.713694 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:00.710132    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-965cc\" (UniqueName: \"kubernetes.io/projected/c957cfe7-2d58-4f98-a5b2-f6b6d45b8a6b-kube-api-access-965cc\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:48:00.713694 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:00.710156    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c957cfe7-2d58-4f98-a5b2-f6b6d45b8a6b-maas-api-tls\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:48:00.713694 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:00.710171    4921 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c957cfe7-2d58-4f98-a5b2-f6b6d45b8a6b-metrics-certs\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:48:01.055696 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:01.052166    4921 generic.go:356] "Generic (PLEG): container finished" podID="c957cfe7-2d58-4f98-a5b2-f6b6d45b8a6b" containerID="d80b4aa4d4b40682cf2b51d1ab657f4c3ae71349666faf81d9055dc5ef76f0eb" exitCode=0
Nov 03 19:48:01.055696 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:01.052307    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-26c00c22-7b7b5fdccd-jsfbk"
Nov 03 19:48:01.060601 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:01.060558    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-26c00c22-7b7b5fdccd-jsfbk" event={"ID":"c957cfe7-2d58-4f98-a5b2-f6b6d45b8a6b","Type":"ContainerDied","Data":"d80b4aa4d4b40682cf2b51d1ab657f4c3ae71349666faf81d9055dc5ef76f0eb"}
Nov 03 19:48:01.060815 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:01.060635    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-26c00c22-7b7b5fdccd-jsfbk" event={"ID":"c957cfe7-2d58-4f98-a5b2-f6b6d45b8a6b","Type":"ContainerDied","Data":"22e6fc1f70015361ad80fba00da5bc16e88353c4e5da172093d09811e05392ea"}
Nov 03 19:48:01.060815 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:01.060684    4921 scope.go:122] "RemoveContainer" containerID="d80b4aa4d4b40682cf2b51d1ab657f4c3ae71349666faf81d9055dc5ef76f0eb"
Nov 03 19:48:01.079122 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:01.079086    4921 scope.go:122] "RemoveContainer" containerID="d80b4aa4d4b40682cf2b51d1ab657f4c3ae71349666faf81d9055dc5ef76f0eb"
Nov 03 19:48:01.081283 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:48:01.079993    4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d80b4aa4d4b40682cf2b51d1ab657f4c3ae71349666faf81d9055dc5ef76f0eb\": container with ID starting with d80b4aa4d4b40682cf2b51d1ab657f4c3ae71349666faf81d9055dc5ef76f0eb not found: ID does not exist" containerID="d80b4aa4d4b40682cf2b51d1ab657f4c3ae71349666faf81d9055dc5ef76f0eb"
Nov 03 19:48:01.081283 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:01.080048    4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d80b4aa4d4b40682cf2b51d1ab657f4c3ae71349666faf81d9055dc5ef76f0eb"} err="failed to get container status \"d80b4aa4d4b40682cf2b51d1ab657f4c3ae71349666faf81d9055dc5ef76f0eb\": rpc error: code = NotFound desc = could not find container \"d80b4aa4d4b40682cf2b51d1ab657f4c3ae71349666faf81d9055dc5ef76f0eb\": container with ID starting with d80b4aa4d4b40682cf2b51d1ab657f4c3ae71349666faf81d9055dc5ef76f0eb not found: ID does not exist"
Nov 03 19:48:01.088705 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:01.088641    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-26c00c22-7b7b5fdccd-jsfbk"]
Nov 03 19:48:01.093695 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:01.093658    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-26c00c22-7b7b5fdccd-jsfbk"]
Nov 03 19:48:02.893011 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:02.892968    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c957cfe7-2d58-4f98-a5b2-f6b6d45b8a6b" path="/var/lib/kubelet/pods/c957cfe7-2d58-4f98-a5b2-f6b6d45b8a6b/volumes"
Nov 03 19:48:02.922075 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:48:02.922034    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 19:48:08.246871 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:08.246836    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c96ee058-v9hcv"]
Nov 03 19:48:08.247710 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:08.247654    4921 cpu_manager.go:414] "containerMap: removing container" podUID="9edd35a2-d2c5-4910-b7fe-aaa3d8b52909" containerName="maas-api"
Nov 03 19:48:08.247710 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:08.247691    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9edd35a2-d2c5-4910-b7fe-aaa3d8b52909" containerName="maas-api"
Nov 03 19:48:08.247884 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:08.247720    4921 cpu_manager.go:414] "containerMap: removing container" podUID="c957cfe7-2d58-4f98-a5b2-f6b6d45b8a6b" containerName="maas-api"
Nov 03 19:48:08.247884 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:08.247727    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c957cfe7-2d58-4f98-a5b2-f6b6d45b8a6b" containerName="maas-api"
Nov 03 19:48:08.247981 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:08.247923    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="9edd35a2-d2c5-4910-b7fe-aaa3d8b52909" containerName="maas-api"
Nov 03 19:48:08.247981 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:08.247939    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="c957cfe7-2d58-4f98-a5b2-f6b6d45b8a6b" containerName="maas-api"
Nov 03 19:48:08.253108 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:08.253090    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c96ee058-v9hcv"
Nov 03 19:48:08.256172 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:08.255720    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:48:08.256172 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:08.255985    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 19:48:08.260913 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:08.260886    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c96ee058-v9hcv"]
Nov 03 19:48:08.359245 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:08.359206    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ghbt\" (UniqueName: \"kubernetes.io/projected/451fdc53-52c3-4f9b-affb-86822bbeac73-kube-api-access-9ghbt\") pod \"maas-api-revoke-keys-e2e-migrate-c96ee058-v9hcv\" (UID: \"451fdc53-52c3-4f9b-affb-86822bbeac73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c96ee058-v9hcv"
Nov 03 19:48:08.359421 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:08.359277    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/451fdc53-52c3-4f9b-affb-86822bbeac73-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c96ee058-v9hcv\" (UID: \"451fdc53-52c3-4f9b-affb-86822bbeac73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c96ee058-v9hcv"
Nov 03 19:48:08.359421 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:08.359349    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/451fdc53-52c3-4f9b-affb-86822bbeac73-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c96ee058-v9hcv\" (UID: \"451fdc53-52c3-4f9b-affb-86822bbeac73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c96ee058-v9hcv"
Nov 03 19:48:08.461372 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:08.461340    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ghbt\" (UniqueName: \"kubernetes.io/projected/451fdc53-52c3-4f9b-affb-86822bbeac73-kube-api-access-9ghbt\") pod \"maas-api-revoke-keys-e2e-migrate-c96ee058-v9hcv\" (UID: \"451fdc53-52c3-4f9b-affb-86822bbeac73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c96ee058-v9hcv"
Nov 03 19:48:08.461535 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:08.461395    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/451fdc53-52c3-4f9b-affb-86822bbeac73-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c96ee058-v9hcv\" (UID: \"451fdc53-52c3-4f9b-affb-86822bbeac73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c96ee058-v9hcv"
Nov 03 19:48:08.461535 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:08.461423    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/451fdc53-52c3-4f9b-affb-86822bbeac73-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c96ee058-v9hcv\" (UID: \"451fdc53-52c3-4f9b-affb-86822bbeac73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c96ee058-v9hcv"
Nov 03 19:48:08.462070 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:08.462045    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/451fdc53-52c3-4f9b-affb-86822bbeac73-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c96ee058-v9hcv\" (UID: \"451fdc53-52c3-4f9b-affb-86822bbeac73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c96ee058-v9hcv"
Nov 03 19:48:08.463715 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:08.463688    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/451fdc53-52c3-4f9b-affb-86822bbeac73-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c96ee058-v9hcv\" (UID: \"451fdc53-52c3-4f9b-affb-86822bbeac73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c96ee058-v9hcv"
Nov 03 19:48:08.468809 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:08.468787    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ghbt\" (UniqueName: \"kubernetes.io/projected/451fdc53-52c3-4f9b-affb-86822bbeac73-kube-api-access-9ghbt\") pod \"maas-api-revoke-keys-e2e-migrate-c96ee058-v9hcv\" (UID: \"451fdc53-52c3-4f9b-affb-86822bbeac73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c96ee058-v9hcv"
Nov 03 19:48:08.591060 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:08.590987    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c96ee058-v9hcv"
Nov 03 19:48:08.892064 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:08.892038    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c96ee058-v9hcv"]
Nov 03 19:48:09.117849 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:09.117816    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c96ee058-v9hcv" event={"ID":"451fdc53-52c3-4f9b-affb-86822bbeac73","Type":"ContainerStarted","Data":"c9d690d0985825b46691168e7a41d0858107153b6d5651d5bab2b0b7c8eba9dc"}
Nov 03 19:48:10.129009 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:10.128975    4921 generic.go:356] "Generic (PLEG): container finished" podID="451fdc53-52c3-4f9b-affb-86822bbeac73" containerID="5b5e81a1a6dd0c7dd8c8d9d89bb3a635b9cc900f7f5022297e4dd0a40bb4dc29" exitCode=0
Nov 03 19:48:10.129460 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:10.129018    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c96ee058-v9hcv" event={"ID":"451fdc53-52c3-4f9b-affb-86822bbeac73","Type":"ContainerDied","Data":"5b5e81a1a6dd0c7dd8c8d9d89bb3a635b9cc900f7f5022297e4dd0a40bb4dc29"}
Nov 03 19:48:12.270294 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:12.270256    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c96ee058-v9hcv" event={"ID":"451fdc53-52c3-4f9b-affb-86822bbeac73","Type":"ContainerDied","Data":"c9d690d0985825b46691168e7a41d0858107153b6d5651d5bab2b0b7c8eba9dc"}
Nov 03 19:48:12.271021 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:12.270999    4921 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9d690d0985825b46691168e7a41d0858107153b6d5651d5bab2b0b7c8eba9dc"
Nov 03 19:48:12.272548 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:12.272193    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c96ee058-v9hcv"
Nov 03 19:48:12.422707 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:12.422598    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/451fdc53-52c3-4f9b-affb-86822bbeac73-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/451fdc53-52c3-4f9b-affb-86822bbeac73-maas-api-openshift-ca\") pod \"451fdc53-52c3-4f9b-affb-86822bbeac73\" (UID: \"451fdc53-52c3-4f9b-affb-86822bbeac73\") "
Nov 03 19:48:12.423046 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:12.423024    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/451fdc53-52c3-4f9b-affb-86822bbeac73-kube-api-access-9ghbt\" (UniqueName: \"kubernetes.io/projected/451fdc53-52c3-4f9b-affb-86822bbeac73-kube-api-access-9ghbt\") pod \"451fdc53-52c3-4f9b-affb-86822bbeac73\" (UID: \"451fdc53-52c3-4f9b-affb-86822bbeac73\") "
Nov 03 19:48:12.423393 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:12.423369    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/451fdc53-52c3-4f9b-affb-86822bbeac73-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/451fdc53-52c3-4f9b-affb-86822bbeac73-maas-api-xks-ca\") pod \"451fdc53-52c3-4f9b-affb-86822bbeac73\" (UID: \"451fdc53-52c3-4f9b-affb-86822bbeac73\") "
Nov 03 19:48:12.437177 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:12.437150    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/451fdc53-52c3-4f9b-affb-86822bbeac73-maas-api-openshift-ca" pod "451fdc53-52c3-4f9b-affb-86822bbeac73" (UID: "451fdc53-52c3-4f9b-affb-86822bbeac73"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:48:12.439917 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:12.439894    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/451fdc53-52c3-4f9b-affb-86822bbeac73-kube-api-access-9ghbt" pod "451fdc53-52c3-4f9b-affb-86822bbeac73" (UID: "451fdc53-52c3-4f9b-affb-86822bbeac73"). InnerVolumeSpecName "kube-api-access-9ghbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:48:12.498109 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:12.498077    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/451fdc53-52c3-4f9b-affb-86822bbeac73-maas-api-xks-ca" pod "451fdc53-52c3-4f9b-affb-86822bbeac73" (UID: "451fdc53-52c3-4f9b-affb-86822bbeac73"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:48:12.549739 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:12.547274    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/451fdc53-52c3-4f9b-affb-86822bbeac73-maas-api-xks-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:48:12.549739 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:12.547335    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/451fdc53-52c3-4f9b-affb-86822bbeac73-maas-api-openshift-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:48:12.549739 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:12.547354    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ghbt\" (UniqueName: \"kubernetes.io/projected/451fdc53-52c3-4f9b-affb-86822bbeac73-kube-api-access-9ghbt\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:48:13.390746 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:13.390195    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c96ee058-v9hcv"
Nov 03 19:48:14.011694 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:14.010929    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9" podUID="e6a71aa3-33b3-42bd-b270-64ce3260cc2a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:48:22.387510 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:22.387474    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3a1058c-sg7h8"]
Nov 03 19:48:22.388136 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:22.388122    4921 cpu_manager.go:414] "containerMap: removing container" podUID="451fdc53-52c3-4f9b-affb-86822bbeac73" containerName="revoke-keys"
Nov 03 19:48:22.388187 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:22.388137    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="451fdc53-52c3-4f9b-affb-86822bbeac73" containerName="revoke-keys"
Nov 03 19:48:22.388269 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:22.388259    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="451fdc53-52c3-4f9b-affb-86822bbeac73" containerName="revoke-keys"
Nov 03 19:48:22.391885 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:22.391866    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3a1058c-sg7h8"
Nov 03 19:48:22.395178 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:22.395156    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:48:22.395310 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:22.395264    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 19:48:22.400763 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:22.400497    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3a1058c-sg7h8"]
Nov 03 19:48:22.424790 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:22.424754    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8916be04-39a2-4524-8e15-37a74aa82819-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3a1058c-sg7h8\" (UID: \"8916be04-39a2-4524-8e15-37a74aa82819\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3a1058c-sg7h8"
Nov 03 19:48:22.424947 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:22.424911    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r54tz\" (UniqueName: \"kubernetes.io/projected/8916be04-39a2-4524-8e15-37a74aa82819-kube-api-access-r54tz\") pod \"maas-api-revoke-keys-e2e-ait-b3a1058c-sg7h8\" (UID: \"8916be04-39a2-4524-8e15-37a74aa82819\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3a1058c-sg7h8"
Nov 03 19:48:22.425111 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:22.425091    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8916be04-39a2-4524-8e15-37a74aa82819-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3a1058c-sg7h8\" (UID: \"8916be04-39a2-4524-8e15-37a74aa82819\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3a1058c-sg7h8"
Nov 03 19:48:22.534822 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:22.534119    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8916be04-39a2-4524-8e15-37a74aa82819-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3a1058c-sg7h8\" (UID: \"8916be04-39a2-4524-8e15-37a74aa82819\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3a1058c-sg7h8"
Nov 03 19:48:22.534822 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:22.534435    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r54tz\" (UniqueName: \"kubernetes.io/projected/8916be04-39a2-4524-8e15-37a74aa82819-kube-api-access-r54tz\") pod \"maas-api-revoke-keys-e2e-ait-b3a1058c-sg7h8\" (UID: \"8916be04-39a2-4524-8e15-37a74aa82819\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3a1058c-sg7h8"
Nov 03 19:48:22.535087 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:22.534962    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8916be04-39a2-4524-8e15-37a74aa82819-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3a1058c-sg7h8\" (UID: \"8916be04-39a2-4524-8e15-37a74aa82819\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3a1058c-sg7h8"
Nov 03 19:48:22.546110 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:22.546073    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8916be04-39a2-4524-8e15-37a74aa82819-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3a1058c-sg7h8\" (UID: \"8916be04-39a2-4524-8e15-37a74aa82819\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3a1058c-sg7h8"
Nov 03 19:48:22.551291 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:22.548492    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8916be04-39a2-4524-8e15-37a74aa82819-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3a1058c-sg7h8\" (UID: \"8916be04-39a2-4524-8e15-37a74aa82819\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3a1058c-sg7h8"
Nov 03 19:48:22.559390 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:22.559356    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r54tz\" (UniqueName: \"kubernetes.io/projected/8916be04-39a2-4524-8e15-37a74aa82819-kube-api-access-r54tz\") pod \"maas-api-revoke-keys-e2e-ait-b3a1058c-sg7h8\" (UID: \"8916be04-39a2-4524-8e15-37a74aa82819\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3a1058c-sg7h8"
Nov 03 19:48:22.708858 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:22.708776    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3a1058c-sg7h8"
Nov 03 19:48:23.428742 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:23.424922    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3a1058c-sg7h8"]
Nov 03 19:48:23.973792 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:23.973664    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3a1058c-sg7h8" event={"ID":"8916be04-39a2-4524-8e15-37a74aa82819","Type":"ContainerStarted","Data":"62346d20ccc3352584d0a7c1ab4f699c54bceb5aebbbc933aeee0386048cccf2"}
Nov 03 19:48:23.973792 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:23.973763    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3a1058c-sg7h8" event={"ID":"8916be04-39a2-4524-8e15-37a74aa82819","Type":"ContainerStarted","Data":"386ed27d851525d5dcda3753142ebf172bdce6df20230d6f3774de640cec9d27"}
Nov 03 19:48:23.988244 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:23.988193    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3a1058c-sg7h8" podStartSLOduration=1.9881806640000002 podStartE2EDuration="1.988180664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:48:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:48:23.986530698 +0000 UTC m=+1017.230496163" watchObservedRunningTime="2026-11-03 19:48:23.988180664 +0000 UTC m=+1017.232146129"
Nov 03 19:48:24.486765 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:24.486718    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c96ee058-v9hcv"]
Nov 03 19:48:24.497951 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:24.497888    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c96ee058-v9hcv"]
Nov 03 19:48:24.904978 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:24.904941    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="451fdc53-52c3-4f9b-affb-86822bbeac73" path="/var/lib/kubelet/pods/451fdc53-52c3-4f9b-affb-86822bbeac73/volumes"
Nov 03 19:48:24.986988 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:24.986952    4921 generic.go:356] "Generic (PLEG): container finished" podID="8916be04-39a2-4524-8e15-37a74aa82819" containerID="62346d20ccc3352584d0a7c1ab4f699c54bceb5aebbbc933aeee0386048cccf2" exitCode=0
Nov 03 19:48:24.987254 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:24.987043    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3a1058c-sg7h8" event={"ID":"8916be04-39a2-4524-8e15-37a74aa82819","Type":"ContainerDied","Data":"62346d20ccc3352584d0a7c1ab4f699c54bceb5aebbbc933aeee0386048cccf2"}
Nov 03 19:48:26.374935 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:26.374906    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3a1058c-sg7h8"
Nov 03 19:48:26.516043 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:26.513473    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8916be04-39a2-4524-8e15-37a74aa82819-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8916be04-39a2-4524-8e15-37a74aa82819-maas-api-xks-ca\") pod \"8916be04-39a2-4524-8e15-37a74aa82819\" (UID: \"8916be04-39a2-4524-8e15-37a74aa82819\") "
Nov 03 19:48:26.516043 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:26.513549    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8916be04-39a2-4524-8e15-37a74aa82819-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8916be04-39a2-4524-8e15-37a74aa82819-maas-api-openshift-ca\") pod \"8916be04-39a2-4524-8e15-37a74aa82819\" (UID: \"8916be04-39a2-4524-8e15-37a74aa82819\") "
Nov 03 19:48:26.516043 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:26.513612    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8916be04-39a2-4524-8e15-37a74aa82819-kube-api-access-r54tz\" (UniqueName: \"kubernetes.io/projected/8916be04-39a2-4524-8e15-37a74aa82819-kube-api-access-r54tz\") pod \"8916be04-39a2-4524-8e15-37a74aa82819\" (UID: \"8916be04-39a2-4524-8e15-37a74aa82819\") "
Nov 03 19:48:26.517401 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:26.517081    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8916be04-39a2-4524-8e15-37a74aa82819-maas-api-openshift-ca" pod "8916be04-39a2-4524-8e15-37a74aa82819" (UID: "8916be04-39a2-4524-8e15-37a74aa82819"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:48:26.553697 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:26.539861    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8916be04-39a2-4524-8e15-37a74aa82819-kube-api-access-r54tz" pod "8916be04-39a2-4524-8e15-37a74aa82819" (UID: "8916be04-39a2-4524-8e15-37a74aa82819"). InnerVolumeSpecName "kube-api-access-r54tz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:48:26.572700 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:26.568842    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8916be04-39a2-4524-8e15-37a74aa82819-maas-api-xks-ca" pod "8916be04-39a2-4524-8e15-37a74aa82819" (UID: "8916be04-39a2-4524-8e15-37a74aa82819"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:48:26.615468 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:26.615375    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8916be04-39a2-4524-8e15-37a74aa82819-maas-api-xks-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:48:26.615468 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:26.615409    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8916be04-39a2-4524-8e15-37a74aa82819-maas-api-openshift-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:48:26.615468 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:26.615424    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r54tz\" (UniqueName: \"kubernetes.io/projected/8916be04-39a2-4524-8e15-37a74aa82819-kube-api-access-r54tz\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:48:27.039996 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:27.039963    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3a1058c-sg7h8"
Nov 03 19:48:27.039996 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:27.039984    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3a1058c-sg7h8" event={"ID":"8916be04-39a2-4524-8e15-37a74aa82819","Type":"ContainerDied","Data":"386ed27d851525d5dcda3753142ebf172bdce6df20230d6f3774de640cec9d27"}
Nov 03 19:48:27.040214 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:27.040035    4921 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="386ed27d851525d5dcda3753142ebf172bdce6df20230d6f3774de640cec9d27"
Nov 03 19:48:32.923115 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:48:32.923070    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 19:48:37.026422 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:37.026304    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c598ec5c-xf4sb"]
Nov 03 19:48:37.031391 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:37.031361    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c598ec5c-xf4sb"]
Nov 03 19:48:38.890573 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:38.890539    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="709ab7af-01f4-4205-bd44-1c1d03ef8136" path="/var/lib/kubelet/pods/709ab7af-01f4-4205-bd44-1c1d03ef8136/volumes"
Nov 03 19:48:39.723476 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:39.723426    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3a1058c-sg7h8"]
Nov 03 19:48:39.728177 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:39.728147    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3a1058c-sg7h8"]
Nov 03 19:48:40.909108 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:40.908749    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8916be04-39a2-4524-8e15-37a74aa82819" path="/var/lib/kubelet/pods/8916be04-39a2-4524-8e15-37a74aa82819/volumes"
Nov 03 19:48:44.034948 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:44.034908    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6db7b021-z5g6p"]
Nov 03 19:48:44.042713 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:44.042646    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6db7b021-z5g6p"]
Nov 03 19:48:44.923464 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:44.923424    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbade572-c672-4955-bdf8-5e47fc2099c7" path="/var/lib/kubelet/pods/dbade572-c672-4955-bdf8-5e47fc2099c7/volumes"
Nov 03 19:48:48.038483 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:48.038433    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b01b13a-qmqfx"]
Nov 03 19:48:48.043174 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:48.043132    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b01b13a-qmqfx"]
Nov 03 19:48:48.895481 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:48.895441    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3990af60-0428-45ec-8f19-715ea138bfdc" path="/var/lib/kubelet/pods/3990af60-0428-45ec-8f19-715ea138bfdc/volumes"
Nov 03 19:48:53.022721 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:53.022647    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab4ea102-fqfq9"]
Nov 03 19:48:53.028095 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:53.028061    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab4ea102-fqfq9"]
Nov 03 19:48:54.957333 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:54.957292    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27f755da-8289-4800-b933-6c147ff53e48" path="/var/lib/kubelet/pods/27f755da-8289-4800-b933-6c147ff53e48/volumes"
Nov 03 19:48:56.096030 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:56.095986    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1c446eb-qkztr"]
Nov 03 19:48:56.097190 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:56.097163    4921 cpu_manager.go:414] "containerMap: removing container" podUID="8916be04-39a2-4524-8e15-37a74aa82819" containerName="revoke-keys"
Nov 03 19:48:56.097386 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:56.097368    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8916be04-39a2-4524-8e15-37a74aa82819" containerName="revoke-keys"
Nov 03 19:48:56.097726 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:56.097710    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="8916be04-39a2-4524-8e15-37a74aa82819" containerName="revoke-keys"
Nov 03 19:48:56.110816 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:56.107814    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1c446eb-qkztr"]
Nov 03 19:48:56.110816 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:56.107905    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1c446eb-qkztr"
Nov 03 19:48:56.112716 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:56.111419    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 19:48:56.112716 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:56.111494    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:48:56.268089 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:56.268055    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkdwx\" (UniqueName: \"kubernetes.io/projected/18b354d5-f500-42c4-9406-55351d7c1d65-kube-api-access-qkdwx\") pod \"maas-api-revoke-sub-a1c446eb-qkztr\" (UID: \"18b354d5-f500-42c4-9406-55351d7c1d65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1c446eb-qkztr"
Nov 03 19:48:56.268248 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:56.268190    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18b354d5-f500-42c4-9406-55351d7c1d65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1c446eb-qkztr\" (UID: \"18b354d5-f500-42c4-9406-55351d7c1d65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1c446eb-qkztr"
Nov 03 19:48:56.268407 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:56.268393    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18b354d5-f500-42c4-9406-55351d7c1d65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1c446eb-qkztr\" (UID: \"18b354d5-f500-42c4-9406-55351d7c1d65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1c446eb-qkztr"
Nov 03 19:48:56.381036 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:56.380944    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18b354d5-f500-42c4-9406-55351d7c1d65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1c446eb-qkztr\" (UID: \"18b354d5-f500-42c4-9406-55351d7c1d65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1c446eb-qkztr"
Nov 03 19:48:56.381553 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:56.381525    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkdwx\" (UniqueName: \"kubernetes.io/projected/18b354d5-f500-42c4-9406-55351d7c1d65-kube-api-access-qkdwx\") pod \"maas-api-revoke-sub-a1c446eb-qkztr\" (UID: \"18b354d5-f500-42c4-9406-55351d7c1d65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1c446eb-qkztr"
Nov 03 19:48:56.381757 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:56.381738    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18b354d5-f500-42c4-9406-55351d7c1d65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1c446eb-qkztr\" (UID: \"18b354d5-f500-42c4-9406-55351d7c1d65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1c446eb-qkztr"
Nov 03 19:48:56.390457 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:56.388685    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18b354d5-f500-42c4-9406-55351d7c1d65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1c446eb-qkztr\" (UID: \"18b354d5-f500-42c4-9406-55351d7c1d65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1c446eb-qkztr"
Nov 03 19:48:56.404763 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:56.397434    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkdwx\" (UniqueName: \"kubernetes.io/projected/18b354d5-f500-42c4-9406-55351d7c1d65-kube-api-access-qkdwx\") pod \"maas-api-revoke-sub-a1c446eb-qkztr\" (UID: \"18b354d5-f500-42c4-9406-55351d7c1d65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1c446eb-qkztr"
Nov 03 19:48:56.421715 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:56.418297    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18b354d5-f500-42c4-9406-55351d7c1d65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1c446eb-qkztr\" (UID: \"18b354d5-f500-42c4-9406-55351d7c1d65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1c446eb-qkztr"
Nov 03 19:48:56.437041 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:56.437013    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1c446eb-qkztr"
Nov 03 19:48:56.720783 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:48:56.720750    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18b354d5_f500_42c4_9406_55351d7c1d65.slice/crio-38dc595cc5e2c6b4629462d16ad19c9212cb61206eac64690319368fd7dfc327 WatchSource:0}: Error finding container 38dc595cc5e2c6b4629462d16ad19c9212cb61206eac64690319368fd7dfc327: Status 404 returned error can't find the container with id 38dc595cc5e2c6b4629462d16ad19c9212cb61206eac64690319368fd7dfc327
Nov 03 19:48:56.723385 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:56.723359    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1c446eb-qkztr"]
Nov 03 19:48:57.038974 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:57.038940    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1c446eb-qkztr" event={"ID":"18b354d5-f500-42c4-9406-55351d7c1d65","Type":"ContainerStarted","Data":"38dc595cc5e2c6b4629462d16ad19c9212cb61206eac64690319368fd7dfc327"}
Nov 03 19:48:58.021525 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:58.021490    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66af4baf-jd5mj"]
Nov 03 19:48:58.025511 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:58.025487    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66af4baf-jd5mj"]
Nov 03 19:48:58.048253 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:58.048224    4921 generic.go:356] "Generic (PLEG): container finished" podID="18b354d5-f500-42c4-9406-55351d7c1d65" containerID="15bf8dd32dceddb8a1e942f90cd8191440a10e8fbeb4d5ab9834da7ba6b59852" exitCode=0
Nov 03 19:48:58.048393 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:58.048295    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1c446eb-qkztr" event={"ID":"18b354d5-f500-42c4-9406-55351d7c1d65","Type":"ContainerDied","Data":"15bf8dd32dceddb8a1e942f90cd8191440a10e8fbeb4d5ab9834da7ba6b59852"}
Nov 03 19:48:58.889567 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:58.889533    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecdf5c52-5f0d-41f4-ba12-9377eadbf76e" path="/var/lib/kubelet/pods/ecdf5c52-5f0d-41f4-ba12-9377eadbf76e/volumes"
Nov 03 19:48:59.235117 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:59.235093    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1c446eb-qkztr"
Nov 03 19:48:59.352866 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:59.352833    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18b354d5-f500-42c4-9406-55351d7c1d65-kube-api-access-qkdwx\" (UniqueName: \"kubernetes.io/projected/18b354d5-f500-42c4-9406-55351d7c1d65-kube-api-access-qkdwx\") pod \"18b354d5-f500-42c4-9406-55351d7c1d65\" (UID: \"18b354d5-f500-42c4-9406-55351d7c1d65\") "
Nov 03 19:48:59.353038 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:59.352887    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18b354d5-f500-42c4-9406-55351d7c1d65-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18b354d5-f500-42c4-9406-55351d7c1d65-maas-api-openshift-ca\") pod \"18b354d5-f500-42c4-9406-55351d7c1d65\" (UID: \"18b354d5-f500-42c4-9406-55351d7c1d65\") "
Nov 03 19:48:59.353207 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:59.353186    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18b354d5-f500-42c4-9406-55351d7c1d65-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18b354d5-f500-42c4-9406-55351d7c1d65-maas-api-xks-ca\") pod \"18b354d5-f500-42c4-9406-55351d7c1d65\" (UID: \"18b354d5-f500-42c4-9406-55351d7c1d65\") "
Nov 03 19:48:59.353315 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:59.353296    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18b354d5-f500-42c4-9406-55351d7c1d65-maas-api-openshift-ca" pod "18b354d5-f500-42c4-9406-55351d7c1d65" (UID: "18b354d5-f500-42c4-9406-55351d7c1d65"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:48:59.353745 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:59.353726    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18b354d5-f500-42c4-9406-55351d7c1d65-maas-api-openshift-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:48:59.355189 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:59.355157    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18b354d5-f500-42c4-9406-55351d7c1d65-kube-api-access-qkdwx" pod "18b354d5-f500-42c4-9406-55351d7c1d65" (UID: "18b354d5-f500-42c4-9406-55351d7c1d65"). InnerVolumeSpecName "kube-api-access-qkdwx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:48:59.355298 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:59.355221    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18b354d5-f500-42c4-9406-55351d7c1d65-maas-api-xks-ca" pod "18b354d5-f500-42c4-9406-55351d7c1d65" (UID: "18b354d5-f500-42c4-9406-55351d7c1d65"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:48:59.462739 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:59.456969    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18b354d5-f500-42c4-9406-55351d7c1d65-maas-api-xks-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:48:59.462739 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:48:59.457005    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkdwx\" (UniqueName: \"kubernetes.io/projected/18b354d5-f500-42c4-9406-55351d7c1d65-kube-api-access-qkdwx\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:49:00.070021 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:00.069988    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1c446eb-qkztr"
Nov 03 19:49:00.070275 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:00.069992    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1c446eb-qkztr" event={"ID":"18b354d5-f500-42c4-9406-55351d7c1d65","Type":"ContainerDied","Data":"38dc595cc5e2c6b4629462d16ad19c9212cb61206eac64690319368fd7dfc327"}
Nov 03 19:49:00.070275 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:00.070113    4921 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="38dc595cc5e2c6b4629462d16ad19c9212cb61206eac64690319368fd7dfc327"
Nov 03 19:49:02.923049 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:49:02.923011    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 19:49:10.023962 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:10.023930    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53ab2d2d-8v9d9"]
Nov 03 19:49:10.028008 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:10.027980    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53ab2d2d-8v9d9"]
Nov 03 19:49:10.894786 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:10.894747    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9b67cca-6629-4c9c-a69f-b4f0dabe2da3" path="/var/lib/kubelet/pods/c9b67cca-6629-4c9c-a69f-b4f0dabe2da3/volumes"
Nov 03 19:49:17.621813 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:17.621777    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2d78ea-6496b8c67-4wtxz"]
Nov 03 19:49:17.622836 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:17.622816    4921 cpu_manager.go:414] "containerMap: removing container" podUID="18b354d5-f500-42c4-9406-55351d7c1d65" containerName="revoke-keys"
Nov 03 19:49:17.622904 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:17.622838    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18b354d5-f500-42c4-9406-55351d7c1d65" containerName="revoke-keys"
Nov 03 19:49:17.623054 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:17.623042    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="18b354d5-f500-42c4-9406-55351d7c1d65" containerName="revoke-keys"
Nov 03 19:49:17.632691 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:17.626879    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2d78ea-6496b8c67-4wtxz"
Nov 03 19:49:17.636908 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:17.634107    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-2d78ea-serving-cert\""
Nov 03 19:49:17.636908 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:17.636211    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2d78ea-6496b8c67-4wtxz"]
Nov 03 19:49:17.777691 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:17.756918    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1fd32d8-1484-4622-86a3-24b16903084b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-2d78ea-6496b8c67-4wtxz\" (UID: \"d1fd32d8-1484-4622-86a3-24b16903084b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2d78ea-6496b8c67-4wtxz"
Nov 03 19:49:17.778087 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:17.778041    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1fd32d8-1484-4622-86a3-24b16903084b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-2d78ea-6496b8c67-4wtxz\" (UID: \"d1fd32d8-1484-4622-86a3-24b16903084b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2d78ea-6496b8c67-4wtxz"
Nov 03 19:49:17.778512 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:17.778486    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zvhh\" (UniqueName: \"kubernetes.io/projected/d1fd32d8-1484-4622-86a3-24b16903084b-kube-api-access-4zvhh\") pod \"maas-api-e2e-shared-b-w1-2d78ea-6496b8c67-4wtxz\" (UID: \"d1fd32d8-1484-4622-86a3-24b16903084b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2d78ea-6496b8c67-4wtxz"
Nov 03 19:49:17.788772 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:17.778713    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1fd32d8-1484-4622-86a3-24b16903084b-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-2d78ea-6496b8c67-4wtxz\" (UID: \"d1fd32d8-1484-4622-86a3-24b16903084b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2d78ea-6496b8c67-4wtxz"
Nov 03 19:49:17.890245 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:17.890150    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1fd32d8-1484-4622-86a3-24b16903084b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-2d78ea-6496b8c67-4wtxz\" (UID: \"d1fd32d8-1484-4622-86a3-24b16903084b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2d78ea-6496b8c67-4wtxz"
Nov 03 19:49:17.890420 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:17.890350    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zvhh\" (UniqueName: \"kubernetes.io/projected/d1fd32d8-1484-4622-86a3-24b16903084b-kube-api-access-4zvhh\") pod \"maas-api-e2e-shared-b-w1-2d78ea-6496b8c67-4wtxz\" (UID: \"d1fd32d8-1484-4622-86a3-24b16903084b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2d78ea-6496b8c67-4wtxz"
Nov 03 19:49:17.890493 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:17.890434    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1fd32d8-1484-4622-86a3-24b16903084b-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-2d78ea-6496b8c67-4wtxz\" (UID: \"d1fd32d8-1484-4622-86a3-24b16903084b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2d78ea-6496b8c67-4wtxz"
Nov 03 19:49:17.890598 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:17.890580    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1fd32d8-1484-4622-86a3-24b16903084b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-2d78ea-6496b8c67-4wtxz\" (UID: \"d1fd32d8-1484-4622-86a3-24b16903084b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2d78ea-6496b8c67-4wtxz"
Nov 03 19:49:17.890771 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:49:17.890751    4921 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2d78ea-serving-cert: secret "maas-api-e2e-shared-b-w1-2d78ea-serving-cert" not found
Nov 03 19:49:17.890850 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:49:17.890833    4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1fd32d8-1484-4622-86a3-24b16903084b-maas-api-tls podName:d1fd32d8-1484-4622-86a3-24b16903084b nodeName:}" failed. No retries permitted until 2026-11-03 19:49:18.390812294 +0000 UTC m=+1071.634777751 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d1fd32d8-1484-4622-86a3-24b16903084b-maas-api-tls") pod "maas-api-e2e-shared-b-w1-2d78ea-6496b8c67-4wtxz" (UID: "d1fd32d8-1484-4622-86a3-24b16903084b") : secret "maas-api-e2e-shared-b-w1-2d78ea-serving-cert" not found
Nov 03 19:49:17.891295 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:17.891276    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1fd32d8-1484-4622-86a3-24b16903084b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-2d78ea-6496b8c67-4wtxz\" (UID: \"d1fd32d8-1484-4622-86a3-24b16903084b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2d78ea-6496b8c67-4wtxz"
Nov 03 19:49:17.893054 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:17.893030    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1fd32d8-1484-4622-86a3-24b16903084b-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-2d78ea-6496b8c67-4wtxz\" (UID: \"d1fd32d8-1484-4622-86a3-24b16903084b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2d78ea-6496b8c67-4wtxz"
Nov 03 19:49:17.901757 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:17.901706    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zvhh\" (UniqueName: \"kubernetes.io/projected/d1fd32d8-1484-4622-86a3-24b16903084b-kube-api-access-4zvhh\") pod \"maas-api-e2e-shared-b-w1-2d78ea-6496b8c67-4wtxz\" (UID: \"d1fd32d8-1484-4622-86a3-24b16903084b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2d78ea-6496b8c67-4wtxz"
Nov 03 19:49:18.414193 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:18.414148    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1fd32d8-1484-4622-86a3-24b16903084b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-2d78ea-6496b8c67-4wtxz\" (UID: \"d1fd32d8-1484-4622-86a3-24b16903084b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2d78ea-6496b8c67-4wtxz"
Nov 03 19:49:18.452748 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:18.452715    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1fd32d8-1484-4622-86a3-24b16903084b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-2d78ea-6496b8c67-4wtxz\" (UID: \"d1fd32d8-1484-4622-86a3-24b16903084b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2d78ea-6496b8c67-4wtxz"
Nov 03 19:49:18.566260 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:18.566226    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2d78ea-6496b8c67-4wtxz"
Nov 03 19:49:19.534103 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:49:19.534021    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1fd32d8_1484_4622_86a3_24b16903084b.slice/crio-45bad26b45693de9d9cf3c88a4f991a75c4ffa7147a762dc8d20f4d5cfaed37a WatchSource:0}: Error finding container 45bad26b45693de9d9cf3c88a4f991a75c4ffa7147a762dc8d20f4d5cfaed37a: Status 404 returned error can't find the container with id 45bad26b45693de9d9cf3c88a4f991a75c4ffa7147a762dc8d20f4d5cfaed37a
Nov 03 19:49:19.567192 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:19.565554    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2d78ea-6496b8c67-4wtxz"]
Nov 03 19:49:20.571682 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:20.571631    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2d78ea-6496b8c67-4wtxz" event={"ID":"d1fd32d8-1484-4622-86a3-24b16903084b","Type":"ContainerStarted","Data":"50f3cc2ac0d7bfafdb4640589b9735df46d142135165830a93db9e8624a81f93"}
Nov 03 19:49:20.572154 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:20.571750    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2d78ea-6496b8c67-4wtxz"
Nov 03 19:49:20.572154 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:20.571775    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2d78ea-6496b8c67-4wtxz" event={"ID":"d1fd32d8-1484-4622-86a3-24b16903084b","Type":"ContainerStarted","Data":"45bad26b45693de9d9cf3c88a4f991a75c4ffa7147a762dc8d20f4d5cfaed37a"}
Nov 03 19:49:20.590411 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:20.590349    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2d78ea-6496b8c67-4wtxz" podStartSLOduration=3.056023408 podStartE2EDuration="3.5903312s" totalImagesPullingTime="534.307792ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:49:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:49:20.584969747 +0000 UTC m=+1073.828935214" watchObservedRunningTime="2026-11-03 19:49:20.5903312 +0000 UTC m=+1073.834296666"
Nov 03 19:49:26.504412 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:26.504375    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd"]
Nov 03 19:49:26.516408 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:26.516346    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd"
Nov 03 19:49:26.524084 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:26.524045    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd"]
Nov 03 19:49:26.526663 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:26.526636    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-fcddeb-openshift-default-dockercfg-cq459\""
Nov 03 19:49:26.570139 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:26.570105    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4736bbc2-086a-4d21-bafa-3abd36c25fa6-workload-certs\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd\" (UID: \"4736bbc2-086a-4d21-bafa-3abd36c25fa6\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd"
Nov 03 19:49:26.570316 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:26.570144    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4736bbc2-086a-4d21-bafa-3abd36c25fa6-workload-socket\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd\" (UID: \"4736bbc2-086a-4d21-bafa-3abd36c25fa6\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd"
Nov 03 19:49:26.570316 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:26.570226    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4736bbc2-086a-4d21-bafa-3abd36c25fa6-istio-envoy\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd\" (UID: \"4736bbc2-086a-4d21-bafa-3abd36c25fa6\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd"
Nov 03 19:49:26.570390 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:26.570341    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4736bbc2-086a-4d21-bafa-3abd36c25fa6-istiod-ca-cert\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd\" (UID: \"4736bbc2-086a-4d21-bafa-3abd36c25fa6\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd"
Nov 03 19:49:26.570425 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:26.570393    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4736bbc2-086a-4d21-bafa-3abd36c25fa6-istio-data\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd\" (UID: \"4736bbc2-086a-4d21-bafa-3abd36c25fa6\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd"
Nov 03 19:49:26.570459 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:26.570421    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4736bbc2-086a-4d21-bafa-3abd36c25fa6-istio-podinfo\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd\" (UID: \"4736bbc2-086a-4d21-bafa-3abd36c25fa6\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd"
Nov 03 19:49:26.570534 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:26.570522    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4736bbc2-086a-4d21-bafa-3abd36c25fa6-istio-token\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd\" (UID: \"4736bbc2-086a-4d21-bafa-3abd36c25fa6\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd"
Nov 03 19:49:26.570654 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:26.570642    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4vdp\" (UniqueName: \"kubernetes.io/projected/4736bbc2-086a-4d21-bafa-3abd36c25fa6-kube-api-access-w4vdp\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd\" (UID: \"4736bbc2-086a-4d21-bafa-3abd36c25fa6\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd"
Nov 03 19:49:26.570737 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:26.570685    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4736bbc2-086a-4d21-bafa-3abd36c25fa6-credential-socket\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd\" (UID: \"4736bbc2-086a-4d21-bafa-3abd36c25fa6\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd"
Nov 03 19:49:26.616990 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:26.616962    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2d78ea-6496b8c67-4wtxz"
Nov 03 19:49:26.672143 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:26.672107    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4736bbc2-086a-4d21-bafa-3abd36c25fa6-istio-token\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd\" (UID: \"4736bbc2-086a-4d21-bafa-3abd36c25fa6\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd"
Nov 03 19:49:26.672348 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:26.672292    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4vdp\" (UniqueName: \"kubernetes.io/projected/4736bbc2-086a-4d21-bafa-3abd36c25fa6-kube-api-access-w4vdp\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd\" (UID: \"4736bbc2-086a-4d21-bafa-3abd36c25fa6\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd"
Nov 03 19:49:26.672348 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:26.672336    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4736bbc2-086a-4d21-bafa-3abd36c25fa6-credential-socket\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd\" (UID: \"4736bbc2-086a-4d21-bafa-3abd36c25fa6\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd"
Nov 03 19:49:26.672476 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:26.672457    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4736bbc2-086a-4d21-bafa-3abd36c25fa6-workload-certs\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd\" (UID: \"4736bbc2-086a-4d21-bafa-3abd36c25fa6\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd"
Nov 03 19:49:26.672534 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:26.672493    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4736bbc2-086a-4d21-bafa-3abd36c25fa6-workload-socket\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd\" (UID: \"4736bbc2-086a-4d21-bafa-3abd36c25fa6\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd"
Nov 03 19:49:26.672616 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:26.672603    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4736bbc2-086a-4d21-bafa-3abd36c25fa6-istio-envoy\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd\" (UID: \"4736bbc2-086a-4d21-bafa-3abd36c25fa6\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd"
Nov 03 19:49:26.672715 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:26.672639    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4736bbc2-086a-4d21-bafa-3abd36c25fa6-istiod-ca-cert\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd\" (UID: \"4736bbc2-086a-4d21-bafa-3abd36c25fa6\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd"
Nov 03 19:49:26.672715 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:26.672708    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4736bbc2-086a-4d21-bafa-3abd36c25fa6-istio-data\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd\" (UID: \"4736bbc2-086a-4d21-bafa-3abd36c25fa6\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd"
Nov 03 19:49:26.672829 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:26.672752    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4736bbc2-086a-4d21-bafa-3abd36c25fa6-istio-podinfo\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd\" (UID: \"4736bbc2-086a-4d21-bafa-3abd36c25fa6\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd"
Nov 03 19:49:26.673775 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:26.673730    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4736bbc2-086a-4d21-bafa-3abd36c25fa6-workload-certs\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd\" (UID: \"4736bbc2-086a-4d21-bafa-3abd36c25fa6\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd"
Nov 03 19:49:26.673942 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:26.673914    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4736bbc2-086a-4d21-bafa-3abd36c25fa6-istio-data\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd\" (UID: \"4736bbc2-086a-4d21-bafa-3abd36c25fa6\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd"
Nov 03 19:49:26.674019 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:26.673663    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4736bbc2-086a-4d21-bafa-3abd36c25fa6-workload-socket\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd\" (UID: \"4736bbc2-086a-4d21-bafa-3abd36c25fa6\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd"
Nov 03 19:49:26.674110 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:26.674019    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4736bbc2-086a-4d21-bafa-3abd36c25fa6-credential-socket\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd\" (UID: \"4736bbc2-086a-4d21-bafa-3abd36c25fa6\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd"
Nov 03 19:49:26.674556 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:26.674531    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4736bbc2-086a-4d21-bafa-3abd36c25fa6-istiod-ca-cert\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd\" (UID: \"4736bbc2-086a-4d21-bafa-3abd36c25fa6\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd"
Nov 03 19:49:26.675660 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:26.675635    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4736bbc2-086a-4d21-bafa-3abd36c25fa6-istio-podinfo\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd\" (UID: \"4736bbc2-086a-4d21-bafa-3abd36c25fa6\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd"
Nov 03 19:49:26.676049 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:26.676029    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4736bbc2-086a-4d21-bafa-3abd36c25fa6-istio-envoy\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd\" (UID: \"4736bbc2-086a-4d21-bafa-3abd36c25fa6\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd"
Nov 03 19:49:26.699864 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:26.699828    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4736bbc2-086a-4d21-bafa-3abd36c25fa6-istio-token\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd\" (UID: \"4736bbc2-086a-4d21-bafa-3abd36c25fa6\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd"
Nov 03 19:49:26.700031 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:26.699927    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4vdp\" (UniqueName: \"kubernetes.io/projected/4736bbc2-086a-4d21-bafa-3abd36c25fa6-kube-api-access-w4vdp\") pod \"e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd\" (UID: \"4736bbc2-086a-4d21-bafa-3abd36c25fa6\") " pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd"
Nov 03 19:49:26.845644 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:26.845557    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd"
Nov 03 19:49:27.341066 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:27.341037    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd"]
Nov 03 19:49:27.341651 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:49:27.341628    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4736bbc2_086a_4d21_bafa_3abd36c25fa6.slice/crio-f18e447ad99cf87d0e8eda8c656f269a66145188b266a0658f85cf9aadca90a2 WatchSource:0}: Error finding container f18e447ad99cf87d0e8eda8c656f269a66145188b266a0658f85cf9aadca90a2: Status 404 returned error can't find the container with id f18e447ad99cf87d0e8eda8c656f269a66145188b266a0658f85cf9aadca90a2
Nov 03 19:49:27.344478 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:27.344440    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:49:27.344598 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:27.344525    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:49:27.344598 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:27.344569    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:49:27.672144 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:27.672058    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd" event={"ID":"4736bbc2-086a-4d21-bafa-3abd36c25fa6","Type":"ContainerStarted","Data":"1d7ffece2c1e3a1e71c9c10f7d5445f3702abd027a7601aefc4b0c0483872c2a"}
Nov 03 19:49:27.672727 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:27.672700    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd" event={"ID":"4736bbc2-086a-4d21-bafa-3abd36c25fa6","Type":"ContainerStarted","Data":"f18e447ad99cf87d0e8eda8c656f269a66145188b266a0658f85cf9aadca90a2"}
Nov 03 19:49:27.737697 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:27.734275    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd" podStartSLOduration=1.734251992 podStartE2EDuration="1.734251992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:49:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:49:27.698130416 +0000 UTC m=+1080.942095883" watchObservedRunningTime="2026-11-03 19:49:27.734251992 +0000 UTC m=+1080.978217458"
Nov 03 19:49:27.871803 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:27.871763    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd"
Nov 03 19:49:28.848014 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:28.847979    4921 patch_prober.go:29] interesting pod/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Nov 03 19:49:28.848482 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:28.848052    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd" podUID="4736bbc2-086a-4d21-bafa-3abd36c25fa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Nov 03 19:49:29.845878 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:29.845845    4921 patch_prober.go:29] interesting pod/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Nov 03 19:49:29.846072 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:29.845898    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd" podUID="4736bbc2-086a-4d21-bafa-3abd36c25fa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Nov 03 19:49:30.845952 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:30.845915    4921 patch_prober.go:29] interesting pod/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Nov 03 19:49:30.846458 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:30.845980    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd" podUID="4736bbc2-086a-4d21-bafa-3abd36c25fa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Nov 03 19:49:31.846161 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:31.846118    4921 patch_prober.go:29] interesting pod/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Nov 03 19:49:31.847059 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:31.846181    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd" podUID="4736bbc2-086a-4d21-bafa-3abd36c25fa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Nov 03 19:49:32.846150 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:32.846116    4921 patch_prober.go:29] interesting pod/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Nov 03 19:49:32.846341 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:32.846170    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd" podUID="4736bbc2-086a-4d21-bafa-3abd36c25fa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Nov 03 19:49:32.934210 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:49:32.934174    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 19:49:33.846615 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:33.846576    4921 patch_prober.go:29] interesting pod/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Nov 03 19:49:33.847027 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:33.846641    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd" podUID="4736bbc2-086a-4d21-bafa-3abd36c25fa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Nov 03 19:49:34.896051 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:34.896015    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd"
Nov 03 19:49:34.926735 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:34.926692    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd"
Nov 03 19:49:34.926921 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:34.926801    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd"
Nov 03 19:49:53.686010 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:53.685927    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv" podUID="01c749b9-d188-4673-9a3d-99b9c4ae0793" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:49:53.820693 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:53.698547    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk" podUID="2d0a934e-4816-44d2-9bb5-e8b4e95dd73c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:49:53.829201 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:53.713156    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6" podUID="1aaee61d-b036-4c2b-bee7-a692b8bed612" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 03 19:49:57.889954 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:57.889906    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e42df580-xjhkn"]
Nov 03 19:49:57.895564 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:57.895541    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e42df580-xjhkn"
Nov 03 19:49:57.899706 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:57.899682    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:49:57.900173 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:57.900130    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 19:49:57.904833 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:57.904810    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e42df580-xjhkn"]
Nov 03 19:49:57.975929 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:57.975893    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0487c9eb-449c-4aa8-b41d-277c67914e58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e42df580-xjhkn\" (UID: \"0487c9eb-449c-4aa8-b41d-277c67914e58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e42df580-xjhkn"
Nov 03 19:49:57.976179 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:57.976153    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-265lm\" (UniqueName: \"kubernetes.io/projected/0487c9eb-449c-4aa8-b41d-277c67914e58-kube-api-access-265lm\") pod \"maas-api-revoke-sub-e42df580-xjhkn\" (UID: \"0487c9eb-449c-4aa8-b41d-277c67914e58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e42df580-xjhkn"
Nov 03 19:49:57.976321 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:57.976306    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0487c9eb-449c-4aa8-b41d-277c67914e58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e42df580-xjhkn\" (UID: \"0487c9eb-449c-4aa8-b41d-277c67914e58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e42df580-xjhkn"
Nov 03 19:49:58.078160 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:58.078122    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-265lm\" (UniqueName: \"kubernetes.io/projected/0487c9eb-449c-4aa8-b41d-277c67914e58-kube-api-access-265lm\") pod \"maas-api-revoke-sub-e42df580-xjhkn\" (UID: \"0487c9eb-449c-4aa8-b41d-277c67914e58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e42df580-xjhkn"
Nov 03 19:49:58.078337 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:58.078239    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0487c9eb-449c-4aa8-b41d-277c67914e58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e42df580-xjhkn\" (UID: \"0487c9eb-449c-4aa8-b41d-277c67914e58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e42df580-xjhkn"
Nov 03 19:49:58.078337 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:58.078320    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0487c9eb-449c-4aa8-b41d-277c67914e58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e42df580-xjhkn\" (UID: \"0487c9eb-449c-4aa8-b41d-277c67914e58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e42df580-xjhkn"
Nov 03 19:49:58.079034 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:58.079007    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0487c9eb-449c-4aa8-b41d-277c67914e58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e42df580-xjhkn\" (UID: \"0487c9eb-449c-4aa8-b41d-277c67914e58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e42df580-xjhkn"
Nov 03 19:49:58.080841 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:58.080816    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0487c9eb-449c-4aa8-b41d-277c67914e58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e42df580-xjhkn\" (UID: \"0487c9eb-449c-4aa8-b41d-277c67914e58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e42df580-xjhkn"
Nov 03 19:49:58.085197 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:58.085172    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-265lm\" (UniqueName: \"kubernetes.io/projected/0487c9eb-449c-4aa8-b41d-277c67914e58-kube-api-access-265lm\") pod \"maas-api-revoke-sub-e42df580-xjhkn\" (UID: \"0487c9eb-449c-4aa8-b41d-277c67914e58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e42df580-xjhkn"
Nov 03 19:49:58.216739 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:58.216640    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e42df580-xjhkn"
Nov 03 19:49:58.510270 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:58.510239    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e42df580-xjhkn"]
Nov 03 19:49:58.510621 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:49:58.510587    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0487c9eb_449c_4aa8_b41d_277c67914e58.slice/crio-8da8bfcf2aab115845516b23a92b8ab0cae7489cc196034fd643e6c86e14e5a4 WatchSource:0}: Error finding container 8da8bfcf2aab115845516b23a92b8ab0cae7489cc196034fd643e6c86e14e5a4: Status 404 returned error can't find the container with id 8da8bfcf2aab115845516b23a92b8ab0cae7489cc196034fd643e6c86e14e5a4
Nov 03 19:49:59.127889 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:59.127803    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e42df580-xjhkn" event={"ID":"0487c9eb-449c-4aa8-b41d-277c67914e58","Type":"ContainerStarted","Data":"62c14dd321b2c5ee0052904551391b13ea36b106e71ca6765f867cbc3ce3a5e8"}
Nov 03 19:49:59.127889 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:59.127882    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e42df580-xjhkn" event={"ID":"0487c9eb-449c-4aa8-b41d-277c67914e58","Type":"ContainerStarted","Data":"8da8bfcf2aab115845516b23a92b8ab0cae7489cc196034fd643e6c86e14e5a4"}
Nov 03 19:49:59.146002 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:49:59.145952    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e42df580-xjhkn" podStartSLOduration=2.14593837 podStartE2EDuration="2.14593837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:49:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:49:59.14079185 +0000 UTC m=+1112.384757314" watchObservedRunningTime="2026-11-03 19:49:59.14593837 +0000 UTC m=+1112.389903901"
Nov 03 19:50:00.137836 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:00.137801    4921 generic.go:356] "Generic (PLEG): container finished" podID="0487c9eb-449c-4aa8-b41d-277c67914e58" containerID="62c14dd321b2c5ee0052904551391b13ea36b106e71ca6765f867cbc3ce3a5e8" exitCode=0
Nov 03 19:50:00.138355 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:00.137903    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e42df580-xjhkn" event={"ID":"0487c9eb-449c-4aa8-b41d-277c67914e58","Type":"ContainerDied","Data":"62c14dd321b2c5ee0052904551391b13ea36b106e71ca6765f867cbc3ce3a5e8"}
Nov 03 19:50:01.444205 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:01.444172    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e42df580-xjhkn"
Nov 03 19:50:01.596065 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:01.596029    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0487c9eb-449c-4aa8-b41d-277c67914e58-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0487c9eb-449c-4aa8-b41d-277c67914e58-maas-api-xks-ca\") pod \"0487c9eb-449c-4aa8-b41d-277c67914e58\" (UID: \"0487c9eb-449c-4aa8-b41d-277c67914e58\") "
Nov 03 19:50:01.596265 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:01.596241    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0487c9eb-449c-4aa8-b41d-277c67914e58-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0487c9eb-449c-4aa8-b41d-277c67914e58-maas-api-openshift-ca\") pod \"0487c9eb-449c-4aa8-b41d-277c67914e58\" (UID: \"0487c9eb-449c-4aa8-b41d-277c67914e58\") "
Nov 03 19:50:01.596410 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:01.596389    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0487c9eb-449c-4aa8-b41d-277c67914e58-kube-api-access-265lm\" (UniqueName: \"kubernetes.io/projected/0487c9eb-449c-4aa8-b41d-277c67914e58-kube-api-access-265lm\") pod \"0487c9eb-449c-4aa8-b41d-277c67914e58\" (UID: \"0487c9eb-449c-4aa8-b41d-277c67914e58\") "
Nov 03 19:50:01.596605 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:01.596582    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0487c9eb-449c-4aa8-b41d-277c67914e58-maas-api-openshift-ca" pod "0487c9eb-449c-4aa8-b41d-277c67914e58" (UID: "0487c9eb-449c-4aa8-b41d-277c67914e58"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:50:01.597060 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:01.597034    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0487c9eb-449c-4aa8-b41d-277c67914e58-maas-api-openshift-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:50:01.598592 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:01.598563    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0487c9eb-449c-4aa8-b41d-277c67914e58-kube-api-access-265lm" pod "0487c9eb-449c-4aa8-b41d-277c67914e58" (UID: "0487c9eb-449c-4aa8-b41d-277c67914e58"). InnerVolumeSpecName "kube-api-access-265lm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:50:01.598717 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:01.598611    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0487c9eb-449c-4aa8-b41d-277c67914e58-maas-api-xks-ca" pod "0487c9eb-449c-4aa8-b41d-277c67914e58" (UID: "0487c9eb-449c-4aa8-b41d-277c67914e58"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:50:01.720745 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:01.718896    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-265lm\" (UniqueName: \"kubernetes.io/projected/0487c9eb-449c-4aa8-b41d-277c67914e58-kube-api-access-265lm\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:50:01.720745 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:01.718941    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0487c9eb-449c-4aa8-b41d-277c67914e58-maas-api-xks-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:50:02.158261 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:02.158228    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e42df580-xjhkn"
Nov 03 19:50:02.158472 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:02.158281    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e42df580-xjhkn" event={"ID":"0487c9eb-449c-4aa8-b41d-277c67914e58","Type":"ContainerDied","Data":"8da8bfcf2aab115845516b23a92b8ab0cae7489cc196034fd643e6c86e14e5a4"}
Nov 03 19:50:02.158472 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:02.158336    4921 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8da8bfcf2aab115845516b23a92b8ab0cae7489cc196034fd643e6c86e14e5a4"
Nov 03 19:50:02.975640 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:50:02.975604    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 19:50:13.743242 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:13.743152    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3c7998-b99c77f94-nn9qk"]
Nov 03 19:50:13.744077 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:13.744055    4921 cpu_manager.go:414] "containerMap: removing container" podUID="0487c9eb-449c-4aa8-b41d-277c67914e58" containerName="revoke-keys"
Nov 03 19:50:13.744199 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:13.744078    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0487c9eb-449c-4aa8-b41d-277c67914e58" containerName="revoke-keys"
Nov 03 19:50:13.744299 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:13.744285    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="0487c9eb-449c-4aa8-b41d-277c67914e58" containerName="revoke-keys"
Nov 03 19:50:13.748095 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:13.748073    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c7998-b99c77f94-nn9qk"
Nov 03 19:50:13.765164 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:13.763594    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-3c7998-serving-cert\""
Nov 03 19:50:13.770618 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:13.770593    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3c7998-b99c77f94-nn9qk"]
Nov 03 19:50:13.837761 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:13.837729    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e12be141-240d-4c12-9b90-900d64537e30-openshift-service-ca\") pod \"maas-api-e2e-api-a-3c7998-b99c77f94-nn9qk\" (UID: \"e12be141-240d-4c12-9b90-900d64537e30\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c7998-b99c77f94-nn9qk"
Nov 03 19:50:13.837953 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:13.837776    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dnkl\" (UniqueName: \"kubernetes.io/projected/e12be141-240d-4c12-9b90-900d64537e30-kube-api-access-8dnkl\") pod \"maas-api-e2e-api-a-3c7998-b99c77f94-nn9qk\" (UID: \"e12be141-240d-4c12-9b90-900d64537e30\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c7998-b99c77f94-nn9qk"
Nov 03 19:50:13.837953 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:13.837941    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e12be141-240d-4c12-9b90-900d64537e30-maas-api-tls\") pod \"maas-api-e2e-api-a-3c7998-b99c77f94-nn9qk\" (UID: \"e12be141-240d-4c12-9b90-900d64537e30\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c7998-b99c77f94-nn9qk"
Nov 03 19:50:13.838097 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:13.838073    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e12be141-240d-4c12-9b90-900d64537e30-metrics-certs\") pod \"maas-api-e2e-api-a-3c7998-b99c77f94-nn9qk\" (UID: \"e12be141-240d-4c12-9b90-900d64537e30\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c7998-b99c77f94-nn9qk"
Nov 03 19:50:13.940947 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:13.940904    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e12be141-240d-4c12-9b90-900d64537e30-metrics-certs\") pod \"maas-api-e2e-api-a-3c7998-b99c77f94-nn9qk\" (UID: \"e12be141-240d-4c12-9b90-900d64537e30\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c7998-b99c77f94-nn9qk"
Nov 03 19:50:13.959643 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:13.959607    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e12be141-240d-4c12-9b90-900d64537e30-openshift-service-ca\") pod \"maas-api-e2e-api-a-3c7998-b99c77f94-nn9qk\" (UID: \"e12be141-240d-4c12-9b90-900d64537e30\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c7998-b99c77f94-nn9qk"
Nov 03 19:50:13.976818 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:13.976779    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dnkl\" (UniqueName: \"kubernetes.io/projected/e12be141-240d-4c12-9b90-900d64537e30-kube-api-access-8dnkl\") pod \"maas-api-e2e-api-a-3c7998-b99c77f94-nn9qk\" (UID: \"e12be141-240d-4c12-9b90-900d64537e30\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c7998-b99c77f94-nn9qk"
Nov 03 19:50:13.987769 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:13.981569    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e12be141-240d-4c12-9b90-900d64537e30-maas-api-tls\") pod \"maas-api-e2e-api-a-3c7998-b99c77f94-nn9qk\" (UID: \"e12be141-240d-4c12-9b90-900d64537e30\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c7998-b99c77f94-nn9qk"
Nov 03 19:50:13.987769 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:13.980632    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e12be141-240d-4c12-9b90-900d64537e30-metrics-certs\") pod \"maas-api-e2e-api-a-3c7998-b99c77f94-nn9qk\" (UID: \"e12be141-240d-4c12-9b90-900d64537e30\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c7998-b99c77f94-nn9qk"
Nov 03 19:50:13.987769 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:13.960804    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e12be141-240d-4c12-9b90-900d64537e30-openshift-service-ca\") pod \"maas-api-e2e-api-a-3c7998-b99c77f94-nn9qk\" (UID: \"e12be141-240d-4c12-9b90-900d64537e30\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c7998-b99c77f94-nn9qk"
Nov 03 19:50:13.987769 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:50:13.982181    4921 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-3c7998-serving-cert: secret "maas-api-e2e-api-a-3c7998-serving-cert" not found
Nov 03 19:50:13.987769 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:50:13.982251    4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e12be141-240d-4c12-9b90-900d64537e30-maas-api-tls podName:e12be141-240d-4c12-9b90-900d64537e30 nodeName:}" failed. No retries permitted until 2026-11-03 19:50:14.482231733 +0000 UTC m=+1127.726197194 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e12be141-240d-4c12-9b90-900d64537e30-maas-api-tls") pod "maas-api-e2e-api-a-3c7998-b99c77f94-nn9qk" (UID: "e12be141-240d-4c12-9b90-900d64537e30") : secret "maas-api-e2e-api-a-3c7998-serving-cert" not found
Nov 03 19:50:13.993221 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:13.989962    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dnkl\" (UniqueName: \"kubernetes.io/projected/e12be141-240d-4c12-9b90-900d64537e30-kube-api-access-8dnkl\") pod \"maas-api-e2e-api-a-3c7998-b99c77f94-nn9qk\" (UID: \"e12be141-240d-4c12-9b90-900d64537e30\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c7998-b99c77f94-nn9qk"
Nov 03 19:50:14.495691 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:14.494794    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e12be141-240d-4c12-9b90-900d64537e30-maas-api-tls\") pod \"maas-api-e2e-api-a-3c7998-b99c77f94-nn9qk\" (UID: \"e12be141-240d-4c12-9b90-900d64537e30\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c7998-b99c77f94-nn9qk"
Nov 03 19:50:14.534705 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:14.531875    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e12be141-240d-4c12-9b90-900d64537e30-maas-api-tls\") pod \"maas-api-e2e-api-a-3c7998-b99c77f94-nn9qk\" (UID: \"e12be141-240d-4c12-9b90-900d64537e30\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c7998-b99c77f94-nn9qk"
Nov 03 19:50:14.686056 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:14.686019    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c7998-b99c77f94-nn9qk"
Nov 03 19:50:15.180218 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:15.180178    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3c7998-b99c77f94-nn9qk"]
Nov 03 19:50:15.190619 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:50:15.190528    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode12be141_240d_4c12_9b90_900d64537e30.slice/crio-fdd75c8d612feca2987d2cfd092453f3fda1408d1b7693deec66b5eef76c986c WatchSource:0}: Error finding container fdd75c8d612feca2987d2cfd092453f3fda1408d1b7693deec66b5eef76c986c: Status 404 returned error can't find the container with id fdd75c8d612feca2987d2cfd092453f3fda1408d1b7693deec66b5eef76c986c
Nov 03 19:50:15.361276 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:15.361173    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c7998-b99c77f94-nn9qk" event={"ID":"e12be141-240d-4c12-9b90-900d64537e30","Type":"ContainerStarted","Data":"fdd75c8d612feca2987d2cfd092453f3fda1408d1b7693deec66b5eef76c986c"}
Nov 03 19:50:16.371079 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:16.371042    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c7998-b99c77f94-nn9qk" event={"ID":"e12be141-240d-4c12-9b90-900d64537e30","Type":"ContainerStarted","Data":"3e0657b5cca9ac115fe99fe9198709bc3df60ef64d9acd12bf0d5c121bf71aca"}
Nov 03 19:50:16.371592 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:16.371143    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c7998-b99c77f94-nn9qk"
Nov 03 19:50:16.389345 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:16.389293    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c7998-b99c77f94-nn9qk" podStartSLOduration=2.878842636 podStartE2EDuration="3.389280856s" totalImagesPullingTime="510.43822ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:50:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:50:16.385762563 +0000 UTC m=+1129.629728028" watchObservedRunningTime="2026-11-03 19:50:16.389280856 +0000 UTC m=+1129.633246338"
Nov 03 19:50:22.385961 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:22.385930    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c7998-b99c77f94-nn9qk"
Nov 03 19:50:28.649632 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:28.649598    4921 scope.go:122] "RemoveContainer" containerID="3eba5649c4cf9ebf9fe6be2fcda57f733fc817660787a36c616dc812581987df"
Nov 03 19:50:28.661521 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:28.661494    4921 scope.go:122] "RemoveContainer" containerID="92a1f022e49fde8cd45cf7dd80b75697ffa1ba6a27907f387ac311ecc33dcd4d"
Nov 03 19:50:28.673709 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:28.673569    4921 scope.go:122] "RemoveContainer" containerID="676d8a7aadecb467cff6c6fcc2eba021efb5f5813c53ee9438bc2964b4fd229d"
Nov 03 19:50:28.744693 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:28.736930    4921 scope.go:122] "RemoveContainer" containerID="16c2994fc09cd5da78af94f3359c0fd049607fae290b486a18f849584d64fa72"
Nov 03 19:50:28.835915 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:28.835881    4921 scope.go:122] "RemoveContainer" containerID="321a599f1e7276c617b29dc72387f7db367c0888c6a95f658f1c4f5ee91efad2"
Nov 03 19:50:28.916246 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:28.916215    4921 scope.go:122] "RemoveContainer" containerID="886ad597e86ca39fb6a8635c7ac666a4c1aa5ff31630dda8a5093a98d89bbd31"
Nov 03 19:50:33.128896 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:50:33.128855    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 19:50:33.803693 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:33.795140    4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-cz4nm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:50:33.803693 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:33.795250    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cz4nm" podUID="5053431a-0bcc-4e99-8776-c226339e3bde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:50:33.851703 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:33.835825    4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-rl9tr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:50:33.851703 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:33.835903    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rl9tr" podUID="cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:50:41.217174 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:41.217138    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-82b108-77f564d699-qnmjh"]
Nov 03 19:50:41.221616 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:41.221593    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-82b108-77f564d699-qnmjh"
Nov 03 19:50:41.239907 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:41.239880    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-82b108-dockercfg-9tb8d\""
Nov 03 19:50:41.240064 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:41.239925    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-82b108\""
Nov 03 19:50:41.243416 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:41.243385    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-82b108-77f564d699-qnmjh"]
Nov 03 19:50:41.249301 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:41.249274    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-82b108-79d7b7dddc-wzqv7"]
Nov 03 19:50:41.254121 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:41.254098    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-82b108-79d7b7dddc-wzqv7"
Nov 03 19:50:41.272689 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:41.272582    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-82b108-79d7b7dddc-wzqv7"]
Nov 03 19:50:41.400701 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:41.398281    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bd7e260-e08c-4413-aea4-eb140c9d8a7e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-82b108-77f564d699-qnmjh\" (UID: \"4bd7e260-e08c-4413-aea4-eb140c9d8a7e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-82b108-77f564d699-qnmjh"
Nov 03 19:50:41.400701 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:41.398434    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2clv\" (UniqueName: \"kubernetes.io/projected/22eb4983-2f08-41d8-b8b6-562c7a151ca2-kube-api-access-c2clv\") pod \"payload-processing-e2e-api-b-82b108-79d7b7dddc-wzqv7\" (UID: \"22eb4983-2f08-41d8-b8b6-562c7a151ca2\") " pod="openshift-ingress/payload-processing-e2e-api-b-82b108-79d7b7dddc-wzqv7"
Nov 03 19:50:41.400701 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:41.398531    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22eb4983-2f08-41d8-b8b6-562c7a151ca2-plugins-config-volume\") pod \"payload-processing-e2e-api-b-82b108-79d7b7dddc-wzqv7\" (UID: \"22eb4983-2f08-41d8-b8b6-562c7a151ca2\") " pod="openshift-ingress/payload-processing-e2e-api-b-82b108-79d7b7dddc-wzqv7"
Nov 03 19:50:41.400701 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:41.398718    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87bw6\" (UniqueName: \"kubernetes.io/projected/4bd7e260-e08c-4413-aea4-eb140c9d8a7e-kube-api-access-87bw6\") pod \"payload-pre-processing-e2e-api-b-82b108-77f564d699-qnmjh\" (UID: \"4bd7e260-e08c-4413-aea4-eb140c9d8a7e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-82b108-77f564d699-qnmjh"
Nov 03 19:50:41.514690 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:41.512279    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2clv\" (UniqueName: \"kubernetes.io/projected/22eb4983-2f08-41d8-b8b6-562c7a151ca2-kube-api-access-c2clv\") pod \"payload-processing-e2e-api-b-82b108-79d7b7dddc-wzqv7\" (UID: \"22eb4983-2f08-41d8-b8b6-562c7a151ca2\") " pod="openshift-ingress/payload-processing-e2e-api-b-82b108-79d7b7dddc-wzqv7"
Nov 03 19:50:41.514690 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:41.512408    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22eb4983-2f08-41d8-b8b6-562c7a151ca2-plugins-config-volume\") pod \"payload-processing-e2e-api-b-82b108-79d7b7dddc-wzqv7\" (UID: \"22eb4983-2f08-41d8-b8b6-562c7a151ca2\") " pod="openshift-ingress/payload-processing-e2e-api-b-82b108-79d7b7dddc-wzqv7"
Nov 03 19:50:41.514690 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:41.512648    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87bw6\" (UniqueName: \"kubernetes.io/projected/4bd7e260-e08c-4413-aea4-eb140c9d8a7e-kube-api-access-87bw6\") pod \"payload-pre-processing-e2e-api-b-82b108-77f564d699-qnmjh\" (UID: \"4bd7e260-e08c-4413-aea4-eb140c9d8a7e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-82b108-77f564d699-qnmjh"
Nov 03 19:50:41.514690 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:41.512779    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bd7e260-e08c-4413-aea4-eb140c9d8a7e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-82b108-77f564d699-qnmjh\" (UID: \"4bd7e260-e08c-4413-aea4-eb140c9d8a7e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-82b108-77f564d699-qnmjh"
Nov 03 19:50:41.591769 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:41.531470    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22eb4983-2f08-41d8-b8b6-562c7a151ca2-plugins-config-volume\") pod \"payload-processing-e2e-api-b-82b108-79d7b7dddc-wzqv7\" (UID: \"22eb4983-2f08-41d8-b8b6-562c7a151ca2\") " pod="openshift-ingress/payload-processing-e2e-api-b-82b108-79d7b7dddc-wzqv7"
Nov 03 19:50:41.602854 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:41.593071    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2clv\" (UniqueName: \"kubernetes.io/projected/22eb4983-2f08-41d8-b8b6-562c7a151ca2-kube-api-access-c2clv\") pod \"payload-processing-e2e-api-b-82b108-79d7b7dddc-wzqv7\" (UID: \"22eb4983-2f08-41d8-b8b6-562c7a151ca2\") " pod="openshift-ingress/payload-processing-e2e-api-b-82b108-79d7b7dddc-wzqv7"
Nov 03 19:50:41.626692 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:41.623513    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bd7e260-e08c-4413-aea4-eb140c9d8a7e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-82b108-77f564d699-qnmjh\" (UID: \"4bd7e260-e08c-4413-aea4-eb140c9d8a7e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-82b108-77f564d699-qnmjh"
Nov 03 19:50:41.635434 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:41.635395    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87bw6\" (UniqueName: \"kubernetes.io/projected/4bd7e260-e08c-4413-aea4-eb140c9d8a7e-kube-api-access-87bw6\") pod \"payload-pre-processing-e2e-api-b-82b108-77f564d699-qnmjh\" (UID: \"4bd7e260-e08c-4413-aea4-eb140c9d8a7e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-82b108-77f564d699-qnmjh"
Nov 03 19:50:41.837073 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:41.836996    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-82b108-77f564d699-qnmjh"
Nov 03 19:50:41.883662 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:41.883624    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-82b108-79d7b7dddc-wzqv7"
Nov 03 19:50:42.429570 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:42.429538    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-82b108-77f564d699-qnmjh"]
Nov 03 19:50:42.430779 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:50:42.430743    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bd7e260_e08c_4413_aea4_eb140c9d8a7e.slice/crio-941b1dbb208ff10a855338e6e16bcdee567157ccdf8e6bb87e0c5afdbfd60f0b WatchSource:0}: Error finding container 941b1dbb208ff10a855338e6e16bcdee567157ccdf8e6bb87e0c5afdbfd60f0b: Status 404 returned error can't find the container with id 941b1dbb208ff10a855338e6e16bcdee567157ccdf8e6bb87e0c5afdbfd60f0b
Nov 03 19:50:42.439933 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:42.439905    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-82b108-79d7b7dddc-wzqv7"]
Nov 03 19:50:42.861426 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:42.861379    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fa9526-7b98b75779-6gbh9"]
Nov 03 19:50:42.865811 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:42.865790    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fa9526-7b98b75779-6gbh9"
Nov 03 19:50:42.869412 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:42.869389    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-fa9526\""
Nov 03 19:50:42.875554 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:42.875528    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fa9526-7b98b75779-6gbh9"]
Nov 03 19:50:42.890224 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:42.890195    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzgnp\" (UniqueName: \"kubernetes.io/projected/aeb289bd-62bd-45a1-9168-6e0e023e504f-kube-api-access-lzgnp\") pod \"payload-pre-processing-e2e-shared-a-w2-fa9526-7b98b75779-6gbh9\" (UID: \"aeb289bd-62bd-45a1-9168-6e0e023e504f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fa9526-7b98b75779-6gbh9"
Nov 03 19:50:42.890394 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:42.890301    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aeb289bd-62bd-45a1-9168-6e0e023e504f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-fa9526-7b98b75779-6gbh9\" (UID: \"aeb289bd-62bd-45a1-9168-6e0e023e504f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fa9526-7b98b75779-6gbh9"
Nov 03 19:50:42.896358 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:42.896332    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-fa9526-dockercfg-rswql\""
Nov 03 19:50:42.920248 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:42.920219    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-fa9526-55fcd67459-dv9g7"]
Nov 03 19:50:42.925113 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:42.925011    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fa9526-55fcd67459-dv9g7"
Nov 03 19:50:42.932277 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:42.932241    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-fa9526-55fcd67459-dv9g7"]
Nov 03 19:50:43.047692 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:43.014975    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzgnp\" (UniqueName: \"kubernetes.io/projected/aeb289bd-62bd-45a1-9168-6e0e023e504f-kube-api-access-lzgnp\") pod \"payload-pre-processing-e2e-shared-a-w2-fa9526-7b98b75779-6gbh9\" (UID: \"aeb289bd-62bd-45a1-9168-6e0e023e504f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fa9526-7b98b75779-6gbh9"
Nov 03 19:50:43.077504 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:43.077203    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aeb289bd-62bd-45a1-9168-6e0e023e504f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-fa9526-7b98b75779-6gbh9\" (UID: \"aeb289bd-62bd-45a1-9168-6e0e023e504f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fa9526-7b98b75779-6gbh9"
Nov 03 19:50:43.085694 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:43.078299    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aeb289bd-62bd-45a1-9168-6e0e023e504f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-fa9526-7b98b75779-6gbh9\" (UID: \"aeb289bd-62bd-45a1-9168-6e0e023e504f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fa9526-7b98b75779-6gbh9"
Nov 03 19:50:43.156773 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:43.148875    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzgnp\" (UniqueName: \"kubernetes.io/projected/aeb289bd-62bd-45a1-9168-6e0e023e504f-kube-api-access-lzgnp\") pod \"payload-pre-processing-e2e-shared-a-w2-fa9526-7b98b75779-6gbh9\" (UID: \"aeb289bd-62bd-45a1-9168-6e0e023e504f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fa9526-7b98b75779-6gbh9"
Nov 03 19:50:43.184629 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:43.181974    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kc8gv\" (UniqueName: \"kubernetes.io/projected/65b8374e-d983-45f0-8b27-8de1000e1f69-kube-api-access-kc8gv\") pod \"payload-processing-e2e-shared-a-w2-fa9526-55fcd67459-dv9g7\" (UID: \"65b8374e-d983-45f0-8b27-8de1000e1f69\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fa9526-55fcd67459-dv9g7"
Nov 03 19:50:43.184629 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:43.182077    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65b8374e-d983-45f0-8b27-8de1000e1f69-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-fa9526-55fcd67459-dv9g7\" (UID: \"65b8374e-d983-45f0-8b27-8de1000e1f69\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fa9526-55fcd67459-dv9g7"
Nov 03 19:50:43.199421 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:43.190724    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fa9526-7b98b75779-6gbh9"
Nov 03 19:50:43.285822 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:43.283885    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kc8gv\" (UniqueName: \"kubernetes.io/projected/65b8374e-d983-45f0-8b27-8de1000e1f69-kube-api-access-kc8gv\") pod \"payload-processing-e2e-shared-a-w2-fa9526-55fcd67459-dv9g7\" (UID: \"65b8374e-d983-45f0-8b27-8de1000e1f69\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fa9526-55fcd67459-dv9g7"
Nov 03 19:50:43.285822 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:43.283990    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65b8374e-d983-45f0-8b27-8de1000e1f69-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-fa9526-55fcd67459-dv9g7\" (UID: \"65b8374e-d983-45f0-8b27-8de1000e1f69\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fa9526-55fcd67459-dv9g7"
Nov 03 19:50:43.285822 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:43.285763    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65b8374e-d983-45f0-8b27-8de1000e1f69-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-fa9526-55fcd67459-dv9g7\" (UID: \"65b8374e-d983-45f0-8b27-8de1000e1f69\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fa9526-55fcd67459-dv9g7"
Nov 03 19:50:43.310288 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:43.310242    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kc8gv\" (UniqueName: \"kubernetes.io/projected/65b8374e-d983-45f0-8b27-8de1000e1f69-kube-api-access-kc8gv\") pod \"payload-processing-e2e-shared-a-w2-fa9526-55fcd67459-dv9g7\" (UID: \"65b8374e-d983-45f0-8b27-8de1000e1f69\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fa9526-55fcd67459-dv9g7"
Nov 03 19:50:43.397159 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:43.396481    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-82b108-77f564d699-qnmjh" event={"ID":"4bd7e260-e08c-4413-aea4-eb140c9d8a7e","Type":"ContainerStarted","Data":"c910b30c76315fcaadf3b7595412a4e10d164bcb2db0ca161256d7c63450e0c0"}
Nov 03 19:50:43.397159 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:43.396550    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-82b108-77f564d699-qnmjh" event={"ID":"4bd7e260-e08c-4413-aea4-eb140c9d8a7e","Type":"ContainerStarted","Data":"941b1dbb208ff10a855338e6e16bcdee567157ccdf8e6bb87e0c5afdbfd60f0b"}
Nov 03 19:50:43.397159 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:43.396599    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-82b108-77f564d699-qnmjh"
Nov 03 19:50:43.400974 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:43.400045    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-82b108-79d7b7dddc-wzqv7" event={"ID":"22eb4983-2f08-41d8-b8b6-562c7a151ca2","Type":"ContainerStarted","Data":"b41ed6b35979ef776a1171d2d92c3a6a36f989af4a5b6c491feabaa65cebb27f"}
Nov 03 19:50:43.400974 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:43.400100    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-82b108-79d7b7dddc-wzqv7" event={"ID":"22eb4983-2f08-41d8-b8b6-562c7a151ca2","Type":"ContainerStarted","Data":"cc7800701b48982619c6cad1919df17f2d3ed4e63c2b85e7013b96dc7fa7c806"}
Nov 03 19:50:43.400974 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:43.400871    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-82b108-79d7b7dddc-wzqv7"
Nov 03 19:50:43.425653 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:43.425459    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-82b108-77f564d699-qnmjh" podStartSLOduration=2.42543607 podStartE2EDuration="2.42543607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:50:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:50:43.415078211 +0000 UTC m=+1156.659043693" watchObservedRunningTime="2026-11-03 19:50:43.42543607 +0000 UTC m=+1156.669401536"
Nov 03 19:50:43.515288 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:43.515213    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-82b108-79d7b7dddc-wzqv7" podStartSLOduration=2.511660757 podStartE2EDuration="2.511660757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:50:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:50:43.430496203 +0000 UTC m=+1156.674461668" watchObservedRunningTime="2026-11-03 19:50:43.511660757 +0000 UTC m=+1156.755626223"
Nov 03 19:50:43.558429 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:43.558379    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fa9526-55fcd67459-dv9g7"
Nov 03 19:50:43.585869 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:43.585803    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fa9526-7b98b75779-6gbh9"]
Nov 03 19:50:43.920308 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:43.920274    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2b73c810-798bdd757f-8k9lr"]
Nov 03 19:50:43.924926 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:43.924905    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2b73c810-798bdd757f-8k9lr"
Nov 03 19:50:43.928617 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:43.928516    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-2b73c810\""
Nov 03 19:50:43.932091 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:43.932065    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2b73c810-798bdd757f-8k9lr"]
Nov 03 19:50:43.973438 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:43.973398    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9070f15b-8ccc-4ea1-812e-82ea26e0a09e-plugins-config-volume\") pod \"payload-processing-e2e-mt-2b73c810-798bdd757f-8k9lr\" (UID: \"9070f15b-8ccc-4ea1-812e-82ea26e0a09e\") " pod="openshift-ingress/payload-processing-e2e-mt-2b73c810-798bdd757f-8k9lr"
Nov 03 19:50:43.973754 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:43.973731    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-222zl\" (UniqueName: \"kubernetes.io/projected/9070f15b-8ccc-4ea1-812e-82ea26e0a09e-kube-api-access-222zl\") pod \"payload-processing-e2e-mt-2b73c810-798bdd757f-8k9lr\" (UID: \"9070f15b-8ccc-4ea1-812e-82ea26e0a09e\") " pod="openshift-ingress/payload-processing-e2e-mt-2b73c810-798bdd757f-8k9lr"
Nov 03 19:50:44.034211 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:44.025138    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-fa9526-55fcd67459-dv9g7"]
Nov 03 19:50:44.093470 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:44.081356    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-222zl\" (UniqueName: \"kubernetes.io/projected/9070f15b-8ccc-4ea1-812e-82ea26e0a09e-kube-api-access-222zl\") pod \"payload-processing-e2e-mt-2b73c810-798bdd757f-8k9lr\" (UID: \"9070f15b-8ccc-4ea1-812e-82ea26e0a09e\") " pod="openshift-ingress/payload-processing-e2e-mt-2b73c810-798bdd757f-8k9lr"
Nov 03 19:50:44.093470 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:44.081644    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9070f15b-8ccc-4ea1-812e-82ea26e0a09e-plugins-config-volume\") pod \"payload-processing-e2e-mt-2b73c810-798bdd757f-8k9lr\" (UID: \"9070f15b-8ccc-4ea1-812e-82ea26e0a09e\") " pod="openshift-ingress/payload-processing-e2e-mt-2b73c810-798bdd757f-8k9lr"
Nov 03 19:50:44.093470 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:44.082972    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9070f15b-8ccc-4ea1-812e-82ea26e0a09e-plugins-config-volume\") pod \"payload-processing-e2e-mt-2b73c810-798bdd757f-8k9lr\" (UID: \"9070f15b-8ccc-4ea1-812e-82ea26e0a09e\") " pod="openshift-ingress/payload-processing-e2e-mt-2b73c810-798bdd757f-8k9lr"
Nov 03 19:50:44.263767 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:44.199290    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-222zl\" (UniqueName: \"kubernetes.io/projected/9070f15b-8ccc-4ea1-812e-82ea26e0a09e-kube-api-access-222zl\") pod \"payload-processing-e2e-mt-2b73c810-798bdd757f-8k9lr\" (UID: \"9070f15b-8ccc-4ea1-812e-82ea26e0a09e\") " pod="openshift-ingress/payload-processing-e2e-mt-2b73c810-798bdd757f-8k9lr"
Nov 03 19:50:44.263767 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:44.249269    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2b73c810-798bdd757f-8k9lr"
Nov 03 19:50:44.469296 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:44.469224    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fa9526-55fcd67459-dv9g7" event={"ID":"65b8374e-d983-45f0-8b27-8de1000e1f69","Type":"ContainerStarted","Data":"48b0423a5eda1c3ce25ce6b3548273ba145f3a601c3e089ea99c25033bd8444b"}
Nov 03 19:50:44.473251 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:44.473217    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fa9526-7b98b75779-6gbh9" event={"ID":"aeb289bd-62bd-45a1-9168-6e0e023e504f","Type":"ContainerStarted","Data":"035b56bfd3b74216aef011d5f8b4f934d6a4af74989501b17a10d359d7c9b9d5"}
Nov 03 19:50:44.473432 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:44.473416    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fa9526-7b98b75779-6gbh9" event={"ID":"aeb289bd-62bd-45a1-9168-6e0e023e504f","Type":"ContainerStarted","Data":"50f2ff14adaf796c6e189d98446d6e5f4b5f17329f6fbc775f563fd4d45bc7eb"}
Nov 03 19:50:44.474149 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:44.474129    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fa9526-7b98b75779-6gbh9"
Nov 03 19:50:44.489964 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:44.489917    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fa9526-7b98b75779-6gbh9" podStartSLOduration=2.489888636 podStartE2EDuration="2.489888636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:50:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:50:44.486304758 +0000 UTC m=+1157.730270223" watchObservedRunningTime="2026-11-03 19:50:44.489888636 +0000 UTC m=+1157.733854134"
Nov 03 19:50:44.746402 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:50:44.746275    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9070f15b_8ccc_4ea1_812e_82ea26e0a09e.slice/crio-d559c9281c39087c4f982aa0bbfde28e08cc2fc1508456b3206320bba3d55fa0 WatchSource:0}: Error finding container d559c9281c39087c4f982aa0bbfde28e08cc2fc1508456b3206320bba3d55fa0: Status 404 returned error can't find the container with id d559c9281c39087c4f982aa0bbfde28e08cc2fc1508456b3206320bba3d55fa0
Nov 03 19:50:44.747094 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:44.746629    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2b73c810-798bdd757f-8k9lr"]
Nov 03 19:50:45.481317 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:45.481268    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fa9526-55fcd67459-dv9g7" event={"ID":"65b8374e-d983-45f0-8b27-8de1000e1f69","Type":"ContainerStarted","Data":"a86dd6bd6ca4e217007ca74ab46dfa2f9622db6fec2fe417c84f13a0376e47c3"}
Nov 03 19:50:45.481529 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:45.481339    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fa9526-55fcd67459-dv9g7"
Nov 03 19:50:45.483026 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:45.482992    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2b73c810-798bdd757f-8k9lr" event={"ID":"9070f15b-8ccc-4ea1-812e-82ea26e0a09e","Type":"ContainerStarted","Data":"860728971f000259874fe7d11dc3ecbd8122abd67587c03a79a08cedcd491440"}
Nov 03 19:50:45.483154 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:45.483047    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2b73c810-798bdd757f-8k9lr" event={"ID":"9070f15b-8ccc-4ea1-812e-82ea26e0a09e","Type":"ContainerStarted","Data":"d559c9281c39087c4f982aa0bbfde28e08cc2fc1508456b3206320bba3d55fa0"}
Nov 03 19:50:45.483430 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:45.483412    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-2b73c810-798bdd757f-8k9lr"
Nov 03 19:50:45.499107 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:45.499046    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fa9526-55fcd67459-dv9g7" podStartSLOduration=3.499033443 podStartE2EDuration="3.499033443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:50:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:50:45.494660097 +0000 UTC m=+1158.738625559" watchObservedRunningTime="2026-11-03 19:50:45.499033443 +0000 UTC m=+1158.742998908"
Nov 03 19:50:45.513198 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:45.513131    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-2b73c810-798bdd757f-8k9lr" podStartSLOduration=2.513111337 podStartE2EDuration="2.513111337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:50:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:50:45.512045685 +0000 UTC m=+1158.756011151" watchObservedRunningTime="2026-11-03 19:50:45.513111337 +0000 UTC m=+1158.757076803"
Nov 03 19:50:51.929479 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:51.929442    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-82b108-6b74cb9cd9-8hw8h"]
Nov 03 19:50:51.946761 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:51.946731    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-82b108-6b74cb9cd9-8hw8h"
Nov 03 19:50:51.950502 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:51.950472    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-82b108-serving-cert\""
Nov 03 19:50:51.979698 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:51.953213    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-82b108-6b74cb9cd9-8hw8h"]
Nov 03 19:50:52.118684 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:52.118283    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13647f3d-08d3-416d-8424-79840b82e398-openshift-service-ca\") pod \"maas-api-e2e-api-b-82b108-6b74cb9cd9-8hw8h\" (UID: \"13647f3d-08d3-416d-8424-79840b82e398\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-82b108-6b74cb9cd9-8hw8h"
Nov 03 19:50:52.118684 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:52.118388    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xp6k\" (UniqueName: \"kubernetes.io/projected/13647f3d-08d3-416d-8424-79840b82e398-kube-api-access-4xp6k\") pod \"maas-api-e2e-api-b-82b108-6b74cb9cd9-8hw8h\" (UID: \"13647f3d-08d3-416d-8424-79840b82e398\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-82b108-6b74cb9cd9-8hw8h"
Nov 03 19:50:52.118684 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:52.118467    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13647f3d-08d3-416d-8424-79840b82e398-metrics-certs\") pod \"maas-api-e2e-api-b-82b108-6b74cb9cd9-8hw8h\" (UID: \"13647f3d-08d3-416d-8424-79840b82e398\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-82b108-6b74cb9cd9-8hw8h"
Nov 03 19:50:52.119337 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:52.119295    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13647f3d-08d3-416d-8424-79840b82e398-maas-api-tls\") pod \"maas-api-e2e-api-b-82b108-6b74cb9cd9-8hw8h\" (UID: \"13647f3d-08d3-416d-8424-79840b82e398\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-82b108-6b74cb9cd9-8hw8h"
Nov 03 19:50:52.226740 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:52.220923    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13647f3d-08d3-416d-8424-79840b82e398-maas-api-tls\") pod \"maas-api-e2e-api-b-82b108-6b74cb9cd9-8hw8h\" (UID: \"13647f3d-08d3-416d-8424-79840b82e398\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-82b108-6b74cb9cd9-8hw8h"
Nov 03 19:50:52.226740 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:52.221121    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13647f3d-08d3-416d-8424-79840b82e398-openshift-service-ca\") pod \"maas-api-e2e-api-b-82b108-6b74cb9cd9-8hw8h\" (UID: \"13647f3d-08d3-416d-8424-79840b82e398\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-82b108-6b74cb9cd9-8hw8h"
Nov 03 19:50:52.226740 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:52.221197    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xp6k\" (UniqueName: \"kubernetes.io/projected/13647f3d-08d3-416d-8424-79840b82e398-kube-api-access-4xp6k\") pod \"maas-api-e2e-api-b-82b108-6b74cb9cd9-8hw8h\" (UID: \"13647f3d-08d3-416d-8424-79840b82e398\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-82b108-6b74cb9cd9-8hw8h"
Nov 03 19:50:52.226740 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:52.221277    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13647f3d-08d3-416d-8424-79840b82e398-metrics-certs\") pod \"maas-api-e2e-api-b-82b108-6b74cb9cd9-8hw8h\" (UID: \"13647f3d-08d3-416d-8424-79840b82e398\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-82b108-6b74cb9cd9-8hw8h"
Nov 03 19:50:52.234739 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:52.233421    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13647f3d-08d3-416d-8424-79840b82e398-openshift-service-ca\") pod \"maas-api-e2e-api-b-82b108-6b74cb9cd9-8hw8h\" (UID: \"13647f3d-08d3-416d-8424-79840b82e398\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-82b108-6b74cb9cd9-8hw8h"
Nov 03 19:50:52.234739 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:52.234404    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13647f3d-08d3-416d-8424-79840b82e398-metrics-certs\") pod \"maas-api-e2e-api-b-82b108-6b74cb9cd9-8hw8h\" (UID: \"13647f3d-08d3-416d-8424-79840b82e398\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-82b108-6b74cb9cd9-8hw8h"
Nov 03 19:50:52.241685 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:52.240735    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xp6k\" (UniqueName: \"kubernetes.io/projected/13647f3d-08d3-416d-8424-79840b82e398-kube-api-access-4xp6k\") pod \"maas-api-e2e-api-b-82b108-6b74cb9cd9-8hw8h\" (UID: \"13647f3d-08d3-416d-8424-79840b82e398\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-82b108-6b74cb9cd9-8hw8h"
Nov 03 19:50:52.245695 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:52.243135    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13647f3d-08d3-416d-8424-79840b82e398-maas-api-tls\") pod \"maas-api-e2e-api-b-82b108-6b74cb9cd9-8hw8h\" (UID: \"13647f3d-08d3-416d-8424-79840b82e398\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-82b108-6b74cb9cd9-8hw8h"
Nov 03 19:50:52.312147 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:52.312109    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-82b108-6b74cb9cd9-8hw8h"
Nov 03 19:50:52.912974 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:52.905099    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-82b108-6b74cb9cd9-8hw8h"]
Nov 03 19:50:53.757870 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:53.744623    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-82b108-6b74cb9cd9-8hw8h" event={"ID":"13647f3d-08d3-416d-8424-79840b82e398","Type":"ContainerStarted","Data":"ce31cc3c97eec393c304c190821a180d618fc37aa65fe9b68cc25d8a04746b02"}
Nov 03 19:50:53.757870 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:53.744714    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-82b108-6b74cb9cd9-8hw8h" event={"ID":"13647f3d-08d3-416d-8424-79840b82e398","Type":"ContainerStarted","Data":"fe77e8c092f562ed5f3e4406721d7ac6ecfb09d4c56bf5134788e925e2b330d9"}
Nov 03 19:50:53.757870 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:53.745937    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-82b108-6b74cb9cd9-8hw8h"
Nov 03 19:50:53.787705 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:53.779911    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-82b108-6b74cb9cd9-8hw8h" podStartSLOduration=2.224103524 podStartE2EDuration="2.779890106s" totalImagesPullingTime="555.786582ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:50:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:50:53.775138871 +0000 UTC m=+1167.019104339" watchObservedRunningTime="2026-11-03 19:50:53.779890106 +0000 UTC m=+1167.023855572"
Nov 03 19:50:55.507130 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:55.507096    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-82b108-77f564d699-qnmjh"
Nov 03 19:50:55.511214 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:55.509283    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-82b108-79d7b7dddc-wzqv7"
Nov 03 19:50:56.530692 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:56.529907    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fa9526-55fcd67459-dv9g7"
Nov 03 19:50:56.537695 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:56.534830    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-2b73c810-798bdd757f-8k9lr"
Nov 03 19:50:56.544825 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:50:56.543869    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fa9526-7b98b75779-6gbh9"
Nov 03 19:51:00.803001 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:00.802972    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-82b108-6b74cb9cd9-8hw8h"
Nov 03 19:51:02.924252 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:51:02.924213    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 19:51:06.915369 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:06.910576    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1754953c-gkm4f"]
Nov 03 19:51:06.923201 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:06.923165    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1754953c-gkm4f"]
Nov 03 19:51:06.923410 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:06.923269    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1754953c-gkm4f"
Nov 03 19:51:06.927152 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:06.927125    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:51:06.927555 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:06.927529    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 19:51:07.006853 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:07.006812    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c1f13c8-7950-4374-9e7a-48ed6007dc77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1754953c-gkm4f\" (UID: \"6c1f13c8-7950-4374-9e7a-48ed6007dc77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1754953c-gkm4f"
Nov 03 19:51:07.007130 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:07.007106    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c1f13c8-7950-4374-9e7a-48ed6007dc77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1754953c-gkm4f\" (UID: \"6c1f13c8-7950-4374-9e7a-48ed6007dc77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1754953c-gkm4f"
Nov 03 19:51:07.007275 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:07.007257    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqb7j\" (UniqueName: \"kubernetes.io/projected/6c1f13c8-7950-4374-9e7a-48ed6007dc77-kube-api-access-fqb7j\") pod \"maas-api-revoke-sub-1754953c-gkm4f\" (UID: \"6c1f13c8-7950-4374-9e7a-48ed6007dc77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1754953c-gkm4f"
Nov 03 19:51:07.108362 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:07.108320    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c1f13c8-7950-4374-9e7a-48ed6007dc77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1754953c-gkm4f\" (UID: \"6c1f13c8-7950-4374-9e7a-48ed6007dc77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1754953c-gkm4f"
Nov 03 19:51:07.108524 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:07.108378    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c1f13c8-7950-4374-9e7a-48ed6007dc77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1754953c-gkm4f\" (UID: \"6c1f13c8-7950-4374-9e7a-48ed6007dc77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1754953c-gkm4f"
Nov 03 19:51:07.108524 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:07.108464    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqb7j\" (UniqueName: \"kubernetes.io/projected/6c1f13c8-7950-4374-9e7a-48ed6007dc77-kube-api-access-fqb7j\") pod \"maas-api-revoke-sub-1754953c-gkm4f\" (UID: \"6c1f13c8-7950-4374-9e7a-48ed6007dc77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1754953c-gkm4f"
Nov 03 19:51:07.109025 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:07.108996    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c1f13c8-7950-4374-9e7a-48ed6007dc77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1754953c-gkm4f\" (UID: \"6c1f13c8-7950-4374-9e7a-48ed6007dc77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1754953c-gkm4f"
Nov 03 19:51:07.110855 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:07.110833    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c1f13c8-7950-4374-9e7a-48ed6007dc77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1754953c-gkm4f\" (UID: \"6c1f13c8-7950-4374-9e7a-48ed6007dc77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1754953c-gkm4f"
Nov 03 19:51:07.115396 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:07.115376    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqb7j\" (UniqueName: \"kubernetes.io/projected/6c1f13c8-7950-4374-9e7a-48ed6007dc77-kube-api-access-fqb7j\") pod \"maas-api-revoke-sub-1754953c-gkm4f\" (UID: \"6c1f13c8-7950-4374-9e7a-48ed6007dc77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1754953c-gkm4f"
Nov 03 19:51:07.256219 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:07.256183    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1754953c-gkm4f"
Nov 03 19:51:07.547476 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:51:07.547446    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c1f13c8_7950_4374_9e7a_48ed6007dc77.slice/crio-82eddd227c47027a38214e1f5a3d5fc3d4cdf0c296ffa746b407e99935b99336 WatchSource:0}: Error finding container 82eddd227c47027a38214e1f5a3d5fc3d4cdf0c296ffa746b407e99935b99336: Status 404 returned error can't find the container with id 82eddd227c47027a38214e1f5a3d5fc3d4cdf0c296ffa746b407e99935b99336
Nov 03 19:51:07.547959 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:07.547931    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1754953c-gkm4f"]
Nov 03 19:51:08.033385 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:08.033350    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1754953c-gkm4f" event={"ID":"6c1f13c8-7950-4374-9e7a-48ed6007dc77","Type":"ContainerStarted","Data":"f2f5191060062a885cdd5037691537542252c87e4d60e861f7f2edf63f475e58"}
Nov 03 19:51:08.033917 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:08.033402    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1754953c-gkm4f" event={"ID":"6c1f13c8-7950-4374-9e7a-48ed6007dc77","Type":"ContainerStarted","Data":"82eddd227c47027a38214e1f5a3d5fc3d4cdf0c296ffa746b407e99935b99336"}
Nov 03 19:51:08.052028 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:08.051978    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1754953c-gkm4f" podStartSLOduration=2.051964352 podStartE2EDuration="2.051964352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:51:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:51:08.047620863 +0000 UTC m=+1181.291586331" watchObservedRunningTime="2026-11-03 19:51:08.051964352 +0000 UTC m=+1181.295929817"
Nov 03 19:51:09.022452 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:09.022420    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa409566-5lzcd"]
Nov 03 19:51:09.027188 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:09.027162    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa409566-5lzcd"]
Nov 03 19:51:09.046728 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:09.046690    4921 generic.go:356] "Generic (PLEG): container finished" podID="6c1f13c8-7950-4374-9e7a-48ed6007dc77" containerID="f2f5191060062a885cdd5037691537542252c87e4d60e861f7f2edf63f475e58" exitCode=0
Nov 03 19:51:09.047136 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:09.046738    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1754953c-gkm4f" event={"ID":"6c1f13c8-7950-4374-9e7a-48ed6007dc77","Type":"ContainerDied","Data":"f2f5191060062a885cdd5037691537542252c87e4d60e861f7f2edf63f475e58"}
Nov 03 19:51:10.248386 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:10.248356    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1754953c-gkm4f"
Nov 03 19:51:10.297156 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:10.297115    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c1f13c8-7950-4374-9e7a-48ed6007dc77-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c1f13c8-7950-4374-9e7a-48ed6007dc77-maas-api-openshift-ca\") pod \"6c1f13c8-7950-4374-9e7a-48ed6007dc77\" (UID: \"6c1f13c8-7950-4374-9e7a-48ed6007dc77\") "
Nov 03 19:51:10.297363 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:10.297205    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c1f13c8-7950-4374-9e7a-48ed6007dc77-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c1f13c8-7950-4374-9e7a-48ed6007dc77-maas-api-xks-ca\") pod \"6c1f13c8-7950-4374-9e7a-48ed6007dc77\" (UID: \"6c1f13c8-7950-4374-9e7a-48ed6007dc77\") "
Nov 03 19:51:10.297447 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:10.297378    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c1f13c8-7950-4374-9e7a-48ed6007dc77-kube-api-access-fqb7j\" (UniqueName: \"kubernetes.io/projected/6c1f13c8-7950-4374-9e7a-48ed6007dc77-kube-api-access-fqb7j\") pod \"6c1f13c8-7950-4374-9e7a-48ed6007dc77\" (UID: \"6c1f13c8-7950-4374-9e7a-48ed6007dc77\") "
Nov 03 19:51:10.299023 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:10.298996    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c1f13c8-7950-4374-9e7a-48ed6007dc77-maas-api-openshift-ca" pod "6c1f13c8-7950-4374-9e7a-48ed6007dc77" (UID: "6c1f13c8-7950-4374-9e7a-48ed6007dc77"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:51:10.301952 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:10.301910    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c1f13c8-7950-4374-9e7a-48ed6007dc77-maas-api-xks-ca" pod "6c1f13c8-7950-4374-9e7a-48ed6007dc77" (UID: "6c1f13c8-7950-4374-9e7a-48ed6007dc77"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:51:10.304276 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:10.304236    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c1f13c8-7950-4374-9e7a-48ed6007dc77-kube-api-access-fqb7j" pod "6c1f13c8-7950-4374-9e7a-48ed6007dc77" (UID: "6c1f13c8-7950-4374-9e7a-48ed6007dc77"). InnerVolumeSpecName "kube-api-access-fqb7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:51:10.436538 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:10.436456    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqb7j\" (UniqueName: \"kubernetes.io/projected/6c1f13c8-7950-4374-9e7a-48ed6007dc77-kube-api-access-fqb7j\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:10.436538 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:10.436491    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c1f13c8-7950-4374-9e7a-48ed6007dc77-maas-api-openshift-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:10.436538 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:10.436506    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c1f13c8-7950-4374-9e7a-48ed6007dc77-maas-api-xks-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:10.896384 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:10.896347    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7790312d-8886-4f05-a13f-01ad360672da" path="/var/lib/kubelet/pods/7790312d-8886-4f05-a13f-01ad360672da/volumes"
Nov 03 19:51:11.065559 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:11.065527    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1754953c-gkm4f"
Nov 03 19:51:11.065764 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:11.065523    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1754953c-gkm4f" event={"ID":"6c1f13c8-7950-4374-9e7a-48ed6007dc77","Type":"ContainerDied","Data":"82eddd227c47027a38214e1f5a3d5fc3d4cdf0c296ffa746b407e99935b99336"}
Nov 03 19:51:11.065764 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:11.065660    4921 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82eddd227c47027a38214e1f5a3d5fc3d4cdf0c296ffa746b407e99935b99336"
Nov 03 19:51:13.421601 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:13.421549    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3c7998-s7656"]
Nov 03 19:51:13.426692 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:13.423354    4921 cpu_manager.go:414] "containerMap: removing container" podUID="6c1f13c8-7950-4374-9e7a-48ed6007dc77" containerName="revoke-keys"
Nov 03 19:51:13.426924 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:13.426903    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c1f13c8-7950-4374-9e7a-48ed6007dc77" containerName="revoke-keys"
Nov 03 19:51:13.427323 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:13.427309    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c1f13c8-7950-4374-9e7a-48ed6007dc77" containerName="revoke-keys"
Nov 03 19:51:13.444209 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:13.444173    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3c7998-s7656"
Nov 03 19:51:13.452237 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:13.452202    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:51:13.452420 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:13.452337    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 19:51:13.486002 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:13.485930    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3c7998-s7656"]
Nov 03 19:51:13.523237 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:13.523205    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b175a76b-b7e8-4d9e-bda2-226b0267fb97-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3c7998-s7656\" (UID: \"b175a76b-b7e8-4d9e-bda2-226b0267fb97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3c7998-s7656"
Nov 03 19:51:13.523411 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:13.523337    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b175a76b-b7e8-4d9e-bda2-226b0267fb97-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3c7998-s7656\" (UID: \"b175a76b-b7e8-4d9e-bda2-226b0267fb97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3c7998-s7656"
Nov 03 19:51:13.523489 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:13.523464    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrhsd\" (UniqueName: \"kubernetes.io/projected/b175a76b-b7e8-4d9e-bda2-226b0267fb97-kube-api-access-zrhsd\") pod \"maas-api-revoke-keys-e2e-api-a-3c7998-s7656\" (UID: \"b175a76b-b7e8-4d9e-bda2-226b0267fb97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3c7998-s7656"
Nov 03 19:51:13.624873 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:13.624832    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b175a76b-b7e8-4d9e-bda2-226b0267fb97-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3c7998-s7656\" (UID: \"b175a76b-b7e8-4d9e-bda2-226b0267fb97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3c7998-s7656"
Nov 03 19:51:13.625069 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:13.624946    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrhsd\" (UniqueName: \"kubernetes.io/projected/b175a76b-b7e8-4d9e-bda2-226b0267fb97-kube-api-access-zrhsd\") pod \"maas-api-revoke-keys-e2e-api-a-3c7998-s7656\" (UID: \"b175a76b-b7e8-4d9e-bda2-226b0267fb97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3c7998-s7656"
Nov 03 19:51:13.625276 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:13.625250    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b175a76b-b7e8-4d9e-bda2-226b0267fb97-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3c7998-s7656\" (UID: \"b175a76b-b7e8-4d9e-bda2-226b0267fb97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3c7998-s7656"
Nov 03 19:51:13.626130 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:13.626104    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b175a76b-b7e8-4d9e-bda2-226b0267fb97-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3c7998-s7656\" (UID: \"b175a76b-b7e8-4d9e-bda2-226b0267fb97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3c7998-s7656"
Nov 03 19:51:13.627640 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:13.627611    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b175a76b-b7e8-4d9e-bda2-226b0267fb97-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3c7998-s7656\" (UID: \"b175a76b-b7e8-4d9e-bda2-226b0267fb97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3c7998-s7656"
Nov 03 19:51:13.641998 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:13.640656    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrhsd\" (UniqueName: \"kubernetes.io/projected/b175a76b-b7e8-4d9e-bda2-226b0267fb97-kube-api-access-zrhsd\") pod \"maas-api-revoke-keys-e2e-api-a-3c7998-s7656\" (UID: \"b175a76b-b7e8-4d9e-bda2-226b0267fb97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3c7998-s7656"
Nov 03 19:51:13.797700 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:13.796969    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3c7998-s7656"
Nov 03 19:51:14.757823 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:14.754688    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3c7998-s7656"]
Nov 03 19:51:15.149709 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:15.142944    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63a98a9b-bkrfb"]
Nov 03 19:51:15.158691 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:15.153722    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63a98a9b-bkrfb"]
Nov 03 19:51:15.279588 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:15.278449    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3c7998-s7656" event={"ID":"b175a76b-b7e8-4d9e-bda2-226b0267fb97","Type":"ContainerStarted","Data":"2dffe7d79f18f671425ccb93932afad2c7d65a55783a4c09dea20d959947394b"}
Nov 03 19:51:16.290082 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:16.290043    4921 generic.go:356] "Generic (PLEG): container finished" podID="b175a76b-b7e8-4d9e-bda2-226b0267fb97" containerID="a5a2f71b5a27308b852fad607e41cc6e90e17a28ef3c7acdd91323a982db80d0" exitCode=0
Nov 03 19:51:16.290540 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:16.290256    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3c7998-s7656" event={"ID":"b175a76b-b7e8-4d9e-bda2-226b0267fb97","Type":"ContainerDied","Data":"a5a2f71b5a27308b852fad607e41cc6e90e17a28ef3c7acdd91323a982db80d0"}
Nov 03 19:51:16.911693 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:16.911572    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a5b26fb-965e-4220-b649-8968cb6fb371" path="/var/lib/kubelet/pods/0a5b26fb-965e-4220-b649-8968cb6fb371/volumes"
Nov 03 19:51:17.495578 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:17.495545    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3c7998-s7656"
Nov 03 19:51:17.554509 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:17.554409    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b175a76b-b7e8-4d9e-bda2-226b0267fb97-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b175a76b-b7e8-4d9e-bda2-226b0267fb97-maas-api-openshift-ca\") pod \"b175a76b-b7e8-4d9e-bda2-226b0267fb97\" (UID: \"b175a76b-b7e8-4d9e-bda2-226b0267fb97\") "
Nov 03 19:51:17.554735 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:17.554571    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b175a76b-b7e8-4d9e-bda2-226b0267fb97-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b175a76b-b7e8-4d9e-bda2-226b0267fb97-maas-api-xks-ca\") pod \"b175a76b-b7e8-4d9e-bda2-226b0267fb97\" (UID: \"b175a76b-b7e8-4d9e-bda2-226b0267fb97\") "
Nov 03 19:51:17.554824 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:17.554806    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b175a76b-b7e8-4d9e-bda2-226b0267fb97-kube-api-access-zrhsd\" (UniqueName: \"kubernetes.io/projected/b175a76b-b7e8-4d9e-bda2-226b0267fb97-kube-api-access-zrhsd\") pod \"b175a76b-b7e8-4d9e-bda2-226b0267fb97\" (UID: \"b175a76b-b7e8-4d9e-bda2-226b0267fb97\") "
Nov 03 19:51:17.554954 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:17.554925    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b175a76b-b7e8-4d9e-bda2-226b0267fb97-maas-api-openshift-ca" pod "b175a76b-b7e8-4d9e-bda2-226b0267fb97" (UID: "b175a76b-b7e8-4d9e-bda2-226b0267fb97"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:51:17.555876 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:17.555854    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b175a76b-b7e8-4d9e-bda2-226b0267fb97-maas-api-openshift-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:17.557359 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:17.557331    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b175a76b-b7e8-4d9e-bda2-226b0267fb97-maas-api-xks-ca" pod "b175a76b-b7e8-4d9e-bda2-226b0267fb97" (UID: "b175a76b-b7e8-4d9e-bda2-226b0267fb97"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:51:17.557505 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:17.557484    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b175a76b-b7e8-4d9e-bda2-226b0267fb97-kube-api-access-zrhsd" pod "b175a76b-b7e8-4d9e-bda2-226b0267fb97" (UID: "b175a76b-b7e8-4d9e-bda2-226b0267fb97"). InnerVolumeSpecName "kube-api-access-zrhsd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:51:17.657565 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:17.657531    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b175a76b-b7e8-4d9e-bda2-226b0267fb97-maas-api-xks-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:17.657565 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:17.657563    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrhsd\" (UniqueName: \"kubernetes.io/projected/b175a76b-b7e8-4d9e-bda2-226b0267fb97-kube-api-access-zrhsd\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:17.986150 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:17.986113    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-10572d-d8dc7767-9lhq7"]
Nov 03 19:51:17.986868 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:17.986853    4921 cpu_manager.go:414] "containerMap: removing container" podUID="b175a76b-b7e8-4d9e-bda2-226b0267fb97" containerName="revoke-keys"
Nov 03 19:51:17.986932 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:17.986870    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b175a76b-b7e8-4d9e-bda2-226b0267fb97" containerName="revoke-keys"
Nov 03 19:51:17.987027 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:17.987018    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="b175a76b-b7e8-4d9e-bda2-226b0267fb97" containerName="revoke-keys"
Nov 03 19:51:17.990828 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:17.990809    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-10572d-d8dc7767-9lhq7"
Nov 03 19:51:17.996602 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:17.996566    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-10572d\""
Nov 03 19:51:17.996783 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:17.996716    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-10572d-dockercfg-5j499\""
Nov 03 19:51:17.999298 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:17.999268    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-10572d-d8dc7767-9lhq7"]
Nov 03 19:51:18.029239 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:18.029197    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-10572d-5bf85ff6df-7gk9c"]
Nov 03 19:51:18.039759 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:18.039728    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-10572d-5bf85ff6df-7gk9c"
Nov 03 19:51:18.045822 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:18.045795    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-10572d-5bf85ff6df-7gk9c"]
Nov 03 19:51:18.063894 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:18.063856    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48ab742a-e2bf-4751-bb31-a962f0996078-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-10572d-d8dc7767-9lhq7\" (UID: \"48ab742a-e2bf-4751-bb31-a962f0996078\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-10572d-d8dc7767-9lhq7"
Nov 03 19:51:18.064093 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:18.064070    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fg45\" (UniqueName: \"kubernetes.io/projected/48ab742a-e2bf-4751-bb31-a962f0996078-kube-api-access-9fg45\") pod \"payload-pre-processing-e2e-shared-b-w2-10572d-d8dc7767-9lhq7\" (UID: \"48ab742a-e2bf-4751-bb31-a962f0996078\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-10572d-d8dc7767-9lhq7"
Nov 03 19:51:18.174694 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:18.167786    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62b4fa76-8b78-4a67-af03-b126b71aa8c6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-10572d-5bf85ff6df-7gk9c\" (UID: \"62b4fa76-8b78-4a67-af03-b126b71aa8c6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-10572d-5bf85ff6df-7gk9c"
Nov 03 19:51:18.174694 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:18.167898    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48ab742a-e2bf-4751-bb31-a962f0996078-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-10572d-d8dc7767-9lhq7\" (UID: \"48ab742a-e2bf-4751-bb31-a962f0996078\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-10572d-d8dc7767-9lhq7"
Nov 03 19:51:18.174694 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:18.167983    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnxnn\" (UniqueName: \"kubernetes.io/projected/62b4fa76-8b78-4a67-af03-b126b71aa8c6-kube-api-access-pnxnn\") pod \"payload-processing-e2e-shared-b-w2-10572d-5bf85ff6df-7gk9c\" (UID: \"62b4fa76-8b78-4a67-af03-b126b71aa8c6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-10572d-5bf85ff6df-7gk9c"
Nov 03 19:51:18.174694 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:18.168110    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fg45\" (UniqueName: \"kubernetes.io/projected/48ab742a-e2bf-4751-bb31-a962f0996078-kube-api-access-9fg45\") pod \"payload-pre-processing-e2e-shared-b-w2-10572d-d8dc7767-9lhq7\" (UID: \"48ab742a-e2bf-4751-bb31-a962f0996078\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-10572d-d8dc7767-9lhq7"
Nov 03 19:51:18.206724 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:18.205219    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48ab742a-e2bf-4751-bb31-a962f0996078-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-10572d-d8dc7767-9lhq7\" (UID: \"48ab742a-e2bf-4751-bb31-a962f0996078\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-10572d-d8dc7767-9lhq7"
Nov 03 19:51:18.218490 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:18.216741    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fg45\" (UniqueName: \"kubernetes.io/projected/48ab742a-e2bf-4751-bb31-a962f0996078-kube-api-access-9fg45\") pod \"payload-pre-processing-e2e-shared-b-w2-10572d-d8dc7767-9lhq7\" (UID: \"48ab742a-e2bf-4751-bb31-a962f0996078\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-10572d-d8dc7767-9lhq7"
Nov 03 19:51:18.270066 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:18.269973    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62b4fa76-8b78-4a67-af03-b126b71aa8c6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-10572d-5bf85ff6df-7gk9c\" (UID: \"62b4fa76-8b78-4a67-af03-b126b71aa8c6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-10572d-5bf85ff6df-7gk9c"
Nov 03 19:51:18.291693 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:18.282516    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnxnn\" (UniqueName: \"kubernetes.io/projected/62b4fa76-8b78-4a67-af03-b126b71aa8c6-kube-api-access-pnxnn\") pod \"payload-processing-e2e-shared-b-w2-10572d-5bf85ff6df-7gk9c\" (UID: \"62b4fa76-8b78-4a67-af03-b126b71aa8c6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-10572d-5bf85ff6df-7gk9c"
Nov 03 19:51:18.292485 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:18.282153    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62b4fa76-8b78-4a67-af03-b126b71aa8c6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-10572d-5bf85ff6df-7gk9c\" (UID: \"62b4fa76-8b78-4a67-af03-b126b71aa8c6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-10572d-5bf85ff6df-7gk9c"
Nov 03 19:51:18.324822 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:18.324448    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-10572d-d8dc7767-9lhq7"
Nov 03 19:51:18.325759 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:18.325731    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnxnn\" (UniqueName: \"kubernetes.io/projected/62b4fa76-8b78-4a67-af03-b126b71aa8c6-kube-api-access-pnxnn\") pod \"payload-processing-e2e-shared-b-w2-10572d-5bf85ff6df-7gk9c\" (UID: \"62b4fa76-8b78-4a67-af03-b126b71aa8c6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-10572d-5bf85ff6df-7gk9c"
Nov 03 19:51:18.429197 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:18.429159    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-10572d-5bf85ff6df-7gk9c"
Nov 03 19:51:18.446646 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:18.446606    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3c7998-s7656" event={"ID":"b175a76b-b7e8-4d9e-bda2-226b0267fb97","Type":"ContainerDied","Data":"2dffe7d79f18f671425ccb93932afad2c7d65a55783a4c09dea20d959947394b"}
Nov 03 19:51:18.446918 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:18.446900    4921 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2dffe7d79f18f671425ccb93932afad2c7d65a55783a4c09dea20d959947394b"
Nov 03 19:51:18.447025 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:18.446826    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3c7998-s7656"
Nov 03 19:51:18.745912 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:51:18.745882    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48ab742a_e2bf_4751_bb31_a962f0996078.slice/crio-3c6ddf4b2009fcc2e64f80f41fca26e5eb18495120b982230a6cd69eda63b4d4 WatchSource:0}: Error finding container 3c6ddf4b2009fcc2e64f80f41fca26e5eb18495120b982230a6cd69eda63b4d4: Status 404 returned error can't find the container with id 3c6ddf4b2009fcc2e64f80f41fca26e5eb18495120b982230a6cd69eda63b4d4
Nov 03 19:51:18.746367 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:18.745930    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-10572d-d8dc7767-9lhq7"]
Nov 03 19:51:18.764155 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:51:18.764122    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62b4fa76_8b78_4a67_af03_b126b71aa8c6.slice/crio-7b556b00169e06b842730c91b4aeffa03db755dc6eb10da4c63ece1dd4c2984e WatchSource:0}: Error finding container 7b556b00169e06b842730c91b4aeffa03db755dc6eb10da4c63ece1dd4c2984e: Status 404 returned error can't find the container with id 7b556b00169e06b842730c91b4aeffa03db755dc6eb10da4c63ece1dd4c2984e
Nov 03 19:51:18.765904 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:18.765875    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-10572d-5bf85ff6df-7gk9c"]
Nov 03 19:51:19.456398 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:19.456360    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-10572d-5bf85ff6df-7gk9c" event={"ID":"62b4fa76-8b78-4a67-af03-b126b71aa8c6","Type":"ContainerStarted","Data":"6cd08e4a269d89aea539436fa2d28446beca64a5ffdb4bcf0082a5f2fe76930a"}
Nov 03 19:51:19.456595 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:19.456415    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-10572d-5bf85ff6df-7gk9c" event={"ID":"62b4fa76-8b78-4a67-af03-b126b71aa8c6","Type":"ContainerStarted","Data":"7b556b00169e06b842730c91b4aeffa03db755dc6eb10da4c63ece1dd4c2984e"}
Nov 03 19:51:19.456595 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:19.456439    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-10572d-5bf85ff6df-7gk9c"
Nov 03 19:51:19.458216 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:19.458189    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-10572d-d8dc7767-9lhq7" event={"ID":"48ab742a-e2bf-4751-bb31-a962f0996078","Type":"ContainerStarted","Data":"836e703e559eb902672de9d1b2f6659080af370b46882fd8609885f193abe10c"}
Nov 03 19:51:19.458343 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:19.458237    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-10572d-d8dc7767-9lhq7" event={"ID":"48ab742a-e2bf-4751-bb31-a962f0996078","Type":"ContainerStarted","Data":"3c6ddf4b2009fcc2e64f80f41fca26e5eb18495120b982230a6cd69eda63b4d4"}
Nov 03 19:51:19.458343 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:19.458264    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-10572d-d8dc7767-9lhq7"
Nov 03 19:51:19.475413 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:19.475370    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-10572d-5bf85ff6df-7gk9c" podStartSLOduration=1.475357161 podStartE2EDuration="1.475357161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:51:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:51:19.470253158 +0000 UTC m=+1192.714218624" watchObservedRunningTime="2026-11-03 19:51:19.475357161 +0000 UTC m=+1192.719322625"
Nov 03 19:51:19.487091 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:19.487049    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-10572d-d8dc7767-9lhq7" podStartSLOduration=2.487037681 podStartE2EDuration="2.487037681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:51:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:51:19.483393516 +0000 UTC m=+1192.727358985" watchObservedRunningTime="2026-11-03 19:51:19.487037681 +0000 UTC m=+1192.731003146"
Nov 03 19:51:30.496708 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:30.489886    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-10572d-5bf85ff6df-7gk9c"
Nov 03 19:51:30.508692 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:30.503864    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-10572d-d8dc7767-9lhq7"
Nov 03 19:51:31.598073 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:31.598043    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gzf8k_6f215b42-0f09-46a0-bc98-392c0fa909c9/console-operator/2.log"
Nov 03 19:51:31.611830 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:31.611803    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gzf8k_6f215b42-0f09-46a0-bc98-392c0fa909c9/console-operator/2.log"
Nov 03 19:51:32.922648 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:51:32.922609    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 19:51:37.025866 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:37.025783    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4006e3d-wdjqb"]
Nov 03 19:51:37.032357 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:37.032319    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4006e3d-wdjqb"]
Nov 03 19:51:38.691274 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:38.690717    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9850de4-89j2h"]
Nov 03 19:51:38.710773 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:38.700549    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9850de4-89j2h"
Nov 03 19:51:38.750318 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:38.750285    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:51:38.774750 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:38.750364    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9850de4-89j2h"]
Nov 03 19:51:38.774750 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:38.751195    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 19:51:38.784731 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:38.784699    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv4hx\" (UniqueName: \"kubernetes.io/projected/da6eb7e1-0dbb-41cb-8c32-d438b2f05ee7-kube-api-access-pv4hx\") pod \"maas-api-revoke-sub-c9850de4-89j2h\" (UID: \"da6eb7e1-0dbb-41cb-8c32-d438b2f05ee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9850de4-89j2h"
Nov 03 19:51:38.784875 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:38.784755    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da6eb7e1-0dbb-41cb-8c32-d438b2f05ee7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9850de4-89j2h\" (UID: \"da6eb7e1-0dbb-41cb-8c32-d438b2f05ee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9850de4-89j2h"
Nov 03 19:51:38.784922 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:38.784897    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da6eb7e1-0dbb-41cb-8c32-d438b2f05ee7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9850de4-89j2h\" (UID: \"da6eb7e1-0dbb-41cb-8c32-d438b2f05ee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9850de4-89j2h"
Nov 03 19:51:38.886157 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:38.886126    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv4hx\" (UniqueName: \"kubernetes.io/projected/da6eb7e1-0dbb-41cb-8c32-d438b2f05ee7-kube-api-access-pv4hx\") pod \"maas-api-revoke-sub-c9850de4-89j2h\" (UID: \"da6eb7e1-0dbb-41cb-8c32-d438b2f05ee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9850de4-89j2h"
Nov 03 19:51:38.886327 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:38.886179    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da6eb7e1-0dbb-41cb-8c32-d438b2f05ee7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9850de4-89j2h\" (UID: \"da6eb7e1-0dbb-41cb-8c32-d438b2f05ee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9850de4-89j2h"
Nov 03 19:51:38.886327 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:38.886316    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da6eb7e1-0dbb-41cb-8c32-d438b2f05ee7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9850de4-89j2h\" (UID: \"da6eb7e1-0dbb-41cb-8c32-d438b2f05ee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9850de4-89j2h"
Nov 03 19:51:38.887030 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:38.887005    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da6eb7e1-0dbb-41cb-8c32-d438b2f05ee7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9850de4-89j2h\" (UID: \"da6eb7e1-0dbb-41cb-8c32-d438b2f05ee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9850de4-89j2h"
Nov 03 19:51:38.889104 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:38.889079    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da6eb7e1-0dbb-41cb-8c32-d438b2f05ee7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9850de4-89j2h\" (UID: \"da6eb7e1-0dbb-41cb-8c32-d438b2f05ee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9850de4-89j2h"
Nov 03 19:51:38.892252 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:38.892224    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e41d2ecd-31d0-4110-af26-8ca6c3f9a593" path="/var/lib/kubelet/pods/e41d2ecd-31d0-4110-af26-8ca6c3f9a593/volumes"
Nov 03 19:51:38.892821 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:38.892800    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv4hx\" (UniqueName: \"kubernetes.io/projected/da6eb7e1-0dbb-41cb-8c32-d438b2f05ee7-kube-api-access-pv4hx\") pod \"maas-api-revoke-sub-c9850de4-89j2h\" (UID: \"da6eb7e1-0dbb-41cb-8c32-d438b2f05ee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9850de4-89j2h"
Nov 03 19:51:39.067972 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:39.067935    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9850de4-89j2h"
Nov 03 19:51:39.441152 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:39.440391    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9850de4-89j2h"]
Nov 03 19:51:39.466118 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:51:39.466083    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda6eb7e1_0dbb_41cb_8c32_d438b2f05ee7.slice/crio-edf79f38e3a4d0175b27a3736cd21296e8e6e6bf282f152abaa5b54782bb51f5 WatchSource:0}: Error finding container edf79f38e3a4d0175b27a3736cd21296e8e6e6bf282f152abaa5b54782bb51f5: Status 404 returned error can't find the container with id edf79f38e3a4d0175b27a3736cd21296e8e6e6bf282f152abaa5b54782bb51f5
Nov 03 19:51:39.788732 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:39.788685    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9850de4-89j2h" event={"ID":"da6eb7e1-0dbb-41cb-8c32-d438b2f05ee7","Type":"ContainerStarted","Data":"edf79f38e3a4d0175b27a3736cd21296e8e6e6bf282f152abaa5b54782bb51f5"}
Nov 03 19:51:40.799101 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:40.799065    4921 generic.go:356] "Generic (PLEG): container finished" podID="da6eb7e1-0dbb-41cb-8c32-d438b2f05ee7" containerID="c2cc150bffaf5d2cf721dcb285e825f3a531d2381ac52e18af67a02af7c6eb2f" exitCode=0
Nov 03 19:51:40.799565 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:40.799535    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9850de4-89j2h" event={"ID":"da6eb7e1-0dbb-41cb-8c32-d438b2f05ee7","Type":"ContainerDied","Data":"c2cc150bffaf5d2cf721dcb285e825f3a531d2381ac52e18af67a02af7c6eb2f"}
Nov 03 19:51:41.987722 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:41.987690    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9850de4-89j2h"
Nov 03 19:51:42.049158 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:42.049115    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da6eb7e1-0dbb-41cb-8c32-d438b2f05ee7-kube-api-access-pv4hx\" (UniqueName: \"kubernetes.io/projected/da6eb7e1-0dbb-41cb-8c32-d438b2f05ee7-kube-api-access-pv4hx\") pod \"da6eb7e1-0dbb-41cb-8c32-d438b2f05ee7\" (UID: \"da6eb7e1-0dbb-41cb-8c32-d438b2f05ee7\") "
Nov 03 19:51:42.049372 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:42.049354    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da6eb7e1-0dbb-41cb-8c32-d438b2f05ee7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da6eb7e1-0dbb-41cb-8c32-d438b2f05ee7-maas-api-openshift-ca\") pod \"da6eb7e1-0dbb-41cb-8c32-d438b2f05ee7\" (UID: \"da6eb7e1-0dbb-41cb-8c32-d438b2f05ee7\") "
Nov 03 19:51:42.049431 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:42.049397    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da6eb7e1-0dbb-41cb-8c32-d438b2f05ee7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da6eb7e1-0dbb-41cb-8c32-d438b2f05ee7-maas-api-xks-ca\") pod \"da6eb7e1-0dbb-41cb-8c32-d438b2f05ee7\" (UID: \"da6eb7e1-0dbb-41cb-8c32-d438b2f05ee7\") "
Nov 03 19:51:42.049841 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:42.049802    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da6eb7e1-0dbb-41cb-8c32-d438b2f05ee7-maas-api-openshift-ca" pod "da6eb7e1-0dbb-41cb-8c32-d438b2f05ee7" (UID: "da6eb7e1-0dbb-41cb-8c32-d438b2f05ee7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:51:42.050173 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:42.050141    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da6eb7e1-0dbb-41cb-8c32-d438b2f05ee7-maas-api-openshift-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:42.051508 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:42.051485    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da6eb7e1-0dbb-41cb-8c32-d438b2f05ee7-kube-api-access-pv4hx" pod "da6eb7e1-0dbb-41cb-8c32-d438b2f05ee7" (UID: "da6eb7e1-0dbb-41cb-8c32-d438b2f05ee7"). InnerVolumeSpecName "kube-api-access-pv4hx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:51:42.051645 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:42.051622    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da6eb7e1-0dbb-41cb-8c32-d438b2f05ee7-maas-api-xks-ca" pod "da6eb7e1-0dbb-41cb-8c32-d438b2f05ee7" (UID: "da6eb7e1-0dbb-41cb-8c32-d438b2f05ee7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:51:42.152086 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:42.152053    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da6eb7e1-0dbb-41cb-8c32-d438b2f05ee7-maas-api-xks-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:42.152086 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:42.152088    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pv4hx\" (UniqueName: \"kubernetes.io/projected/da6eb7e1-0dbb-41cb-8c32-d438b2f05ee7-kube-api-access-pv4hx\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:42.818321 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:42.818281    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9850de4-89j2h" event={"ID":"da6eb7e1-0dbb-41cb-8c32-d438b2f05ee7","Type":"ContainerDied","Data":"edf79f38e3a4d0175b27a3736cd21296e8e6e6bf282f152abaa5b54782bb51f5"}
Nov 03 19:51:42.818321 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:42.818323    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9850de4-89j2h"
Nov 03 19:51:42.818538 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:42.818341    4921 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="edf79f38e3a4d0175b27a3736cd21296e8e6e6bf282f152abaa5b54782bb51f5"
Nov 03 19:51:49.028345 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:49.028308    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efb8207d-87fjr"]
Nov 03 19:51:49.032976 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:49.032947    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efb8207d-87fjr"]
Nov 03 19:51:50.890375 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:50.890322    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3d6e5c0-7246-4aea-9419-f0f5199c5474" path="/var/lib/kubelet/pods/e3d6e5c0-7246-4aea-9419-f0f5199c5474/volumes"
Nov 03 19:51:52.705295 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:52.705253    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3c7998-b99c77f94-nn9qk"]
Nov 03 19:51:52.705844 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:52.705516    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c7998-b99c77f94-nn9qk" podUID="e12be141-240d-4c12-9b90-900d64537e30" containerName="maas-api" containerID="cri-o://3e0657b5cca9ac115fe99fe9198709bc3df60ef64d9acd12bf0d5c121bf71aca" gracePeriod=30
Nov 03 19:51:53.374087 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:53.374034    4921 generic.go:356] "Generic (PLEG): container finished" podID="e12be141-240d-4c12-9b90-900d64537e30" containerID="3e0657b5cca9ac115fe99fe9198709bc3df60ef64d9acd12bf0d5c121bf71aca" exitCode=0
Nov 03 19:51:53.374707 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:53.374682    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c7998-b99c77f94-nn9qk" event={"ID":"e12be141-240d-4c12-9b90-900d64537e30","Type":"ContainerDied","Data":"3e0657b5cca9ac115fe99fe9198709bc3df60ef64d9acd12bf0d5c121bf71aca"}
Nov 03 19:51:53.441881 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:53.441845    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3c7998-s7656"]
Nov 03 19:51:53.543344 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:53.539747    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3c7998-s7656"]
Nov 03 19:51:53.566944 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:53.566908    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c7998-b99c77f94-nn9qk"
Nov 03 19:51:53.695297 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:53.695208    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e12be141-240d-4c12-9b90-900d64537e30-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e12be141-240d-4c12-9b90-900d64537e30-openshift-service-ca\") pod \"e12be141-240d-4c12-9b90-900d64537e30\" (UID: \"e12be141-240d-4c12-9b90-900d64537e30\") "
Nov 03 19:51:53.695464 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:53.695427    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e12be141-240d-4c12-9b90-900d64537e30-kube-api-access-8dnkl\" (UniqueName: \"kubernetes.io/projected/e12be141-240d-4c12-9b90-900d64537e30-kube-api-access-8dnkl\") pod \"e12be141-240d-4c12-9b90-900d64537e30\" (UID: \"e12be141-240d-4c12-9b90-900d64537e30\") "
Nov 03 19:51:53.695512 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:53.695463    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e12be141-240d-4c12-9b90-900d64537e30-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e12be141-240d-4c12-9b90-900d64537e30-maas-api-tls\") pod \"e12be141-240d-4c12-9b90-900d64537e30\" (UID: \"e12be141-240d-4c12-9b90-900d64537e30\") "
Nov 03 19:51:53.695632 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:53.695612    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e12be141-240d-4c12-9b90-900d64537e30-openshift-service-ca" pod "e12be141-240d-4c12-9b90-900d64537e30" (UID: "e12be141-240d-4c12-9b90-900d64537e30"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:51:53.695695 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:53.695641    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e12be141-240d-4c12-9b90-900d64537e30-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e12be141-240d-4c12-9b90-900d64537e30-metrics-certs\") pod \"e12be141-240d-4c12-9b90-900d64537e30\" (UID: \"e12be141-240d-4c12-9b90-900d64537e30\") "
Nov 03 19:51:53.696302 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:53.696279    4921 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e12be141-240d-4c12-9b90-900d64537e30-openshift-service-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:53.697783 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:53.697758    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e12be141-240d-4c12-9b90-900d64537e30-kube-api-access-8dnkl" pod "e12be141-240d-4c12-9b90-900d64537e30" (UID: "e12be141-240d-4c12-9b90-900d64537e30"). InnerVolumeSpecName "kube-api-access-8dnkl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:51:53.697883 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:53.697803    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e12be141-240d-4c12-9b90-900d64537e30-maas-api-tls" pod "e12be141-240d-4c12-9b90-900d64537e30" (UID: "e12be141-240d-4c12-9b90-900d64537e30"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:51:53.698176 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:53.698157    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e12be141-240d-4c12-9b90-900d64537e30-metrics-certs" pod "e12be141-240d-4c12-9b90-900d64537e30" (UID: "e12be141-240d-4c12-9b90-900d64537e30"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:51:53.771004 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:53.770966    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9f84de-678d8f9dc5-588n8"]
Nov 03 19:51:53.771814 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:53.771794    4921 cpu_manager.go:414] "containerMap: removing container" podUID="e12be141-240d-4c12-9b90-900d64537e30" containerName="maas-api"
Nov 03 19:51:53.771914 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:53.771814    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e12be141-240d-4c12-9b90-900d64537e30" containerName="maas-api"
Nov 03 19:51:53.771914 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:53.771881    4921 cpu_manager.go:414] "containerMap: removing container" podUID="da6eb7e1-0dbb-41cb-8c32-d438b2f05ee7" containerName="revoke-keys"
Nov 03 19:51:53.771914 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:53.771887    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da6eb7e1-0dbb-41cb-8c32-d438b2f05ee7" containerName="revoke-keys"
Nov 03 19:51:53.772019 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:53.771995    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="da6eb7e1-0dbb-41cb-8c32-d438b2f05ee7" containerName="revoke-keys"
Nov 03 19:51:53.772019 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:53.772005    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="e12be141-240d-4c12-9b90-900d64537e30" containerName="maas-api"
Nov 03 19:51:53.776046 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:53.776028    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9f84de-678d8f9dc5-588n8"
Nov 03 19:51:53.779433 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:53.779407    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-9f84de-dockercfg-s5b7q\""
Nov 03 19:51:53.783454 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:53.783430    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9f84de-678d8f9dc5-588n8"]
Nov 03 19:51:53.798070 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:53.798033    4921 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e12be141-240d-4c12-9b90-900d64537e30-metrics-certs\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:53.798218 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:53.798073    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dnkl\" (UniqueName: \"kubernetes.io/projected/e12be141-240d-4c12-9b90-900d64537e30-kube-api-access-8dnkl\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:53.798218 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:53.798090    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e12be141-240d-4c12-9b90-900d64537e30-maas-api-tls\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:53.806138 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:53.806110    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9f84de-867b56cbc5-5qmtd"]
Nov 03 19:51:53.810899 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:53.810877    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9f84de-867b56cbc5-5qmtd"
Nov 03 19:51:53.818379 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:53.818345    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9f84de-867b56cbc5-5qmtd"]
Nov 03 19:51:53.939689 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:53.934434    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn8gf\" (UniqueName: \"kubernetes.io/projected/2ac3398b-799f-4287-abf7-bfca2392e357-kube-api-access-zn8gf\") pod \"payload-pre-processing-e2e-api-a-9f84de-678d8f9dc5-588n8\" (UID: \"2ac3398b-799f-4287-abf7-bfca2392e357\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9f84de-678d8f9dc5-588n8"
Nov 03 19:51:53.939689 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:53.934510    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c547aaa-645d-4f0a-805c-885095934756-plugins-config-volume\") pod \"payload-processing-e2e-api-a-9f84de-867b56cbc5-5qmtd\" (UID: \"3c547aaa-645d-4f0a-805c-885095934756\") " pod="openshift-ingress/payload-processing-e2e-api-a-9f84de-867b56cbc5-5qmtd"
Nov 03 19:51:53.939689 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:53.934696    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9zh7\" (UniqueName: \"kubernetes.io/projected/3c547aaa-645d-4f0a-805c-885095934756-kube-api-access-n9zh7\") pod \"payload-processing-e2e-api-a-9f84de-867b56cbc5-5qmtd\" (UID: \"3c547aaa-645d-4f0a-805c-885095934756\") " pod="openshift-ingress/payload-processing-e2e-api-a-9f84de-867b56cbc5-5qmtd"
Nov 03 19:51:53.939689 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:53.934752    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ac3398b-799f-4287-abf7-bfca2392e357-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-9f84de-678d8f9dc5-588n8\" (UID: \"2ac3398b-799f-4287-abf7-bfca2392e357\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9f84de-678d8f9dc5-588n8"
Nov 03 19:51:53.967739 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:53.964973    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-9f84de\""
Nov 03 19:51:54.048746 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:54.045096    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9zh7\" (UniqueName: \"kubernetes.io/projected/3c547aaa-645d-4f0a-805c-885095934756-kube-api-access-n9zh7\") pod \"payload-processing-e2e-api-a-9f84de-867b56cbc5-5qmtd\" (UID: \"3c547aaa-645d-4f0a-805c-885095934756\") " pod="openshift-ingress/payload-processing-e2e-api-a-9f84de-867b56cbc5-5qmtd"
Nov 03 19:51:54.048746 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:54.045178    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ac3398b-799f-4287-abf7-bfca2392e357-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-9f84de-678d8f9dc5-588n8\" (UID: \"2ac3398b-799f-4287-abf7-bfca2392e357\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9f84de-678d8f9dc5-588n8"
Nov 03 19:51:54.048746 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:54.045358    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn8gf\" (UniqueName: \"kubernetes.io/projected/2ac3398b-799f-4287-abf7-bfca2392e357-kube-api-access-zn8gf\") pod \"payload-pre-processing-e2e-api-a-9f84de-678d8f9dc5-588n8\" (UID: \"2ac3398b-799f-4287-abf7-bfca2392e357\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9f84de-678d8f9dc5-588n8"
Nov 03 19:51:54.048746 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:54.045418    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c547aaa-645d-4f0a-805c-885095934756-plugins-config-volume\") pod \"payload-processing-e2e-api-a-9f84de-867b56cbc5-5qmtd\" (UID: \"3c547aaa-645d-4f0a-805c-885095934756\") " pod="openshift-ingress/payload-processing-e2e-api-a-9f84de-867b56cbc5-5qmtd"
Nov 03 19:51:54.048746 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:54.046533    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c547aaa-645d-4f0a-805c-885095934756-plugins-config-volume\") pod \"payload-processing-e2e-api-a-9f84de-867b56cbc5-5qmtd\" (UID: \"3c547aaa-645d-4f0a-805c-885095934756\") " pod="openshift-ingress/payload-processing-e2e-api-a-9f84de-867b56cbc5-5qmtd"
Nov 03 19:51:54.056691 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:54.053754    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ac3398b-799f-4287-abf7-bfca2392e357-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-9f84de-678d8f9dc5-588n8\" (UID: \"2ac3398b-799f-4287-abf7-bfca2392e357\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9f84de-678d8f9dc5-588n8"
Nov 03 19:51:54.121578 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:54.121543    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn8gf\" (UniqueName: \"kubernetes.io/projected/2ac3398b-799f-4287-abf7-bfca2392e357-kube-api-access-zn8gf\") pod \"payload-pre-processing-e2e-api-a-9f84de-678d8f9dc5-588n8\" (UID: \"2ac3398b-799f-4287-abf7-bfca2392e357\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9f84de-678d8f9dc5-588n8"
Nov 03 19:51:54.135543 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:54.135504    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9zh7\" (UniqueName: \"kubernetes.io/projected/3c547aaa-645d-4f0a-805c-885095934756-kube-api-access-n9zh7\") pod \"payload-processing-e2e-api-a-9f84de-867b56cbc5-5qmtd\" (UID: \"3c547aaa-645d-4f0a-805c-885095934756\") " pod="openshift-ingress/payload-processing-e2e-api-a-9f84de-867b56cbc5-5qmtd"
Nov 03 19:51:54.417184 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:54.417152    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9f84de-678d8f9dc5-588n8"
Nov 03 19:51:54.449429 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:54.434658    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9f84de-867b56cbc5-5qmtd"
Nov 03 19:51:54.602151 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:54.602002    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c7998-b99c77f94-nn9qk"
Nov 03 19:51:54.602151 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:54.602017    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3c7998-b99c77f94-nn9qk" event={"ID":"e12be141-240d-4c12-9b90-900d64537e30","Type":"ContainerDied","Data":"fdd75c8d612feca2987d2cfd092453f3fda1408d1b7693deec66b5eef76c986c"}
Nov 03 19:51:54.602151 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:54.602091    4921 scope.go:122] "RemoveContainer" containerID="3e0657b5cca9ac115fe99fe9198709bc3df60ef64d9acd12bf0d5c121bf71aca"
Nov 03 19:51:54.707422 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:54.706647    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3c7998-b99c77f94-nn9qk"]
Nov 03 19:51:54.707422 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:54.706715    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3c7998-b99c77f94-nn9qk"]
Nov 03 19:51:54.903731 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:54.902910    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b175a76b-b7e8-4d9e-bda2-226b0267fb97" path="/var/lib/kubelet/pods/b175a76b-b7e8-4d9e-bda2-226b0267fb97/volumes"
Nov 03 19:51:54.904199 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:54.903747    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e12be141-240d-4c12-9b90-900d64537e30" path="/var/lib/kubelet/pods/e12be141-240d-4c12-9b90-900d64537e30/volumes"
Nov 03 19:51:54.929130 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:54.928036    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9f84de-678d8f9dc5-588n8"]
Nov 03 19:51:55.035315 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:55.035282    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a536f35b-dqm2d"]
Nov 03 19:51:55.041016 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:55.040986    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a536f35b-dqm2d"
Nov 03 19:51:55.046227 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:55.046197    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a536f35b-dqm2d"]
Nov 03 19:51:55.055070 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:55.055048    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:51:55.057201 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:55.057153    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 19:51:55.203468 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:51:55.203433    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c547aaa_645d_4f0a_805c_885095934756.slice/crio-32f3da5755e60dd3408060a55acfac58884e87e0307ca8a122a76db53e9a77cf WatchSource:0}: Error finding container 32f3da5755e60dd3408060a55acfac58884e87e0307ca8a122a76db53e9a77cf: Status 404 returned error can't find the container with id 32f3da5755e60dd3408060a55acfac58884e87e0307ca8a122a76db53e9a77cf
Nov 03 19:51:55.203616 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:55.203516    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9f84de-867b56cbc5-5qmtd"]
Nov 03 19:51:55.236317 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:55.235748    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qblx5\" (UniqueName: \"kubernetes.io/projected/67014765-bdcd-4e2e-80e9-ecba29891abb-kube-api-access-qblx5\") pod \"maas-api-revoke-keys-e2e-mt-a536f35b-dqm2d\" (UID: \"67014765-bdcd-4e2e-80e9-ecba29891abb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a536f35b-dqm2d"
Nov 03 19:51:55.236317 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:55.235816    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67014765-bdcd-4e2e-80e9-ecba29891abb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-a536f35b-dqm2d\" (UID: \"67014765-bdcd-4e2e-80e9-ecba29891abb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a536f35b-dqm2d"
Nov 03 19:51:55.236317 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:55.236046    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67014765-bdcd-4e2e-80e9-ecba29891abb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-a536f35b-dqm2d\" (UID: \"67014765-bdcd-4e2e-80e9-ecba29891abb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a536f35b-dqm2d"
Nov 03 19:51:55.337510 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:55.337472    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67014765-bdcd-4e2e-80e9-ecba29891abb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-a536f35b-dqm2d\" (UID: \"67014765-bdcd-4e2e-80e9-ecba29891abb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a536f35b-dqm2d"
Nov 03 19:51:55.337737 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:55.337718    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qblx5\" (UniqueName: \"kubernetes.io/projected/67014765-bdcd-4e2e-80e9-ecba29891abb-kube-api-access-qblx5\") pod \"maas-api-revoke-keys-e2e-mt-a536f35b-dqm2d\" (UID: \"67014765-bdcd-4e2e-80e9-ecba29891abb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a536f35b-dqm2d"
Nov 03 19:51:55.337787 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:55.337768    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67014765-bdcd-4e2e-80e9-ecba29891abb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-a536f35b-dqm2d\" (UID: \"67014765-bdcd-4e2e-80e9-ecba29891abb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a536f35b-dqm2d"
Nov 03 19:51:55.338222 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:55.338204    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67014765-bdcd-4e2e-80e9-ecba29891abb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-a536f35b-dqm2d\" (UID: \"67014765-bdcd-4e2e-80e9-ecba29891abb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a536f35b-dqm2d"
Nov 03 19:51:55.340609 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:55.340579    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67014765-bdcd-4e2e-80e9-ecba29891abb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-a536f35b-dqm2d\" (UID: \"67014765-bdcd-4e2e-80e9-ecba29891abb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a536f35b-dqm2d"
Nov 03 19:51:55.345143 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:55.345121    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qblx5\" (UniqueName: \"kubernetes.io/projected/67014765-bdcd-4e2e-80e9-ecba29891abb-kube-api-access-qblx5\") pod \"maas-api-revoke-keys-e2e-mt-a536f35b-dqm2d\" (UID: \"67014765-bdcd-4e2e-80e9-ecba29891abb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a536f35b-dqm2d"
Nov 03 19:51:55.374514 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:55.374481    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a536f35b-dqm2d"
Nov 03 19:51:55.745618 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:55.731349    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9f84de-867b56cbc5-5qmtd" event={"ID":"3c547aaa-645d-4f0a-805c-885095934756","Type":"ContainerStarted","Data":"a954e350ab3968e2b166df995f76fb5694d8c47172867e1ad7efa000390882d4"}
Nov 03 19:51:55.745618 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:55.731448    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9f84de-867b56cbc5-5qmtd" event={"ID":"3c547aaa-645d-4f0a-805c-885095934756","Type":"ContainerStarted","Data":"32f3da5755e60dd3408060a55acfac58884e87e0307ca8a122a76db53e9a77cf"}
Nov 03 19:51:55.745618 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:55.732655    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-9f84de-867b56cbc5-5qmtd"
Nov 03 19:51:55.745618 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:55.736227    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9f84de-678d8f9dc5-588n8" event={"ID":"2ac3398b-799f-4287-abf7-bfca2392e357","Type":"ContainerStarted","Data":"b8b96a02c4351513550fe98a3d7a870c0329f93ad889682207e8050ccc7670d7"}
Nov 03 19:51:55.745618 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:55.736303    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9f84de-678d8f9dc5-588n8" event={"ID":"2ac3398b-799f-4287-abf7-bfca2392e357","Type":"ContainerStarted","Data":"1fd9561fa2a27f39c6e4e84cb492f27ef99d2debe7e3513b8f1f6f8f6196be11"}
Nov 03 19:51:55.745618 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:55.737519    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9f84de-678d8f9dc5-588n8"
Nov 03 19:51:55.773688 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:55.770961    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-9f84de-867b56cbc5-5qmtd" podStartSLOduration=2.770943204 podStartE2EDuration="2.770943204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:51:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:51:55.757444646 +0000 UTC m=+1229.001410112" watchObservedRunningTime="2026-11-03 19:51:55.770943204 +0000 UTC m=+1229.014908681"
Nov 03 19:51:55.989499 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:55.985422    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9f84de-678d8f9dc5-588n8" podStartSLOduration=2.985348633 podStartE2EDuration="2.985348633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:51:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:51:55.781033189 +0000 UTC m=+1229.024998654" watchObservedRunningTime="2026-11-03 19:51:55.985348633 +0000 UTC m=+1229.229314147"
Nov 03 19:51:55.989499 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:55.988751    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a536f35b-dqm2d"]
Nov 03 19:51:55.991883 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:51:55.991840    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67014765_bdcd_4e2e_80e9_ecba29891abb.slice/crio-fd886fa0b6c6c89136531f48d889da4b42d55552b7befab319f7f252c701c5bb WatchSource:0}: Error finding container fd886fa0b6c6c89136531f48d889da4b42d55552b7befab319f7f252c701c5bb: Status 404 returned error can't find the container with id fd886fa0b6c6c89136531f48d889da4b42d55552b7befab319f7f252c701c5bb
Nov 03 19:51:56.758087 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:56.758014    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a536f35b-dqm2d" event={"ID":"67014765-bdcd-4e2e-80e9-ecba29891abb","Type":"ContainerStarted","Data":"f912e347d6458e55ab77c911f137bd6d0f4c88ab21d54c050f613e9845988436"}
Nov 03 19:51:56.758375 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:56.758349    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a536f35b-dqm2d" event={"ID":"67014765-bdcd-4e2e-80e9-ecba29891abb","Type":"ContainerStarted","Data":"fd886fa0b6c6c89136531f48d889da4b42d55552b7befab319f7f252c701c5bb"}
Nov 03 19:51:56.782141 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:56.782052    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a536f35b-dqm2d" podStartSLOduration=1.782028585 podStartE2EDuration="1.782028585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:51:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:51:56.774568918 +0000 UTC m=+1230.018534379" watchObservedRunningTime="2026-11-03 19:51:56.782028585 +0000 UTC m=+1230.025994050"
Nov 03 19:51:57.824734 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:57.823324    4921 generic.go:356] "Generic (PLEG): container finished" podID="67014765-bdcd-4e2e-80e9-ecba29891abb" containerID="f912e347d6458e55ab77c911f137bd6d0f4c88ab21d54c050f613e9845988436" exitCode=0
Nov 03 19:51:57.824734 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:57.824063    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a536f35b-dqm2d" event={"ID":"67014765-bdcd-4e2e-80e9-ecba29891abb","Type":"ContainerDied","Data":"f912e347d6458e55ab77c911f137bd6d0f4c88ab21d54c050f613e9845988436"}
Nov 03 19:51:59.307696 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:59.305873    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a536f35b-dqm2d"
Nov 03 19:51:59.442067 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:59.442033    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67014765-bdcd-4e2e-80e9-ecba29891abb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67014765-bdcd-4e2e-80e9-ecba29891abb-maas-api-openshift-ca\") pod \"67014765-bdcd-4e2e-80e9-ecba29891abb\" (UID: \"67014765-bdcd-4e2e-80e9-ecba29891abb\") "
Nov 03 19:51:59.442236 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:59.442194    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67014765-bdcd-4e2e-80e9-ecba29891abb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67014765-bdcd-4e2e-80e9-ecba29891abb-maas-api-xks-ca\") pod \"67014765-bdcd-4e2e-80e9-ecba29891abb\" (UID: \"67014765-bdcd-4e2e-80e9-ecba29891abb\") "
Nov 03 19:51:59.442429 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:59.442412    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67014765-bdcd-4e2e-80e9-ecba29891abb-kube-api-access-qblx5\" (UniqueName: \"kubernetes.io/projected/67014765-bdcd-4e2e-80e9-ecba29891abb-kube-api-access-qblx5\") pod \"67014765-bdcd-4e2e-80e9-ecba29891abb\" (UID: \"67014765-bdcd-4e2e-80e9-ecba29891abb\") "
Nov 03 19:51:59.442511 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:59.442482    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67014765-bdcd-4e2e-80e9-ecba29891abb-maas-api-openshift-ca" pod "67014765-bdcd-4e2e-80e9-ecba29891abb" (UID: "67014765-bdcd-4e2e-80e9-ecba29891abb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:51:59.443152 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:59.443126    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67014765-bdcd-4e2e-80e9-ecba29891abb-maas-api-openshift-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:59.444501 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:59.444477    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67014765-bdcd-4e2e-80e9-ecba29891abb-maas-api-xks-ca" pod "67014765-bdcd-4e2e-80e9-ecba29891abb" (UID: "67014765-bdcd-4e2e-80e9-ecba29891abb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:51:59.444857 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:59.444832    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67014765-bdcd-4e2e-80e9-ecba29891abb-kube-api-access-qblx5" pod "67014765-bdcd-4e2e-80e9-ecba29891abb" (UID: "67014765-bdcd-4e2e-80e9-ecba29891abb"). InnerVolumeSpecName "kube-api-access-qblx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:51:59.552746 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:59.550364    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67014765-bdcd-4e2e-80e9-ecba29891abb-maas-api-xks-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:59.552746 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:51:59.550403    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qblx5\" (UniqueName: \"kubernetes.io/projected/67014765-bdcd-4e2e-80e9-ecba29891abb-kube-api-access-qblx5\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:00.095700 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:00.093405    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a536f35b-dqm2d" event={"ID":"67014765-bdcd-4e2e-80e9-ecba29891abb","Type":"ContainerDied","Data":"fd886fa0b6c6c89136531f48d889da4b42d55552b7befab319f7f252c701c5bb"}
Nov 03 19:52:00.095700 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:00.093489    4921 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd886fa0b6c6c89136531f48d889da4b42d55552b7befab319f7f252c701c5bb"
Nov 03 19:52:00.095700 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:00.093559    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a536f35b-dqm2d"
Nov 03 19:52:00.247576 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:00.247534    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-82b108-6pjd8"]
Nov 03 19:52:00.248611 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:00.248587    4921 cpu_manager.go:414] "containerMap: removing container" podUID="67014765-bdcd-4e2e-80e9-ecba29891abb" containerName="revoke-keys"
Nov 03 19:52:00.248747 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:00.248613    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67014765-bdcd-4e2e-80e9-ecba29891abb" containerName="revoke-keys"
Nov 03 19:52:00.248844 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:00.248828    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="67014765-bdcd-4e2e-80e9-ecba29891abb" containerName="revoke-keys"
Nov 03 19:52:00.252896 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:00.252875    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-82b108-6pjd8"
Nov 03 19:52:00.258611 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:00.256801    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:52:00.258611 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:00.257439    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 19:52:00.260426 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:00.260398    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-82b108-6pjd8"]
Nov 03 19:52:00.337461 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:00.337410    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vksxj\" (UniqueName: \"kubernetes.io/projected/5f2cd2c7-0827-42a6-9369-a13e8c19bafd-kube-api-access-vksxj\") pod \"maas-api-revoke-keys-e2e-api-b-82b108-6pjd8\" (UID: \"5f2cd2c7-0827-42a6-9369-a13e8c19bafd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-82b108-6pjd8"
Nov 03 19:52:00.337972 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:00.337484    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f2cd2c7-0827-42a6-9369-a13e8c19bafd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-82b108-6pjd8\" (UID: \"5f2cd2c7-0827-42a6-9369-a13e8c19bafd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-82b108-6pjd8"
Nov 03 19:52:00.337972 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:00.337731    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f2cd2c7-0827-42a6-9369-a13e8c19bafd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-82b108-6pjd8\" (UID: \"5f2cd2c7-0827-42a6-9369-a13e8c19bafd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-82b108-6pjd8"
Nov 03 19:52:00.439735 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:00.439625    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vksxj\" (UniqueName: \"kubernetes.io/projected/5f2cd2c7-0827-42a6-9369-a13e8c19bafd-kube-api-access-vksxj\") pod \"maas-api-revoke-keys-e2e-api-b-82b108-6pjd8\" (UID: \"5f2cd2c7-0827-42a6-9369-a13e8c19bafd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-82b108-6pjd8"
Nov 03 19:52:00.439735 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:00.439695    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f2cd2c7-0827-42a6-9369-a13e8c19bafd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-82b108-6pjd8\" (UID: \"5f2cd2c7-0827-42a6-9369-a13e8c19bafd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-82b108-6pjd8"
Nov 03 19:52:00.439956 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:00.439793    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f2cd2c7-0827-42a6-9369-a13e8c19bafd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-82b108-6pjd8\" (UID: \"5f2cd2c7-0827-42a6-9369-a13e8c19bafd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-82b108-6pjd8"
Nov 03 19:52:00.440390 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:00.440370    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f2cd2c7-0827-42a6-9369-a13e8c19bafd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-82b108-6pjd8\" (UID: \"5f2cd2c7-0827-42a6-9369-a13e8c19bafd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-82b108-6pjd8"
Nov 03 19:52:00.442229 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:00.442158    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f2cd2c7-0827-42a6-9369-a13e8c19bafd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-82b108-6pjd8\" (UID: \"5f2cd2c7-0827-42a6-9369-a13e8c19bafd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-82b108-6pjd8"
Nov 03 19:52:00.459834 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:00.452971    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vksxj\" (UniqueName: \"kubernetes.io/projected/5f2cd2c7-0827-42a6-9369-a13e8c19bafd-kube-api-access-vksxj\") pod \"maas-api-revoke-keys-e2e-api-b-82b108-6pjd8\" (UID: \"5f2cd2c7-0827-42a6-9369-a13e8c19bafd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-82b108-6pjd8"
Nov 03 19:52:00.585348 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:00.585308    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-82b108-6pjd8"
Nov 03 19:52:00.879863 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:00.879835    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-82b108-6pjd8"]
Nov 03 19:52:01.103247 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:01.103210    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-82b108-6pjd8" event={"ID":"5f2cd2c7-0827-42a6-9369-a13e8c19bafd","Type":"ContainerStarted","Data":"ec14f9056f3741a1c3215698226ac4b0b7b40d2e289b866e731a20f832aa74df"}
Nov 03 19:52:02.113551 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:02.113512    4921 generic.go:356] "Generic (PLEG): container finished" podID="5f2cd2c7-0827-42a6-9369-a13e8c19bafd" containerID="1e4ab45d04a590f407ac437132822221f2c874b33043ec73c6fcb6de0f96490a" exitCode=0
Nov 03 19:52:02.113960 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:02.113594    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-82b108-6pjd8" event={"ID":"5f2cd2c7-0827-42a6-9369-a13e8c19bafd","Type":"ContainerDied","Data":"1e4ab45d04a590f407ac437132822221f2c874b33043ec73c6fcb6de0f96490a"}
Nov 03 19:52:02.949925 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:52:02.949889    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 19:52:03.388516 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:03.388490    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-82b108-6pjd8"
Nov 03 19:52:03.468783 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:03.468747    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f2cd2c7-0827-42a6-9369-a13e8c19bafd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f2cd2c7-0827-42a6-9369-a13e8c19bafd-maas-api-openshift-ca\") pod \"5f2cd2c7-0827-42a6-9369-a13e8c19bafd\" (UID: \"5f2cd2c7-0827-42a6-9369-a13e8c19bafd\") "
Nov 03 19:52:03.468984 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:03.468862    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f2cd2c7-0827-42a6-9369-a13e8c19bafd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f2cd2c7-0827-42a6-9369-a13e8c19bafd-maas-api-xks-ca\") pod \"5f2cd2c7-0827-42a6-9369-a13e8c19bafd\" (UID: \"5f2cd2c7-0827-42a6-9369-a13e8c19bafd\") "
Nov 03 19:52:03.469064 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:03.468980    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f2cd2c7-0827-42a6-9369-a13e8c19bafd-kube-api-access-vksxj\" (UniqueName: \"kubernetes.io/projected/5f2cd2c7-0827-42a6-9369-a13e8c19bafd-kube-api-access-vksxj\") pod \"5f2cd2c7-0827-42a6-9369-a13e8c19bafd\" (UID: \"5f2cd2c7-0827-42a6-9369-a13e8c19bafd\") "
Nov 03 19:52:03.469179 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:03.469157    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f2cd2c7-0827-42a6-9369-a13e8c19bafd-maas-api-openshift-ca" pod "5f2cd2c7-0827-42a6-9369-a13e8c19bafd" (UID: "5f2cd2c7-0827-42a6-9369-a13e8c19bafd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:52:03.469948 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:03.469920    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f2cd2c7-0827-42a6-9369-a13e8c19bafd-maas-api-openshift-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:03.471295 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:03.471268    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f2cd2c7-0827-42a6-9369-a13e8c19bafd-maas-api-xks-ca" pod "5f2cd2c7-0827-42a6-9369-a13e8c19bafd" (UID: "5f2cd2c7-0827-42a6-9369-a13e8c19bafd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:52:03.471798 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:03.471775    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f2cd2c7-0827-42a6-9369-a13e8c19bafd-kube-api-access-vksxj" pod "5f2cd2c7-0827-42a6-9369-a13e8c19bafd" (UID: "5f2cd2c7-0827-42a6-9369-a13e8c19bafd"). InnerVolumeSpecName "kube-api-access-vksxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:52:03.571462 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:03.571377    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f2cd2c7-0827-42a6-9369-a13e8c19bafd-maas-api-xks-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:03.571462 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:03.571413    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vksxj\" (UniqueName: \"kubernetes.io/projected/5f2cd2c7-0827-42a6-9369-a13e8c19bafd-kube-api-access-vksxj\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:04.134564 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:04.134537    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-82b108-6pjd8"
Nov 03 19:52:04.134778 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:04.134569    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-82b108-6pjd8" event={"ID":"5f2cd2c7-0827-42a6-9369-a13e8c19bafd","Type":"ContainerDied","Data":"ec14f9056f3741a1c3215698226ac4b0b7b40d2e289b866e731a20f832aa74df"}
Nov 03 19:52:04.134778 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:04.134624    4921 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec14f9056f3741a1c3215698226ac4b0b7b40d2e289b866e731a20f832aa74df"
Nov 03 19:52:07.827425 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:07.827395    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-9f84de-867b56cbc5-5qmtd"
Nov 03 19:52:07.827927 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:07.827575    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9f84de-678d8f9dc5-588n8"
Nov 03 19:52:08.771354 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:08.771309    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9f84de-6656857fd9-vr2c8"]
Nov 03 19:52:08.772100 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:08.772084    4921 cpu_manager.go:414] "containerMap: removing container" podUID="5f2cd2c7-0827-42a6-9369-a13e8c19bafd" containerName="revoke-keys"
Nov 03 19:52:08.772171 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:08.772101    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f2cd2c7-0827-42a6-9369-a13e8c19bafd" containerName="revoke-keys"
Nov 03 19:52:08.772270 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:08.772260    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f2cd2c7-0827-42a6-9369-a13e8c19bafd" containerName="revoke-keys"
Nov 03 19:52:08.776367 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:08.776342    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9f84de-6656857fd9-vr2c8"
Nov 03 19:52:08.781255 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:08.780480    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-9f84de-serving-cert\""
Nov 03 19:52:08.787856 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:08.787826    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9f84de-6656857fd9-vr2c8"]
Nov 03 19:52:08.890304 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:08.890271    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b3fa065-548c-4546-9948-c6e8647b1f40-maas-api-tls\") pod \"maas-api-e2e-api-a-9f84de-6656857fd9-vr2c8\" (UID: \"5b3fa065-548c-4546-9948-c6e8647b1f40\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9f84de-6656857fd9-vr2c8"
Nov 03 19:52:08.890727 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:08.890463    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlnb6\" (UniqueName: \"kubernetes.io/projected/5b3fa065-548c-4546-9948-c6e8647b1f40-kube-api-access-zlnb6\") pod \"maas-api-e2e-api-a-9f84de-6656857fd9-vr2c8\" (UID: \"5b3fa065-548c-4546-9948-c6e8647b1f40\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9f84de-6656857fd9-vr2c8"
Nov 03 19:52:08.890727 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:08.890507    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b3fa065-548c-4546-9948-c6e8647b1f40-openshift-service-ca\") pod \"maas-api-e2e-api-a-9f84de-6656857fd9-vr2c8\" (UID: \"5b3fa065-548c-4546-9948-c6e8647b1f40\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9f84de-6656857fd9-vr2c8"
Nov 03 19:52:08.890727 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:08.890570    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b3fa065-548c-4546-9948-c6e8647b1f40-metrics-certs\") pod \"maas-api-e2e-api-a-9f84de-6656857fd9-vr2c8\" (UID: \"5b3fa065-548c-4546-9948-c6e8647b1f40\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9f84de-6656857fd9-vr2c8"
Nov 03 19:52:09.010182 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:09.003862    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b3fa065-548c-4546-9948-c6e8647b1f40-maas-api-tls\") pod \"maas-api-e2e-api-a-9f84de-6656857fd9-vr2c8\" (UID: \"5b3fa065-548c-4546-9948-c6e8647b1f40\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9f84de-6656857fd9-vr2c8"
Nov 03 19:52:09.010182 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:09.004199    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlnb6\" (UniqueName: \"kubernetes.io/projected/5b3fa065-548c-4546-9948-c6e8647b1f40-kube-api-access-zlnb6\") pod \"maas-api-e2e-api-a-9f84de-6656857fd9-vr2c8\" (UID: \"5b3fa065-548c-4546-9948-c6e8647b1f40\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9f84de-6656857fd9-vr2c8"
Nov 03 19:52:09.010182 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:09.004271    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b3fa065-548c-4546-9948-c6e8647b1f40-openshift-service-ca\") pod \"maas-api-e2e-api-a-9f84de-6656857fd9-vr2c8\" (UID: \"5b3fa065-548c-4546-9948-c6e8647b1f40\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9f84de-6656857fd9-vr2c8"
Nov 03 19:52:09.010182 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:09.004306    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b3fa065-548c-4546-9948-c6e8647b1f40-metrics-certs\") pod \"maas-api-e2e-api-a-9f84de-6656857fd9-vr2c8\" (UID: \"5b3fa065-548c-4546-9948-c6e8647b1f40\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9f84de-6656857fd9-vr2c8"
Nov 03 19:52:09.010182 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:09.008881    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b3fa065-548c-4546-9948-c6e8647b1f40-openshift-service-ca\") pod \"maas-api-e2e-api-a-9f84de-6656857fd9-vr2c8\" (UID: \"5b3fa065-548c-4546-9948-c6e8647b1f40\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9f84de-6656857fd9-vr2c8"
Nov 03 19:52:09.018718 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:09.016303    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b3fa065-548c-4546-9948-c6e8647b1f40-metrics-certs\") pod \"maas-api-e2e-api-a-9f84de-6656857fd9-vr2c8\" (UID: \"5b3fa065-548c-4546-9948-c6e8647b1f40\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9f84de-6656857fd9-vr2c8"
Nov 03 19:52:09.031733 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:09.029437    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlnb6\" (UniqueName: \"kubernetes.io/projected/5b3fa065-548c-4546-9948-c6e8647b1f40-kube-api-access-zlnb6\") pod \"maas-api-e2e-api-a-9f84de-6656857fd9-vr2c8\" (UID: \"5b3fa065-548c-4546-9948-c6e8647b1f40\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9f84de-6656857fd9-vr2c8"
Nov 03 19:52:09.054694 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:09.048996    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b3fa065-548c-4546-9948-c6e8647b1f40-maas-api-tls\") pod \"maas-api-e2e-api-a-9f84de-6656857fd9-vr2c8\" (UID: \"5b3fa065-548c-4546-9948-c6e8647b1f40\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9f84de-6656857fd9-vr2c8"
Nov 03 19:52:09.104615 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:09.104580    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9f84de-6656857fd9-vr2c8"
Nov 03 19:52:09.710698 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:09.710547    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9f84de-6656857fd9-vr2c8"]
Nov 03 19:52:09.735315 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:52:09.711038    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b3fa065_548c_4546_9948_c6e8647b1f40.slice/crio-5c49517716959e73919efdeda02b8ec938568578aef4455fae9a746e5d045e9c WatchSource:0}: Error finding container 5c49517716959e73919efdeda02b8ec938568578aef4455fae9a746e5d045e9c: Status 404 returned error can't find the container with id 5c49517716959e73919efdeda02b8ec938568578aef4455fae9a746e5d045e9c
Nov 03 19:52:10.289704 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:10.288389    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9f84de-6656857fd9-vr2c8" event={"ID":"5b3fa065-548c-4546-9948-c6e8647b1f40","Type":"ContainerStarted","Data":"5c49517716959e73919efdeda02b8ec938568578aef4455fae9a746e5d045e9c"}
Nov 03 19:52:10.549029 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:10.548945    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-82b108-79d7b7dddc-wzqv7"]
Nov 03 19:52:10.564068 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:10.564024    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-82b108-77f564d699-qnmjh"]
Nov 03 19:52:11.298359 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:11.298324    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9f84de-6656857fd9-vr2c8" event={"ID":"5b3fa065-548c-4546-9948-c6e8647b1f40","Type":"ContainerStarted","Data":"ef1c9faa47aac6ce2cf8f3364ed1a0d2073eb93d61ebbd4adc19293a4790e0bf"}
Nov 03 19:52:11.298852 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:11.298420    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9f84de-6656857fd9-vr2c8"
Nov 03 19:52:11.316956 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:11.316904    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9f84de-6656857fd9-vr2c8" podStartSLOduration=2.74684784 podStartE2EDuration="3.316888643s" totalImagesPullingTime="570.040803ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:52:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:52:11.312409298 +0000 UTC m=+1244.556374763" watchObservedRunningTime="2026-11-03 19:52:11.316888643 +0000 UTC m=+1244.560854107"
Nov 03 19:52:15.549537 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:15.549494    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-82b108-79d7b7dddc-wzqv7" podUID="22eb4983-2f08-41d8-b8b6-562c7a151ca2" containerName="payload-processing" containerID="cri-o://b41ed6b35979ef776a1171d2d92c3a6a36f989af4a5b6c491feabaa65cebb27f" gracePeriod=25
Nov 03 19:52:15.565125 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:15.565093    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-82b108-77f564d699-qnmjh" podUID="4bd7e260-e08c-4413-aea4-eb140c9d8a7e" containerName="payload-pre-processing" containerID="cri-o://c910b30c76315fcaadf3b7595412a4e10d164bcb2db0ca161256d7c63450e0c0" gracePeriod=25
Nov 03 19:52:15.880985 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:15.880944    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-82b108-79d7b7dddc-wzqv7"
Nov 03 19:52:15.918791 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:15.918763    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-82b108-77f564d699-qnmjh"
Nov 03 19:52:16.033993 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:16.033955    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bd7e260-e08c-4413-aea4-eb140c9d8a7e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bd7e260-e08c-4413-aea4-eb140c9d8a7e-plugins-config-volume\") pod \"4bd7e260-e08c-4413-aea4-eb140c9d8a7e\" (UID: \"4bd7e260-e08c-4413-aea4-eb140c9d8a7e\") "
Nov 03 19:52:16.034189 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:16.034033    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22eb4983-2f08-41d8-b8b6-562c7a151ca2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22eb4983-2f08-41d8-b8b6-562c7a151ca2-plugins-config-volume\") pod \"22eb4983-2f08-41d8-b8b6-562c7a151ca2\" (UID: \"22eb4983-2f08-41d8-b8b6-562c7a151ca2\") "
Nov 03 19:52:16.034250 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:16.034227    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bd7e260-e08c-4413-aea4-eb140c9d8a7e-kube-api-access-87bw6\" (UniqueName: \"kubernetes.io/projected/4bd7e260-e08c-4413-aea4-eb140c9d8a7e-kube-api-access-87bw6\") pod \"4bd7e260-e08c-4413-aea4-eb140c9d8a7e\" (UID: \"4bd7e260-e08c-4413-aea4-eb140c9d8a7e\") "
Nov 03 19:52:16.034300 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:16.034260    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22eb4983-2f08-41d8-b8b6-562c7a151ca2-kube-api-access-c2clv\" (UniqueName: \"kubernetes.io/projected/22eb4983-2f08-41d8-b8b6-562c7a151ca2-kube-api-access-c2clv\") pod \"22eb4983-2f08-41d8-b8b6-562c7a151ca2\" (UID: \"22eb4983-2f08-41d8-b8b6-562c7a151ca2\") "
Nov 03 19:52:16.034510 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:16.034487    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bd7e260-e08c-4413-aea4-eb140c9d8a7e-plugins-config-volume" pod "4bd7e260-e08c-4413-aea4-eb140c9d8a7e" (UID: "4bd7e260-e08c-4413-aea4-eb140c9d8a7e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:52:16.035018 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:16.034997    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22eb4983-2f08-41d8-b8b6-562c7a151ca2-plugins-config-volume" pod "22eb4983-2f08-41d8-b8b6-562c7a151ca2" (UID: "22eb4983-2f08-41d8-b8b6-562c7a151ca2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:52:16.035136 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:16.035119    4921 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bd7e260-e08c-4413-aea4-eb140c9d8a7e-plugins-config-volume\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:16.035210 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:16.035141    4921 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22eb4983-2f08-41d8-b8b6-562c7a151ca2-plugins-config-volume\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:16.040279 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:16.040249    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bd7e260-e08c-4413-aea4-eb140c9d8a7e-kube-api-access-87bw6" pod "4bd7e260-e08c-4413-aea4-eb140c9d8a7e" (UID: "4bd7e260-e08c-4413-aea4-eb140c9d8a7e"). InnerVolumeSpecName "kube-api-access-87bw6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:52:16.042588 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:16.042413    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22eb4983-2f08-41d8-b8b6-562c7a151ca2-kube-api-access-c2clv" pod "22eb4983-2f08-41d8-b8b6-562c7a151ca2" (UID: "22eb4983-2f08-41d8-b8b6-562c7a151ca2"). InnerVolumeSpecName "kube-api-access-c2clv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:52:16.137224 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:16.137123    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87bw6\" (UniqueName: \"kubernetes.io/projected/4bd7e260-e08c-4413-aea4-eb140c9d8a7e-kube-api-access-87bw6\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:16.137224 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:16.137163    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2clv\" (UniqueName: \"kubernetes.io/projected/22eb4983-2f08-41d8-b8b6-562c7a151ca2-kube-api-access-c2clv\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:16.346153 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:16.346123    4921 generic.go:356] "Generic (PLEG): container finished" podID="4bd7e260-e08c-4413-aea4-eb140c9d8a7e" containerID="c910b30c76315fcaadf3b7595412a4e10d164bcb2db0ca161256d7c63450e0c0" exitCode=0
Nov 03 19:52:16.346302 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:16.346170    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-82b108-77f564d699-qnmjh"
Nov 03 19:52:16.346302 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:16.346210    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-82b108-77f564d699-qnmjh" event={"ID":"4bd7e260-e08c-4413-aea4-eb140c9d8a7e","Type":"ContainerDied","Data":"c910b30c76315fcaadf3b7595412a4e10d164bcb2db0ca161256d7c63450e0c0"}
Nov 03 19:52:16.346302 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:16.346264    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-82b108-77f564d699-qnmjh" event={"ID":"4bd7e260-e08c-4413-aea4-eb140c9d8a7e","Type":"ContainerDied","Data":"941b1dbb208ff10a855338e6e16bcdee567157ccdf8e6bb87e0c5afdbfd60f0b"}
Nov 03 19:52:16.346302 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:16.346289    4921 scope.go:122] "RemoveContainer" containerID="c910b30c76315fcaadf3b7595412a4e10d164bcb2db0ca161256d7c63450e0c0"
Nov 03 19:52:16.347949 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:16.347927    4921 generic.go:356] "Generic (PLEG): container finished" podID="22eb4983-2f08-41d8-b8b6-562c7a151ca2" containerID="b41ed6b35979ef776a1171d2d92c3a6a36f989af4a5b6c491feabaa65cebb27f" exitCode=0
Nov 03 19:52:16.348067 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:16.347968    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-82b108-79d7b7dddc-wzqv7"
Nov 03 19:52:16.348067 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:16.348006    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-82b108-79d7b7dddc-wzqv7" event={"ID":"22eb4983-2f08-41d8-b8b6-562c7a151ca2","Type":"ContainerDied","Data":"b41ed6b35979ef776a1171d2d92c3a6a36f989af4a5b6c491feabaa65cebb27f"}
Nov 03 19:52:16.348067 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:16.348049    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-82b108-79d7b7dddc-wzqv7" event={"ID":"22eb4983-2f08-41d8-b8b6-562c7a151ca2","Type":"ContainerDied","Data":"cc7800701b48982619c6cad1919df17f2d3ed4e63c2b85e7013b96dc7fa7c806"}
Nov 03 19:52:16.358615 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:16.358600    4921 scope.go:122] "RemoveContainer" containerID="c910b30c76315fcaadf3b7595412a4e10d164bcb2db0ca161256d7c63450e0c0"
Nov 03 19:52:16.358923 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:52:16.358906    4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c910b30c76315fcaadf3b7595412a4e10d164bcb2db0ca161256d7c63450e0c0\": container with ID starting with c910b30c76315fcaadf3b7595412a4e10d164bcb2db0ca161256d7c63450e0c0 not found: ID does not exist" containerID="c910b30c76315fcaadf3b7595412a4e10d164bcb2db0ca161256d7c63450e0c0"
Nov 03 19:52:16.358983 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:16.358930    4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c910b30c76315fcaadf3b7595412a4e10d164bcb2db0ca161256d7c63450e0c0"} err="failed to get container status \"c910b30c76315fcaadf3b7595412a4e10d164bcb2db0ca161256d7c63450e0c0\": rpc error: code = NotFound desc = could not find container \"c910b30c76315fcaadf3b7595412a4e10d164bcb2db0ca161256d7c63450e0c0\": container with ID starting with c910b30c76315fcaadf3b7595412a4e10d164bcb2db0ca161256d7c63450e0c0 not found: ID does not exist"
Nov 03 19:52:16.358983 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:16.358947    4921 scope.go:122] "RemoveContainer" containerID="b41ed6b35979ef776a1171d2d92c3a6a36f989af4a5b6c491feabaa65cebb27f"
Nov 03 19:52:16.372269 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:16.372242    4921 scope.go:122] "RemoveContainer" containerID="b41ed6b35979ef776a1171d2d92c3a6a36f989af4a5b6c491feabaa65cebb27f"
Nov 03 19:52:16.372656 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:52:16.372633    4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b41ed6b35979ef776a1171d2d92c3a6a36f989af4a5b6c491feabaa65cebb27f\": container with ID starting with b41ed6b35979ef776a1171d2d92c3a6a36f989af4a5b6c491feabaa65cebb27f not found: ID does not exist" containerID="b41ed6b35979ef776a1171d2d92c3a6a36f989af4a5b6c491feabaa65cebb27f"
Nov 03 19:52:16.372754 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:16.372681    4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b41ed6b35979ef776a1171d2d92c3a6a36f989af4a5b6c491feabaa65cebb27f"} err="failed to get container status \"b41ed6b35979ef776a1171d2d92c3a6a36f989af4a5b6c491feabaa65cebb27f\": rpc error: code = NotFound desc = could not find container \"b41ed6b35979ef776a1171d2d92c3a6a36f989af4a5b6c491feabaa65cebb27f\": container with ID starting with b41ed6b35979ef776a1171d2d92c3a6a36f989af4a5b6c491feabaa65cebb27f not found: ID does not exist"
Nov 03 19:52:16.373782 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:16.373766    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-82b108-77f564d699-qnmjh"]
Nov 03 19:52:16.378816 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:16.378781    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-82b108-77f564d699-qnmjh"]
Nov 03 19:52:16.383756 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:16.383733    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-82b108-79d7b7dddc-wzqv7"]
Nov 03 19:52:16.387629 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:16.387581    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-82b108-79d7b7dddc-wzqv7"]
Nov 03 19:52:16.893231 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:16.893185    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22eb4983-2f08-41d8-b8b6-562c7a151ca2" path="/var/lib/kubelet/pods/22eb4983-2f08-41d8-b8b6-562c7a151ca2/volumes"
Nov 03 19:52:16.894019 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:16.893996    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bd7e260-e08c-4413-aea4-eb140c9d8a7e" path="/var/lib/kubelet/pods/4bd7e260-e08c-4413-aea4-eb140c9d8a7e/volumes"
Nov 03 19:52:17.353608 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:17.353571    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9f84de-6656857fd9-vr2c8"
Nov 03 19:52:17.781601 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:17.781570    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db72c9bb-sdzcv"]
Nov 03 19:52:17.782340 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:17.782321    4921 cpu_manager.go:414] "containerMap: removing container" podUID="22eb4983-2f08-41d8-b8b6-562c7a151ca2" containerName="payload-processing"
Nov 03 19:52:17.782404 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:17.782341    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22eb4983-2f08-41d8-b8b6-562c7a151ca2" containerName="payload-processing"
Nov 03 19:52:17.782404 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:17.782364    4921 cpu_manager.go:414] "containerMap: removing container" podUID="4bd7e260-e08c-4413-aea4-eb140c9d8a7e" containerName="payload-pre-processing"
Nov 03 19:52:17.782404 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:17.782370    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bd7e260-e08c-4413-aea4-eb140c9d8a7e" containerName="payload-pre-processing"
Nov 03 19:52:17.782532 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:17.782522    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="22eb4983-2f08-41d8-b8b6-562c7a151ca2" containerName="payload-processing"
Nov 03 19:52:17.782566 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:17.782536    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bd7e260-e08c-4413-aea4-eb140c9d8a7e" containerName="payload-pre-processing"
Nov 03 19:52:17.787966 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:17.787941    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db72c9bb-sdzcv"
Nov 03 19:52:17.794712 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:17.791598    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 19:52:17.794712 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:17.791790    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:52:17.795054 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:17.795033    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db72c9bb-sdzcv"]
Nov 03 19:52:17.937586 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:17.937545    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/758e8ecf-b0c0-4133-889f-6aa948dc7a89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db72c9bb-sdzcv\" (UID: \"758e8ecf-b0c0-4133-889f-6aa948dc7a89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db72c9bb-sdzcv"
Nov 03 19:52:17.938008 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:17.937622    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hbq7\" (UniqueName: \"kubernetes.io/projected/758e8ecf-b0c0-4133-889f-6aa948dc7a89-kube-api-access-8hbq7\") pod \"maas-api-revoke-sub-db72c9bb-sdzcv\" (UID: \"758e8ecf-b0c0-4133-889f-6aa948dc7a89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db72c9bb-sdzcv"
Nov 03 19:52:17.938008 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:17.937796    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/758e8ecf-b0c0-4133-889f-6aa948dc7a89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db72c9bb-sdzcv\" (UID: \"758e8ecf-b0c0-4133-889f-6aa948dc7a89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db72c9bb-sdzcv"
Nov 03 19:52:18.039606 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:18.039516    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/758e8ecf-b0c0-4133-889f-6aa948dc7a89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db72c9bb-sdzcv\" (UID: \"758e8ecf-b0c0-4133-889f-6aa948dc7a89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db72c9bb-sdzcv"
Nov 03 19:52:18.039606 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:18.039568    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hbq7\" (UniqueName: \"kubernetes.io/projected/758e8ecf-b0c0-4133-889f-6aa948dc7a89-kube-api-access-8hbq7\") pod \"maas-api-revoke-sub-db72c9bb-sdzcv\" (UID: \"758e8ecf-b0c0-4133-889f-6aa948dc7a89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db72c9bb-sdzcv"
Nov 03 19:52:18.039890 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:18.039641    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/758e8ecf-b0c0-4133-889f-6aa948dc7a89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db72c9bb-sdzcv\" (UID: \"758e8ecf-b0c0-4133-889f-6aa948dc7a89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db72c9bb-sdzcv"
Nov 03 19:52:18.040340 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:18.040314    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/758e8ecf-b0c0-4133-889f-6aa948dc7a89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db72c9bb-sdzcv\" (UID: \"758e8ecf-b0c0-4133-889f-6aa948dc7a89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db72c9bb-sdzcv"
Nov 03 19:52:18.042143 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:18.042123    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/758e8ecf-b0c0-4133-889f-6aa948dc7a89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db72c9bb-sdzcv\" (UID: \"758e8ecf-b0c0-4133-889f-6aa948dc7a89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db72c9bb-sdzcv"
Nov 03 19:52:18.047008 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:18.046988    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hbq7\" (UniqueName: \"kubernetes.io/projected/758e8ecf-b0c0-4133-889f-6aa948dc7a89-kube-api-access-8hbq7\") pod \"maas-api-revoke-sub-db72c9bb-sdzcv\" (UID: \"758e8ecf-b0c0-4133-889f-6aa948dc7a89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db72c9bb-sdzcv"
Nov 03 19:52:18.108660 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:18.108629    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db72c9bb-sdzcv"
Nov 03 19:52:18.485000 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:52:18.484971    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod758e8ecf_b0c0_4133_889f_6aa948dc7a89.slice/crio-10213b590d5f9c15491e3e89cfc6fe4c4ea62bd6a140ddb7b5564627a8dd0257 WatchSource:0}: Error finding container 10213b590d5f9c15491e3e89cfc6fe4c4ea62bd6a140ddb7b5564627a8dd0257: Status 404 returned error can't find the container with id 10213b590d5f9c15491e3e89cfc6fe4c4ea62bd6a140ddb7b5564627a8dd0257
Nov 03 19:52:18.486177 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:18.486154    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db72c9bb-sdzcv"]
Nov 03 19:52:18.668007 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:18.667980    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2"]
Nov 03 19:52:18.668234 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:18.668209    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2" podUID="476b08ee-e24d-4f0c-a5ae-5cb23d1fa105" containerName="istio-proxy" containerID="cri-o://63ed9099e84a399a347bdf1a5a4b7f19d524546abb5eef5a2e43ab00455c0e23" gracePeriod=30
Nov 03 19:52:19.482320 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:19.482291    4921 generic.go:356] "Generic (PLEG): container finished" podID="758e8ecf-b0c0-4133-889f-6aa948dc7a89" containerID="1e0fdf2b7b3fc4d1f41ced9bb3d946cc8a3d6a230eec30b852daa3607a02a10a" exitCode=0
Nov 03 19:52:19.482695 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:19.482372    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db72c9bb-sdzcv" event={"ID":"758e8ecf-b0c0-4133-889f-6aa948dc7a89","Type":"ContainerDied","Data":"1e0fdf2b7b3fc4d1f41ced9bb3d946cc8a3d6a230eec30b852daa3607a02a10a"}
Nov 03 19:52:19.482695 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:19.482432    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db72c9bb-sdzcv" event={"ID":"758e8ecf-b0c0-4133-889f-6aa948dc7a89","Type":"ContainerStarted","Data":"10213b590d5f9c15491e3e89cfc6fe4c4ea62bd6a140ddb7b5564627a8dd0257"}
Nov 03 19:52:20.947900 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:20.926016    4921 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 19:52:20.947900 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:20.926074    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2" podUID="476b08ee-e24d-4f0c-a5ae-5cb23d1fa105" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 19:52:21.239419 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:21.239088    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db72c9bb-sdzcv"
Nov 03 19:52:21.287897 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:21.286574    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/758e8ecf-b0c0-4133-889f-6aa948dc7a89-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/758e8ecf-b0c0-4133-889f-6aa948dc7a89-maas-api-openshift-ca\") pod \"758e8ecf-b0c0-4133-889f-6aa948dc7a89\" (UID: \"758e8ecf-b0c0-4133-889f-6aa948dc7a89\") "
Nov 03 19:52:21.287897 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:21.286637    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/758e8ecf-b0c0-4133-889f-6aa948dc7a89-kube-api-access-8hbq7\" (UniqueName: \"kubernetes.io/projected/758e8ecf-b0c0-4133-889f-6aa948dc7a89-kube-api-access-8hbq7\") pod \"758e8ecf-b0c0-4133-889f-6aa948dc7a89\" (UID: \"758e8ecf-b0c0-4133-889f-6aa948dc7a89\") "
Nov 03 19:52:21.287897 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:21.286811    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/758e8ecf-b0c0-4133-889f-6aa948dc7a89-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/758e8ecf-b0c0-4133-889f-6aa948dc7a89-maas-api-xks-ca\") pod \"758e8ecf-b0c0-4133-889f-6aa948dc7a89\" (UID: \"758e8ecf-b0c0-4133-889f-6aa948dc7a89\") "
Nov 03 19:52:21.312717 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:21.309146    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/758e8ecf-b0c0-4133-889f-6aa948dc7a89-maas-api-openshift-ca" pod "758e8ecf-b0c0-4133-889f-6aa948dc7a89" (UID: "758e8ecf-b0c0-4133-889f-6aa948dc7a89"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:52:21.342716 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:21.342621    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/758e8ecf-b0c0-4133-889f-6aa948dc7a89-maas-api-xks-ca" pod "758e8ecf-b0c0-4133-889f-6aa948dc7a89" (UID: "758e8ecf-b0c0-4133-889f-6aa948dc7a89"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:52:21.343018 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:21.342900    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/758e8ecf-b0c0-4133-889f-6aa948dc7a89-kube-api-access-8hbq7" pod "758e8ecf-b0c0-4133-889f-6aa948dc7a89" (UID: "758e8ecf-b0c0-4133-889f-6aa948dc7a89"). InnerVolumeSpecName "kube-api-access-8hbq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:52:21.416692 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:21.406085    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/758e8ecf-b0c0-4133-889f-6aa948dc7a89-maas-api-openshift-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:21.416692 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:21.406131    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hbq7\" (UniqueName: \"kubernetes.io/projected/758e8ecf-b0c0-4133-889f-6aa948dc7a89-kube-api-access-8hbq7\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:21.416692 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:21.406149    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/758e8ecf-b0c0-4133-889f-6aa948dc7a89-maas-api-xks-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:21.535558 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:21.535518    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db72c9bb-sdzcv" event={"ID":"758e8ecf-b0c0-4133-889f-6aa948dc7a89","Type":"ContainerDied","Data":"10213b590d5f9c15491e3e89cfc6fe4c4ea62bd6a140ddb7b5564627a8dd0257"}
Nov 03 19:52:21.535744 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:21.535579    4921 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10213b590d5f9c15491e3e89cfc6fe4c4ea62bd6a140ddb7b5564627a8dd0257"
Nov 03 19:52:21.535744 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:21.535645    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db72c9bb-sdzcv"
Nov 03 19:52:24.149441 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:24.149412    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2"
Nov 03 19:52:24.258438 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:24.258401    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/476b08ee-e24d-4f0c-a5ae-5cb23d1fa105-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/476b08ee-e24d-4f0c-a5ae-5cb23d1fa105-istiod-ca-cert\") pod \"476b08ee-e24d-4f0c-a5ae-5cb23d1fa105\" (UID: \"476b08ee-e24d-4f0c-a5ae-5cb23d1fa105\") "
Nov 03 19:52:24.258648 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:24.258486    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/476b08ee-e24d-4f0c-a5ae-5cb23d1fa105-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/476b08ee-e24d-4f0c-a5ae-5cb23d1fa105-workload-socket\") pod \"476b08ee-e24d-4f0c-a5ae-5cb23d1fa105\" (UID: \"476b08ee-e24d-4f0c-a5ae-5cb23d1fa105\") "
Nov 03 19:52:24.258648 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:24.258633    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/476b08ee-e24d-4f0c-a5ae-5cb23d1fa105-istio-token\" (UniqueName: \"kubernetes.io/projected/476b08ee-e24d-4f0c-a5ae-5cb23d1fa105-istio-token\") pod \"476b08ee-e24d-4f0c-a5ae-5cb23d1fa105\" (UID: \"476b08ee-e24d-4f0c-a5ae-5cb23d1fa105\") "
Nov 03 19:52:24.258794 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:24.258698    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/476b08ee-e24d-4f0c-a5ae-5cb23d1fa105-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/476b08ee-e24d-4f0c-a5ae-5cb23d1fa105-credential-socket\") pod \"476b08ee-e24d-4f0c-a5ae-5cb23d1fa105\" (UID: \"476b08ee-e24d-4f0c-a5ae-5cb23d1fa105\") "
Nov 03 19:52:24.258845 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:24.258792    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/476b08ee-e24d-4f0c-a5ae-5cb23d1fa105-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/476b08ee-e24d-4f0c-a5ae-5cb23d1fa105-istio-podinfo\") pod \"476b08ee-e24d-4f0c-a5ae-5cb23d1fa105\" (UID: \"476b08ee-e24d-4f0c-a5ae-5cb23d1fa105\") "
Nov 03 19:52:24.258845 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:24.258830    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/476b08ee-e24d-4f0c-a5ae-5cb23d1fa105-workload-socket" pod "476b08ee-e24d-4f0c-a5ae-5cb23d1fa105" (UID: "476b08ee-e24d-4f0c-a5ae-5cb23d1fa105"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:24.258947 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:24.258859    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/476b08ee-e24d-4f0c-a5ae-5cb23d1fa105-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/476b08ee-e24d-4f0c-a5ae-5cb23d1fa105-istio-data\") pod \"476b08ee-e24d-4f0c-a5ae-5cb23d1fa105\" (UID: \"476b08ee-e24d-4f0c-a5ae-5cb23d1fa105\") "
Nov 03 19:52:24.258947 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:24.258860    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/476b08ee-e24d-4f0c-a5ae-5cb23d1fa105-istiod-ca-cert" pod "476b08ee-e24d-4f0c-a5ae-5cb23d1fa105" (UID: "476b08ee-e24d-4f0c-a5ae-5cb23d1fa105"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:52:24.259050 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:24.258963    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/476b08ee-e24d-4f0c-a5ae-5cb23d1fa105-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/476b08ee-e24d-4f0c-a5ae-5cb23d1fa105-workload-certs\") pod \"476b08ee-e24d-4f0c-a5ae-5cb23d1fa105\" (UID: \"476b08ee-e24d-4f0c-a5ae-5cb23d1fa105\") "
Nov 03 19:52:24.259104 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:24.259079    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/476b08ee-e24d-4f0c-a5ae-5cb23d1fa105-kube-api-access-fl7bj\" (UniqueName: \"kubernetes.io/projected/476b08ee-e24d-4f0c-a5ae-5cb23d1fa105-kube-api-access-fl7bj\") pod \"476b08ee-e24d-4f0c-a5ae-5cb23d1fa105\" (UID: \"476b08ee-e24d-4f0c-a5ae-5cb23d1fa105\") "
Nov 03 19:52:24.259160 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:24.259120    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/476b08ee-e24d-4f0c-a5ae-5cb23d1fa105-istio-data" pod "476b08ee-e24d-4f0c-a5ae-5cb23d1fa105" (UID: "476b08ee-e24d-4f0c-a5ae-5cb23d1fa105"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:24.259210 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:24.259181    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/476b08ee-e24d-4f0c-a5ae-5cb23d1fa105-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/476b08ee-e24d-4f0c-a5ae-5cb23d1fa105-istio-envoy\") pod \"476b08ee-e24d-4f0c-a5ae-5cb23d1fa105\" (UID: \"476b08ee-e24d-4f0c-a5ae-5cb23d1fa105\") "
Nov 03 19:52:24.259711 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:24.259640    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/476b08ee-e24d-4f0c-a5ae-5cb23d1fa105-workload-certs" pod "476b08ee-e24d-4f0c-a5ae-5cb23d1fa105" (UID: "476b08ee-e24d-4f0c-a5ae-5cb23d1fa105"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:24.260210 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:24.260187    4921 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/476b08ee-e24d-4f0c-a5ae-5cb23d1fa105-istiod-ca-cert\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:24.260295 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:24.260217    4921 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/476b08ee-e24d-4f0c-a5ae-5cb23d1fa105-workload-socket\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:24.260295 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:24.260231    4921 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/476b08ee-e24d-4f0c-a5ae-5cb23d1fa105-istio-data\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:24.260295 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:24.260245    4921 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/476b08ee-e24d-4f0c-a5ae-5cb23d1fa105-workload-certs\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:24.260468 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:24.260447    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/476b08ee-e24d-4f0c-a5ae-5cb23d1fa105-credential-socket" pod "476b08ee-e24d-4f0c-a5ae-5cb23d1fa105" (UID: "476b08ee-e24d-4f0c-a5ae-5cb23d1fa105"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:24.261500 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:24.261471    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/476b08ee-e24d-4f0c-a5ae-5cb23d1fa105-istio-podinfo" pod "476b08ee-e24d-4f0c-a5ae-5cb23d1fa105" (UID: "476b08ee-e24d-4f0c-a5ae-5cb23d1fa105"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:52:24.261694 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:24.261658    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/476b08ee-e24d-4f0c-a5ae-5cb23d1fa105-kube-api-access-fl7bj" pod "476b08ee-e24d-4f0c-a5ae-5cb23d1fa105" (UID: "476b08ee-e24d-4f0c-a5ae-5cb23d1fa105"). InnerVolumeSpecName "kube-api-access-fl7bj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:52:24.261776 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:24.261716    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/476b08ee-e24d-4f0c-a5ae-5cb23d1fa105-istio-token" pod "476b08ee-e24d-4f0c-a5ae-5cb23d1fa105" (UID: "476b08ee-e24d-4f0c-a5ae-5cb23d1fa105"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:52:24.262416 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:24.262393    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/476b08ee-e24d-4f0c-a5ae-5cb23d1fa105-istio-envoy" pod "476b08ee-e24d-4f0c-a5ae-5cb23d1fa105" (UID: "476b08ee-e24d-4f0c-a5ae-5cb23d1fa105"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:24.361750 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:24.361694    4921 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/476b08ee-e24d-4f0c-a5ae-5cb23d1fa105-istio-token\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:24.361750 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:24.361735    4921 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/476b08ee-e24d-4f0c-a5ae-5cb23d1fa105-credential-socket\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:24.361750 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:24.361745    4921 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/476b08ee-e24d-4f0c-a5ae-5cb23d1fa105-istio-podinfo\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:24.361750 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:24.361755    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fl7bj\" (UniqueName: \"kubernetes.io/projected/476b08ee-e24d-4f0c-a5ae-5cb23d1fa105-kube-api-access-fl7bj\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:24.362021 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:24.361766    4921 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/476b08ee-e24d-4f0c-a5ae-5cb23d1fa105-istio-envoy\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:24.599910 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:24.599823    4921 generic.go:356] "Generic (PLEG): container finished" podID="476b08ee-e24d-4f0c-a5ae-5cb23d1fa105" containerID="63ed9099e84a399a347bdf1a5a4b7f19d524546abb5eef5a2e43ab00455c0e23" exitCode=0
Nov 03 19:52:24.599910 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:24.599876    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2"
Nov 03 19:52:24.599910 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:24.599896    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2" event={"ID":"476b08ee-e24d-4f0c-a5ae-5cb23d1fa105","Type":"ContainerDied","Data":"63ed9099e84a399a347bdf1a5a4b7f19d524546abb5eef5a2e43ab00455c0e23"}
Nov 03 19:52:24.600161 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:24.599954    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2" event={"ID":"476b08ee-e24d-4f0c-a5ae-5cb23d1fa105","Type":"ContainerDied","Data":"37e81b910f6fa8843353e301c913b4411f793791d5860d5f6e60b1512b77a352"}
Nov 03 19:52:24.600161 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:24.599978    4921 scope.go:122] "RemoveContainer" containerID="63ed9099e84a399a347bdf1a5a4b7f19d524546abb5eef5a2e43ab00455c0e23"
Nov 03 19:52:24.615061 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:24.615040    4921 scope.go:122] "RemoveContainer" containerID="63ed9099e84a399a347bdf1a5a4b7f19d524546abb5eef5a2e43ab00455c0e23"
Nov 03 19:52:24.615414 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:52:24.615396    4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63ed9099e84a399a347bdf1a5a4b7f19d524546abb5eef5a2e43ab00455c0e23\": container with ID starting with 63ed9099e84a399a347bdf1a5a4b7f19d524546abb5eef5a2e43ab00455c0e23 not found: ID does not exist" containerID="63ed9099e84a399a347bdf1a5a4b7f19d524546abb5eef5a2e43ab00455c0e23"
Nov 03 19:52:24.615478 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:24.615429    4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63ed9099e84a399a347bdf1a5a4b7f19d524546abb5eef5a2e43ab00455c0e23"} err="failed to get container status \"63ed9099e84a399a347bdf1a5a4b7f19d524546abb5eef5a2e43ab00455c0e23\": rpc error: code = NotFound desc = could not find container \"63ed9099e84a399a347bdf1a5a4b7f19d524546abb5eef5a2e43ab00455c0e23\": container with ID starting with 63ed9099e84a399a347bdf1a5a4b7f19d524546abb5eef5a2e43ab00455c0e23 not found: ID does not exist"
Nov 03 19:52:24.627254 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:24.627213    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2"]
Nov 03 19:52:24.632422 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:24.632387    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-8zhk2"]
Nov 03 19:52:24.890380 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:24.890294    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="476b08ee-e24d-4f0c-a5ae-5cb23d1fa105" path="/var/lib/kubelet/pods/476b08ee-e24d-4f0c-a5ae-5cb23d1fa105/volumes"
Nov 03 19:52:29.083476 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:29.083439    4921 scope.go:122] "RemoveContainer" containerID="5042d5effeb93fb51af9bcb7402e687fa6725b846b9cd947c1e14519e3086c84"
Nov 03 19:52:29.118146 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:29.116373    4921 scope.go:122] "RemoveContainer" containerID="a2729f3f40ebc255abd98c542ee106d02ff07c2549f52976ff62421db1c72510"
Nov 03 19:52:32.922120 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:52:32.922085    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 19:52:33.811098 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:33.811056    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a9cd5e-69bb85bc55-zv8d9"]
Nov 03 19:52:33.811829 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:33.811811    4921 cpu_manager.go:414] "containerMap: removing container" podUID="476b08ee-e24d-4f0c-a5ae-5cb23d1fa105" containerName="istio-proxy"
Nov 03 19:52:33.811829 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:33.811828    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="476b08ee-e24d-4f0c-a5ae-5cb23d1fa105" containerName="istio-proxy"
Nov 03 19:52:33.811993 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:33.811841    4921 cpu_manager.go:414] "containerMap: removing container" podUID="758e8ecf-b0c0-4133-889f-6aa948dc7a89" containerName="revoke-keys"
Nov 03 19:52:33.811993 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:33.811846    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="758e8ecf-b0c0-4133-889f-6aa948dc7a89" containerName="revoke-keys"
Nov 03 19:52:33.812097 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:33.811999    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="476b08ee-e24d-4f0c-a5ae-5cb23d1fa105" containerName="istio-proxy"
Nov 03 19:52:33.812097 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:33.812010    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="758e8ecf-b0c0-4133-889f-6aa948dc7a89" containerName="revoke-keys"
Nov 03 19:52:33.818271 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:33.818241    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9cd5e-69bb85bc55-zv8d9"
Nov 03 19:52:33.821598 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:33.821574    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-a9cd5e\""
Nov 03 19:52:33.821842 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:33.821640    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-a9cd5e-dockercfg-v6drw\""
Nov 03 19:52:33.823355 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:33.823318    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a9cd5e-69bb85bc55-zv8d9"]
Nov 03 19:52:33.850658 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:33.850622    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a9cd5e-58c95dbbdc-7pnq6"]
Nov 03 19:52:33.857645 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:33.857614    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a9cd5e-58c95dbbdc-7pnq6"
Nov 03 19:52:33.862408 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:33.862375    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a9cd5e-58c95dbbdc-7pnq6"]
Nov 03 19:52:33.994694 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:33.989957    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f095c43-12af-4dc0-beb7-b2d6189bf544-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a9cd5e-58c95dbbdc-7pnq6\" (UID: \"6f095c43-12af-4dc0-beb7-b2d6189bf544\") " pod="openshift-ingress/payload-processing-e2e-api-b-a9cd5e-58c95dbbdc-7pnq6"
Nov 03 19:52:33.994694 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:33.990021    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khq82\" (UniqueName: \"kubernetes.io/projected/6f095c43-12af-4dc0-beb7-b2d6189bf544-kube-api-access-khq82\") pod \"payload-processing-e2e-api-b-a9cd5e-58c95dbbdc-7pnq6\" (UID: \"6f095c43-12af-4dc0-beb7-b2d6189bf544\") " pod="openshift-ingress/payload-processing-e2e-api-b-a9cd5e-58c95dbbdc-7pnq6"
Nov 03 19:52:33.994694 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:33.990221    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q2mt\" (UniqueName: \"kubernetes.io/projected/cfad7089-f35f-41e3-8800-525f4fd7c2fb-kube-api-access-8q2mt\") pod \"payload-pre-processing-e2e-api-b-a9cd5e-69bb85bc55-zv8d9\" (UID: \"cfad7089-f35f-41e3-8800-525f4fd7c2fb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9cd5e-69bb85bc55-zv8d9"
Nov 03 19:52:33.994694 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:33.990294    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfad7089-f35f-41e3-8800-525f4fd7c2fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a9cd5e-69bb85bc55-zv8d9\" (UID: \"cfad7089-f35f-41e3-8800-525f4fd7c2fb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9cd5e-69bb85bc55-zv8d9"
Nov 03 19:52:34.097749 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:34.091943    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f095c43-12af-4dc0-beb7-b2d6189bf544-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a9cd5e-58c95dbbdc-7pnq6\" (UID: \"6f095c43-12af-4dc0-beb7-b2d6189bf544\") " pod="openshift-ingress/payload-processing-e2e-api-b-a9cd5e-58c95dbbdc-7pnq6"
Nov 03 19:52:34.097749 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:34.091999    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khq82\" (UniqueName: \"kubernetes.io/projected/6f095c43-12af-4dc0-beb7-b2d6189bf544-kube-api-access-khq82\") pod \"payload-processing-e2e-api-b-a9cd5e-58c95dbbdc-7pnq6\" (UID: \"6f095c43-12af-4dc0-beb7-b2d6189bf544\") " pod="openshift-ingress/payload-processing-e2e-api-b-a9cd5e-58c95dbbdc-7pnq6"
Nov 03 19:52:34.097749 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:34.092162    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q2mt\" (UniqueName: \"kubernetes.io/projected/cfad7089-f35f-41e3-8800-525f4fd7c2fb-kube-api-access-8q2mt\") pod \"payload-pre-processing-e2e-api-b-a9cd5e-69bb85bc55-zv8d9\" (UID: \"cfad7089-f35f-41e3-8800-525f4fd7c2fb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9cd5e-69bb85bc55-zv8d9"
Nov 03 19:52:34.097749 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:34.092226    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfad7089-f35f-41e3-8800-525f4fd7c2fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a9cd5e-69bb85bc55-zv8d9\" (UID: \"cfad7089-f35f-41e3-8800-525f4fd7c2fb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9cd5e-69bb85bc55-zv8d9"
Nov 03 19:52:34.097749 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:34.093222    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfad7089-f35f-41e3-8800-525f4fd7c2fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a9cd5e-69bb85bc55-zv8d9\" (UID: \"cfad7089-f35f-41e3-8800-525f4fd7c2fb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9cd5e-69bb85bc55-zv8d9"
Nov 03 19:52:34.108700 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:34.105751    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f095c43-12af-4dc0-beb7-b2d6189bf544-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a9cd5e-58c95dbbdc-7pnq6\" (UID: \"6f095c43-12af-4dc0-beb7-b2d6189bf544\") " pod="openshift-ingress/payload-processing-e2e-api-b-a9cd5e-58c95dbbdc-7pnq6"
Nov 03 19:52:34.134143 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:34.128095    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q2mt\" (UniqueName: \"kubernetes.io/projected/cfad7089-f35f-41e3-8800-525f4fd7c2fb-kube-api-access-8q2mt\") pod \"payload-pre-processing-e2e-api-b-a9cd5e-69bb85bc55-zv8d9\" (UID: \"cfad7089-f35f-41e3-8800-525f4fd7c2fb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9cd5e-69bb85bc55-zv8d9"
Nov 03 19:52:34.134143 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:34.132335    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khq82\" (UniqueName: \"kubernetes.io/projected/6f095c43-12af-4dc0-beb7-b2d6189bf544-kube-api-access-khq82\") pod \"payload-processing-e2e-api-b-a9cd5e-58c95dbbdc-7pnq6\" (UID: \"6f095c43-12af-4dc0-beb7-b2d6189bf544\") " pod="openshift-ingress/payload-processing-e2e-api-b-a9cd5e-58c95dbbdc-7pnq6"
Nov 03 19:52:34.151692 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:34.148130    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9cd5e-69bb85bc55-zv8d9"
Nov 03 19:52:34.180037 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:34.179997    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a9cd5e-58c95dbbdc-7pnq6"
Nov 03 19:52:34.573811 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:52:34.573778    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfad7089_f35f_41e3_8800_525f4fd7c2fb.slice/crio-5427c42f3a8efc49075777d02ad45353aff70f11c788b68ce724012e1a1d9c12 WatchSource:0}: Error finding container 5427c42f3a8efc49075777d02ad45353aff70f11c788b68ce724012e1a1d9c12: Status 404 returned error can't find the container with id 5427c42f3a8efc49075777d02ad45353aff70f11c788b68ce724012e1a1d9c12
Nov 03 19:52:34.574129 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:34.574104    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a9cd5e-69bb85bc55-zv8d9"]
Nov 03 19:52:34.628834 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:34.628805    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a9cd5e-58c95dbbdc-7pnq6"]
Nov 03 19:52:34.639112 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:52:34.639085    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f095c43_12af_4dc0_beb7_b2d6189bf544.slice/crio-d5a472f25419df168515e25dc6edb6512671bca1bc4c2141fa49e8257a199922 WatchSource:0}: Error finding container d5a472f25419df168515e25dc6edb6512671bca1bc4c2141fa49e8257a199922: Status 404 returned error can't find the container with id d5a472f25419df168515e25dc6edb6512671bca1bc4c2141fa49e8257a199922
Nov 03 19:52:34.776725 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:34.776688    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a9cd5e-58c95dbbdc-7pnq6" event={"ID":"6f095c43-12af-4dc0-beb7-b2d6189bf544","Type":"ContainerStarted","Data":"d5a472f25419df168515e25dc6edb6512671bca1bc4c2141fa49e8257a199922"}
Nov 03 19:52:34.780602 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:34.780570    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9cd5e-69bb85bc55-zv8d9" event={"ID":"cfad7089-f35f-41e3-8800-525f4fd7c2fb","Type":"ContainerStarted","Data":"7f0a98adc2d4ab4a792bbd6759d668a036765be4f072fd815bed8d60102369ed"}
Nov 03 19:52:34.780978 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:34.780962    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9cd5e-69bb85bc55-zv8d9"
Nov 03 19:52:34.781089 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:34.781076    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9cd5e-69bb85bc55-zv8d9" event={"ID":"cfad7089-f35f-41e3-8800-525f4fd7c2fb","Type":"ContainerStarted","Data":"5427c42f3a8efc49075777d02ad45353aff70f11c788b68ce724012e1a1d9c12"}
Nov 03 19:52:34.811209 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:34.798168    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9cd5e-69bb85bc55-zv8d9" podStartSLOduration=1.798148081 podStartE2EDuration="1.798148081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:52:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:52:34.793597092 +0000 UTC m=+1268.037562568" watchObservedRunningTime="2026-11-03 19:52:34.798148081 +0000 UTC m=+1268.042113547"
Nov 03 19:52:35.789282 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:35.789248    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a9cd5e-58c95dbbdc-7pnq6" event={"ID":"6f095c43-12af-4dc0-beb7-b2d6189bf544","Type":"ContainerStarted","Data":"2c6841d100c7ad5f9a1560f9e6232f925a7c100f14b20e0eaa01f3af41e31108"}
Nov 03 19:52:35.789857 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:35.789337    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-a9cd5e-58c95dbbdc-7pnq6"
Nov 03 19:52:35.807624 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:35.807581    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-a9cd5e-58c95dbbdc-7pnq6" podStartSLOduration=2.807566291 podStartE2EDuration="2.807566291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:52:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:52:35.803314438 +0000 UTC m=+1269.047279903" watchObservedRunningTime="2026-11-03 19:52:35.807566291 +0000 UTC m=+1269.051531755"
Nov 03 19:52:37.079107 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:37.079074    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a536f35b-dqm2d"]
Nov 03 19:52:37.087023 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:37.086986    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a536f35b-dqm2d"]
Nov 03 19:52:38.139993 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:38.139090    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-82b108-6b74cb9cd9-8hw8h"]
Nov 03 19:52:38.139993 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:38.139359    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-82b108-6b74cb9cd9-8hw8h" podUID="13647f3d-08d3-416d-8424-79840b82e398" containerName="maas-api" containerID="cri-o://ce31cc3c97eec393c304c190821a180d618fc37aa65fe9b68cc25d8a04746b02" gracePeriod=30
Nov 03 19:52:38.672704 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:38.672343    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-82b108-6pjd8"]
Nov 03 19:52:38.709697 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:38.708316    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-82b108-6pjd8"]
Nov 03 19:52:38.732962 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:38.731318    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-82b108-6b74cb9cd9-8hw8h"
Nov 03 19:52:38.813689 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:38.806066    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13647f3d-08d3-416d-8424-79840b82e398-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13647f3d-08d3-416d-8424-79840b82e398-openshift-service-ca\") pod \"13647f3d-08d3-416d-8424-79840b82e398\" (UID: \"13647f3d-08d3-416d-8424-79840b82e398\") "
Nov 03 19:52:38.813689 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:38.806162    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13647f3d-08d3-416d-8424-79840b82e398-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13647f3d-08d3-416d-8424-79840b82e398-maas-api-tls\") pod \"13647f3d-08d3-416d-8424-79840b82e398\" (UID: \"13647f3d-08d3-416d-8424-79840b82e398\") "
Nov 03 19:52:38.813689 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:38.806256    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13647f3d-08d3-416d-8424-79840b82e398-kube-api-access-4xp6k\" (UniqueName: \"kubernetes.io/projected/13647f3d-08d3-416d-8424-79840b82e398-kube-api-access-4xp6k\") pod \"13647f3d-08d3-416d-8424-79840b82e398\" (UID: \"13647f3d-08d3-416d-8424-79840b82e398\") "
Nov 03 19:52:38.813689 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:38.806403    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13647f3d-08d3-416d-8424-79840b82e398-metrics-certs\" (UniqueName: \"kubernetes.io/secret/13647f3d-08d3-416d-8424-79840b82e398-metrics-certs\") pod \"13647f3d-08d3-416d-8424-79840b82e398\" (UID: \"13647f3d-08d3-416d-8424-79840b82e398\") "
Nov 03 19:52:38.816656 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:38.815054    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13647f3d-08d3-416d-8424-79840b82e398-openshift-service-ca" pod "13647f3d-08d3-416d-8424-79840b82e398" (UID: "13647f3d-08d3-416d-8424-79840b82e398"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:52:38.816656 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:38.815515    4921 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13647f3d-08d3-416d-8424-79840b82e398-openshift-service-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:38.850689 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:38.840088    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13647f3d-08d3-416d-8424-79840b82e398-metrics-certs" pod "13647f3d-08d3-416d-8424-79840b82e398" (UID: "13647f3d-08d3-416d-8424-79840b82e398"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:52:38.854692 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:38.851242    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13647f3d-08d3-416d-8424-79840b82e398-maas-api-tls" pod "13647f3d-08d3-416d-8424-79840b82e398" (UID: "13647f3d-08d3-416d-8424-79840b82e398"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:52:38.871692 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:38.865628    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13647f3d-08d3-416d-8424-79840b82e398-kube-api-access-4xp6k" pod "13647f3d-08d3-416d-8424-79840b82e398" (UID: "13647f3d-08d3-416d-8424-79840b82e398"). InnerVolumeSpecName "kube-api-access-4xp6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:52:38.917741 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:38.917646    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xp6k\" (UniqueName: \"kubernetes.io/projected/13647f3d-08d3-416d-8424-79840b82e398-kube-api-access-4xp6k\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:38.917741 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:38.917701    4921 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13647f3d-08d3-416d-8424-79840b82e398-metrics-certs\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:38.917741 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:38.917715    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13647f3d-08d3-416d-8424-79840b82e398-maas-api-tls\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:38.953692 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:38.950020    4921 generic.go:356] "Generic (PLEG): container finished" podID="13647f3d-08d3-416d-8424-79840b82e398" containerID="ce31cc3c97eec393c304c190821a180d618fc37aa65fe9b68cc25d8a04746b02" exitCode=0
Nov 03 19:52:38.953692 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:38.950304    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-82b108-6b74cb9cd9-8hw8h"
Nov 03 19:52:38.965463 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:38.965417    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f2cd2c7-0827-42a6-9369-a13e8c19bafd" path="/var/lib/kubelet/pods/5f2cd2c7-0827-42a6-9369-a13e8c19bafd/volumes"
Nov 03 19:52:38.966761 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:38.966735    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67014765-bdcd-4e2e-80e9-ecba29891abb" path="/var/lib/kubelet/pods/67014765-bdcd-4e2e-80e9-ecba29891abb/volumes"
Nov 03 19:52:38.968983 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:38.968956    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-82b108-6b74cb9cd9-8hw8h" event={"ID":"13647f3d-08d3-416d-8424-79840b82e398","Type":"ContainerDied","Data":"ce31cc3c97eec393c304c190821a180d618fc37aa65fe9b68cc25d8a04746b02"}
Nov 03 19:52:38.974764 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:38.974732    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-82b108-6b74cb9cd9-8hw8h" event={"ID":"13647f3d-08d3-416d-8424-79840b82e398","Type":"ContainerDied","Data":"fe77e8c092f562ed5f3e4406721d7ac6ecfb09d4c56bf5134788e925e2b330d9"}
Nov 03 19:52:38.975223 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:38.975114    4921 scope.go:122] "RemoveContainer" containerID="ce31cc3c97eec393c304c190821a180d618fc37aa65fe9b68cc25d8a04746b02"
Nov 03 19:52:39.011364 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:39.011221    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-82b108-6b74cb9cd9-8hw8h"]
Nov 03 19:52:39.041979 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:39.041949    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-82b108-6b74cb9cd9-8hw8h"]
Nov 03 19:52:39.059442 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:39.059419    4921 scope.go:122] "RemoveContainer" containerID="ce31cc3c97eec393c304c190821a180d618fc37aa65fe9b68cc25d8a04746b02"
Nov 03 19:52:39.060698 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:52:39.060652    4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce31cc3c97eec393c304c190821a180d618fc37aa65fe9b68cc25d8a04746b02\": container with ID starting with ce31cc3c97eec393c304c190821a180d618fc37aa65fe9b68cc25d8a04746b02 not found: ID does not exist" containerID="ce31cc3c97eec393c304c190821a180d618fc37aa65fe9b68cc25d8a04746b02"
Nov 03 19:52:39.060868 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:39.060839    4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce31cc3c97eec393c304c190821a180d618fc37aa65fe9b68cc25d8a04746b02"} err="failed to get container status \"ce31cc3c97eec393c304c190821a180d618fc37aa65fe9b68cc25d8a04746b02\": rpc error: code = NotFound desc = could not find container \"ce31cc3c97eec393c304c190821a180d618fc37aa65fe9b68cc25d8a04746b02\": container with ID starting with ce31cc3c97eec393c304c190821a180d618fc37aa65fe9b68cc25d8a04746b02 not found: ID does not exist"
Nov 03 19:52:40.891003 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:40.890971    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13647f3d-08d3-416d-8424-79840b82e398" path="/var/lib/kubelet/pods/13647f3d-08d3-416d-8424-79840b82e398/volumes"
Nov 03 19:52:44.031037 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:44.031003    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2b73c810-ptmpm"]
Nov 03 19:52:44.031794 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:44.031777    4921 cpu_manager.go:414] "containerMap: removing container" podUID="13647f3d-08d3-416d-8424-79840b82e398" containerName="maas-api"
Nov 03 19:52:44.031851 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:44.031794    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13647f3d-08d3-416d-8424-79840b82e398" containerName="maas-api"
Nov 03 19:52:44.031952 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:44.031942    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="13647f3d-08d3-416d-8424-79840b82e398" containerName="maas-api"
Nov 03 19:52:44.035542 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:44.035528    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2b73c810-ptmpm"
Nov 03 19:52:44.041190 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:44.041161    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:52:44.041475 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:44.041453    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 19:52:44.043386 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:44.043355    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2b73c810-ptmpm"]
Nov 03 19:52:44.141533 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:44.141472    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7b8m\" (UniqueName: \"kubernetes.io/projected/bb482f3b-80c8-4ba4-b25b-02d9fa97ef3d-kube-api-access-z7b8m\") pod \"maas-api-revoke-keys-e2e-mt-2b73c810-ptmpm\" (UID: \"bb482f3b-80c8-4ba4-b25b-02d9fa97ef3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2b73c810-ptmpm"
Nov 03 19:52:44.141784 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:44.141645    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb482f3b-80c8-4ba4-b25b-02d9fa97ef3d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-2b73c810-ptmpm\" (UID: \"bb482f3b-80c8-4ba4-b25b-02d9fa97ef3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2b73c810-ptmpm"
Nov 03 19:52:44.141784 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:44.141775    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb482f3b-80c8-4ba4-b25b-02d9fa97ef3d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-2b73c810-ptmpm\" (UID: \"bb482f3b-80c8-4ba4-b25b-02d9fa97ef3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2b73c810-ptmpm"
Nov 03 19:52:44.243446 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:44.243397    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7b8m\" (UniqueName: \"kubernetes.io/projected/bb482f3b-80c8-4ba4-b25b-02d9fa97ef3d-kube-api-access-z7b8m\") pod \"maas-api-revoke-keys-e2e-mt-2b73c810-ptmpm\" (UID: \"bb482f3b-80c8-4ba4-b25b-02d9fa97ef3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2b73c810-ptmpm"
Nov 03 19:52:44.243636 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:44.243507    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb482f3b-80c8-4ba4-b25b-02d9fa97ef3d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-2b73c810-ptmpm\" (UID: \"bb482f3b-80c8-4ba4-b25b-02d9fa97ef3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2b73c810-ptmpm"
Nov 03 19:52:44.243636 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:44.243544    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb482f3b-80c8-4ba4-b25b-02d9fa97ef3d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-2b73c810-ptmpm\" (UID: \"bb482f3b-80c8-4ba4-b25b-02d9fa97ef3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2b73c810-ptmpm"
Nov 03 19:52:44.244279 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:44.244253    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb482f3b-80c8-4ba4-b25b-02d9fa97ef3d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-2b73c810-ptmpm\" (UID: \"bb482f3b-80c8-4ba4-b25b-02d9fa97ef3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2b73c810-ptmpm"
Nov 03 19:52:44.247294 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:44.247266    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb482f3b-80c8-4ba4-b25b-02d9fa97ef3d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-2b73c810-ptmpm\" (UID: \"bb482f3b-80c8-4ba4-b25b-02d9fa97ef3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2b73c810-ptmpm"
Nov 03 19:52:44.253582 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:44.253555    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7b8m\" (UniqueName: \"kubernetes.io/projected/bb482f3b-80c8-4ba4-b25b-02d9fa97ef3d-kube-api-access-z7b8m\") pod \"maas-api-revoke-keys-e2e-mt-2b73c810-ptmpm\" (UID: \"bb482f3b-80c8-4ba4-b25b-02d9fa97ef3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2b73c810-ptmpm"
Nov 03 19:52:44.359599 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:44.359512    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2b73c810-ptmpm"
Nov 03 19:52:44.724764 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:44.724729    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2b73c810-ptmpm"]
Nov 03 19:52:45.004744 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:45.004659    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2b73c810-ptmpm" event={"ID":"bb482f3b-80c8-4ba4-b25b-02d9fa97ef3d","Type":"ContainerStarted","Data":"00669926dfd5f79d506e35b5a14c027626f2dd243ca4f94933a584b7874dc751"}
Nov 03 19:52:45.334579 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:45.334506    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a9cd5e-59b5db6cdf-j95k7"]
Nov 03 19:52:45.340015 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:45.339979    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9cd5e-59b5db6cdf-j95k7"
Nov 03 19:52:45.358724 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:45.351446    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-a9cd5e-serving-cert\""
Nov 03 19:52:45.358724 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:45.353810    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a9cd5e-59b5db6cdf-j95k7"]
Nov 03 19:52:45.483218 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:45.483175    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d27071d-8535-4b88-9e91-e99c3c74abe2-metrics-certs\") pod \"maas-api-e2e-api-b-a9cd5e-59b5db6cdf-j95k7\" (UID: \"9d27071d-8535-4b88-9e91-e99c3c74abe2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9cd5e-59b5db6cdf-j95k7"
Nov 03 19:52:45.483477 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:45.483451    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d27071d-8535-4b88-9e91-e99c3c74abe2-maas-api-tls\") pod \"maas-api-e2e-api-b-a9cd5e-59b5db6cdf-j95k7\" (UID: \"9d27071d-8535-4b88-9e91-e99c3c74abe2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9cd5e-59b5db6cdf-j95k7"
Nov 03 19:52:45.483540 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:45.483514    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d27071d-8535-4b88-9e91-e99c3c74abe2-openshift-service-ca\") pod \"maas-api-e2e-api-b-a9cd5e-59b5db6cdf-j95k7\" (UID: \"9d27071d-8535-4b88-9e91-e99c3c74abe2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9cd5e-59b5db6cdf-j95k7"
Nov 03 19:52:45.483630 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:45.483612    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5c5dk\" (UniqueName: \"kubernetes.io/projected/9d27071d-8535-4b88-9e91-e99c3c74abe2-kube-api-access-5c5dk\") pod \"maas-api-e2e-api-b-a9cd5e-59b5db6cdf-j95k7\" (UID: \"9d27071d-8535-4b88-9e91-e99c3c74abe2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9cd5e-59b5db6cdf-j95k7"
Nov 03 19:52:45.585554 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:45.585458    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d27071d-8535-4b88-9e91-e99c3c74abe2-metrics-certs\") pod \"maas-api-e2e-api-b-a9cd5e-59b5db6cdf-j95k7\" (UID: \"9d27071d-8535-4b88-9e91-e99c3c74abe2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9cd5e-59b5db6cdf-j95k7"
Nov 03 19:52:45.585755 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:45.585741    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d27071d-8535-4b88-9e91-e99c3c74abe2-maas-api-tls\") pod \"maas-api-e2e-api-b-a9cd5e-59b5db6cdf-j95k7\" (UID: \"9d27071d-8535-4b88-9e91-e99c3c74abe2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9cd5e-59b5db6cdf-j95k7"
Nov 03 19:52:45.585836 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:45.585815    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d27071d-8535-4b88-9e91-e99c3c74abe2-openshift-service-ca\") pod \"maas-api-e2e-api-b-a9cd5e-59b5db6cdf-j95k7\" (UID: \"9d27071d-8535-4b88-9e91-e99c3c74abe2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9cd5e-59b5db6cdf-j95k7"
Nov 03 19:52:45.585913 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:45.585890    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c5dk\" (UniqueName: \"kubernetes.io/projected/9d27071d-8535-4b88-9e91-e99c3c74abe2-kube-api-access-5c5dk\") pod \"maas-api-e2e-api-b-a9cd5e-59b5db6cdf-j95k7\" (UID: \"9d27071d-8535-4b88-9e91-e99c3c74abe2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9cd5e-59b5db6cdf-j95k7"
Nov 03 19:52:45.587496 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:45.587402    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d27071d-8535-4b88-9e91-e99c3c74abe2-openshift-service-ca\") pod \"maas-api-e2e-api-b-a9cd5e-59b5db6cdf-j95k7\" (UID: \"9d27071d-8535-4b88-9e91-e99c3c74abe2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9cd5e-59b5db6cdf-j95k7"
Nov 03 19:52:45.597690 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:45.597646    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5c5dk\" (UniqueName: \"kubernetes.io/projected/9d27071d-8535-4b88-9e91-e99c3c74abe2-kube-api-access-5c5dk\") pod \"maas-api-e2e-api-b-a9cd5e-59b5db6cdf-j95k7\" (UID: \"9d27071d-8535-4b88-9e91-e99c3c74abe2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9cd5e-59b5db6cdf-j95k7"
Nov 03 19:52:45.601579 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:45.601551    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d27071d-8535-4b88-9e91-e99c3c74abe2-metrics-certs\") pod \"maas-api-e2e-api-b-a9cd5e-59b5db6cdf-j95k7\" (UID: \"9d27071d-8535-4b88-9e91-e99c3c74abe2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9cd5e-59b5db6cdf-j95k7"
Nov 03 19:52:45.602974 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:45.602951    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d27071d-8535-4b88-9e91-e99c3c74abe2-maas-api-tls\") pod \"maas-api-e2e-api-b-a9cd5e-59b5db6cdf-j95k7\" (UID: \"9d27071d-8535-4b88-9e91-e99c3c74abe2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9cd5e-59b5db6cdf-j95k7"
Nov 03 19:52:45.671402 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:45.671368    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9cd5e-59b5db6cdf-j95k7"
Nov 03 19:52:45.824364 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:45.814858    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9cd5e-69bb85bc55-zv8d9"
Nov 03 19:52:46.471831 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:46.459302    4921 generic.go:356] "Generic (PLEG): container finished" podID="bb482f3b-80c8-4ba4-b25b-02d9fa97ef3d" containerID="1d5ff4531378f990e85ba09a2c4c5aee33634273659e00a6dad962a5fc178e02" exitCode=0
Nov 03 19:52:46.471831 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:46.459634    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2b73c810-ptmpm" event={"ID":"bb482f3b-80c8-4ba4-b25b-02d9fa97ef3d","Type":"ContainerDied","Data":"1d5ff4531378f990e85ba09a2c4c5aee33634273659e00a6dad962a5fc178e02"}
Nov 03 19:52:46.536690 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:46.519655    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a9cd5e-59b5db6cdf-j95k7"]
Nov 03 19:52:46.536690 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:46.528527    4921 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 19:52:46.857022 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:46.856990    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-a9cd5e-58c95dbbdc-7pnq6"
Nov 03 19:52:47.481619 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:47.481430    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9cd5e-59b5db6cdf-j95k7" event={"ID":"9d27071d-8535-4b88-9e91-e99c3c74abe2","Type":"ContainerStarted","Data":"58db1dd44518ccd26d1b40cfbb9a2a520014b9f6f8db7975a1185a5a136c0e6e"}
Nov 03 19:52:47.482179 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:47.481663    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9cd5e-59b5db6cdf-j95k7" event={"ID":"9d27071d-8535-4b88-9e91-e99c3c74abe2","Type":"ContainerStarted","Data":"acf75bea503dc0aa88a9c9072705d23ff69095bbdb6431b536e436ef7a211514"}
Nov 03 19:52:47.501083 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:47.501016    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9cd5e-59b5db6cdf-j95k7" podStartSLOduration=1.790957828 podStartE2EDuration="2.500996632s" totalImagesPullingTime="710.038804ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:52:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:52:47.496232321 +0000 UTC m=+1280.740197787" watchObservedRunningTime="2026-11-03 19:52:47.500996632 +0000 UTC m=+1280.744962098"
Nov 03 19:52:47.756759 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:47.750906    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2b73c810-ptmpm"
Nov 03 19:52:47.842756 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:47.842722    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb482f3b-80c8-4ba4-b25b-02d9fa97ef3d-kube-api-access-z7b8m\" (UniqueName: \"kubernetes.io/projected/bb482f3b-80c8-4ba4-b25b-02d9fa97ef3d-kube-api-access-z7b8m\") pod \"bb482f3b-80c8-4ba4-b25b-02d9fa97ef3d\" (UID: \"bb482f3b-80c8-4ba4-b25b-02d9fa97ef3d\") "
Nov 03 19:52:47.842947 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:47.842930    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb482f3b-80c8-4ba4-b25b-02d9fa97ef3d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb482f3b-80c8-4ba4-b25b-02d9fa97ef3d-maas-api-xks-ca\") pod \"bb482f3b-80c8-4ba4-b25b-02d9fa97ef3d\" (UID: \"bb482f3b-80c8-4ba4-b25b-02d9fa97ef3d\") "
Nov 03 19:52:47.843090 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:47.843075    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb482f3b-80c8-4ba4-b25b-02d9fa97ef3d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb482f3b-80c8-4ba4-b25b-02d9fa97ef3d-maas-api-openshift-ca\") pod \"bb482f3b-80c8-4ba4-b25b-02d9fa97ef3d\" (UID: \"bb482f3b-80c8-4ba4-b25b-02d9fa97ef3d\") "
Nov 03 19:52:47.843613 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:47.843589    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb482f3b-80c8-4ba4-b25b-02d9fa97ef3d-maas-api-openshift-ca" pod "bb482f3b-80c8-4ba4-b25b-02d9fa97ef3d" (UID: "bb482f3b-80c8-4ba4-b25b-02d9fa97ef3d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:52:47.845655 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:47.845627    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb482f3b-80c8-4ba4-b25b-02d9fa97ef3d-maas-api-xks-ca" pod "bb482f3b-80c8-4ba4-b25b-02d9fa97ef3d" (UID: "bb482f3b-80c8-4ba4-b25b-02d9fa97ef3d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:52:47.846827 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:47.846805    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb482f3b-80c8-4ba4-b25b-02d9fa97ef3d-kube-api-access-z7b8m" pod "bb482f3b-80c8-4ba4-b25b-02d9fa97ef3d" (UID: "bb482f3b-80c8-4ba4-b25b-02d9fa97ef3d"). InnerVolumeSpecName "kube-api-access-z7b8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:52:47.945525 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:47.945495    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb482f3b-80c8-4ba4-b25b-02d9fa97ef3d-maas-api-xks-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:47.945525 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:47.945521    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb482f3b-80c8-4ba4-b25b-02d9fa97ef3d-maas-api-openshift-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:47.945750 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:47.945532    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7b8m\" (UniqueName: \"kubernetes.io/projected/bb482f3b-80c8-4ba4-b25b-02d9fa97ef3d-kube-api-access-z7b8m\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:48.537688 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:48.535008    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2b73c810-ptmpm"
Nov 03 19:52:48.538991 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:48.538732    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2b73c810-ptmpm" event={"ID":"bb482f3b-80c8-4ba4-b25b-02d9fa97ef3d","Type":"ContainerDied","Data":"00669926dfd5f79d506e35b5a14c027626f2dd243ca4f94933a584b7874dc751"}
Nov 03 19:52:48.538991 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:48.538803    4921 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="00669926dfd5f79d506e35b5a14c027626f2dd243ca4f94933a584b7874dc751"
Nov 03 19:52:48.538991 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:48.538840    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9cd5e-59b5db6cdf-j95k7"
Nov 03 19:52:48.936991 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:48.936906    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt"]
Nov 03 19:52:48.937140 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:48.937124    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt" podUID="5c2fc702-9338-4af0-a044-810fafccbfad" containerName="istio-proxy" containerID="cri-o://f1a832e2f6fe841a3a93619900e9716bc01f3ec890d822bcc590edf8b761deae" gracePeriod=30
Nov 03 19:52:51.166030 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:51.165992    4921 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 19:52:51.166432 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:51.166046    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt" podUID="5c2fc702-9338-4af0-a044-810fafccbfad" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 19:52:54.331695 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:54.251876    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2b73c810-798bdd757f-8k9lr"]
Nov 03 19:52:54.623418 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:54.623339    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9cd5e-59b5db6cdf-j95k7"
Nov 03 19:52:54.720422 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:54.719657    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt"
Nov 03 19:52:54.770241 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:54.770207    4921 generic.go:356] "Generic (PLEG): container finished" podID="5c2fc702-9338-4af0-a044-810fafccbfad" containerID="f1a832e2f6fe841a3a93619900e9716bc01f3ec890d822bcc590edf8b761deae" exitCode=0
Nov 03 19:52:54.770433 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:54.770289    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt" event={"ID":"5c2fc702-9338-4af0-a044-810fafccbfad","Type":"ContainerDied","Data":"f1a832e2f6fe841a3a93619900e9716bc01f3ec890d822bcc590edf8b761deae"}
Nov 03 19:52:54.770433 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:54.770360    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt" event={"ID":"5c2fc702-9338-4af0-a044-810fafccbfad","Type":"ContainerDied","Data":"2bb4aa349d5304dba35ddf1710fd50959efd89f9f7164dbe4f04c94f3a681ee8"}
Nov 03 19:52:54.770433 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:54.770388    4921 scope.go:122] "RemoveContainer" containerID="f1a832e2f6fe841a3a93619900e9716bc01f3ec890d822bcc590edf8b761deae"
Nov 03 19:52:54.770642 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:54.770502    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt"
Nov 03 19:52:54.790405 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:54.790365    4921 scope.go:122] "RemoveContainer" containerID="f1a832e2f6fe841a3a93619900e9716bc01f3ec890d822bcc590edf8b761deae"
Nov 03 19:52:54.791919 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:52:54.791879    4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1a832e2f6fe841a3a93619900e9716bc01f3ec890d822bcc590edf8b761deae\": container with ID starting with f1a832e2f6fe841a3a93619900e9716bc01f3ec890d822bcc590edf8b761deae not found: ID does not exist" containerID="f1a832e2f6fe841a3a93619900e9716bc01f3ec890d822bcc590edf8b761deae"
Nov 03 19:52:54.792012 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:54.791928    4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1a832e2f6fe841a3a93619900e9716bc01f3ec890d822bcc590edf8b761deae"} err="failed to get container status \"f1a832e2f6fe841a3a93619900e9716bc01f3ec890d822bcc590edf8b761deae\": rpc error: code = NotFound desc = could not find container \"f1a832e2f6fe841a3a93619900e9716bc01f3ec890d822bcc590edf8b761deae\": container with ID starting with f1a832e2f6fe841a3a93619900e9716bc01f3ec890d822bcc590edf8b761deae not found: ID does not exist"
Nov 03 19:52:54.874745 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:54.865289    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c2fc702-9338-4af0-a044-810fafccbfad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c2fc702-9338-4af0-a044-810fafccbfad-istiod-ca-cert\") pod \"5c2fc702-9338-4af0-a044-810fafccbfad\" (UID: \"5c2fc702-9338-4af0-a044-810fafccbfad\") "
Nov 03 19:52:54.874745 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:54.865486    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c2fc702-9338-4af0-a044-810fafccbfad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c2fc702-9338-4af0-a044-810fafccbfad-istio-envoy\") pod \"5c2fc702-9338-4af0-a044-810fafccbfad\" (UID: \"5c2fc702-9338-4af0-a044-810fafccbfad\") "
Nov 03 19:52:54.874745 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:54.865559    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c2fc702-9338-4af0-a044-810fafccbfad-istio-token\" (UniqueName: \"kubernetes.io/projected/5c2fc702-9338-4af0-a044-810fafccbfad-istio-token\") pod \"5c2fc702-9338-4af0-a044-810fafccbfad\" (UID: \"5c2fc702-9338-4af0-a044-810fafccbfad\") "
Nov 03 19:52:54.874745 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:54.865649    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c2fc702-9338-4af0-a044-810fafccbfad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c2fc702-9338-4af0-a044-810fafccbfad-workload-certs\") pod \"5c2fc702-9338-4af0-a044-810fafccbfad\" (UID: \"5c2fc702-9338-4af0-a044-810fafccbfad\") "
Nov 03 19:52:54.874745 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:54.865705    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c2fc702-9338-4af0-a044-810fafccbfad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c2fc702-9338-4af0-a044-810fafccbfad-workload-socket\") pod \"5c2fc702-9338-4af0-a044-810fafccbfad\" (UID: \"5c2fc702-9338-4af0-a044-810fafccbfad\") "
Nov 03 19:52:54.874745 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:54.865815    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c2fc702-9338-4af0-a044-810fafccbfad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c2fc702-9338-4af0-a044-810fafccbfad-istio-data\") pod \"5c2fc702-9338-4af0-a044-810fafccbfad\" (UID: \"5c2fc702-9338-4af0-a044-810fafccbfad\") "
Nov 03 19:52:54.874745 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:54.865860    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c2fc702-9338-4af0-a044-810fafccbfad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c2fc702-9338-4af0-a044-810fafccbfad-credential-socket\") pod \"5c2fc702-9338-4af0-a044-810fafccbfad\" (UID: \"5c2fc702-9338-4af0-a044-810fafccbfad\") "
Nov 03 19:52:54.874745 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:54.866000    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5c2fc702-9338-4af0-a044-810fafccbfad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c2fc702-9338-4af0-a044-810fafccbfad-istio-podinfo\") pod \"5c2fc702-9338-4af0-a044-810fafccbfad\" (UID: \"5c2fc702-9338-4af0-a044-810fafccbfad\") "
Nov 03 19:52:54.874745 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:54.866043    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c2fc702-9338-4af0-a044-810fafccbfad-kube-api-access-czgdq\" (UniqueName: \"kubernetes.io/projected/5c2fc702-9338-4af0-a044-810fafccbfad-kube-api-access-czgdq\") pod \"5c2fc702-9338-4af0-a044-810fafccbfad\" (UID: \"5c2fc702-9338-4af0-a044-810fafccbfad\") "
Nov 03 19:52:54.874745 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:54.866328    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c2fc702-9338-4af0-a044-810fafccbfad-istiod-ca-cert" pod "5c2fc702-9338-4af0-a044-810fafccbfad" (UID: "5c2fc702-9338-4af0-a044-810fafccbfad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:52:54.874745 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:54.867574    4921 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c2fc702-9338-4af0-a044-810fafccbfad-istiod-ca-cert\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:54.874745 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:54.868076    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c2fc702-9338-4af0-a044-810fafccbfad-workload-socket" pod "5c2fc702-9338-4af0-a044-810fafccbfad" (UID: "5c2fc702-9338-4af0-a044-810fafccbfad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:55.052539 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:55.039986    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c2fc702-9338-4af0-a044-810fafccbfad-istio-data" pod "5c2fc702-9338-4af0-a044-810fafccbfad" (UID: "5c2fc702-9338-4af0-a044-810fafccbfad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:55.052539 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:55.045316    4921 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c2fc702-9338-4af0-a044-810fafccbfad-workload-socket\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:55.052539 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:55.045591    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c2fc702-9338-4af0-a044-810fafccbfad-credential-socket" pod "5c2fc702-9338-4af0-a044-810fafccbfad" (UID: "5c2fc702-9338-4af0-a044-810fafccbfad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:55.070691 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:55.067411    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c2fc702-9338-4af0-a044-810fafccbfad-workload-certs" pod "5c2fc702-9338-4af0-a044-810fafccbfad" (UID: "5c2fc702-9338-4af0-a044-810fafccbfad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:55.096896 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:55.096854    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c2fc702-9338-4af0-a044-810fafccbfad-istio-token" pod "5c2fc702-9338-4af0-a044-810fafccbfad" (UID: "5c2fc702-9338-4af0-a044-810fafccbfad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:52:55.116738 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:55.108744    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c2fc702-9338-4af0-a044-810fafccbfad-istio-envoy" pod "5c2fc702-9338-4af0-a044-810fafccbfad" (UID: "5c2fc702-9338-4af0-a044-810fafccbfad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:55.117416 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:55.117377    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c2fc702-9338-4af0-a044-810fafccbfad-kube-api-access-czgdq" pod "5c2fc702-9338-4af0-a044-810fafccbfad" (UID: "5c2fc702-9338-4af0-a044-810fafccbfad"). InnerVolumeSpecName "kube-api-access-czgdq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:52:55.118270 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:55.118060    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c2fc702-9338-4af0-a044-810fafccbfad-istio-podinfo" pod "5c2fc702-9338-4af0-a044-810fafccbfad" (UID: "5c2fc702-9338-4af0-a044-810fafccbfad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:52:55.148747 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:55.148718    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czgdq\" (UniqueName: \"kubernetes.io/projected/5c2fc702-9338-4af0-a044-810fafccbfad-kube-api-access-czgdq\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:55.148947 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:55.148931    4921 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c2fc702-9338-4af0-a044-810fafccbfad-istio-envoy\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:55.149039 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:55.149027    4921 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c2fc702-9338-4af0-a044-810fafccbfad-istio-token\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:55.149117 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:55.149107    4921 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c2fc702-9338-4af0-a044-810fafccbfad-workload-certs\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:55.149196 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:55.149186    4921 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c2fc702-9338-4af0-a044-810fafccbfad-istio-data\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:55.149298 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:55.149286    4921 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c2fc702-9338-4af0-a044-810fafccbfad-credential-socket\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:55.149383 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:55.149372    4921 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c2fc702-9338-4af0-a044-810fafccbfad-istio-podinfo\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:55.467567 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:55.464264    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt"]
Nov 03 19:52:55.470939 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:55.469340    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-zlfkt"]
Nov 03 19:52:56.890782 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:56.890741    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c2fc702-9338-4af0-a044-810fafccbfad" path="/var/lib/kubelet/pods/5c2fc702-9338-4af0-a044-810fafccbfad/volumes"
Nov 03 19:52:59.379691 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:59.379328    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-2b73c810-798bdd757f-8k9lr" podUID="9070f15b-8ccc-4ea1-812e-82ea26e0a09e" containerName="payload-processing" containerID="cri-o://860728971f000259874fe7d11dc3ecbd8122abd67587c03a79a08cedcd491440" gracePeriod=25
Nov 03 19:52:59.754685 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:59.754646    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2b73c810-798bdd757f-8k9lr"
Nov 03 19:52:59.817825 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:59.817795    4921 generic.go:356] "Generic (PLEG): container finished" podID="9070f15b-8ccc-4ea1-812e-82ea26e0a09e" containerID="860728971f000259874fe7d11dc3ecbd8122abd67587c03a79a08cedcd491440" exitCode=0
Nov 03 19:52:59.818094 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:59.817859    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2b73c810-798bdd757f-8k9lr"
Nov 03 19:52:59.818094 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:59.817891    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2b73c810-798bdd757f-8k9lr" event={"ID":"9070f15b-8ccc-4ea1-812e-82ea26e0a09e","Type":"ContainerDied","Data":"860728971f000259874fe7d11dc3ecbd8122abd67587c03a79a08cedcd491440"}
Nov 03 19:52:59.818094 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:59.817979    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2b73c810-798bdd757f-8k9lr" event={"ID":"9070f15b-8ccc-4ea1-812e-82ea26e0a09e","Type":"ContainerDied","Data":"d559c9281c39087c4f982aa0bbfde28e08cc2fc1508456b3206320bba3d55fa0"}
Nov 03 19:52:59.818094 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:59.818006    4921 scope.go:122] "RemoveContainer" containerID="860728971f000259874fe7d11dc3ecbd8122abd67587c03a79a08cedcd491440"
Nov 03 19:52:59.820643 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:59.820607    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9070f15b-8ccc-4ea1-812e-82ea26e0a09e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9070f15b-8ccc-4ea1-812e-82ea26e0a09e-plugins-config-volume\") pod \"9070f15b-8ccc-4ea1-812e-82ea26e0a09e\" (UID: \"9070f15b-8ccc-4ea1-812e-82ea26e0a09e\") "
Nov 03 19:52:59.820907 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:59.820886    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9070f15b-8ccc-4ea1-812e-82ea26e0a09e-kube-api-access-222zl\" (UniqueName: \"kubernetes.io/projected/9070f15b-8ccc-4ea1-812e-82ea26e0a09e-kube-api-access-222zl\") pod \"9070f15b-8ccc-4ea1-812e-82ea26e0a09e\" (UID: \"9070f15b-8ccc-4ea1-812e-82ea26e0a09e\") "
Nov 03 19:52:59.821190 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:59.821159    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9070f15b-8ccc-4ea1-812e-82ea26e0a09e-plugins-config-volume" pod "9070f15b-8ccc-4ea1-812e-82ea26e0a09e" (UID: "9070f15b-8ccc-4ea1-812e-82ea26e0a09e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:52:59.821744 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:59.821718    4921 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9070f15b-8ccc-4ea1-812e-82ea26e0a09e-plugins-config-volume\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:59.823383 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:59.823359    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9070f15b-8ccc-4ea1-812e-82ea26e0a09e-kube-api-access-222zl" pod "9070f15b-8ccc-4ea1-812e-82ea26e0a09e" (UID: "9070f15b-8ccc-4ea1-812e-82ea26e0a09e"). InnerVolumeSpecName "kube-api-access-222zl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:52:59.924863 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:59.924825    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-222zl\" (UniqueName: \"kubernetes.io/projected/9070f15b-8ccc-4ea1-812e-82ea26e0a09e-kube-api-access-222zl\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:59.941240 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:59.941205    4921 scope.go:122] "RemoveContainer" containerID="860728971f000259874fe7d11dc3ecbd8122abd67587c03a79a08cedcd491440"
Nov 03 19:52:59.941597 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:52:59.941572    4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"860728971f000259874fe7d11dc3ecbd8122abd67587c03a79a08cedcd491440\": container with ID starting with 860728971f000259874fe7d11dc3ecbd8122abd67587c03a79a08cedcd491440 not found: ID does not exist" containerID="860728971f000259874fe7d11dc3ecbd8122abd67587c03a79a08cedcd491440"
Nov 03 19:52:59.941748 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:52:59.941609    4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"860728971f000259874fe7d11dc3ecbd8122abd67587c03a79a08cedcd491440"} err="failed to get container status \"860728971f000259874fe7d11dc3ecbd8122abd67587c03a79a08cedcd491440\": rpc error: code = NotFound desc = could not find container \"860728971f000259874fe7d11dc3ecbd8122abd67587c03a79a08cedcd491440\": container with ID starting with 860728971f000259874fe7d11dc3ecbd8122abd67587c03a79a08cedcd491440 not found: ID does not exist"
Nov 03 19:53:00.143237 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:00.143195    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2b73c810-798bdd757f-8k9lr"]
Nov 03 19:53:00.147797 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:00.147769    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2b73c810-798bdd757f-8k9lr"]
Nov 03 19:53:00.955587 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:00.955096    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9070f15b-8ccc-4ea1-812e-82ea26e0a09e" path="/var/lib/kubelet/pods/9070f15b-8ccc-4ea1-812e-82ea26e0a09e/volumes"
Nov 03 19:53:02.923807 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:53:02.923773    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 19:53:03.597049 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:03.597009    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5"]
Nov 03 19:53:03.597740 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:03.597703    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5" podUID="0da622a1-45f9-4311-b668-d27eb7ec2e43" containerName="istio-proxy" containerID="cri-o://133dcce05548df4da2f2b1b43cf49f4bc0e88577b8ee7203c5eeafa476f93aea" gracePeriod=30
Nov 03 19:53:07.317083 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:07.316996    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bdb694-57f5479559-fv5m8"]
Nov 03 19:53:07.318142 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:07.318119    4921 cpu_manager.go:414] "containerMap: removing container" podUID="9070f15b-8ccc-4ea1-812e-82ea26e0a09e" containerName="payload-processing"
Nov 03 19:53:07.318142 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:07.318142    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9070f15b-8ccc-4ea1-812e-82ea26e0a09e" containerName="payload-processing"
Nov 03 19:53:07.318334 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:07.318179    4921 cpu_manager.go:414] "containerMap: removing container" podUID="bb482f3b-80c8-4ba4-b25b-02d9fa97ef3d" containerName="revoke-keys"
Nov 03 19:53:07.318334 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:07.318188    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb482f3b-80c8-4ba4-b25b-02d9fa97ef3d" containerName="revoke-keys"
Nov 03 19:53:07.318334 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:07.318231    4921 cpu_manager.go:414] "containerMap: removing container" podUID="5c2fc702-9338-4af0-a044-810fafccbfad" containerName="istio-proxy"
Nov 03 19:53:07.318334 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:07.318238    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c2fc702-9338-4af0-a044-810fafccbfad" containerName="istio-proxy"
Nov 03 19:53:07.318540 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:07.318391    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb482f3b-80c8-4ba4-b25b-02d9fa97ef3d" containerName="revoke-keys"
Nov 03 19:53:07.318540 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:07.318407    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c2fc702-9338-4af0-a044-810fafccbfad" containerName="istio-proxy"
Nov 03 19:53:07.318540 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:07.318424    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="9070f15b-8ccc-4ea1-812e-82ea26e0a09e" containerName="payload-processing"
Nov 03 19:53:07.351429 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:07.351390    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bdb694-57f5479559-fv5m8"]
Nov 03 19:53:07.351611 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:07.351498    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bdb694-57f5479559-fv5m8"
Nov 03 19:53:07.354815 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:07.354788    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-bdb694-serving-cert\""
Nov 03 19:53:07.444866 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:07.444832    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e75a177a-f6a6-49c7-87a2-297154d3b7f3-openshift-service-ca\") pod \"maas-api-e2e-inf-a-bdb694-57f5479559-fv5m8\" (UID: \"e75a177a-f6a6-49c7-87a2-297154d3b7f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bdb694-57f5479559-fv5m8"
Nov 03 19:53:07.444866 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:07.444878    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh8k7\" (UniqueName: \"kubernetes.io/projected/e75a177a-f6a6-49c7-87a2-297154d3b7f3-kube-api-access-lh8k7\") pod \"maas-api-e2e-inf-a-bdb694-57f5479559-fv5m8\" (UID: \"e75a177a-f6a6-49c7-87a2-297154d3b7f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bdb694-57f5479559-fv5m8"
Nov 03 19:53:07.445082 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:07.444996    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e75a177a-f6a6-49c7-87a2-297154d3b7f3-metrics-certs\") pod \"maas-api-e2e-inf-a-bdb694-57f5479559-fv5m8\" (UID: \"e75a177a-f6a6-49c7-87a2-297154d3b7f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bdb694-57f5479559-fv5m8"
Nov 03 19:53:07.445170 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:07.445154    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e75a177a-f6a6-49c7-87a2-297154d3b7f3-maas-api-tls\") pod \"maas-api-e2e-inf-a-bdb694-57f5479559-fv5m8\" (UID: \"e75a177a-f6a6-49c7-87a2-297154d3b7f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bdb694-57f5479559-fv5m8"
Nov 03 19:53:07.547475 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:07.547431    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e75a177a-f6a6-49c7-87a2-297154d3b7f3-openshift-service-ca\") pod \"maas-api-e2e-inf-a-bdb694-57f5479559-fv5m8\" (UID: \"e75a177a-f6a6-49c7-87a2-297154d3b7f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bdb694-57f5479559-fv5m8"
Nov 03 19:53:07.548638 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:07.547774    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh8k7\" (UniqueName: \"kubernetes.io/projected/e75a177a-f6a6-49c7-87a2-297154d3b7f3-kube-api-access-lh8k7\") pod \"maas-api-e2e-inf-a-bdb694-57f5479559-fv5m8\" (UID: \"e75a177a-f6a6-49c7-87a2-297154d3b7f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bdb694-57f5479559-fv5m8"
Nov 03 19:53:07.548638 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:07.547882    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e75a177a-f6a6-49c7-87a2-297154d3b7f3-metrics-certs\") pod \"maas-api-e2e-inf-a-bdb694-57f5479559-fv5m8\" (UID: \"e75a177a-f6a6-49c7-87a2-297154d3b7f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bdb694-57f5479559-fv5m8"
Nov 03 19:53:07.548638 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:07.548065    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e75a177a-f6a6-49c7-87a2-297154d3b7f3-maas-api-tls\") pod \"maas-api-e2e-inf-a-bdb694-57f5479559-fv5m8\" (UID: \"e75a177a-f6a6-49c7-87a2-297154d3b7f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bdb694-57f5479559-fv5m8"
Nov 03 19:53:07.548990 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:07.548917    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e75a177a-f6a6-49c7-87a2-297154d3b7f3-openshift-service-ca\") pod \"maas-api-e2e-inf-a-bdb694-57f5479559-fv5m8\" (UID: \"e75a177a-f6a6-49c7-87a2-297154d3b7f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bdb694-57f5479559-fv5m8"
Nov 03 19:53:07.581738 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:07.559280    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e75a177a-f6a6-49c7-87a2-297154d3b7f3-maas-api-tls\") pod \"maas-api-e2e-inf-a-bdb694-57f5479559-fv5m8\" (UID: \"e75a177a-f6a6-49c7-87a2-297154d3b7f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bdb694-57f5479559-fv5m8"
Nov 03 19:53:07.587255 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:07.585579    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh8k7\" (UniqueName: \"kubernetes.io/projected/e75a177a-f6a6-49c7-87a2-297154d3b7f3-kube-api-access-lh8k7\") pod \"maas-api-e2e-inf-a-bdb694-57f5479559-fv5m8\" (UID: \"e75a177a-f6a6-49c7-87a2-297154d3b7f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bdb694-57f5479559-fv5m8"
Nov 03 19:53:07.595720 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:07.587895    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e75a177a-f6a6-49c7-87a2-297154d3b7f3-metrics-certs\") pod \"maas-api-e2e-inf-a-bdb694-57f5479559-fv5m8\" (UID: \"e75a177a-f6a6-49c7-87a2-297154d3b7f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bdb694-57f5479559-fv5m8"
Nov 03 19:53:07.672716 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:07.668977    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bdb694-57f5479559-fv5m8"
Nov 03 19:53:08.150718 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:08.123090    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bdb694-57f5479559-fv5m8"]
Nov 03 19:53:08.920828 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:08.920806    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5"
Nov 03 19:53:08.950031 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:08.949997    4921 generic.go:356] "Generic (PLEG): container finished" podID="0da622a1-45f9-4311-b668-d27eb7ec2e43" containerID="133dcce05548df4da2f2b1b43cf49f4bc0e88577b8ee7203c5eeafa476f93aea" exitCode=0
Nov 03 19:53:08.950319 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:08.950295    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5" event={"ID":"0da622a1-45f9-4311-b668-d27eb7ec2e43","Type":"ContainerDied","Data":"133dcce05548df4da2f2b1b43cf49f4bc0e88577b8ee7203c5eeafa476f93aea"}
Nov 03 19:53:08.950385 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:08.950346    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5" event={"ID":"0da622a1-45f9-4311-b668-d27eb7ec2e43","Type":"ContainerDied","Data":"7417e896df1c434063e535487a6305e74f48a6a9ff980793855164f79558d703"}
Nov 03 19:53:08.950385 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:08.950368    4921 scope.go:122] "RemoveContainer" containerID="133dcce05548df4da2f2b1b43cf49f4bc0e88577b8ee7203c5eeafa476f93aea"
Nov 03 19:53:08.950503 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:08.950492    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5"
Nov 03 19:53:08.953252 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:08.953224    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bdb694-57f5479559-fv5m8" event={"ID":"e75a177a-f6a6-49c7-87a2-297154d3b7f3","Type":"ContainerStarted","Data":"98ee120bff9dc8970104391f9853b68b175605463c2d423f4f2bc857e416c072"}
Nov 03 19:53:08.953402 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:08.953277    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bdb694-57f5479559-fv5m8" event={"ID":"e75a177a-f6a6-49c7-87a2-297154d3b7f3","Type":"ContainerStarted","Data":"16112ef31c12ef818658d1e813c51e43b958e35b95fe0fe12fc32c8b0403da9d"}
Nov 03 19:53:08.954213 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:08.954189    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bdb694-57f5479559-fv5m8"
Nov 03 19:53:08.969134 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:08.969110    4921 scope.go:122] "RemoveContainer" containerID="133dcce05548df4da2f2b1b43cf49f4bc0e88577b8ee7203c5eeafa476f93aea"
Nov 03 19:53:08.969516 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:53:08.969493    4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"133dcce05548df4da2f2b1b43cf49f4bc0e88577b8ee7203c5eeafa476f93aea\": container with ID starting with 133dcce05548df4da2f2b1b43cf49f4bc0e88577b8ee7203c5eeafa476f93aea not found: ID does not exist" containerID="133dcce05548df4da2f2b1b43cf49f4bc0e88577b8ee7203c5eeafa476f93aea"
Nov 03 19:53:08.969608 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:08.969535    4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"133dcce05548df4da2f2b1b43cf49f4bc0e88577b8ee7203c5eeafa476f93aea"} err="failed to get container status \"133dcce05548df4da2f2b1b43cf49f4bc0e88577b8ee7203c5eeafa476f93aea\": rpc error: code = NotFound desc = could not find container \"133dcce05548df4da2f2b1b43cf49f4bc0e88577b8ee7203c5eeafa476f93aea\": container with ID starting with 133dcce05548df4da2f2b1b43cf49f4bc0e88577b8ee7203c5eeafa476f93aea not found: ID does not exist"
Nov 03 19:53:08.975213 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:08.975171    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bdb694-57f5479559-fv5m8" podStartSLOduration=1.484581425 podStartE2EDuration="1.975156817s" totalImagesPullingTime="490.575392ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:53:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:53:08.970810166 +0000 UTC m=+1302.214775633" watchObservedRunningTime="2026-11-03 19:53:08.975156817 +0000 UTC m=+1302.219122314"
Nov 03 19:53:08.989190 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:08.989159    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0da622a1-45f9-4311-b668-d27eb7ec2e43-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0da622a1-45f9-4311-b668-d27eb7ec2e43-istio-data\") pod \"0da622a1-45f9-4311-b668-d27eb7ec2e43\" (UID: \"0da622a1-45f9-4311-b668-d27eb7ec2e43\") "
Nov 03 19:53:08.989547 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:08.989522    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0da622a1-45f9-4311-b668-d27eb7ec2e43-kube-api-access-z5r7m\" (UniqueName: \"kubernetes.io/projected/0da622a1-45f9-4311-b668-d27eb7ec2e43-kube-api-access-z5r7m\") pod \"0da622a1-45f9-4311-b668-d27eb7ec2e43\" (UID: \"0da622a1-45f9-4311-b668-d27eb7ec2e43\") "
Nov 03 19:53:08.989836 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:08.989811    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da622a1-45f9-4311-b668-d27eb7ec2e43-istio-data" pod "0da622a1-45f9-4311-b668-d27eb7ec2e43" (UID: "0da622a1-45f9-4311-b668-d27eb7ec2e43"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:08.989991 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:08.989960    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0da622a1-45f9-4311-b668-d27eb7ec2e43-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0da622a1-45f9-4311-b668-d27eb7ec2e43-istio-podinfo\") pod \"0da622a1-45f9-4311-b668-d27eb7ec2e43\" (UID: \"0da622a1-45f9-4311-b668-d27eb7ec2e43\") "
Nov 03 19:53:08.990102 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:08.990045    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0da622a1-45f9-4311-b668-d27eb7ec2e43-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0da622a1-45f9-4311-b668-d27eb7ec2e43-istiod-ca-cert\") pod \"0da622a1-45f9-4311-b668-d27eb7ec2e43\" (UID: \"0da622a1-45f9-4311-b668-d27eb7ec2e43\") "
Nov 03 19:53:08.990102 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:08.990080    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0da622a1-45f9-4311-b668-d27eb7ec2e43-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0da622a1-45f9-4311-b668-d27eb7ec2e43-workload-certs\") pod \"0da622a1-45f9-4311-b668-d27eb7ec2e43\" (UID: \"0da622a1-45f9-4311-b668-d27eb7ec2e43\") "
Nov 03 19:53:08.990213 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:08.990120    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0da622a1-45f9-4311-b668-d27eb7ec2e43-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da622a1-45f9-4311-b668-d27eb7ec2e43-workload-socket\") pod \"0da622a1-45f9-4311-b668-d27eb7ec2e43\" (UID: \"0da622a1-45f9-4311-b668-d27eb7ec2e43\") "
Nov 03 19:53:08.990213 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:08.990162    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0da622a1-45f9-4311-b668-d27eb7ec2e43-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0da622a1-45f9-4311-b668-d27eb7ec2e43-istio-envoy\") pod \"0da622a1-45f9-4311-b668-d27eb7ec2e43\" (UID: \"0da622a1-45f9-4311-b668-d27eb7ec2e43\") "
Nov 03 19:53:08.990213 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:08.990208    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0da622a1-45f9-4311-b668-d27eb7ec2e43-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da622a1-45f9-4311-b668-d27eb7ec2e43-credential-socket\") pod \"0da622a1-45f9-4311-b668-d27eb7ec2e43\" (UID: \"0da622a1-45f9-4311-b668-d27eb7ec2e43\") "
Nov 03 19:53:08.990376 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:08.990261    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0da622a1-45f9-4311-b668-d27eb7ec2e43-istio-token\" (UniqueName: \"kubernetes.io/projected/0da622a1-45f9-4311-b668-d27eb7ec2e43-istio-token\") pod \"0da622a1-45f9-4311-b668-d27eb7ec2e43\" (UID: \"0da622a1-45f9-4311-b668-d27eb7ec2e43\") "
Nov 03 19:53:08.991181 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:08.991157    4921 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0da622a1-45f9-4311-b668-d27eb7ec2e43-istio-data\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:08.995907 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:08.992204    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da622a1-45f9-4311-b668-d27eb7ec2e43-workload-socket" pod "0da622a1-45f9-4311-b668-d27eb7ec2e43" (UID: "0da622a1-45f9-4311-b668-d27eb7ec2e43"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:08.995907 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:08.992613    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0da622a1-45f9-4311-b668-d27eb7ec2e43-kube-api-access-z5r7m" pod "0da622a1-45f9-4311-b668-d27eb7ec2e43" (UID: "0da622a1-45f9-4311-b668-d27eb7ec2e43"). InnerVolumeSpecName "kube-api-access-z5r7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:53:08.995907 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:08.993274    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da622a1-45f9-4311-b668-d27eb7ec2e43-credential-socket" pod "0da622a1-45f9-4311-b668-d27eb7ec2e43" (UID: "0da622a1-45f9-4311-b668-d27eb7ec2e43"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:08.995907 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:08.993411    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da622a1-45f9-4311-b668-d27eb7ec2e43-workload-certs" pod "0da622a1-45f9-4311-b668-d27eb7ec2e43" (UID: "0da622a1-45f9-4311-b668-d27eb7ec2e43"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:08.995907 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:08.995194    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da622a1-45f9-4311-b668-d27eb7ec2e43-istio-envoy" pod "0da622a1-45f9-4311-b668-d27eb7ec2e43" (UID: "0da622a1-45f9-4311-b668-d27eb7ec2e43"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:08.995907 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:08.995427    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0da622a1-45f9-4311-b668-d27eb7ec2e43-istio-podinfo" pod "0da622a1-45f9-4311-b668-d27eb7ec2e43" (UID: "0da622a1-45f9-4311-b668-d27eb7ec2e43"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:53:08.995907 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:08.995871    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0da622a1-45f9-4311-b668-d27eb7ec2e43-istiod-ca-cert" pod "0da622a1-45f9-4311-b668-d27eb7ec2e43" (UID: "0da622a1-45f9-4311-b668-d27eb7ec2e43"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:53:08.998188 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:08.998163    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0da622a1-45f9-4311-b668-d27eb7ec2e43-istio-token" pod "0da622a1-45f9-4311-b668-d27eb7ec2e43" (UID: "0da622a1-45f9-4311-b668-d27eb7ec2e43"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:53:09.106697 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:09.094129    4921 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0da622a1-45f9-4311-b668-d27eb7ec2e43-istio-token\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.106697 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:09.094172    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5r7m\" (UniqueName: \"kubernetes.io/projected/0da622a1-45f9-4311-b668-d27eb7ec2e43-kube-api-access-z5r7m\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.106697 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:09.094188    4921 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0da622a1-45f9-4311-b668-d27eb7ec2e43-istio-podinfo\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.106697 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:09.094202    4921 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0da622a1-45f9-4311-b668-d27eb7ec2e43-istiod-ca-cert\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.106697 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:09.094218    4921 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0da622a1-45f9-4311-b668-d27eb7ec2e43-workload-certs\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.106697 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:09.094231    4921 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da622a1-45f9-4311-b668-d27eb7ec2e43-workload-socket\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.106697 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:09.094244    4921 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0da622a1-45f9-4311-b668-d27eb7ec2e43-istio-envoy\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.106697 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:09.094263    4921 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da622a1-45f9-4311-b668-d27eb7ec2e43-credential-socket\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:09.284704 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:09.284499    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5"]
Nov 03 19:53:09.293068 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:09.293028    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-6cwn5"]
Nov 03 19:53:10.936702 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:10.934628    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0da622a1-45f9-4311-b668-d27eb7ec2e43" path="/var/lib/kubelet/pods/0da622a1-45f9-4311-b668-d27eb7ec2e43/volumes"
Nov 03 19:53:13.017692 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:13.012943    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2b73c810-ptmpm"]
Nov 03 19:53:13.032346 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:13.032306    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2b73c810-ptmpm"]
Nov 03 19:53:14.891455 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:14.891423    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb482f3b-80c8-4ba4-b25b-02d9fa97ef3d" path="/var/lib/kubelet/pods/bb482f3b-80c8-4ba4-b25b-02d9fa97ef3d/volumes"
Nov 03 19:53:16.107689 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:16.104858    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bdb694-57f5479559-fv5m8"
Nov 03 19:53:18.794905 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:18.794872    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n"]
Nov 03 19:53:18.795318 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:18.795122    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n" podUID="4ee1b6ae-046b-4402-a36e-bbd523e73463" containerName="istio-proxy" containerID="cri-o://5491e2aed27e2d6b7ee87fbabfeb281b70e3021351b666975202e6f44dccf880" gracePeriod=30
Nov 03 19:53:19.195742 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:19.192921    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9f84de-867b56cbc5-5qmtd"]
Nov 03 19:53:19.201999 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:19.201588    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9f84de-678d8f9dc5-588n8"]
Nov 03 19:53:20.682692 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:20.682146    4921 patch_prober.go:29] interesting pod/e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 19:53:20.682692 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:20.682226    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n" podUID="4ee1b6ae-046b-4402-a36e-bbd523e73463" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 19:53:24.193945 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:24.193900    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-9f84de-867b56cbc5-5qmtd" podUID="3c547aaa-645d-4f0a-805c-885095934756" containerName="payload-processing" containerID="cri-o://a954e350ab3968e2b166df995f76fb5694d8c47172867e1ad7efa000390882d4" gracePeriod=25
Nov 03 19:53:24.203620 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:24.203584    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9f84de-678d8f9dc5-588n8" podUID="2ac3398b-799f-4287-abf7-bfca2392e357" containerName="payload-pre-processing" containerID="cri-o://b8b96a02c4351513550fe98a3d7a870c0329f93ad889682207e8050ccc7670d7" gracePeriod=25
Nov 03 19:53:24.327639 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:24.327605    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n"
Nov 03 19:53:24.396646 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:24.395927    4921 generic.go:356] "Generic (PLEG): container finished" podID="4ee1b6ae-046b-4402-a36e-bbd523e73463" containerID="5491e2aed27e2d6b7ee87fbabfeb281b70e3021351b666975202e6f44dccf880" exitCode=0
Nov 03 19:53:24.396646 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:24.396177    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ee1b6ae-046b-4402-a36e-bbd523e73463-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ee1b6ae-046b-4402-a36e-bbd523e73463-workload-certs\") pod \"4ee1b6ae-046b-4402-a36e-bbd523e73463\" (UID: \"4ee1b6ae-046b-4402-a36e-bbd523e73463\") "
Nov 03 19:53:24.396646 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:24.396278    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n" event={"ID":"4ee1b6ae-046b-4402-a36e-bbd523e73463","Type":"ContainerDied","Data":"5491e2aed27e2d6b7ee87fbabfeb281b70e3021351b666975202e6f44dccf880"}
Nov 03 19:53:24.396646 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:24.396333    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n" event={"ID":"4ee1b6ae-046b-4402-a36e-bbd523e73463","Type":"ContainerDied","Data":"102015d3a30ff0467aec55158078e1232799bd1db22e4898653afd0279ca4508"}
Nov 03 19:53:24.396646 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:24.396366    4921 scope.go:122] "RemoveContainer" containerID="5491e2aed27e2d6b7ee87fbabfeb281b70e3021351b666975202e6f44dccf880"
Nov 03 19:53:24.396646 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:24.396386    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ee1b6ae-046b-4402-a36e-bbd523e73463-kube-api-access-wdpzb\" (UniqueName: \"kubernetes.io/projected/4ee1b6ae-046b-4402-a36e-bbd523e73463-kube-api-access-wdpzb\") pod \"4ee1b6ae-046b-4402-a36e-bbd523e73463\" (UID: \"4ee1b6ae-046b-4402-a36e-bbd523e73463\") "
Nov 03 19:53:24.396646 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:24.396501    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ee1b6ae-046b-4402-a36e-bbd523e73463-istio-token\" (UniqueName: \"kubernetes.io/projected/4ee1b6ae-046b-4402-a36e-bbd523e73463-istio-token\") pod \"4ee1b6ae-046b-4402-a36e-bbd523e73463\" (UID: \"4ee1b6ae-046b-4402-a36e-bbd523e73463\") "
Nov 03 19:53:24.396646 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:24.396532    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n"
Nov 03 19:53:24.396646 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:24.396569    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ee1b6ae-046b-4402-a36e-bbd523e73463-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ee1b6ae-046b-4402-a36e-bbd523e73463-credential-socket\") pod \"4ee1b6ae-046b-4402-a36e-bbd523e73463\" (UID: \"4ee1b6ae-046b-4402-a36e-bbd523e73463\") "
Nov 03 19:53:24.396646 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:24.396625    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ee1b6ae-046b-4402-a36e-bbd523e73463-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ee1b6ae-046b-4402-a36e-bbd523e73463-istio-envoy\") pod \"4ee1b6ae-046b-4402-a36e-bbd523e73463\" (UID: \"4ee1b6ae-046b-4402-a36e-bbd523e73463\") "
Nov 03 19:53:24.397273 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:24.396686    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4ee1b6ae-046b-4402-a36e-bbd523e73463-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ee1b6ae-046b-4402-a36e-bbd523e73463-istio-podinfo\") pod \"4ee1b6ae-046b-4402-a36e-bbd523e73463\" (UID: \"4ee1b6ae-046b-4402-a36e-bbd523e73463\") "
Nov 03 19:53:24.397273 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:24.396722    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ee1b6ae-046b-4402-a36e-bbd523e73463-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ee1b6ae-046b-4402-a36e-bbd523e73463-workload-socket\") pod \"4ee1b6ae-046b-4402-a36e-bbd523e73463\" (UID: \"4ee1b6ae-046b-4402-a36e-bbd523e73463\") "
Nov 03 19:53:24.397273 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:24.396767    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ee1b6ae-046b-4402-a36e-bbd523e73463-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ee1b6ae-046b-4402-a36e-bbd523e73463-istio-data\") pod \"4ee1b6ae-046b-4402-a36e-bbd523e73463\" (UID: \"4ee1b6ae-046b-4402-a36e-bbd523e73463\") "
Nov 03 19:53:24.397273 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:24.396844    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ee1b6ae-046b-4402-a36e-bbd523e73463-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ee1b6ae-046b-4402-a36e-bbd523e73463-istiod-ca-cert\") pod \"4ee1b6ae-046b-4402-a36e-bbd523e73463\" (UID: \"4ee1b6ae-046b-4402-a36e-bbd523e73463\") "
Nov 03 19:53:24.397893 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:24.397799    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ee1b6ae-046b-4402-a36e-bbd523e73463-credential-socket" pod "4ee1b6ae-046b-4402-a36e-bbd523e73463" (UID: "4ee1b6ae-046b-4402-a36e-bbd523e73463"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:24.398224 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:24.398033    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ee1b6ae-046b-4402-a36e-bbd523e73463-istiod-ca-cert" pod "4ee1b6ae-046b-4402-a36e-bbd523e73463" (UID: "4ee1b6ae-046b-4402-a36e-bbd523e73463"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:53:24.398336 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:24.398309    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ee1b6ae-046b-4402-a36e-bbd523e73463-workload-socket" pod "4ee1b6ae-046b-4402-a36e-bbd523e73463" (UID: "4ee1b6ae-046b-4402-a36e-bbd523e73463"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:24.398869 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:24.398500    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ee1b6ae-046b-4402-a36e-bbd523e73463-workload-certs" pod "4ee1b6ae-046b-4402-a36e-bbd523e73463" (UID: "4ee1b6ae-046b-4402-a36e-bbd523e73463"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:24.398869 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:24.398587    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ee1b6ae-046b-4402-a36e-bbd523e73463-istio-data" pod "4ee1b6ae-046b-4402-a36e-bbd523e73463" (UID: "4ee1b6ae-046b-4402-a36e-bbd523e73463"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:24.409580 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:24.401179    4921 generic.go:356] "Generic (PLEG): container finished" podID="3c547aaa-645d-4f0a-805c-885095934756" containerID="a954e350ab3968e2b166df995f76fb5694d8c47172867e1ad7efa000390882d4" exitCode=0
Nov 03 19:53:24.409580 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:24.401280    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9f84de-867b56cbc5-5qmtd" event={"ID":"3c547aaa-645d-4f0a-805c-885095934756","Type":"ContainerDied","Data":"a954e350ab3968e2b166df995f76fb5694d8c47172867e1ad7efa000390882d4"}
Nov 03 19:53:24.416621 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:24.416235    4921 generic.go:356] "Generic (PLEG): container finished" podID="2ac3398b-799f-4287-abf7-bfca2392e357" containerID="b8b96a02c4351513550fe98a3d7a870c0329f93ad889682207e8050ccc7670d7" exitCode=0
Nov 03 19:53:24.416621 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:24.416293    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9f84de-678d8f9dc5-588n8" event={"ID":"2ac3398b-799f-4287-abf7-bfca2392e357","Type":"ContainerDied","Data":"b8b96a02c4351513550fe98a3d7a870c0329f93ad889682207e8050ccc7670d7"}
Nov 03 19:53:24.425555 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:24.425414    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4ee1b6ae-046b-4402-a36e-bbd523e73463-istio-podinfo" pod "4ee1b6ae-046b-4402-a36e-bbd523e73463" (UID: "4ee1b6ae-046b-4402-a36e-bbd523e73463"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:53:24.425912 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:24.425832    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ee1b6ae-046b-4402-a36e-bbd523e73463-istio-envoy" pod "4ee1b6ae-046b-4402-a36e-bbd523e73463" (UID: "4ee1b6ae-046b-4402-a36e-bbd523e73463"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:53:24.425912 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:24.425866    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ee1b6ae-046b-4402-a36e-bbd523e73463-kube-api-access-wdpzb" pod "4ee1b6ae-046b-4402-a36e-bbd523e73463" (UID: "4ee1b6ae-046b-4402-a36e-bbd523e73463"). InnerVolumeSpecName "kube-api-access-wdpzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:53:24.426074 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:24.425973    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ee1b6ae-046b-4402-a36e-bbd523e73463-istio-token" pod "4ee1b6ae-046b-4402-a36e-bbd523e73463" (UID: "4ee1b6ae-046b-4402-a36e-bbd523e73463"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:53:24.434629 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:24.434596    4921 scope.go:122] "RemoveContainer" containerID="5491e2aed27e2d6b7ee87fbabfeb281b70e3021351b666975202e6f44dccf880"
Nov 03 19:53:24.435615 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:53:24.435531    4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5491e2aed27e2d6b7ee87fbabfeb281b70e3021351b666975202e6f44dccf880\": container with ID starting with 5491e2aed27e2d6b7ee87fbabfeb281b70e3021351b666975202e6f44dccf880 not found: ID does not exist" containerID="5491e2aed27e2d6b7ee87fbabfeb281b70e3021351b666975202e6f44dccf880"
Nov 03 19:53:24.435615 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:24.435571    4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5491e2aed27e2d6b7ee87fbabfeb281b70e3021351b666975202e6f44dccf880"} err="failed to get container status \"5491e2aed27e2d6b7ee87fbabfeb281b70e3021351b666975202e6f44dccf880\": rpc error: code = NotFound desc = could not find container \"5491e2aed27e2d6b7ee87fbabfeb281b70e3021351b666975202e6f44dccf880\": container with ID starting with 5491e2aed27e2d6b7ee87fbabfeb281b70e3021351b666975202e6f44dccf880 not found: ID does not exist"
Nov 03 19:53:24.498820 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:24.498732    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdpzb\" (UniqueName: \"kubernetes.io/projected/4ee1b6ae-046b-4402-a36e-bbd523e73463-kube-api-access-wdpzb\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:24.498820 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:24.498766    4921 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ee1b6ae-046b-4402-a36e-bbd523e73463-istio-token\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:24.498820 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:24.498777    4921 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ee1b6ae-046b-4402-a36e-bbd523e73463-credential-socket\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:24.498820 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:24.498786    4921 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ee1b6ae-046b-4402-a36e-bbd523e73463-istio-envoy\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:24.498820 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:24.498798    4921 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ee1b6ae-046b-4402-a36e-bbd523e73463-istio-podinfo\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:24.498820 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:24.498807    4921 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ee1b6ae-046b-4402-a36e-bbd523e73463-workload-socket\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:24.498820 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:24.498816    4921 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ee1b6ae-046b-4402-a36e-bbd523e73463-istio-data\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:24.498820 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:24.498824    4921 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ee1b6ae-046b-4402-a36e-bbd523e73463-istiod-ca-cert\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:24.499179 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:24.498833    4921 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ee1b6ae-046b-4402-a36e-bbd523e73463-workload-certs\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:24.517885 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:24.517852    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9f84de-867b56cbc5-5qmtd"
Nov 03 19:53:24.584243 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:24.584213    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9f84de-678d8f9dc5-588n8"
Nov 03 19:53:24.599845 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:24.599802    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c547aaa-645d-4f0a-805c-885095934756-kube-api-access-n9zh7\" (UniqueName: \"kubernetes.io/projected/3c547aaa-645d-4f0a-805c-885095934756-kube-api-access-n9zh7\") pod \"3c547aaa-645d-4f0a-805c-885095934756\" (UID: \"3c547aaa-645d-4f0a-805c-885095934756\") "
Nov 03 19:53:24.600248 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:24.600213    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c547aaa-645d-4f0a-805c-885095934756-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c547aaa-645d-4f0a-805c-885095934756-plugins-config-volume\") pod \"3c547aaa-645d-4f0a-805c-885095934756\" (UID: \"3c547aaa-645d-4f0a-805c-885095934756\") "
Nov 03 19:53:24.601480 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:24.601456    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c547aaa-645d-4f0a-805c-885095934756-plugins-config-volume" pod "3c547aaa-645d-4f0a-805c-885095934756" (UID: "3c547aaa-645d-4f0a-805c-885095934756"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:53:24.607231 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:24.607207    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c547aaa-645d-4f0a-805c-885095934756-kube-api-access-n9zh7" pod "3c547aaa-645d-4f0a-805c-885095934756" (UID: "3c547aaa-645d-4f0a-805c-885095934756"). InnerVolumeSpecName "kube-api-access-n9zh7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:53:24.702130 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:24.702040    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ac3398b-799f-4287-abf7-bfca2392e357-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ac3398b-799f-4287-abf7-bfca2392e357-plugins-config-volume\") pod \"2ac3398b-799f-4287-abf7-bfca2392e357\" (UID: \"2ac3398b-799f-4287-abf7-bfca2392e357\") "
Nov 03 19:53:24.702130 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:24.702116    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ac3398b-799f-4287-abf7-bfca2392e357-kube-api-access-zn8gf\" (UniqueName: \"kubernetes.io/projected/2ac3398b-799f-4287-abf7-bfca2392e357-kube-api-access-zn8gf\") pod \"2ac3398b-799f-4287-abf7-bfca2392e357\" (UID: \"2ac3398b-799f-4287-abf7-bfca2392e357\") "
Nov 03 19:53:24.702474 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:24.702446    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ac3398b-799f-4287-abf7-bfca2392e357-plugins-config-volume" pod "2ac3398b-799f-4287-abf7-bfca2392e357" (UID: "2ac3398b-799f-4287-abf7-bfca2392e357"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:53:24.702762 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:24.702732    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9zh7\" (UniqueName: \"kubernetes.io/projected/3c547aaa-645d-4f0a-805c-885095934756-kube-api-access-n9zh7\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:24.702762 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:24.702758    4921 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c547aaa-645d-4f0a-805c-885095934756-plugins-config-volume\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:24.702882 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:24.702772    4921 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ac3398b-799f-4287-abf7-bfca2392e357-plugins-config-volume\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:24.704428 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:24.704404    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ac3398b-799f-4287-abf7-bfca2392e357-kube-api-access-zn8gf" pod "2ac3398b-799f-4287-abf7-bfca2392e357" (UID: "2ac3398b-799f-4287-abf7-bfca2392e357"). InnerVolumeSpecName "kube-api-access-zn8gf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:53:24.721866 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:24.721835    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n"]
Nov 03 19:53:24.729784 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:24.729750    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-t729n"]
Nov 03 19:53:24.808590 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:24.808557    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zn8gf\" (UniqueName: \"kubernetes.io/projected/2ac3398b-799f-4287-abf7-bfca2392e357-kube-api-access-zn8gf\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:24.891637 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:24.891594    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ee1b6ae-046b-4402-a36e-bbd523e73463" path="/var/lib/kubelet/pods/4ee1b6ae-046b-4402-a36e-bbd523e73463/volumes"
Nov 03 19:53:25.426703 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:25.426638    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9f84de-867b56cbc5-5qmtd" event={"ID":"3c547aaa-645d-4f0a-805c-885095934756","Type":"ContainerDied","Data":"32f3da5755e60dd3408060a55acfac58884e87e0307ca8a122a76db53e9a77cf"}
Nov 03 19:53:25.427175 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:25.426641    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9f84de-867b56cbc5-5qmtd"
Nov 03 19:53:25.427175 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:25.426833    4921 scope.go:122] "RemoveContainer" containerID="a954e350ab3968e2b166df995f76fb5694d8c47172867e1ad7efa000390882d4"
Nov 03 19:53:25.428292 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:25.428168    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9f84de-678d8f9dc5-588n8" event={"ID":"2ac3398b-799f-4287-abf7-bfca2392e357","Type":"ContainerDied","Data":"1fd9561fa2a27f39c6e4e84cb492f27ef99d2debe7e3513b8f1f6f8f6196be11"}
Nov 03 19:53:25.428292 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:25.428204    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9f84de-678d8f9dc5-588n8"
Nov 03 19:53:25.440333 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:25.440308    4921 scope.go:122] "RemoveContainer" containerID="b8b96a02c4351513550fe98a3d7a870c0329f93ad889682207e8050ccc7670d7"
Nov 03 19:53:25.467330 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:25.467296    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9f84de-678d8f9dc5-588n8"]
Nov 03 19:53:25.475783 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:25.475758    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9f84de-678d8f9dc5-588n8"]
Nov 03 19:53:25.484228 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:25.484204    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9f84de-867b56cbc5-5qmtd"]
Nov 03 19:53:25.488942 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:25.488920    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9f84de-867b56cbc5-5qmtd"]
Nov 03 19:53:26.890027 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:26.889997    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ac3398b-799f-4287-abf7-bfca2392e357" path="/var/lib/kubelet/pods/2ac3398b-799f-4287-abf7-bfca2392e357/volumes"
Nov 03 19:53:26.890562 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:26.890543    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c547aaa-645d-4f0a-805c-885095934756" path="/var/lib/kubelet/pods/3c547aaa-645d-4f0a-805c-885095934756/volumes"
Nov 03 19:53:29.304987 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:29.304953    4921 scope.go:122] "RemoveContainer" containerID="c5a18174b1497959dd9d9a2ab3fb437a63b880e479be0bd215b93e9e7320211c"
Nov 03 19:53:29.319784 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:29.319761    4921 scope.go:122] "RemoveContainer" containerID="9ff0c21a509c7a3081189b7e3595123b1cd0074aa4bb2232ccc9172ee7a688de"
Nov 03 19:53:33.105119 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:53:33.103736    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 19:53:44.657227 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:44.657175    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0c8eec-fc8bcf68c-bnkz8"]
Nov 03 19:53:44.658294 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:44.658272    4921 cpu_manager.go:414] "containerMap: removing container" podUID="0da622a1-45f9-4311-b668-d27eb7ec2e43" containerName="istio-proxy"
Nov 03 19:53:44.658294 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:44.658294    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0da622a1-45f9-4311-b668-d27eb7ec2e43" containerName="istio-proxy"
Nov 03 19:53:44.658496 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:44.658329    4921 cpu_manager.go:414] "containerMap: removing container" podUID="2ac3398b-799f-4287-abf7-bfca2392e357" containerName="payload-pre-processing"
Nov 03 19:53:44.658496 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:44.658338    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ac3398b-799f-4287-abf7-bfca2392e357" containerName="payload-pre-processing"
Nov 03 19:53:44.658496 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:44.658348    4921 cpu_manager.go:414] "containerMap: removing container" podUID="3c547aaa-645d-4f0a-805c-885095934756" containerName="payload-processing"
Nov 03 19:53:44.658496 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:44.658356    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c547aaa-645d-4f0a-805c-885095934756" containerName="payload-processing"
Nov 03 19:53:44.658496 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:44.658395    4921 cpu_manager.go:414] "containerMap: removing container" podUID="4ee1b6ae-046b-4402-a36e-bbd523e73463" containerName="istio-proxy"
Nov 03 19:53:44.658496 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:44.658404    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ee1b6ae-046b-4402-a36e-bbd523e73463" containerName="istio-proxy"
Nov 03 19:53:44.658833 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:44.658569    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ee1b6ae-046b-4402-a36e-bbd523e73463" containerName="istio-proxy"
Nov 03 19:53:44.658833 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:44.658588    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="0da622a1-45f9-4311-b668-d27eb7ec2e43" containerName="istio-proxy"
Nov 03 19:53:44.658833 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:44.658601    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ac3398b-799f-4287-abf7-bfca2392e357" containerName="payload-pre-processing"
Nov 03 19:53:44.658833 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:44.658611    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c547aaa-645d-4f0a-805c-885095934756" containerName="payload-processing"
Nov 03 19:53:44.679202 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:44.663303    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c8eec-fc8bcf68c-bnkz8"
Nov 03 19:53:44.683707 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:44.681591    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-0c8eec-serving-cert\""
Nov 03 19:53:44.693154 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:44.692707    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0c8eec-fc8bcf68c-bnkz8"]
Nov 03 19:53:44.833379 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:44.833336    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f4ba087-9ef9-45b1-adb0-337db91e1fed-openshift-service-ca\") pod \"maas-api-e2e-api-a-0c8eec-fc8bcf68c-bnkz8\" (UID: \"1f4ba087-9ef9-45b1-adb0-337db91e1fed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c8eec-fc8bcf68c-bnkz8"
Nov 03 19:53:44.833910 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:44.833556    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f4ba087-9ef9-45b1-adb0-337db91e1fed-metrics-certs\") pod \"maas-api-e2e-api-a-0c8eec-fc8bcf68c-bnkz8\" (UID: \"1f4ba087-9ef9-45b1-adb0-337db91e1fed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c8eec-fc8bcf68c-bnkz8"
Nov 03 19:53:44.833910 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:44.833610    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f4ba087-9ef9-45b1-adb0-337db91e1fed-maas-api-tls\") pod \"maas-api-e2e-api-a-0c8eec-fc8bcf68c-bnkz8\" (UID: \"1f4ba087-9ef9-45b1-adb0-337db91e1fed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c8eec-fc8bcf68c-bnkz8"
Nov 03 19:53:44.833910 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:44.833714    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwsgb\" (UniqueName: \"kubernetes.io/projected/1f4ba087-9ef9-45b1-adb0-337db91e1fed-kube-api-access-gwsgb\") pod \"maas-api-e2e-api-a-0c8eec-fc8bcf68c-bnkz8\" (UID: \"1f4ba087-9ef9-45b1-adb0-337db91e1fed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c8eec-fc8bcf68c-bnkz8"
Nov 03 19:53:44.935118 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:44.935019    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f4ba087-9ef9-45b1-adb0-337db91e1fed-metrics-certs\") pod \"maas-api-e2e-api-a-0c8eec-fc8bcf68c-bnkz8\" (UID: \"1f4ba087-9ef9-45b1-adb0-337db91e1fed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c8eec-fc8bcf68c-bnkz8"
Nov 03 19:53:44.935118 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:44.935069    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f4ba087-9ef9-45b1-adb0-337db91e1fed-maas-api-tls\") pod \"maas-api-e2e-api-a-0c8eec-fc8bcf68c-bnkz8\" (UID: \"1f4ba087-9ef9-45b1-adb0-337db91e1fed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c8eec-fc8bcf68c-bnkz8"
Nov 03 19:53:44.935118 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:44.935111    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwsgb\" (UniqueName: \"kubernetes.io/projected/1f4ba087-9ef9-45b1-adb0-337db91e1fed-kube-api-access-gwsgb\") pod \"maas-api-e2e-api-a-0c8eec-fc8bcf68c-bnkz8\" (UID: \"1f4ba087-9ef9-45b1-adb0-337db91e1fed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c8eec-fc8bcf68c-bnkz8"
Nov 03 19:53:44.935445 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:44.935238    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f4ba087-9ef9-45b1-adb0-337db91e1fed-openshift-service-ca\") pod \"maas-api-e2e-api-a-0c8eec-fc8bcf68c-bnkz8\" (UID: \"1f4ba087-9ef9-45b1-adb0-337db91e1fed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c8eec-fc8bcf68c-bnkz8"
Nov 03 19:53:44.935445 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:53:44.935254    4921 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-0c8eec-serving-cert: secret "maas-api-e2e-api-a-0c8eec-serving-cert" not found
Nov 03 19:53:44.935445 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:53:44.935343    4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f4ba087-9ef9-45b1-adb0-337db91e1fed-maas-api-tls podName:1f4ba087-9ef9-45b1-adb0-337db91e1fed nodeName:}" failed. No retries permitted until 2026-11-03 19:53:45.435321484 +0000 UTC m=+1338.679286941 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1f4ba087-9ef9-45b1-adb0-337db91e1fed-maas-api-tls") pod "maas-api-e2e-api-a-0c8eec-fc8bcf68c-bnkz8" (UID: "1f4ba087-9ef9-45b1-adb0-337db91e1fed") : secret "maas-api-e2e-api-a-0c8eec-serving-cert" not found
Nov 03 19:53:44.936035 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:44.936014    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f4ba087-9ef9-45b1-adb0-337db91e1fed-openshift-service-ca\") pod \"maas-api-e2e-api-a-0c8eec-fc8bcf68c-bnkz8\" (UID: \"1f4ba087-9ef9-45b1-adb0-337db91e1fed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c8eec-fc8bcf68c-bnkz8"
Nov 03 19:53:44.938313 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:44.938288    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f4ba087-9ef9-45b1-adb0-337db91e1fed-metrics-certs\") pod \"maas-api-e2e-api-a-0c8eec-fc8bcf68c-bnkz8\" (UID: \"1f4ba087-9ef9-45b1-adb0-337db91e1fed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c8eec-fc8bcf68c-bnkz8"
Nov 03 19:53:44.967897 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:44.967835    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwsgb\" (UniqueName: \"kubernetes.io/projected/1f4ba087-9ef9-45b1-adb0-337db91e1fed-kube-api-access-gwsgb\") pod \"maas-api-e2e-api-a-0c8eec-fc8bcf68c-bnkz8\" (UID: \"1f4ba087-9ef9-45b1-adb0-337db91e1fed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c8eec-fc8bcf68c-bnkz8"
Nov 03 19:53:45.442362 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:45.442318    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f4ba087-9ef9-45b1-adb0-337db91e1fed-maas-api-tls\") pod \"maas-api-e2e-api-a-0c8eec-fc8bcf68c-bnkz8\" (UID: \"1f4ba087-9ef9-45b1-adb0-337db91e1fed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c8eec-fc8bcf68c-bnkz8"
Nov 03 19:53:45.445055 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:45.445024    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f4ba087-9ef9-45b1-adb0-337db91e1fed-maas-api-tls\") pod \"maas-api-e2e-api-a-0c8eec-fc8bcf68c-bnkz8\" (UID: \"1f4ba087-9ef9-45b1-adb0-337db91e1fed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c8eec-fc8bcf68c-bnkz8"
Nov 03 19:53:45.629983 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:45.629949    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c8eec-fc8bcf68c-bnkz8"
Nov 03 19:53:46.226647 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:46.212955    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0c8eec-fc8bcf68c-bnkz8"]
Nov 03 19:53:46.980311 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:46.980269    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c8eec-fc8bcf68c-bnkz8" event={"ID":"1f4ba087-9ef9-45b1-adb0-337db91e1fed","Type":"ContainerStarted","Data":"8e5a9850d278e261ed914d643928f5e1d7dac0b1a0d4eab0933142bb0d4e9b59"}
Nov 03 19:53:46.980486 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:46.980344    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c8eec-fc8bcf68c-bnkz8" event={"ID":"1f4ba087-9ef9-45b1-adb0-337db91e1fed","Type":"ContainerStarted","Data":"f340f623857e5c9cf4b52f2969b48eac148fd89dab01539bd87bbbffd009243f"}
Nov 03 19:53:46.980486 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:46.980373    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c8eec-fc8bcf68c-bnkz8"
Nov 03 19:53:47.003011 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:47.002937    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c8eec-fc8bcf68c-bnkz8" podStartSLOduration=2.570592969 podStartE2EDuration="3.002916701s" totalImagesPullingTime="432.323732ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:53:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:53:46.997054448 +0000 UTC m=+1340.241019914" watchObservedRunningTime="2026-11-03 19:53:47.002916701 +0000 UTC m=+1340.246882172"
Nov 03 19:53:53.013599 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:53.013565    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c8eec-fc8bcf68c-bnkz8"
Nov 03 19:53:56.070692 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:56.058637    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9f84de-6656857fd9-vr2c8"]
Nov 03 19:53:56.070692 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:56.060659    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9f84de-6656857fd9-vr2c8" podUID="5b3fa065-548c-4546-9948-c6e8647b1f40" containerName="maas-api" containerID="cri-o://ef1c9faa47aac6ce2cf8f3364ed1a0d2073eb93d61ebbd4adc19293a4790e0bf" gracePeriod=30
Nov 03 19:53:57.104733 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:57.102200    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9f84de-6656857fd9-vr2c8"
Nov 03 19:53:57.169195 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:57.169159    4921 generic.go:356] "Generic (PLEG): container finished" podID="5b3fa065-548c-4546-9948-c6e8647b1f40" containerID="ef1c9faa47aac6ce2cf8f3364ed1a0d2073eb93d61ebbd4adc19293a4790e0bf" exitCode=0
Nov 03 19:53:57.169403 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:57.169211    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9f84de-6656857fd9-vr2c8"
Nov 03 19:53:57.169403 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:57.169247    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9f84de-6656857fd9-vr2c8" event={"ID":"5b3fa065-548c-4546-9948-c6e8647b1f40","Type":"ContainerDied","Data":"ef1c9faa47aac6ce2cf8f3364ed1a0d2073eb93d61ebbd4adc19293a4790e0bf"}
Nov 03 19:53:57.169403 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:57.169319    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9f84de-6656857fd9-vr2c8" event={"ID":"5b3fa065-548c-4546-9948-c6e8647b1f40","Type":"ContainerDied","Data":"5c49517716959e73919efdeda02b8ec938568578aef4455fae9a746e5d045e9c"}
Nov 03 19:53:57.169403 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:57.169346    4921 scope.go:122] "RemoveContainer" containerID="ef1c9faa47aac6ce2cf8f3364ed1a0d2073eb93d61ebbd4adc19293a4790e0bf"
Nov 03 19:53:57.187784 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:57.187756    4921 scope.go:122] "RemoveContainer" containerID="ef1c9faa47aac6ce2cf8f3364ed1a0d2073eb93d61ebbd4adc19293a4790e0bf"
Nov 03 19:53:57.188202 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:53:57.188177    4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef1c9faa47aac6ce2cf8f3364ed1a0d2073eb93d61ebbd4adc19293a4790e0bf\": container with ID starting with ef1c9faa47aac6ce2cf8f3364ed1a0d2073eb93d61ebbd4adc19293a4790e0bf not found: ID does not exist" containerID="ef1c9faa47aac6ce2cf8f3364ed1a0d2073eb93d61ebbd4adc19293a4790e0bf"
Nov 03 19:53:57.188263 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:57.188216    4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef1c9faa47aac6ce2cf8f3364ed1a0d2073eb93d61ebbd4adc19293a4790e0bf"} err="failed to get container status \"ef1c9faa47aac6ce2cf8f3364ed1a0d2073eb93d61ebbd4adc19293a4790e0bf\": rpc error: code = NotFound desc = could not find container \"ef1c9faa47aac6ce2cf8f3364ed1a0d2073eb93d61ebbd4adc19293a4790e0bf\": container with ID starting with ef1c9faa47aac6ce2cf8f3364ed1a0d2073eb93d61ebbd4adc19293a4790e0bf not found: ID does not exist"
Nov 03 19:53:57.249924 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:57.249889    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b3fa065-548c-4546-9948-c6e8647b1f40-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b3fa065-548c-4546-9948-c6e8647b1f40-metrics-certs\") pod \"5b3fa065-548c-4546-9948-c6e8647b1f40\" (UID: \"5b3fa065-548c-4546-9948-c6e8647b1f40\") "
Nov 03 19:53:57.249924 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:57.249931    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b3fa065-548c-4546-9948-c6e8647b1f40-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b3fa065-548c-4546-9948-c6e8647b1f40-maas-api-tls\") pod \"5b3fa065-548c-4546-9948-c6e8647b1f40\" (UID: \"5b3fa065-548c-4546-9948-c6e8647b1f40\") "
Nov 03 19:53:57.250147 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:57.249964    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b3fa065-548c-4546-9948-c6e8647b1f40-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b3fa065-548c-4546-9948-c6e8647b1f40-openshift-service-ca\") pod \"5b3fa065-548c-4546-9948-c6e8647b1f40\" (UID: \"5b3fa065-548c-4546-9948-c6e8647b1f40\") "
Nov 03 19:53:57.250225 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:57.250210    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b3fa065-548c-4546-9948-c6e8647b1f40-kube-api-access-zlnb6\" (UniqueName: \"kubernetes.io/projected/5b3fa065-548c-4546-9948-c6e8647b1f40-kube-api-access-zlnb6\") pod \"5b3fa065-548c-4546-9948-c6e8647b1f40\" (UID: \"5b3fa065-548c-4546-9948-c6e8647b1f40\") "
Nov 03 19:53:57.250356 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:57.250332    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b3fa065-548c-4546-9948-c6e8647b1f40-openshift-service-ca" pod "5b3fa065-548c-4546-9948-c6e8647b1f40" (UID: "5b3fa065-548c-4546-9948-c6e8647b1f40"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:53:57.250846 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:57.250822    4921 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b3fa065-548c-4546-9948-c6e8647b1f40-openshift-service-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:57.252495 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:57.252467    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b3fa065-548c-4546-9948-c6e8647b1f40-metrics-certs" pod "5b3fa065-548c-4546-9948-c6e8647b1f40" (UID: "5b3fa065-548c-4546-9948-c6e8647b1f40"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:53:57.252751 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:57.252726    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b3fa065-548c-4546-9948-c6e8647b1f40-maas-api-tls" pod "5b3fa065-548c-4546-9948-c6e8647b1f40" (UID: "5b3fa065-548c-4546-9948-c6e8647b1f40"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:53:57.252847 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:57.252793    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b3fa065-548c-4546-9948-c6e8647b1f40-kube-api-access-zlnb6" pod "5b3fa065-548c-4546-9948-c6e8647b1f40" (UID: "5b3fa065-548c-4546-9948-c6e8647b1f40"). InnerVolumeSpecName "kube-api-access-zlnb6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:53:57.352204 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:57.352164    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlnb6\" (UniqueName: \"kubernetes.io/projected/5b3fa065-548c-4546-9948-c6e8647b1f40-kube-api-access-zlnb6\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:57.352204 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:57.352198    4921 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b3fa065-548c-4546-9948-c6e8647b1f40-metrics-certs\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:57.352204 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:57.352208    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b3fa065-548c-4546-9948-c6e8647b1f40-maas-api-tls\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:57.497260 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:57.497223    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9f84de-6656857fd9-vr2c8"]
Nov 03 19:53:57.501696 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:57.501609    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9f84de-6656857fd9-vr2c8"]
Nov 03 19:53:58.034151 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:58.034115    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-32a584-5bc48c4649-xdrrx"]
Nov 03 19:53:58.035179 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:58.035149    4921 cpu_manager.go:414] "containerMap: removing container" podUID="5b3fa065-548c-4546-9948-c6e8647b1f40" containerName="maas-api"
Nov 03 19:53:58.035179 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:58.035175    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b3fa065-548c-4546-9948-c6e8647b1f40" containerName="maas-api"
Nov 03 19:53:58.035401 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:58.035340    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b3fa065-548c-4546-9948-c6e8647b1f40" containerName="maas-api"
Nov 03 19:53:58.041067 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:58.041037    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-32a584-5bc48c4649-xdrrx"
Nov 03 19:53:58.046316 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:58.046282    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-32a584-serving-cert\""
Nov 03 19:53:58.049165 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:58.049133    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-32a584-5bc48c4649-xdrrx"]
Nov 03 19:53:58.164217 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:58.164177    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9n5w\" (UniqueName: \"kubernetes.io/projected/1068a9b9-6312-4246-b139-679922ea7a5a-kube-api-access-b9n5w\") pod \"maas-api-e2e-shared-b-w3-32a584-5bc48c4649-xdrrx\" (UID: \"1068a9b9-6312-4246-b139-679922ea7a5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-32a584-5bc48c4649-xdrrx"
Nov 03 19:53:58.164651 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:58.164245    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1068a9b9-6312-4246-b139-679922ea7a5a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-32a584-5bc48c4649-xdrrx\" (UID: \"1068a9b9-6312-4246-b139-679922ea7a5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-32a584-5bc48c4649-xdrrx"
Nov 03 19:53:58.164651 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:58.164420    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1068a9b9-6312-4246-b139-679922ea7a5a-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-32a584-5bc48c4649-xdrrx\" (UID: \"1068a9b9-6312-4246-b139-679922ea7a5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-32a584-5bc48c4649-xdrrx"
Nov 03 19:53:58.164651 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:58.164473    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1068a9b9-6312-4246-b139-679922ea7a5a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-32a584-5bc48c4649-xdrrx\" (UID: \"1068a9b9-6312-4246-b139-679922ea7a5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-32a584-5bc48c4649-xdrrx"
Nov 03 19:53:58.280695 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:58.270095    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9n5w\" (UniqueName: \"kubernetes.io/projected/1068a9b9-6312-4246-b139-679922ea7a5a-kube-api-access-b9n5w\") pod \"maas-api-e2e-shared-b-w3-32a584-5bc48c4649-xdrrx\" (UID: \"1068a9b9-6312-4246-b139-679922ea7a5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-32a584-5bc48c4649-xdrrx"
Nov 03 19:53:58.281048 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:58.281020    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1068a9b9-6312-4246-b139-679922ea7a5a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-32a584-5bc48c4649-xdrrx\" (UID: \"1068a9b9-6312-4246-b139-679922ea7a5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-32a584-5bc48c4649-xdrrx"
Nov 03 19:53:58.281923 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:58.281896    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1068a9b9-6312-4246-b139-679922ea7a5a-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-32a584-5bc48c4649-xdrrx\" (UID: \"1068a9b9-6312-4246-b139-679922ea7a5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-32a584-5bc48c4649-xdrrx"
Nov 03 19:53:58.282631 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:58.282608    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1068a9b9-6312-4246-b139-679922ea7a5a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-32a584-5bc48c4649-xdrrx\" (UID: \"1068a9b9-6312-4246-b139-679922ea7a5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-32a584-5bc48c4649-xdrrx"
Nov 03 19:53:58.324220 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:58.283498    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1068a9b9-6312-4246-b139-679922ea7a5a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-32a584-5bc48c4649-xdrrx\" (UID: \"1068a9b9-6312-4246-b139-679922ea7a5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-32a584-5bc48c4649-xdrrx"
Nov 03 19:53:58.333890 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:58.333813    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1068a9b9-6312-4246-b139-679922ea7a5a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-32a584-5bc48c4649-xdrrx\" (UID: \"1068a9b9-6312-4246-b139-679922ea7a5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-32a584-5bc48c4649-xdrrx"
Nov 03 19:53:58.341267 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:58.341213    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9n5w\" (UniqueName: \"kubernetes.io/projected/1068a9b9-6312-4246-b139-679922ea7a5a-kube-api-access-b9n5w\") pod \"maas-api-e2e-shared-b-w3-32a584-5bc48c4649-xdrrx\" (UID: \"1068a9b9-6312-4246-b139-679922ea7a5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-32a584-5bc48c4649-xdrrx"
Nov 03 19:53:58.353141 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:58.349432    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1068a9b9-6312-4246-b139-679922ea7a5a-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-32a584-5bc48c4649-xdrrx\" (UID: \"1068a9b9-6312-4246-b139-679922ea7a5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-32a584-5bc48c4649-xdrrx"
Nov 03 19:53:58.362897 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:58.362861    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-32a584-5bc48c4649-xdrrx"
Nov 03 19:53:58.707051 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:58.702999    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-32a584-5bc48c4649-xdrrx"]
Nov 03 19:53:58.732786 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:53:58.732747    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1068a9b9_6312_4246_b139_679922ea7a5a.slice/crio-9faaf6861c6f1411f1ed70b245b4dc295778a045eb720a030b8ea7ef48528556 WatchSource:0}: Error finding container 9faaf6861c6f1411f1ed70b245b4dc295778a045eb720a030b8ea7ef48528556: Status 404 returned error can't find the container with id 9faaf6861c6f1411f1ed70b245b4dc295778a045eb720a030b8ea7ef48528556
Nov 03 19:53:58.932305 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:58.932261    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b3fa065-548c-4546-9948-c6e8647b1f40" path="/var/lib/kubelet/pods/5b3fa065-548c-4546-9948-c6e8647b1f40/volumes"
Nov 03 19:53:59.190550 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:53:59.190380    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-32a584-5bc48c4649-xdrrx" event={"ID":"1068a9b9-6312-4246-b139-679922ea7a5a","Type":"ContainerStarted","Data":"9faaf6861c6f1411f1ed70b245b4dc295778a045eb720a030b8ea7ef48528556"}
Nov 03 19:54:00.026425 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:00.026388    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1c446eb-qkztr"]
Nov 03 19:54:00.030698 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:00.030653    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1c446eb-qkztr"]
Nov 03 19:54:00.205920 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:00.205871    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-32a584-5bc48c4649-xdrrx" event={"ID":"1068a9b9-6312-4246-b139-679922ea7a5a","Type":"ContainerStarted","Data":"2d7518a66331807813225c4e6e9cee8b4cef2220a7c2cfbcf41dbafee7dafa7c"}
Nov 03 19:54:00.206427 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:00.206279    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-32a584-5bc48c4649-xdrrx"
Nov 03 19:54:00.224142 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:00.224084    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-32a584-5bc48c4649-xdrrx" podStartSLOduration=1.7952236 podStartE2EDuration="2.224067781s" totalImagesPullingTime="428.844181ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:53:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:54:00.222880593 +0000 UTC m=+1353.466846058" watchObservedRunningTime="2026-11-03 19:54:00.224067781 +0000 UTC m=+1353.468033247"
Nov 03 19:54:00.890309 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:00.890267    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18b354d5-f500-42c4-9406-55351d7c1d65" path="/var/lib/kubelet/pods/18b354d5-f500-42c4-9406-55351d7c1d65/volumes"
Nov 03 19:54:02.988797 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:54:02.988756    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 19:54:06.220114 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:06.220084    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-32a584-5bc48c4649-xdrrx"
Nov 03 19:54:14.380104 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:14.380066    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a9cd5e-58c95dbbdc-7pnq6"]
Nov 03 19:54:14.393002 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:14.392968    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a9cd5e-69bb85bc55-zv8d9"]
Nov 03 19:54:19.380805 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:19.380742    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-a9cd5e-58c95dbbdc-7pnq6" podUID="6f095c43-12af-4dc0-beb7-b2d6189bf544" containerName="payload-processing" containerID="cri-o://2c6841d100c7ad5f9a1560f9e6232f925a7c100f14b20e0eaa01f3af41e31108" gracePeriod=25
Nov 03 19:54:19.394284 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:19.394248    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9cd5e-69bb85bc55-zv8d9" podUID="cfad7089-f35f-41e3-8800-525f4fd7c2fb" containerName="payload-pre-processing" containerID="cri-o://7f0a98adc2d4ab4a792bbd6759d668a036765be4f072fd815bed8d60102369ed" gracePeriod=25
Nov 03 19:54:19.748455 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:19.748418    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a9cd5e-58c95dbbdc-7pnq6"
Nov 03 19:54:19.843449 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:19.843420    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9cd5e-69bb85bc55-zv8d9"
Nov 03 19:54:19.873793 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:19.873758    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f095c43-12af-4dc0-beb7-b2d6189bf544-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f095c43-12af-4dc0-beb7-b2d6189bf544-plugins-config-volume\") pod \"6f095c43-12af-4dc0-beb7-b2d6189bf544\" (UID: \"6f095c43-12af-4dc0-beb7-b2d6189bf544\") "
Nov 03 19:54:19.874050 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:19.874026    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f095c43-12af-4dc0-beb7-b2d6189bf544-kube-api-access-khq82\" (UniqueName: \"kubernetes.io/projected/6f095c43-12af-4dc0-beb7-b2d6189bf544-kube-api-access-khq82\") pod \"6f095c43-12af-4dc0-beb7-b2d6189bf544\" (UID: \"6f095c43-12af-4dc0-beb7-b2d6189bf544\") "
Nov 03 19:54:19.874189 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:19.874157    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f095c43-12af-4dc0-beb7-b2d6189bf544-plugins-config-volume" pod "6f095c43-12af-4dc0-beb7-b2d6189bf544" (UID: "6f095c43-12af-4dc0-beb7-b2d6189bf544"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:54:19.874828 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:19.874806    4921 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f095c43-12af-4dc0-beb7-b2d6189bf544-plugins-config-volume\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:54:19.876191 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:19.876168    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f095c43-12af-4dc0-beb7-b2d6189bf544-kube-api-access-khq82" pod "6f095c43-12af-4dc0-beb7-b2d6189bf544" (UID: "6f095c43-12af-4dc0-beb7-b2d6189bf544"). InnerVolumeSpecName "kube-api-access-khq82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:54:19.975630 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:19.975557    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfad7089-f35f-41e3-8800-525f4fd7c2fb-kube-api-access-8q2mt\" (UniqueName: \"kubernetes.io/projected/cfad7089-f35f-41e3-8800-525f4fd7c2fb-kube-api-access-8q2mt\") pod \"cfad7089-f35f-41e3-8800-525f4fd7c2fb\" (UID: \"cfad7089-f35f-41e3-8800-525f4fd7c2fb\") "
Nov 03 19:54:19.975630 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:19.975625    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfad7089-f35f-41e3-8800-525f4fd7c2fb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfad7089-f35f-41e3-8800-525f4fd7c2fb-plugins-config-volume\") pod \"cfad7089-f35f-41e3-8800-525f4fd7c2fb\" (UID: \"cfad7089-f35f-41e3-8800-525f4fd7c2fb\") "
Nov 03 19:54:19.976071 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:19.976045    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfad7089-f35f-41e3-8800-525f4fd7c2fb-plugins-config-volume" pod "cfad7089-f35f-41e3-8800-525f4fd7c2fb" (UID: "cfad7089-f35f-41e3-8800-525f4fd7c2fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:54:19.976298 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:19.976282    4921 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfad7089-f35f-41e3-8800-525f4fd7c2fb-plugins-config-volume\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:54:19.976338 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:19.976301    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khq82\" (UniqueName: \"kubernetes.io/projected/6f095c43-12af-4dc0-beb7-b2d6189bf544-kube-api-access-khq82\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:54:19.977840 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:19.977823    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfad7089-f35f-41e3-8800-525f4fd7c2fb-kube-api-access-8q2mt" pod "cfad7089-f35f-41e3-8800-525f4fd7c2fb" (UID: "cfad7089-f35f-41e3-8800-525f4fd7c2fb"). InnerVolumeSpecName "kube-api-access-8q2mt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:54:20.090690 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:20.089951    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8q2mt\" (UniqueName: \"kubernetes.io/projected/cfad7089-f35f-41e3-8800-525f4fd7c2fb-kube-api-access-8q2mt\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:54:20.514338 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:20.514301    4921 generic.go:356] "Generic (PLEG): container finished" podID="6f095c43-12af-4dc0-beb7-b2d6189bf544" containerID="2c6841d100c7ad5f9a1560f9e6232f925a7c100f14b20e0eaa01f3af41e31108" exitCode=0
Nov 03 19:54:20.526153 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:20.526119    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a9cd5e-58c95dbbdc-7pnq6" event={"ID":"6f095c43-12af-4dc0-beb7-b2d6189bf544","Type":"ContainerDied","Data":"2c6841d100c7ad5f9a1560f9e6232f925a7c100f14b20e0eaa01f3af41e31108"}
Nov 03 19:54:20.527386 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:20.527357    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a9cd5e-58c95dbbdc-7pnq6" event={"ID":"6f095c43-12af-4dc0-beb7-b2d6189bf544","Type":"ContainerDied","Data":"d5a472f25419df168515e25dc6edb6512671bca1bc4c2141fa49e8257a199922"}
Nov 03 19:54:20.527559 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:20.527547    4921 scope.go:122] "RemoveContainer" containerID="2c6841d100c7ad5f9a1560f9e6232f925a7c100f14b20e0eaa01f3af41e31108"
Nov 03 19:54:20.527844 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:20.527826    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a9cd5e-58c95dbbdc-7pnq6"
Nov 03 19:54:20.531416 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:20.531391    4921 generic.go:356] "Generic (PLEG): container finished" podID="cfad7089-f35f-41e3-8800-525f4fd7c2fb" containerID="7f0a98adc2d4ab4a792bbd6759d668a036765be4f072fd815bed8d60102369ed" exitCode=0
Nov 03 19:54:20.532148 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:20.532119    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9cd5e-69bb85bc55-zv8d9" event={"ID":"cfad7089-f35f-41e3-8800-525f4fd7c2fb","Type":"ContainerDied","Data":"7f0a98adc2d4ab4a792bbd6759d668a036765be4f072fd815bed8d60102369ed"}
Nov 03 19:54:20.532396 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:20.532378    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9cd5e-69bb85bc55-zv8d9" event={"ID":"cfad7089-f35f-41e3-8800-525f4fd7c2fb","Type":"ContainerDied","Data":"5427c42f3a8efc49075777d02ad45353aff70f11c788b68ce724012e1a1d9c12"}
Nov 03 19:54:20.536825 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:20.536807    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a9cd5e-69bb85bc55-zv8d9"
Nov 03 19:54:20.704747 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:20.697830    4921 scope.go:122] "RemoveContainer" containerID="2c6841d100c7ad5f9a1560f9e6232f925a7c100f14b20e0eaa01f3af41e31108"
Nov 03 19:54:20.704747 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:54:20.699349    4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c6841d100c7ad5f9a1560f9e6232f925a7c100f14b20e0eaa01f3af41e31108\": container with ID starting with 2c6841d100c7ad5f9a1560f9e6232f925a7c100f14b20e0eaa01f3af41e31108 not found: ID does not exist" containerID="2c6841d100c7ad5f9a1560f9e6232f925a7c100f14b20e0eaa01f3af41e31108"
Nov 03 19:54:20.704747 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:20.699391    4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c6841d100c7ad5f9a1560f9e6232f925a7c100f14b20e0eaa01f3af41e31108"} err="failed to get container status \"2c6841d100c7ad5f9a1560f9e6232f925a7c100f14b20e0eaa01f3af41e31108\": rpc error: code = NotFound desc = could not find container \"2c6841d100c7ad5f9a1560f9e6232f925a7c100f14b20e0eaa01f3af41e31108\": container with ID starting with 2c6841d100c7ad5f9a1560f9e6232f925a7c100f14b20e0eaa01f3af41e31108 not found: ID does not exist"
Nov 03 19:54:20.704747 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:20.699415    4921 scope.go:122] "RemoveContainer" containerID="7f0a98adc2d4ab4a792bbd6759d668a036765be4f072fd815bed8d60102369ed"
Nov 03 19:54:20.718924 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:20.718892    4921 scope.go:122] "RemoveContainer" containerID="7f0a98adc2d4ab4a792bbd6759d668a036765be4f072fd815bed8d60102369ed"
Nov 03 19:54:20.719047 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:20.719022    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a9cd5e-58c95dbbdc-7pnq6"]
Nov 03 19:54:20.728583 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:54:20.727977    4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f0a98adc2d4ab4a792bbd6759d668a036765be4f072fd815bed8d60102369ed\": container with ID starting with 7f0a98adc2d4ab4a792bbd6759d668a036765be4f072fd815bed8d60102369ed not found: ID does not exist" containerID="7f0a98adc2d4ab4a792bbd6759d668a036765be4f072fd815bed8d60102369ed"
Nov 03 19:54:20.728583 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:20.728012    4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f0a98adc2d4ab4a792bbd6759d668a036765be4f072fd815bed8d60102369ed"} err="failed to get container status \"7f0a98adc2d4ab4a792bbd6759d668a036765be4f072fd815bed8d60102369ed\": rpc error: code = NotFound desc = could not find container \"7f0a98adc2d4ab4a792bbd6759d668a036765be4f072fd815bed8d60102369ed\": container with ID starting with 7f0a98adc2d4ab4a792bbd6759d668a036765be4f072fd815bed8d60102369ed not found: ID does not exist"
Nov 03 19:54:20.729178 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:20.729154    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a9cd5e-58c95dbbdc-7pnq6"]
Nov 03 19:54:20.734685 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:20.734645    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a9cd5e-69bb85bc55-zv8d9"]
Nov 03 19:54:20.739309 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:20.739275    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a9cd5e-69bb85bc55-zv8d9"]
Nov 03 19:54:20.893101 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:20.893068    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f095c43-12af-4dc0-beb7-b2d6189bf544" path="/var/lib/kubelet/pods/6f095c43-12af-4dc0-beb7-b2d6189bf544/volumes"
Nov 03 19:54:20.893558 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:20.893544    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfad7089-f35f-41e3-8800-525f4fd7c2fb" path="/var/lib/kubelet/pods/cfad7089-f35f-41e3-8800-525f4fd7c2fb/volumes"
Nov 03 19:54:29.646213 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:29.646177    4921 scope.go:122] "RemoveContainer" containerID="5b5e81a1a6dd0c7dd8c8d9d89bb3a635b9cc900f7f5022297e4dd0a40bb4dc29"
Nov 03 19:54:29.657578 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:29.657555    4921 scope.go:122] "RemoveContainer" containerID="62346d20ccc3352584d0a7c1ab4f699c54bceb5aebbbc933aeee0386048cccf2"
Nov 03 19:54:30.507869 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:30.507834    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-225668-65675d6655-qdr5l"]
Nov 03 19:54:30.508679 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:30.508623    4921 cpu_manager.go:414] "containerMap: removing container" podUID="6f095c43-12af-4dc0-beb7-b2d6189bf544" containerName="payload-processing"
Nov 03 19:54:30.508679 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:30.508650    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f095c43-12af-4dc0-beb7-b2d6189bf544" containerName="payload-processing"
Nov 03 19:54:30.508889 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:30.508688    4921 cpu_manager.go:414] "containerMap: removing container" podUID="cfad7089-f35f-41e3-8800-525f4fd7c2fb" containerName="payload-pre-processing"
Nov 03 19:54:30.508889 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:30.508697    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfad7089-f35f-41e3-8800-525f4fd7c2fb" containerName="payload-pre-processing"
Nov 03 19:54:30.508889 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:30.508844    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f095c43-12af-4dc0-beb7-b2d6189bf544" containerName="payload-processing"
Nov 03 19:54:30.508889 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:30.508854    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfad7089-f35f-41e3-8800-525f4fd7c2fb" containerName="payload-pre-processing"
Nov 03 19:54:30.516712 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:30.516141    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-225668-65675d6655-qdr5l"]
Nov 03 19:54:30.516712 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:30.516258    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-225668-65675d6655-qdr5l"
Nov 03 19:54:30.521575 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:30.520587    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-225668\""
Nov 03 19:54:30.521575 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:30.520734    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-225668-dockercfg-zpbrh\""
Nov 03 19:54:30.522652 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:30.522631    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-225668-6d7866c8dc-29bbm"]
Nov 03 19:54:30.529968 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:30.529939    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-225668-6d7866c8dc-29bbm"
Nov 03 19:54:30.530463 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:30.530435    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-225668-6d7866c8dc-29bbm"]
Nov 03 19:54:30.760748 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:30.741732    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5lkl\" (UniqueName: \"kubernetes.io/projected/2105ca77-5b9c-45b2-9256-782b998a0041-kube-api-access-d5lkl\") pod \"payload-processing-e2e-inf-a-225668-65675d6655-qdr5l\" (UID: \"2105ca77-5b9c-45b2-9256-782b998a0041\") " pod="openshift-ingress/payload-processing-e2e-inf-a-225668-65675d6655-qdr5l"
Nov 03 19:54:30.760748 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:30.741811    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phljb\" (UniqueName: \"kubernetes.io/projected/e2c9f88f-de8e-406e-ad7d-2038e0faba39-kube-api-access-phljb\") pod \"payload-pre-processing-e2e-inf-a-225668-6d7866c8dc-29bbm\" (UID: \"e2c9f88f-de8e-406e-ad7d-2038e0faba39\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-225668-6d7866c8dc-29bbm"
Nov 03 19:54:30.760748 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:30.741863    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2105ca77-5b9c-45b2-9256-782b998a0041-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-225668-65675d6655-qdr5l\" (UID: \"2105ca77-5b9c-45b2-9256-782b998a0041\") " pod="openshift-ingress/payload-processing-e2e-inf-a-225668-65675d6655-qdr5l"
Nov 03 19:54:30.760748 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:30.742056    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2c9f88f-de8e-406e-ad7d-2038e0faba39-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-225668-6d7866c8dc-29bbm\" (UID: \"e2c9f88f-de8e-406e-ad7d-2038e0faba39\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-225668-6d7866c8dc-29bbm"
Nov 03 19:54:30.870010 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:30.849755    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2c9f88f-de8e-406e-ad7d-2038e0faba39-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-225668-6d7866c8dc-29bbm\" (UID: \"e2c9f88f-de8e-406e-ad7d-2038e0faba39\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-225668-6d7866c8dc-29bbm"
Nov 03 19:54:30.870010 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:30.849910    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5lkl\" (UniqueName: \"kubernetes.io/projected/2105ca77-5b9c-45b2-9256-782b998a0041-kube-api-access-d5lkl\") pod \"payload-processing-e2e-inf-a-225668-65675d6655-qdr5l\" (UID: \"2105ca77-5b9c-45b2-9256-782b998a0041\") " pod="openshift-ingress/payload-processing-e2e-inf-a-225668-65675d6655-qdr5l"
Nov 03 19:54:30.870010 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:30.849983    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phljb\" (UniqueName: \"kubernetes.io/projected/e2c9f88f-de8e-406e-ad7d-2038e0faba39-kube-api-access-phljb\") pod \"payload-pre-processing-e2e-inf-a-225668-6d7866c8dc-29bbm\" (UID: \"e2c9f88f-de8e-406e-ad7d-2038e0faba39\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-225668-6d7866c8dc-29bbm"
Nov 03 19:54:30.870010 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:30.850033    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2105ca77-5b9c-45b2-9256-782b998a0041-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-225668-65675d6655-qdr5l\" (UID: \"2105ca77-5b9c-45b2-9256-782b998a0041\") " pod="openshift-ingress/payload-processing-e2e-inf-a-225668-65675d6655-qdr5l"
Nov 03 19:54:30.870010 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:30.851615    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2c9f88f-de8e-406e-ad7d-2038e0faba39-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-225668-6d7866c8dc-29bbm\" (UID: \"e2c9f88f-de8e-406e-ad7d-2038e0faba39\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-225668-6d7866c8dc-29bbm"
Nov 03 19:54:30.893756 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:30.893619    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5lkl\" (UniqueName: \"kubernetes.io/projected/2105ca77-5b9c-45b2-9256-782b998a0041-kube-api-access-d5lkl\") pod \"payload-processing-e2e-inf-a-225668-65675d6655-qdr5l\" (UID: \"2105ca77-5b9c-45b2-9256-782b998a0041\") " pod="openshift-ingress/payload-processing-e2e-inf-a-225668-65675d6655-qdr5l"
Nov 03 19:54:30.906942 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:30.898095    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phljb\" (UniqueName: \"kubernetes.io/projected/e2c9f88f-de8e-406e-ad7d-2038e0faba39-kube-api-access-phljb\") pod \"payload-pre-processing-e2e-inf-a-225668-6d7866c8dc-29bbm\" (UID: \"e2c9f88f-de8e-406e-ad7d-2038e0faba39\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-225668-6d7866c8dc-29bbm"
Nov 03 19:54:30.906942 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:30.906710    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2105ca77-5b9c-45b2-9256-782b998a0041-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-225668-65675d6655-qdr5l\" (UID: \"2105ca77-5b9c-45b2-9256-782b998a0041\") " pod="openshift-ingress/payload-processing-e2e-inf-a-225668-65675d6655-qdr5l"
Nov 03 19:54:31.119636 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:31.118196    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-225668-6d7866c8dc-29bbm"
Nov 03 19:54:31.163209 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:31.161995    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-225668-65675d6655-qdr5l"
Nov 03 19:54:31.781015 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:54:31.775429    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2c9f88f_de8e_406e_ad7d_2038e0faba39.slice/crio-03a9883bb8a17a31e79fa45397a76987fa72bd89caad6c083a44fbcfca515e4e WatchSource:0}: Error finding container 03a9883bb8a17a31e79fa45397a76987fa72bd89caad6c083a44fbcfca515e4e: Status 404 returned error can't find the container with id 03a9883bb8a17a31e79fa45397a76987fa72bd89caad6c083a44fbcfca515e4e
Nov 03 19:54:31.781015 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:31.776270    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-225668-6d7866c8dc-29bbm"]
Nov 03 19:54:31.892600 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:31.892556    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-225668-6d7866c8dc-29bbm" event={"ID":"e2c9f88f-de8e-406e-ad7d-2038e0faba39","Type":"ContainerStarted","Data":"3b0c6a0e30cb1fcb599d3a582876cbaaa9dcd837ed8e58ed55f26a4e6d29ec73"}
Nov 03 19:54:31.893801 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:31.893765    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-225668-6d7866c8dc-29bbm" event={"ID":"e2c9f88f-de8e-406e-ad7d-2038e0faba39","Type":"ContainerStarted","Data":"03a9883bb8a17a31e79fa45397a76987fa72bd89caad6c083a44fbcfca515e4e"}
Nov 03 19:54:31.893932 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:31.893833    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-225668-6d7866c8dc-29bbm"
Nov 03 19:54:31.911866 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:31.911754    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-225668-6d7866c8dc-29bbm" podStartSLOduration=1.91173874 podStartE2EDuration="1.91173874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:54:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:54:31.907405953 +0000 UTC m=+1385.151371420" watchObservedRunningTime="2026-11-03 19:54:31.91173874 +0000 UTC m=+1385.155704206"
Nov 03 19:54:31.925760 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:31.925720    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-225668-65675d6655-qdr5l"]
Nov 03 19:54:31.925911 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:54:31.925820    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2105ca77_5b9c_45b2_9256_782b998a0041.slice/crio-35e9259dd57d1adefb45a4b4f5fb61eab195ae0d50f7e316ed23a9eccfc63eea WatchSource:0}: Error finding container 35e9259dd57d1adefb45a4b4f5fb61eab195ae0d50f7e316ed23a9eccfc63eea: Status 404 returned error can't find the container with id 35e9259dd57d1adefb45a4b4f5fb61eab195ae0d50f7e316ed23a9eccfc63eea
Nov 03 19:54:33.570965 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:54:33.570923    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 19:54:33.590167 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:33.588377    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-225668-65675d6655-qdr5l" event={"ID":"2105ca77-5b9c-45b2-9256-782b998a0041","Type":"ContainerStarted","Data":"925ce0d76832c7f770773101fd88e6ee410b0ce690e8577656f6fb1702ed5cb3"}
Nov 03 19:54:33.590167 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:33.588459    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-225668-65675d6655-qdr5l" event={"ID":"2105ca77-5b9c-45b2-9256-782b998a0041","Type":"ContainerStarted","Data":"35e9259dd57d1adefb45a4b4f5fb61eab195ae0d50f7e316ed23a9eccfc63eea"}
Nov 03 19:54:33.590167 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:33.588486    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-225668-65675d6655-qdr5l"
Nov 03 19:54:33.611743 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:33.611646    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-225668-65675d6655-qdr5l" podStartSLOduration=3.611627153 podStartE2EDuration="3.611627153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:54:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:54:33.605967291 +0000 UTC m=+1386.849932794" watchObservedRunningTime="2026-11-03 19:54:33.611627153 +0000 UTC m=+1386.855592619"
Nov 03 19:54:36.084005 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:36.079872    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9" podUID="e6a71aa3-33b3-42bd-b270-64ce3260cc2a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:54:36.584866 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:36.584812    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv" podUID="96b73342-0079-4709-8570-3dbce57e5243" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:54:36.620999 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:36.617767    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt" podUID="17dc0ed2-5164-449d-be12-a38bdfeb946d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:54:36.679820 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:36.679761    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk" podUID="2d0a934e-4816-44d2-9bb5-e8b4e95dd73c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:54:37.101442 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:37.097809    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9" podUID="e6a71aa3-33b3-42bd-b270-64ce3260cc2a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:54:37.153693 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:37.146015    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9"
Nov 03 19:54:37.375766 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:37.373533    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9"
Nov 03 19:54:40.893869 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:40.893832    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0d0987-6f7b54c8cb-8s9ds"]
Nov 03 19:54:40.904192 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:40.904165    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0d0987-6f7b54c8cb-8s9ds"]
Nov 03 19:54:40.904487 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:40.904472    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0d0987-6f7b54c8cb-8s9ds"
Nov 03 19:54:40.911283 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:40.911225    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-0d0987-serving-cert\""
Nov 03 19:54:41.002767 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:41.002720    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfvsx\" (UniqueName: \"kubernetes.io/projected/4fcdf540-fd42-4f5e-bb3a-422d726c3281-kube-api-access-jfvsx\") pod \"maas-api-e2e-api-b-0d0987-6f7b54c8cb-8s9ds\" (UID: \"4fcdf540-fd42-4f5e-bb3a-422d726c3281\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0d0987-6f7b54c8cb-8s9ds"
Nov 03 19:54:41.002767 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:41.002768    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fcdf540-fd42-4f5e-bb3a-422d726c3281-maas-api-tls\") pod \"maas-api-e2e-api-b-0d0987-6f7b54c8cb-8s9ds\" (UID: \"4fcdf540-fd42-4f5e-bb3a-422d726c3281\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0d0987-6f7b54c8cb-8s9ds"
Nov 03 19:54:41.002979 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:41.002866    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fcdf540-fd42-4f5e-bb3a-422d726c3281-metrics-certs\") pod \"maas-api-e2e-api-b-0d0987-6f7b54c8cb-8s9ds\" (UID: \"4fcdf540-fd42-4f5e-bb3a-422d726c3281\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0d0987-6f7b54c8cb-8s9ds"
Nov 03 19:54:41.002979 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:41.002927    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fcdf540-fd42-4f5e-bb3a-422d726c3281-openshift-service-ca\") pod \"maas-api-e2e-api-b-0d0987-6f7b54c8cb-8s9ds\" (UID: \"4fcdf540-fd42-4f5e-bb3a-422d726c3281\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0d0987-6f7b54c8cb-8s9ds"
Nov 03 19:54:41.105780 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:41.104969    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfvsx\" (UniqueName: \"kubernetes.io/projected/4fcdf540-fd42-4f5e-bb3a-422d726c3281-kube-api-access-jfvsx\") pod \"maas-api-e2e-api-b-0d0987-6f7b54c8cb-8s9ds\" (UID: \"4fcdf540-fd42-4f5e-bb3a-422d726c3281\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0d0987-6f7b54c8cb-8s9ds"
Nov 03 19:54:41.105780 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:41.105028    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fcdf540-fd42-4f5e-bb3a-422d726c3281-maas-api-tls\") pod \"maas-api-e2e-api-b-0d0987-6f7b54c8cb-8s9ds\" (UID: \"4fcdf540-fd42-4f5e-bb3a-422d726c3281\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0d0987-6f7b54c8cb-8s9ds"
Nov 03 19:54:41.105780 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:41.105085    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fcdf540-fd42-4f5e-bb3a-422d726c3281-metrics-certs\") pod \"maas-api-e2e-api-b-0d0987-6f7b54c8cb-8s9ds\" (UID: \"4fcdf540-fd42-4f5e-bb3a-422d726c3281\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0d0987-6f7b54c8cb-8s9ds"
Nov 03 19:54:41.105780 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:41.105162    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fcdf540-fd42-4f5e-bb3a-422d726c3281-openshift-service-ca\") pod \"maas-api-e2e-api-b-0d0987-6f7b54c8cb-8s9ds\" (UID: \"4fcdf540-fd42-4f5e-bb3a-422d726c3281\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0d0987-6f7b54c8cb-8s9ds"
Nov 03 19:54:41.106841 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:54:41.106817    4921 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-0d0987-serving-cert: secret "maas-api-e2e-api-b-0d0987-serving-cert" not found
Nov 03 19:54:41.107112 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:54:41.107098    4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fcdf540-fd42-4f5e-bb3a-422d726c3281-maas-api-tls podName:4fcdf540-fd42-4f5e-bb3a-422d726c3281 nodeName:}" failed. No retries permitted until 2026-11-03 19:54:41.607075353 +0000 UTC m=+1394.851040810 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4fcdf540-fd42-4f5e-bb3a-422d726c3281-maas-api-tls") pod "maas-api-e2e-api-b-0d0987-6f7b54c8cb-8s9ds" (UID: "4fcdf540-fd42-4f5e-bb3a-422d726c3281") : secret "maas-api-e2e-api-b-0d0987-serving-cert" not found
Nov 03 19:54:41.109304 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:41.109278    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fcdf540-fd42-4f5e-bb3a-422d726c3281-metrics-certs\") pod \"maas-api-e2e-api-b-0d0987-6f7b54c8cb-8s9ds\" (UID: \"4fcdf540-fd42-4f5e-bb3a-422d726c3281\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0d0987-6f7b54c8cb-8s9ds"
Nov 03 19:54:41.109741 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:41.109717    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fcdf540-fd42-4f5e-bb3a-422d726c3281-openshift-service-ca\") pod \"maas-api-e2e-api-b-0d0987-6f7b54c8cb-8s9ds\" (UID: \"4fcdf540-fd42-4f5e-bb3a-422d726c3281\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0d0987-6f7b54c8cb-8s9ds"
Nov 03 19:54:41.120545 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:41.120431    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfvsx\" (UniqueName: \"kubernetes.io/projected/4fcdf540-fd42-4f5e-bb3a-422d726c3281-kube-api-access-jfvsx\") pod \"maas-api-e2e-api-b-0d0987-6f7b54c8cb-8s9ds\" (UID: \"4fcdf540-fd42-4f5e-bb3a-422d726c3281\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0d0987-6f7b54c8cb-8s9ds"
Nov 03 19:54:41.622557 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:41.619491    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fcdf540-fd42-4f5e-bb3a-422d726c3281-maas-api-tls\") pod \"maas-api-e2e-api-b-0d0987-6f7b54c8cb-8s9ds\" (UID: \"4fcdf540-fd42-4f5e-bb3a-422d726c3281\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0d0987-6f7b54c8cb-8s9ds"
Nov 03 19:54:41.656626 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:41.656554    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fcdf540-fd42-4f5e-bb3a-422d726c3281-maas-api-tls\") pod \"maas-api-e2e-api-b-0d0987-6f7b54c8cb-8s9ds\" (UID: \"4fcdf540-fd42-4f5e-bb3a-422d726c3281\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0d0987-6f7b54c8cb-8s9ds"
Nov 03 19:54:41.834962 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:41.834926    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0d0987-6f7b54c8cb-8s9ds"
Nov 03 19:54:42.392657 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:42.392621    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0d0987-6f7b54c8cb-8s9ds"]
Nov 03 19:54:42.945477 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:42.945438    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0d0987-6f7b54c8cb-8s9ds" event={"ID":"4fcdf540-fd42-4f5e-bb3a-422d726c3281","Type":"ContainerStarted","Data":"5d6520597ecef5f2e98b68cb493d5cf092a2e9bdfa1da40bc65a3840582732dc"}
Nov 03 19:54:42.945657 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:42.945503    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0d0987-6f7b54c8cb-8s9ds" event={"ID":"4fcdf540-fd42-4f5e-bb3a-422d726c3281","Type":"ContainerStarted","Data":"471b9222e2be05c6afce2e16d3597579c2ccff649546d50184313eb560c7f88c"}
Nov 03 19:54:42.945657 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:42.945530    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0d0987-6f7b54c8cb-8s9ds"
Nov 03 19:54:42.963036 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:42.962986    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0d0987-6f7b54c8cb-8s9ds" podStartSLOduration=2.519108126 podStartE2EDuration="2.962971423s" totalImagesPullingTime="443.863297ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:54:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:54:42.960032874 +0000 UTC m=+1396.203998380" watchObservedRunningTime="2026-11-03 19:54:42.962971423 +0000 UTC m=+1396.206936888"
Nov 03 19:54:44.644654 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:44.644617    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-225668-65675d6655-qdr5l"
Nov 03 19:54:44.650341 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:44.650315    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-225668-6d7866c8dc-29bbm"
Nov 03 19:54:46.126739 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:46.126704    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a9cd5e-59b5db6cdf-j95k7"]
Nov 03 19:54:46.127153 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:46.126906    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9cd5e-59b5db6cdf-j95k7" podUID="9d27071d-8535-4b88-9e91-e99c3c74abe2" containerName="maas-api" containerID="cri-o://58db1dd44518ccd26d1b40cfbb9a2a520014b9f6f8db7975a1185a5a136c0e6e" gracePeriod=30
Nov 03 19:54:46.861641 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:46.861608    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9cd5e-59b5db6cdf-j95k7"
Nov 03 19:54:46.920591 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:46.920548    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d27071d-8535-4b88-9e91-e99c3c74abe2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d27071d-8535-4b88-9e91-e99c3c74abe2-openshift-service-ca\") pod \"9d27071d-8535-4b88-9e91-e99c3c74abe2\" (UID: \"9d27071d-8535-4b88-9e91-e99c3c74abe2\") "
Nov 03 19:54:46.920837 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:46.920635    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d27071d-8535-4b88-9e91-e99c3c74abe2-kube-api-access-5c5dk\" (UniqueName: \"kubernetes.io/projected/9d27071d-8535-4b88-9e91-e99c3c74abe2-kube-api-access-5c5dk\") pod \"9d27071d-8535-4b88-9e91-e99c3c74abe2\" (UID: \"9d27071d-8535-4b88-9e91-e99c3c74abe2\") "
Nov 03 19:54:46.920837 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:46.920713    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d27071d-8535-4b88-9e91-e99c3c74abe2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d27071d-8535-4b88-9e91-e99c3c74abe2-metrics-certs\") pod \"9d27071d-8535-4b88-9e91-e99c3c74abe2\" (UID: \"9d27071d-8535-4b88-9e91-e99c3c74abe2\") "
Nov 03 19:54:46.920980 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:46.920880    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d27071d-8535-4b88-9e91-e99c3c74abe2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d27071d-8535-4b88-9e91-e99c3c74abe2-maas-api-tls\") pod \"9d27071d-8535-4b88-9e91-e99c3c74abe2\" (UID: \"9d27071d-8535-4b88-9e91-e99c3c74abe2\") "
Nov 03 19:54:46.921818 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:46.921791    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d27071d-8535-4b88-9e91-e99c3c74abe2-openshift-service-ca" pod "9d27071d-8535-4b88-9e91-e99c3c74abe2" (UID: "9d27071d-8535-4b88-9e91-e99c3c74abe2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:54:46.922359 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:46.922322    4921 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d27071d-8535-4b88-9e91-e99c3c74abe2-openshift-service-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:54:46.925058 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:46.925014    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d27071d-8535-4b88-9e91-e99c3c74abe2-metrics-certs" pod "9d27071d-8535-4b88-9e91-e99c3c74abe2" (UID: "9d27071d-8535-4b88-9e91-e99c3c74abe2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:54:46.925982 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:46.925954    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d27071d-8535-4b88-9e91-e99c3c74abe2-kube-api-access-5c5dk" pod "9d27071d-8535-4b88-9e91-e99c3c74abe2" (UID: "9d27071d-8535-4b88-9e91-e99c3c74abe2"). InnerVolumeSpecName "kube-api-access-5c5dk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:54:46.940294 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:46.940252    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d27071d-8535-4b88-9e91-e99c3c74abe2-maas-api-tls" pod "9d27071d-8535-4b88-9e91-e99c3c74abe2" (UID: "9d27071d-8535-4b88-9e91-e99c3c74abe2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:54:47.051016 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:47.043134    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d27071d-8535-4b88-9e91-e99c3c74abe2-maas-api-tls\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:54:47.051016 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:47.043166    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5c5dk\" (UniqueName: \"kubernetes.io/projected/9d27071d-8535-4b88-9e91-e99c3c74abe2-kube-api-access-5c5dk\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:54:47.051016 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:47.043182    4921 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d27071d-8535-4b88-9e91-e99c3c74abe2-metrics-certs\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:54:47.051016 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:47.048818    4921 generic.go:356] "Generic (PLEG): container finished" podID="9d27071d-8535-4b88-9e91-e99c3c74abe2" containerID="58db1dd44518ccd26d1b40cfbb9a2a520014b9f6f8db7975a1185a5a136c0e6e" exitCode=0
Nov 03 19:54:47.051016 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:47.048938    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9cd5e-59b5db6cdf-j95k7" event={"ID":"9d27071d-8535-4b88-9e91-e99c3c74abe2","Type":"ContainerDied","Data":"58db1dd44518ccd26d1b40cfbb9a2a520014b9f6f8db7975a1185a5a136c0e6e"}
Nov 03 19:54:47.051016 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:47.048981    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9cd5e-59b5db6cdf-j95k7" event={"ID":"9d27071d-8535-4b88-9e91-e99c3c74abe2","Type":"ContainerDied","Data":"acf75bea503dc0aa88a9c9072705d23ff69095bbdb6431b536e436ef7a211514"}
Nov 03 19:54:47.051016 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:47.049003    4921 scope.go:122] "RemoveContainer" containerID="58db1dd44518ccd26d1b40cfbb9a2a520014b9f6f8db7975a1185a5a136c0e6e"
Nov 03 19:54:47.051016 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:47.049134    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9cd5e-59b5db6cdf-j95k7"
Nov 03 19:54:47.063799 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:47.063780    4921 scope.go:122] "RemoveContainer" containerID="58db1dd44518ccd26d1b40cfbb9a2a520014b9f6f8db7975a1185a5a136c0e6e"
Nov 03 19:54:47.064140 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:54:47.064118    4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58db1dd44518ccd26d1b40cfbb9a2a520014b9f6f8db7975a1185a5a136c0e6e\": container with ID starting with 58db1dd44518ccd26d1b40cfbb9a2a520014b9f6f8db7975a1185a5a136c0e6e not found: ID does not exist" containerID="58db1dd44518ccd26d1b40cfbb9a2a520014b9f6f8db7975a1185a5a136c0e6e"
Nov 03 19:54:47.064207 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:47.064151    4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58db1dd44518ccd26d1b40cfbb9a2a520014b9f6f8db7975a1185a5a136c0e6e"} err="failed to get container status \"58db1dd44518ccd26d1b40cfbb9a2a520014b9f6f8db7975a1185a5a136c0e6e\": rpc error: code = NotFound desc = could not find container \"58db1dd44518ccd26d1b40cfbb9a2a520014b9f6f8db7975a1185a5a136c0e6e\": container with ID starting with 58db1dd44518ccd26d1b40cfbb9a2a520014b9f6f8db7975a1185a5a136c0e6e not found: ID does not exist"
Nov 03 19:54:47.076786 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:47.076761    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a9cd5e-59b5db6cdf-j95k7"]
Nov 03 19:54:47.086053 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:47.086017    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a9cd5e-59b5db6cdf-j95k7"]
Nov 03 19:54:47.756225 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:47.756171    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-x4frm"]
Nov 03 19:54:47.771915 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:47.757925    4921 cpu_manager.go:414] "containerMap: removing container" podUID="9d27071d-8535-4b88-9e91-e99c3c74abe2" containerName="maas-api"
Nov 03 19:54:47.780483 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:47.780449    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d27071d-8535-4b88-9e91-e99c3c74abe2" containerName="maas-api"
Nov 03 19:54:47.781047 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:47.781014    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d27071d-8535-4b88-9e91-e99c3c74abe2" containerName="maas-api"
Nov 03 19:54:47.800434 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:47.800393    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-x4frm"
Nov 03 19:54:47.805315 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:47.805262    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-332694-openshift-default-dockercfg-pvj4g\""
Nov 03 19:54:47.807547 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:47.807512    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-x4frm"]
Nov 03 19:54:47.893366 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:47.893317    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dec4941-e24a-4b8b-852c-a9d0128c533c-istio-data\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-x4frm\" (UID: \"6dec4941-e24a-4b8b-852c-a9d0128c533c\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-x4frm"
Nov 03 19:54:47.893537 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:47.893376    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dec4941-e24a-4b8b-852c-a9d0128c533c-istio-podinfo\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-x4frm\" (UID: \"6dec4941-e24a-4b8b-852c-a9d0128c533c\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-x4frm"
Nov 03 19:54:47.893537 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:47.893437    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dec4941-e24a-4b8b-852c-a9d0128c533c-workload-socket\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-x4frm\" (UID: \"6dec4941-e24a-4b8b-852c-a9d0128c533c\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-x4frm"
Nov 03 19:54:47.893537 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:47.893467    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dec4941-e24a-4b8b-852c-a9d0128c533c-credential-socket\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-x4frm\" (UID: \"6dec4941-e24a-4b8b-852c-a9d0128c533c\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-x4frm"
Nov 03 19:54:47.893752 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:47.893601    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dec4941-e24a-4b8b-852c-a9d0128c533c-istio-token\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-x4frm\" (UID: \"6dec4941-e24a-4b8b-852c-a9d0128c533c\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-x4frm"
Nov 03 19:54:47.893752 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:47.893628    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dec4941-e24a-4b8b-852c-a9d0128c533c-istiod-ca-cert\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-x4frm\" (UID: \"6dec4941-e24a-4b8b-852c-a9d0128c533c\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-x4frm"
Nov 03 19:54:47.893752 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:47.893705    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dec4941-e24a-4b8b-852c-a9d0128c533c-istio-envoy\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-x4frm\" (UID: \"6dec4941-e24a-4b8b-852c-a9d0128c533c\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-x4frm"
Nov 03 19:54:47.893934 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:47.893751    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dec4941-e24a-4b8b-852c-a9d0128c533c-workload-certs\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-x4frm\" (UID: \"6dec4941-e24a-4b8b-852c-a9d0128c533c\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-x4frm"
Nov 03 19:54:47.893934 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:47.893784    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rzzc\" (UniqueName: \"kubernetes.io/projected/6dec4941-e24a-4b8b-852c-a9d0128c533c-kube-api-access-2rzzc\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-x4frm\" (UID: \"6dec4941-e24a-4b8b-852c-a9d0128c533c\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-x4frm"
Nov 03 19:54:48.004692 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:47.995766    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dec4941-e24a-4b8b-852c-a9d0128c533c-workload-socket\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-x4frm\" (UID: \"6dec4941-e24a-4b8b-852c-a9d0128c533c\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-x4frm"
Nov 03 19:54:48.004692 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:47.995851    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dec4941-e24a-4b8b-852c-a9d0128c533c-credential-socket\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-x4frm\" (UID: \"6dec4941-e24a-4b8b-852c-a9d0128c533c\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-x4frm"
Nov 03 19:54:48.004692 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:47.996144    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dec4941-e24a-4b8b-852c-a9d0128c533c-istio-token\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-x4frm\" (UID: \"6dec4941-e24a-4b8b-852c-a9d0128c533c\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-x4frm"
Nov 03 19:54:48.004692 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:47.996204    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dec4941-e24a-4b8b-852c-a9d0128c533c-istiod-ca-cert\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-x4frm\" (UID: \"6dec4941-e24a-4b8b-852c-a9d0128c533c\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-x4frm"
Nov 03 19:54:48.004692 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:47.996293    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dec4941-e24a-4b8b-852c-a9d0128c533c-istio-envoy\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-x4frm\" (UID: \"6dec4941-e24a-4b8b-852c-a9d0128c533c\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-x4frm"
Nov 03 19:54:48.004692 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:47.996353    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dec4941-e24a-4b8b-852c-a9d0128c533c-workload-certs\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-x4frm\" (UID: \"6dec4941-e24a-4b8b-852c-a9d0128c533c\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-x4frm"
Nov 03 19:54:48.004692 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:47.996389    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rzzc\" (UniqueName: \"kubernetes.io/projected/6dec4941-e24a-4b8b-852c-a9d0128c533c-kube-api-access-2rzzc\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-x4frm\" (UID: \"6dec4941-e24a-4b8b-852c-a9d0128c533c\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-x4frm"
Nov 03 19:54:48.004692 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:47.996618    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dec4941-e24a-4b8b-852c-a9d0128c533c-istio-data\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-x4frm\" (UID: \"6dec4941-e24a-4b8b-852c-a9d0128c533c\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-x4frm"
Nov 03 19:54:48.004692 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:47.996661    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dec4941-e24a-4b8b-852c-a9d0128c533c-istio-podinfo\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-x4frm\" (UID: \"6dec4941-e24a-4b8b-852c-a9d0128c533c\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-x4frm"
Nov 03 19:54:48.004692 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:47.998253    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dec4941-e24a-4b8b-852c-a9d0128c533c-workload-socket\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-x4frm\" (UID: \"6dec4941-e24a-4b8b-852c-a9d0128c533c\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-x4frm"
Nov 03 19:54:48.004692 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:47.999703    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dec4941-e24a-4b8b-852c-a9d0128c533c-workload-certs\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-x4frm\" (UID: \"6dec4941-e24a-4b8b-852c-a9d0128c533c\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-x4frm"
Nov 03 19:54:48.004692 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:48.000052    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dec4941-e24a-4b8b-852c-a9d0128c533c-istio-data\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-x4frm\" (UID: \"6dec4941-e24a-4b8b-852c-a9d0128c533c\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-x4frm"
Nov 03 19:54:48.004692 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:48.000561    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dec4941-e24a-4b8b-852c-a9d0128c533c-credential-socket\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-x4frm\" (UID: \"6dec4941-e24a-4b8b-852c-a9d0128c533c\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-x4frm"
Nov 03 19:54:48.004692 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:48.000885    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dec4941-e24a-4b8b-852c-a9d0128c533c-istiod-ca-cert\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-x4frm\" (UID: \"6dec4941-e24a-4b8b-852c-a9d0128c533c\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-x4frm"
Nov 03 19:54:48.004692 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:48.003366    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dec4941-e24a-4b8b-852c-a9d0128c533c-istio-podinfo\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-x4frm\" (UID: \"6dec4941-e24a-4b8b-852c-a9d0128c533c\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-x4frm"
Nov 03 19:54:48.008061 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:48.006621    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dec4941-e24a-4b8b-852c-a9d0128c533c-istio-envoy\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-x4frm\" (UID: \"6dec4941-e24a-4b8b-852c-a9d0128c533c\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-x4frm"
Nov 03 19:54:48.008858 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:48.008827    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rzzc\" (UniqueName: \"kubernetes.io/projected/6dec4941-e24a-4b8b-852c-a9d0128c533c-kube-api-access-2rzzc\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-x4frm\" (UID: \"6dec4941-e24a-4b8b-852c-a9d0128c533c\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-x4frm"
Nov 03 19:54:48.009905 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:48.009887    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dec4941-e24a-4b8b-852c-a9d0128c533c-istio-token\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-x4frm\" (UID: \"6dec4941-e24a-4b8b-852c-a9d0128c533c\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-x4frm"
Nov 03 19:54:48.145278 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:48.145241    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-x4frm"
Nov 03 19:54:48.438360 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:54:48.438328    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dec4941_e24a_4b8b_852c_a9d0128c533c.slice/crio-03aef56d98b20a9a730fe5e4c4fbf2509b9deaf08d9677a79ab25168faeb1ac2 WatchSource:0}: Error finding container 03aef56d98b20a9a730fe5e4c4fbf2509b9deaf08d9677a79ab25168faeb1ac2: Status 404 returned error can't find the container with id 03aef56d98b20a9a730fe5e4c4fbf2509b9deaf08d9677a79ab25168faeb1ac2
Nov 03 19:54:48.438954 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:48.438927    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-x4frm"]
Nov 03 19:54:48.442170 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:48.442128    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:54:48.442265 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:48.442216    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:54:48.442311 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:48.442260    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:54:48.891326 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:48.891290    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d27071d-8535-4b88-9e91-e99c3c74abe2" path="/var/lib/kubelet/pods/9d27071d-8535-4b88-9e91-e99c3c74abe2/volumes"
Nov 03 19:54:49.002381 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:49.002354    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0d0987-6f7b54c8cb-8s9ds"
Nov 03 19:54:49.075413 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:49.075374    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-x4frm" event={"ID":"6dec4941-e24a-4b8b-852c-a9d0128c533c","Type":"ContainerStarted","Data":"fad4cd1770166848ac8e89407e7eefd32fda359488ceff800282bebf483caea8"}
Nov 03 19:54:49.075648 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:49.075448    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-x4frm" event={"ID":"6dec4941-e24a-4b8b-852c-a9d0128c533c","Type":"ContainerStarted","Data":"03aef56d98b20a9a730fe5e4c4fbf2509b9deaf08d9677a79ab25168faeb1ac2"}
Nov 03 19:54:49.095760 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:49.095711    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-x4frm" podStartSLOduration=2.095697139 podStartE2EDuration="2.095697139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:54:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:54:49.093447195 +0000 UTC m=+1402.337412678" watchObservedRunningTime="2026-11-03 19:54:49.095697139 +0000 UTC m=+1402.339662603"
Nov 03 19:54:49.146152 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:49.146063    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-x4frm"
Nov 03 19:54:49.148022 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:49.147995    4921 patch_prober.go:29] interesting pod/e2e-inf-b-332694-openshift-default-5c5687c879-x4frm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 03 19:54:49.148155 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:49.148043    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-x4frm" podUID="6dec4941-e24a-4b8b-852c-a9d0128c533c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 03 19:54:50.146349 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:50.146317    4921 patch_prober.go:29] interesting pod/e2e-inf-b-332694-openshift-default-5c5687c879-x4frm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 03 19:54:50.146776 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:50.146373    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-x4frm" podUID="6dec4941-e24a-4b8b-852c-a9d0128c533c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 03 19:54:51.145540 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:51.145503    4921 patch_prober.go:29] interesting pod/e2e-inf-b-332694-openshift-default-5c5687c879-x4frm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 03 19:54:51.145756 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:51.145556    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-x4frm" podUID="6dec4941-e24a-4b8b-852c-a9d0128c533c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 03 19:54:52.145771 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:52.145730    4921 patch_prober.go:29] interesting pod/e2e-inf-b-332694-openshift-default-5c5687c879-x4frm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 03 19:54:52.146159 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:52.145788    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-x4frm" podUID="6dec4941-e24a-4b8b-852c-a9d0128c533c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 03 19:54:53.146144 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:53.146109    4921 patch_prober.go:29] interesting pod/e2e-inf-b-332694-openshift-default-5c5687c879-x4frm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 03 19:54:53.146653 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:53.146162    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-x4frm" podUID="6dec4941-e24a-4b8b-852c-a9d0128c533c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 03 19:54:54.189694 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:54.188827    4921 patch_prober.go:29] interesting pod/e2e-inf-b-332694-openshift-default-5c5687c879-x4frm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 03 19:54:54.189694 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:54.188890    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-x4frm" podUID="6dec4941-e24a-4b8b-852c-a9d0128c533c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 03 19:54:55.147107 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:55.147070    4921 patch_prober.go:29] interesting pod/e2e-inf-b-332694-openshift-default-5c5687c879-x4frm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 03 19:54:55.147349 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:55.147325    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-x4frm" podUID="6dec4941-e24a-4b8b-852c-a9d0128c533c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 03 19:54:56.151571 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:56.151532    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-x4frm"
Nov 03 19:54:56.152080 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:56.151964    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-x4frm"
Nov 03 19:54:56.152764 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:56.152745    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-x4frm"
Nov 03 19:54:56.410252 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:56.406544    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-225668-758c694f89-tsbbm"]
Nov 03 19:54:56.431695 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:56.414170    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-225668-758c694f89-tsbbm"
Nov 03 19:54:56.431695 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:56.423215    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-225668-serving-cert\""
Nov 03 19:54:56.448744 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:56.442414    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-225668-758c694f89-tsbbm"]
Nov 03 19:54:56.488571 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:56.488530    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f79e6cd-82c7-4e8b-b40b-67f5c8948173-metrics-certs\") pod \"maas-api-e2e-inf-a-225668-758c694f89-tsbbm\" (UID: \"6f79e6cd-82c7-4e8b-b40b-67f5c8948173\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-225668-758c694f89-tsbbm"
Nov 03 19:54:56.488781 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:56.488653    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f79e6cd-82c7-4e8b-b40b-67f5c8948173-maas-api-tls\") pod \"maas-api-e2e-inf-a-225668-758c694f89-tsbbm\" (UID: \"6f79e6cd-82c7-4e8b-b40b-67f5c8948173\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-225668-758c694f89-tsbbm"
Nov 03 19:54:56.488781 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:56.488747    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f79e6cd-82c7-4e8b-b40b-67f5c8948173-openshift-service-ca\") pod \"maas-api-e2e-inf-a-225668-758c694f89-tsbbm\" (UID: \"6f79e6cd-82c7-4e8b-b40b-67f5c8948173\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-225668-758c694f89-tsbbm"
Nov 03 19:54:56.488942 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:56.488917    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlngx\" (UniqueName: \"kubernetes.io/projected/6f79e6cd-82c7-4e8b-b40b-67f5c8948173-kube-api-access-zlngx\") pod \"maas-api-e2e-inf-a-225668-758c694f89-tsbbm\" (UID: \"6f79e6cd-82c7-4e8b-b40b-67f5c8948173\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-225668-758c694f89-tsbbm"
Nov 03 19:54:56.597691 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:56.597220    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f79e6cd-82c7-4e8b-b40b-67f5c8948173-openshift-service-ca\") pod \"maas-api-e2e-inf-a-225668-758c694f89-tsbbm\" (UID: \"6f79e6cd-82c7-4e8b-b40b-67f5c8948173\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-225668-758c694f89-tsbbm"
Nov 03 19:54:56.597691 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:56.597425    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlngx\" (UniqueName: \"kubernetes.io/projected/6f79e6cd-82c7-4e8b-b40b-67f5c8948173-kube-api-access-zlngx\") pod \"maas-api-e2e-inf-a-225668-758c694f89-tsbbm\" (UID: \"6f79e6cd-82c7-4e8b-b40b-67f5c8948173\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-225668-758c694f89-tsbbm"
Nov 03 19:54:56.597691 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:56.597550    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f79e6cd-82c7-4e8b-b40b-67f5c8948173-metrics-certs\") pod \"maas-api-e2e-inf-a-225668-758c694f89-tsbbm\" (UID: \"6f79e6cd-82c7-4e8b-b40b-67f5c8948173\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-225668-758c694f89-tsbbm"
Nov 03 19:54:56.597691 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:56.597636    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f79e6cd-82c7-4e8b-b40b-67f5c8948173-maas-api-tls\") pod \"maas-api-e2e-inf-a-225668-758c694f89-tsbbm\" (UID: \"6f79e6cd-82c7-4e8b-b40b-67f5c8948173\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-225668-758c694f89-tsbbm"
Nov 03 19:54:56.615755 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:56.598888    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f79e6cd-82c7-4e8b-b40b-67f5c8948173-openshift-service-ca\") pod \"maas-api-e2e-inf-a-225668-758c694f89-tsbbm\" (UID: \"6f79e6cd-82c7-4e8b-b40b-67f5c8948173\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-225668-758c694f89-tsbbm"
Nov 03 19:54:56.655791 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:56.655756    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f79e6cd-82c7-4e8b-b40b-67f5c8948173-metrics-certs\") pod \"maas-api-e2e-inf-a-225668-758c694f89-tsbbm\" (UID: \"6f79e6cd-82c7-4e8b-b40b-67f5c8948173\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-225668-758c694f89-tsbbm"
Nov 03 19:54:56.658962 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:56.658869    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlngx\" (UniqueName: \"kubernetes.io/projected/6f79e6cd-82c7-4e8b-b40b-67f5c8948173-kube-api-access-zlngx\") pod \"maas-api-e2e-inf-a-225668-758c694f89-tsbbm\" (UID: \"6f79e6cd-82c7-4e8b-b40b-67f5c8948173\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-225668-758c694f89-tsbbm"
Nov 03 19:54:56.719309 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:56.719233    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f79e6cd-82c7-4e8b-b40b-67f5c8948173-maas-api-tls\") pod \"maas-api-e2e-inf-a-225668-758c694f89-tsbbm\" (UID: \"6f79e6cd-82c7-4e8b-b40b-67f5c8948173\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-225668-758c694f89-tsbbm"
Nov 03 19:54:56.760329 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:56.760301    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-225668-758c694f89-tsbbm"
Nov 03 19:54:57.081402 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:54:57.081357    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f79e6cd_82c7_4e8b_b40b_67f5c8948173.slice/crio-a52bceac1e55c6f2e878419ddecb1834e2d5011d80659ca1cc8c98adc84f4d85 WatchSource:0}: Error finding container a52bceac1e55c6f2e878419ddecb1834e2d5011d80659ca1cc8c98adc84f4d85: Status 404 returned error can't find the container with id a52bceac1e55c6f2e878419ddecb1834e2d5011d80659ca1cc8c98adc84f4d85
Nov 03 19:54:57.083818 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:57.083786    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-225668-758c694f89-tsbbm"]
Nov 03 19:54:57.397752 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:57.395164    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-225668-758c694f89-tsbbm" event={"ID":"6f79e6cd-82c7-4e8b-b40b-67f5c8948173","Type":"ContainerStarted","Data":"a52bceac1e55c6f2e878419ddecb1834e2d5011d80659ca1cc8c98adc84f4d85"}
Nov 03 19:54:58.406150 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:58.406053    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-225668-758c694f89-tsbbm" event={"ID":"6f79e6cd-82c7-4e8b-b40b-67f5c8948173","Type":"ContainerStarted","Data":"ecc1defb828c6022397a390a95c73e043debe3d0adc2740336f071cce34439f7"}
Nov 03 19:54:58.407428 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:58.407365    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-225668-758c694f89-tsbbm"
Nov 03 19:54:58.425864 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:54:58.425742    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-225668-758c694f89-tsbbm" podStartSLOduration=1.75806763 podStartE2EDuration="2.425721683s" totalImagesPullingTime="667.654053ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:54:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:54:58.422883541 +0000 UTC m=+1411.666849007" watchObservedRunningTime="2026-11-03 19:54:58.425721683 +0000 UTC m=+1411.669687148"
Nov 03 19:55:01.516645 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:01.516608    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-11c1a8-598fbd7d99-fzmxn"]
Nov 03 19:55:01.521528 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:01.521505    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-11c1a8-598fbd7d99-fzmxn"
Nov 03 19:55:01.526027 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:01.525975    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-11c1a8\""
Nov 03 19:55:01.531649 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:01.531616    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-11c1a8-598fbd7d99-fzmxn"]
Nov 03 19:55:01.594932 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:01.593976    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c76485a0-b175-405c-8b6e-ff94c6b9ae1f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-11c1a8-598fbd7d99-fzmxn\" (UID: \"c76485a0-b175-405c-8b6e-ff94c6b9ae1f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-11c1a8-598fbd7d99-fzmxn"
Nov 03 19:55:01.594932 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:01.594075    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbwtd\" (UniqueName: \"kubernetes.io/projected/c76485a0-b175-405c-8b6e-ff94c6b9ae1f-kube-api-access-sbwtd\") pod \"payload-processing-e2e-inf-a-11c1a8-598fbd7d99-fzmxn\" (UID: \"c76485a0-b175-405c-8b6e-ff94c6b9ae1f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-11c1a8-598fbd7d99-fzmxn"
Nov 03 19:55:01.696697 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:01.696467    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c76485a0-b175-405c-8b6e-ff94c6b9ae1f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-11c1a8-598fbd7d99-fzmxn\" (UID: \"c76485a0-b175-405c-8b6e-ff94c6b9ae1f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-11c1a8-598fbd7d99-fzmxn"
Nov 03 19:55:01.696697 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:01.696565    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbwtd\" (UniqueName: \"kubernetes.io/projected/c76485a0-b175-405c-8b6e-ff94c6b9ae1f-kube-api-access-sbwtd\") pod \"payload-processing-e2e-inf-a-11c1a8-598fbd7d99-fzmxn\" (UID: \"c76485a0-b175-405c-8b6e-ff94c6b9ae1f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-11c1a8-598fbd7d99-fzmxn"
Nov 03 19:55:01.706841 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:01.698257    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c76485a0-b175-405c-8b6e-ff94c6b9ae1f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-11c1a8-598fbd7d99-fzmxn\" (UID: \"c76485a0-b175-405c-8b6e-ff94c6b9ae1f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-11c1a8-598fbd7d99-fzmxn"
Nov 03 19:55:01.732698 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:01.732527    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbwtd\" (UniqueName: \"kubernetes.io/projected/c76485a0-b175-405c-8b6e-ff94c6b9ae1f-kube-api-access-sbwtd\") pod \"payload-processing-e2e-inf-a-11c1a8-598fbd7d99-fzmxn\" (UID: \"c76485a0-b175-405c-8b6e-ff94c6b9ae1f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-11c1a8-598fbd7d99-fzmxn"
Nov 03 19:55:01.844743 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:01.843954    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-11c1a8-598fbd7d99-fzmxn"
Nov 03 19:55:02.025314 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:02.025264    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e42df580-xjhkn"]
Nov 03 19:55:02.031288 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:02.031251    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e42df580-xjhkn"]
Nov 03 19:55:02.266693 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:02.266648    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-11c1a8-598fbd7d99-fzmxn"]
Nov 03 19:55:02.267726 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:55:02.267684    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc76485a0_b175_405c_8b6e_ff94c6b9ae1f.slice/crio-520849455f16408a9934b9e72b908262bc1579b34ecaa5eff2051e069897270b WatchSource:0}: Error finding container 520849455f16408a9934b9e72b908262bc1579b34ecaa5eff2051e069897270b: Status 404 returned error can't find the container with id 520849455f16408a9934b9e72b908262bc1579b34ecaa5eff2051e069897270b
Nov 03 19:55:02.449979 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:02.449936    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-11c1a8-598fbd7d99-fzmxn" event={"ID":"c76485a0-b175-405c-8b6e-ff94c6b9ae1f","Type":"ContainerStarted","Data":"6eb4de70602cb3c2e9acfddb47ea90fa306e2aa5113764982b51720f862f44f7"}
Nov 03 19:55:02.450151 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:02.450012    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-11c1a8-598fbd7d99-fzmxn"
Nov 03 19:55:02.450151 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:02.450038    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-11c1a8-598fbd7d99-fzmxn" event={"ID":"c76485a0-b175-405c-8b6e-ff94c6b9ae1f","Type":"ContainerStarted","Data":"520849455f16408a9934b9e72b908262bc1579b34ecaa5eff2051e069897270b"}
Nov 03 19:55:02.469536 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:02.469464    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-11c1a8-598fbd7d99-fzmxn" podStartSLOduration=1.469448198 podStartE2EDuration="1.469448198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:55:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:55:02.46544192 +0000 UTC m=+1415.709407386" watchObservedRunningTime="2026-11-03 19:55:02.469448198 +0000 UTC m=+1415.713413694"
Nov 03 19:55:02.892063 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:02.892027    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0487c9eb-449c-4aa8-b41d-277c67914e58" path="/var/lib/kubelet/pods/0487c9eb-449c-4aa8-b41d-277c67914e58/volumes"
Nov 03 19:55:02.928950 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:55:02.928912    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 19:55:05.477691 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:05.471837    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-225668-758c694f89-tsbbm"
Nov 03 19:55:13.462061 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:13.462028    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-11c1a8-598fbd7d99-fzmxn"
Nov 03 19:55:18.460955 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:18.460919    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-11c1a8-6c664c46cf-9pcm6"]
Nov 03 19:55:18.466487 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:18.466070    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11c1a8-6c664c46cf-9pcm6"
Nov 03 19:55:18.485911 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:18.484205    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-11c1a8-serving-cert\""
Nov 03 19:55:18.499810 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:18.498599    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-11c1a8-6c664c46cf-9pcm6"]
Nov 03 19:55:18.500330 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:18.499815    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgdzs\" (UniqueName: \"kubernetes.io/projected/63c60ac9-edc9-41ff-88d8-80197876106c-kube-api-access-kgdzs\") pod \"maas-api-e2e-inf-a-11c1a8-6c664c46cf-9pcm6\" (UID: \"63c60ac9-edc9-41ff-88d8-80197876106c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11c1a8-6c664c46cf-9pcm6"
Nov 03 19:55:18.500330 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:18.499956    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63c60ac9-edc9-41ff-88d8-80197876106c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-11c1a8-6c664c46cf-9pcm6\" (UID: \"63c60ac9-edc9-41ff-88d8-80197876106c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11c1a8-6c664c46cf-9pcm6"
Nov 03 19:55:18.500330 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:18.500064    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63c60ac9-edc9-41ff-88d8-80197876106c-maas-api-tls\") pod \"maas-api-e2e-inf-a-11c1a8-6c664c46cf-9pcm6\" (UID: \"63c60ac9-edc9-41ff-88d8-80197876106c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11c1a8-6c664c46cf-9pcm6"
Nov 03 19:55:18.500330 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:18.500091    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63c60ac9-edc9-41ff-88d8-80197876106c-metrics-certs\") pod \"maas-api-e2e-inf-a-11c1a8-6c664c46cf-9pcm6\" (UID: \"63c60ac9-edc9-41ff-88d8-80197876106c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11c1a8-6c664c46cf-9pcm6"
Nov 03 19:55:18.601119 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:18.601067    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63c60ac9-edc9-41ff-88d8-80197876106c-maas-api-tls\") pod \"maas-api-e2e-inf-a-11c1a8-6c664c46cf-9pcm6\" (UID: \"63c60ac9-edc9-41ff-88d8-80197876106c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11c1a8-6c664c46cf-9pcm6"
Nov 03 19:55:18.601119 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:18.601110    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63c60ac9-edc9-41ff-88d8-80197876106c-metrics-certs\") pod \"maas-api-e2e-inf-a-11c1a8-6c664c46cf-9pcm6\" (UID: \"63c60ac9-edc9-41ff-88d8-80197876106c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11c1a8-6c664c46cf-9pcm6"
Nov 03 19:55:18.601514 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:18.601493    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgdzs\" (UniqueName: \"kubernetes.io/projected/63c60ac9-edc9-41ff-88d8-80197876106c-kube-api-access-kgdzs\") pod \"maas-api-e2e-inf-a-11c1a8-6c664c46cf-9pcm6\" (UID: \"63c60ac9-edc9-41ff-88d8-80197876106c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11c1a8-6c664c46cf-9pcm6"
Nov 03 19:55:18.601679 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:18.601652    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63c60ac9-edc9-41ff-88d8-80197876106c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-11c1a8-6c664c46cf-9pcm6\" (UID: \"63c60ac9-edc9-41ff-88d8-80197876106c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11c1a8-6c664c46cf-9pcm6"
Nov 03 19:55:18.602568 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:18.602540    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63c60ac9-edc9-41ff-88d8-80197876106c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-11c1a8-6c664c46cf-9pcm6\" (UID: \"63c60ac9-edc9-41ff-88d8-80197876106c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11c1a8-6c664c46cf-9pcm6"
Nov 03 19:55:18.603872 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:18.603853    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63c60ac9-edc9-41ff-88d8-80197876106c-maas-api-tls\") pod \"maas-api-e2e-inf-a-11c1a8-6c664c46cf-9pcm6\" (UID: \"63c60ac9-edc9-41ff-88d8-80197876106c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11c1a8-6c664c46cf-9pcm6"
Nov 03 19:55:18.604057 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:18.604039    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63c60ac9-edc9-41ff-88d8-80197876106c-metrics-certs\") pod \"maas-api-e2e-inf-a-11c1a8-6c664c46cf-9pcm6\" (UID: \"63c60ac9-edc9-41ff-88d8-80197876106c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11c1a8-6c664c46cf-9pcm6"
Nov 03 19:55:18.640692 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:18.621686    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgdzs\" (UniqueName: \"kubernetes.io/projected/63c60ac9-edc9-41ff-88d8-80197876106c-kube-api-access-kgdzs\") pod \"maas-api-e2e-inf-a-11c1a8-6c664c46cf-9pcm6\" (UID: \"63c60ac9-edc9-41ff-88d8-80197876106c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11c1a8-6c664c46cf-9pcm6"
Nov 03 19:55:18.822535 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:18.816567    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11c1a8-6c664c46cf-9pcm6"
Nov 03 19:55:19.699386 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:55:19.699339    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63c60ac9_edc9_41ff_88d8_80197876106c.slice/crio-c96d5778d5cfb82d9915b6bcb5fe9a206a3fc9e295db3b9b4747f81478607f13 WatchSource:0}: Error finding container c96d5778d5cfb82d9915b6bcb5fe9a206a3fc9e295db3b9b4747f81478607f13: Status 404 returned error can't find the container with id c96d5778d5cfb82d9915b6bcb5fe9a206a3fc9e295db3b9b4747f81478607f13
Nov 03 19:55:19.699920 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:19.699895    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-11c1a8-6c664c46cf-9pcm6"]
Nov 03 19:55:19.873691 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:19.873128    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11c1a8-6c664c46cf-9pcm6" event={"ID":"63c60ac9-edc9-41ff-88d8-80197876106c","Type":"ContainerStarted","Data":"c96d5778d5cfb82d9915b6bcb5fe9a206a3fc9e295db3b9b4747f81478607f13"}
Nov 03 19:55:20.893212 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:20.893170    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11c1a8-6c664c46cf-9pcm6" event={"ID":"63c60ac9-edc9-41ff-88d8-80197876106c","Type":"ContainerStarted","Data":"1149d38940fc9b963f7de4fe41a69155868d54ab0534244da4c2c67bb9b96c68"}
Nov 03 19:55:20.893611 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:20.893255    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11c1a8-6c664c46cf-9pcm6"
Nov 03 19:55:20.905535 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:20.905475    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11c1a8-6c664c46cf-9pcm6" podStartSLOduration=2.377919859 podStartE2EDuration="2.905458743s" totalImagesPullingTime="527.538884ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:55:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:55:20.904152839 +0000 UTC m=+1434.148118305" watchObservedRunningTime="2026-11-03 19:55:20.905458743 +0000 UTC m=+1434.149424225"
Nov 03 19:55:27.063256 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:27.063226    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11c1a8-6c664c46cf-9pcm6"
Nov 03 19:55:29.716038 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:29.715819    4921 scope.go:122] "RemoveContainer" containerID="15bf8dd32dceddb8a1e942f90cd8191440a10e8fbeb4d5ab9834da7ba6b59852"
Nov 03 19:55:32.806956 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:32.806912    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-546fr"]
Nov 03 19:55:32.819091 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:32.819042    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-546fr"]
Nov 03 19:55:32.819355 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:32.819193    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-546fr"
Nov 03 19:55:32.847609 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:32.847569    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca28dda5-c003-4070-b331-43c5a51d9f87-workload-certs\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-546fr\" (UID: \"ca28dda5-c003-4070-b331-43c5a51d9f87\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-546fr"
Nov 03 19:55:32.847897 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:32.847716    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca28dda5-c003-4070-b331-43c5a51d9f87-istio-podinfo\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-546fr\" (UID: \"ca28dda5-c003-4070-b331-43c5a51d9f87\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-546fr"
Nov 03 19:55:32.847897 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:32.847768    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pr7s\" (UniqueName: \"kubernetes.io/projected/ca28dda5-c003-4070-b331-43c5a51d9f87-kube-api-access-7pr7s\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-546fr\" (UID: \"ca28dda5-c003-4070-b331-43c5a51d9f87\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-546fr"
Nov 03 19:55:32.847897 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:32.847820    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca28dda5-c003-4070-b331-43c5a51d9f87-istio-data\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-546fr\" (UID: \"ca28dda5-c003-4070-b331-43c5a51d9f87\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-546fr"
Nov 03 19:55:32.848139 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:32.848116    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca28dda5-c003-4070-b331-43c5a51d9f87-istio-token\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-546fr\" (UID: \"ca28dda5-c003-4070-b331-43c5a51d9f87\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-546fr"
Nov 03 19:55:32.848212 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:32.848184    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca28dda5-c003-4070-b331-43c5a51d9f87-workload-socket\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-546fr\" (UID: \"ca28dda5-c003-4070-b331-43c5a51d9f87\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-546fr"
Nov 03 19:55:32.848277 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:32.848243    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca28dda5-c003-4070-b331-43c5a51d9f87-credential-socket\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-546fr\" (UID: \"ca28dda5-c003-4070-b331-43c5a51d9f87\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-546fr"
Nov 03 19:55:32.848336 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:32.848274    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca28dda5-c003-4070-b331-43c5a51d9f87-istio-envoy\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-546fr\" (UID: \"ca28dda5-c003-4070-b331-43c5a51d9f87\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-546fr"
Nov 03 19:55:32.848407 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:32.848389    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca28dda5-c003-4070-b331-43c5a51d9f87-istiod-ca-cert\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-546fr\" (UID: \"ca28dda5-c003-4070-b331-43c5a51d9f87\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-546fr"
Nov 03 19:55:32.924219 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:55:32.924185    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 19:55:32.950257 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:32.950217    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca28dda5-c003-4070-b331-43c5a51d9f87-workload-socket\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-546fr\" (UID: \"ca28dda5-c003-4070-b331-43c5a51d9f87\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-546fr"
Nov 03 19:55:32.950257 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:32.950257    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca28dda5-c003-4070-b331-43c5a51d9f87-credential-socket\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-546fr\" (UID: \"ca28dda5-c003-4070-b331-43c5a51d9f87\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-546fr"
Nov 03 19:55:32.950503 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:32.950285    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca28dda5-c003-4070-b331-43c5a51d9f87-istio-envoy\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-546fr\" (UID: \"ca28dda5-c003-4070-b331-43c5a51d9f87\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-546fr"
Nov 03 19:55:32.950503 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:32.950390    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca28dda5-c003-4070-b331-43c5a51d9f87-istiod-ca-cert\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-546fr\" (UID: \"ca28dda5-c003-4070-b331-43c5a51d9f87\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-546fr"
Nov 03 19:55:32.950615 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:32.950546    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca28dda5-c003-4070-b331-43c5a51d9f87-workload-certs\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-546fr\" (UID: \"ca28dda5-c003-4070-b331-43c5a51d9f87\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-546fr"
Nov 03 19:55:32.950615 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:32.950583    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca28dda5-c003-4070-b331-43c5a51d9f87-istio-podinfo\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-546fr\" (UID: \"ca28dda5-c003-4070-b331-43c5a51d9f87\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-546fr"
Nov 03 19:55:32.950782 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:32.950614    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pr7s\" (UniqueName: \"kubernetes.io/projected/ca28dda5-c003-4070-b331-43c5a51d9f87-kube-api-access-7pr7s\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-546fr\" (UID: \"ca28dda5-c003-4070-b331-43c5a51d9f87\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-546fr"
Nov 03 19:55:32.950782 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:32.950662    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca28dda5-c003-4070-b331-43c5a51d9f87-istio-data\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-546fr\" (UID: \"ca28dda5-c003-4070-b331-43c5a51d9f87\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-546fr"
Nov 03 19:55:32.950782 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:32.950735    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca28dda5-c003-4070-b331-43c5a51d9f87-workload-socket\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-546fr\" (UID: \"ca28dda5-c003-4070-b331-43c5a51d9f87\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-546fr"
Nov 03 19:55:32.950951 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:32.950825    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca28dda5-c003-4070-b331-43c5a51d9f87-credential-socket\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-546fr\" (UID: \"ca28dda5-c003-4070-b331-43c5a51d9f87\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-546fr"
Nov 03 19:55:32.950951 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:32.950884    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca28dda5-c003-4070-b331-43c5a51d9f87-istio-token\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-546fr\" (UID: \"ca28dda5-c003-4070-b331-43c5a51d9f87\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-546fr"
Nov 03 19:55:32.951146 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:32.951120    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca28dda5-c003-4070-b331-43c5a51d9f87-workload-certs\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-546fr\" (UID: \"ca28dda5-c003-4070-b331-43c5a51d9f87\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-546fr"
Nov 03 19:55:32.951434 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:32.951242    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca28dda5-c003-4070-b331-43c5a51d9f87-istiod-ca-cert\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-546fr\" (UID: \"ca28dda5-c003-4070-b331-43c5a51d9f87\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-546fr"
Nov 03 19:55:32.951573 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:32.951461    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca28dda5-c003-4070-b331-43c5a51d9f87-istio-data\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-546fr\" (UID: \"ca28dda5-c003-4070-b331-43c5a51d9f87\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-546fr"
Nov 03 19:55:32.953288 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:32.953267    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca28dda5-c003-4070-b331-43c5a51d9f87-istio-envoy\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-546fr\" (UID: \"ca28dda5-c003-4070-b331-43c5a51d9f87\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-546fr"
Nov 03 19:55:32.953403 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:32.953380    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca28dda5-c003-4070-b331-43c5a51d9f87-istio-podinfo\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-546fr\" (UID: \"ca28dda5-c003-4070-b331-43c5a51d9f87\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-546fr"
Nov 03 19:55:32.958167 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:32.958138    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca28dda5-c003-4070-b331-43c5a51d9f87-istio-token\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-546fr\" (UID: \"ca28dda5-c003-4070-b331-43c5a51d9f87\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-546fr"
Nov 03 19:55:32.958364 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:32.958338    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pr7s\" (UniqueName: \"kubernetes.io/projected/ca28dda5-c003-4070-b331-43c5a51d9f87-kube-api-access-7pr7s\") pod \"e2e-inf-b-332694-openshift-default-5c5687c879-546fr\" (UID: \"ca28dda5-c003-4070-b331-43c5a51d9f87\") " pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-546fr"
Nov 03 19:55:33.141626 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:33.141543    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-546fr"
Nov 03 19:55:33.445346 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:55:33.445306    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca28dda5_c003_4070_b331_43c5a51d9f87.slice/crio-0bf90900eeaeaaba4911349b3e8a79144fec40569e74fb732fdb5d9b1c09c7ec WatchSource:0}: Error finding container 0bf90900eeaeaaba4911349b3e8a79144fec40569e74fb732fdb5d9b1c09c7ec: Status 404 returned error can't find the container with id 0bf90900eeaeaaba4911349b3e8a79144fec40569e74fb732fdb5d9b1c09c7ec
Nov 03 19:55:33.446187 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:33.446161    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-546fr"]
Nov 03 19:55:33.448682 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:33.448626    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:55:33.448790 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:33.448732    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:55:33.448790 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:33.448783    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:55:34.213361 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:34.213303    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-546fr" event={"ID":"ca28dda5-c003-4070-b331-43c5a51d9f87","Type":"ContainerStarted","Data":"490d991e7f3e1faabfeb7d453d9ef637234a4215fc3d2a4d35d8f352c181338e"}
Nov 03 19:55:34.213834 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:34.213388    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-546fr" event={"ID":"ca28dda5-c003-4070-b331-43c5a51d9f87","Type":"ContainerStarted","Data":"0bf90900eeaeaaba4911349b3e8a79144fec40569e74fb732fdb5d9b1c09c7ec"}
Nov 03 19:55:34.231761 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:34.231701    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-546fr" podStartSLOduration=2.231682668 podStartE2EDuration="2.231682668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:55:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:55:34.230585992 +0000 UTC m=+1447.474551494" watchObservedRunningTime="2026-11-03 19:55:34.231682668 +0000 UTC m=+1447.475648146"
Nov 03 19:55:35.142280 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:35.142249    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-546fr"
Nov 03 19:55:35.144291 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:35.144265    4921 patch_prober.go:29] interesting pod/e2e-inf-b-332694-openshift-default-5c5687c879-546fr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Nov 03 19:55:35.144425 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:35.144318    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-546fr" podUID="ca28dda5-c003-4070-b331-43c5a51d9f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Nov 03 19:55:36.149697 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:36.143163    4921 patch_prober.go:29] interesting pod/e2e-inf-b-332694-openshift-default-5c5687c879-546fr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Nov 03 19:55:36.149697 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:36.143234    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-546fr" podUID="ca28dda5-c003-4070-b331-43c5a51d9f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Nov 03 19:55:37.142919 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:37.142884    4921 patch_prober.go:29] interesting pod/e2e-inf-b-332694-openshift-default-5c5687c879-546fr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Nov 03 19:55:37.143107 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:37.142939    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-546fr" podUID="ca28dda5-c003-4070-b331-43c5a51d9f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Nov 03 19:55:38.142705 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:38.142658    4921 patch_prober.go:29] interesting pod/e2e-inf-b-332694-openshift-default-5c5687c879-546fr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Nov 03 19:55:38.143112 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:38.142765    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-546fr" podUID="ca28dda5-c003-4070-b331-43c5a51d9f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Nov 03 19:55:39.142396 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:39.142356    4921 patch_prober.go:29] interesting pod/e2e-inf-b-332694-openshift-default-5c5687c879-546fr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Nov 03 19:55:39.142588 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:39.142415    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-546fr" podUID="ca28dda5-c003-4070-b331-43c5a51d9f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Nov 03 19:55:40.142177 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:40.142141    4921 patch_prober.go:29] interesting pod/e2e-inf-b-332694-openshift-default-5c5687c879-546fr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Nov 03 19:55:40.142595 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:40.142195    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-546fr" podUID="ca28dda5-c003-4070-b331-43c5a51d9f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Nov 03 19:55:41.147480 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:41.147451    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-546fr"
Nov 03 19:55:41.147972 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:41.147942    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-546fr"
Nov 03 19:55:41.148747 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:41.148722    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-546fr"
Nov 03 19:55:50.627158 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:50.627121    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3cee6a-544fd5876c-56hwf"]
Nov 03 19:55:50.631969 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:50.631946    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3cee6a-544fd5876c-56hwf"
Nov 03 19:55:50.638238 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:50.638180    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-3cee6a-dockercfg-s2kfb\""
Nov 03 19:55:50.643911 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:50.643887    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-3cee6a\""
Nov 03 19:55:50.644257 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:50.644235    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3cee6a-544fd5876c-56hwf"]
Nov 03 19:55:50.658768 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:50.658724    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26379d10-3da7-4549-8b8c-41730a41792b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-3cee6a-544fd5876c-56hwf\" (UID: \"26379d10-3da7-4549-8b8c-41730a41792b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3cee6a-544fd5876c-56hwf"
Nov 03 19:55:50.658768 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:50.658765    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr6rd\" (UniqueName: \"kubernetes.io/projected/26379d10-3da7-4549-8b8c-41730a41792b-kube-api-access-tr6rd\") pod \"payload-pre-processing-e2e-inf-a-3cee6a-544fd5876c-56hwf\" (UID: \"26379d10-3da7-4549-8b8c-41730a41792b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3cee6a-544fd5876c-56hwf"
Nov 03 19:55:50.680717 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:50.680639    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3cee6a-799d7c674c-fj9vk"]
Nov 03 19:55:50.685351 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:50.685327    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3cee6a-799d7c674c-fj9vk"
Nov 03 19:55:50.708082 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:50.708053    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3cee6a-799d7c674c-fj9vk"]
Nov 03 19:55:50.775445 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:50.775333    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26379d10-3da7-4549-8b8c-41730a41792b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-3cee6a-544fd5876c-56hwf\" (UID: \"26379d10-3da7-4549-8b8c-41730a41792b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3cee6a-544fd5876c-56hwf"
Nov 03 19:55:50.850612 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:50.846765    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr6rd\" (UniqueName: \"kubernetes.io/projected/26379d10-3da7-4549-8b8c-41730a41792b-kube-api-access-tr6rd\") pod \"payload-pre-processing-e2e-inf-a-3cee6a-544fd5876c-56hwf\" (UID: \"26379d10-3da7-4549-8b8c-41730a41792b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3cee6a-544fd5876c-56hwf"
Nov 03 19:55:50.903004 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:50.902922    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26379d10-3da7-4549-8b8c-41730a41792b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-3cee6a-544fd5876c-56hwf\" (UID: \"26379d10-3da7-4549-8b8c-41730a41792b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3cee6a-544fd5876c-56hwf"
Nov 03 19:55:51.047147 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:50.949876    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zngl8\" (UniqueName: \"kubernetes.io/projected/4aa9c292-f06c-45d9-9a58-8aeae3b7b6ab-kube-api-access-zngl8\") pod \"payload-processing-e2e-inf-a-3cee6a-799d7c674c-fj9vk\" (UID: \"4aa9c292-f06c-45d9-9a58-8aeae3b7b6ab\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3cee6a-799d7c674c-fj9vk"
Nov 03 19:55:51.047943 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:51.047910    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4aa9c292-f06c-45d9-9a58-8aeae3b7b6ab-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-3cee6a-799d7c674c-fj9vk\" (UID: \"4aa9c292-f06c-45d9-9a58-8aeae3b7b6ab\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3cee6a-799d7c674c-fj9vk"
Nov 03 19:55:51.063764 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:50.993492    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr6rd\" (UniqueName: \"kubernetes.io/projected/26379d10-3da7-4549-8b8c-41730a41792b-kube-api-access-tr6rd\") pod \"payload-pre-processing-e2e-inf-a-3cee6a-544fd5876c-56hwf\" (UID: \"26379d10-3da7-4549-8b8c-41730a41792b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3cee6a-544fd5876c-56hwf"
Nov 03 19:55:51.150216 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:51.150178    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4aa9c292-f06c-45d9-9a58-8aeae3b7b6ab-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-3cee6a-799d7c674c-fj9vk\" (UID: \"4aa9c292-f06c-45d9-9a58-8aeae3b7b6ab\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3cee6a-799d7c674c-fj9vk"
Nov 03 19:55:51.150443 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:51.150425    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zngl8\" (UniqueName: \"kubernetes.io/projected/4aa9c292-f06c-45d9-9a58-8aeae3b7b6ab-kube-api-access-zngl8\") pod \"payload-processing-e2e-inf-a-3cee6a-799d7c674c-fj9vk\" (UID: \"4aa9c292-f06c-45d9-9a58-8aeae3b7b6ab\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3cee6a-799d7c674c-fj9vk"
Nov 03 19:55:51.151026 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:51.151006    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4aa9c292-f06c-45d9-9a58-8aeae3b7b6ab-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-3cee6a-799d7c674c-fj9vk\" (UID: \"4aa9c292-f06c-45d9-9a58-8aeae3b7b6ab\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3cee6a-799d7c674c-fj9vk"
Nov 03 19:55:51.161086 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:51.161021    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zngl8\" (UniqueName: \"kubernetes.io/projected/4aa9c292-f06c-45d9-9a58-8aeae3b7b6ab-kube-api-access-zngl8\") pod \"payload-processing-e2e-inf-a-3cee6a-799d7c674c-fj9vk\" (UID: \"4aa9c292-f06c-45d9-9a58-8aeae3b7b6ab\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3cee6a-799d7c674c-fj9vk"
Nov 03 19:55:51.273126 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:51.272966    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3cee6a-544fd5876c-56hwf"
Nov 03 19:55:51.300782 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:51.300655    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3cee6a-799d7c674c-fj9vk"
Nov 03 19:55:51.607262 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:55:51.605777    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26379d10_3da7_4549_8b8c_41730a41792b.slice/crio-5764911b8d3bcac9aa028b8b60f2394946bea26c43b2473d9d3d84459b269027 WatchSource:0}: Error finding container 5764911b8d3bcac9aa028b8b60f2394946bea26c43b2473d9d3d84459b269027: Status 404 returned error can't find the container with id 5764911b8d3bcac9aa028b8b60f2394946bea26c43b2473d9d3d84459b269027
Nov 03 19:55:51.607262 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:51.606973    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3cee6a-544fd5876c-56hwf"]
Nov 03 19:55:51.623413 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:51.623385    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3cee6a-799d7c674c-fj9vk"]
Nov 03 19:55:52.738439 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:52.737168    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3cee6a-544fd5876c-56hwf" event={"ID":"26379d10-3da7-4549-8b8c-41730a41792b","Type":"ContainerStarted","Data":"03ff696cb9d480ac235973dcde8e5a58b5d22ac3a8144b098460f74873cc28a4"}
Nov 03 19:55:52.738439 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:52.737314    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3cee6a-544fd5876c-56hwf" event={"ID":"26379d10-3da7-4549-8b8c-41730a41792b","Type":"ContainerStarted","Data":"5764911b8d3bcac9aa028b8b60f2394946bea26c43b2473d9d3d84459b269027"}
Nov 03 19:55:52.738439 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:52.737388    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3cee6a-544fd5876c-56hwf"
Nov 03 19:55:52.794855 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:52.794768    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3cee6a-544fd5876c-56hwf" podStartSLOduration=2.7947308619999998 podStartE2EDuration="2.794730862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:55:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:55:52.767530196 +0000 UTC m=+1466.011495661" watchObservedRunningTime="2026-11-03 19:55:52.794730862 +0000 UTC m=+1466.038696343"
Nov 03 19:55:52.811043 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:52.810991    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3cee6a-799d7c674c-fj9vk" event={"ID":"4aa9c292-f06c-45d9-9a58-8aeae3b7b6ab","Type":"ContainerStarted","Data":"c22dd48c7c79a0e292ef4e7ee62a718aa5c150899e124577e4bea7f0abf2ea3c"}
Nov 03 19:55:52.811360 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:52.811341    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3cee6a-799d7c674c-fj9vk" event={"ID":"4aa9c292-f06c-45d9-9a58-8aeae3b7b6ab","Type":"ContainerStarted","Data":"c2fd821d3aec5c8c2dcf1cc062aadf8e7e79675575e5bfb1750f9178e1182eb1"}
Nov 03 19:55:52.817025 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:52.812187    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-3cee6a-799d7c674c-fj9vk"
Nov 03 19:55:52.835732 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:52.835633    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-3cee6a-799d7c674c-fj9vk" podStartSLOduration=2.835616323 podStartE2EDuration="2.835616323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:55:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:55:52.830716326 +0000 UTC m=+1466.074681792" watchObservedRunningTime="2026-11-03 19:55:52.835616323 +0000 UTC m=+1466.079581789"
Nov 03 19:55:56.140228 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:56.140190    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-79c47f-5dfb89f897-kwzwd"]
Nov 03 19:55:56.145274 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:56.145249    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-79c47f-5dfb89f897-kwzwd"
Nov 03 19:55:56.152697 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:56.152362    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-79c47f-serving-cert\""
Nov 03 19:55:56.167259 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:56.167214    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-79c47f-5dfb89f897-kwzwd"]
Nov 03 19:55:56.243197 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:56.243160    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7wnp\" (UniqueName: \"kubernetes.io/projected/c38ce101-aa00-4700-8a03-854d3e56ee34-kube-api-access-q7wnp\") pod \"maas-api-e2e-inf-b-79c47f-5dfb89f897-kwzwd\" (UID: \"c38ce101-aa00-4700-8a03-854d3e56ee34\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-79c47f-5dfb89f897-kwzwd"
Nov 03 19:55:56.243380 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:56.243305    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c38ce101-aa00-4700-8a03-854d3e56ee34-maas-api-tls\") pod \"maas-api-e2e-inf-b-79c47f-5dfb89f897-kwzwd\" (UID: \"c38ce101-aa00-4700-8a03-854d3e56ee34\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-79c47f-5dfb89f897-kwzwd"
Nov 03 19:55:56.243485 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:56.243458    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c38ce101-aa00-4700-8a03-854d3e56ee34-metrics-certs\") pod \"maas-api-e2e-inf-b-79c47f-5dfb89f897-kwzwd\" (UID: \"c38ce101-aa00-4700-8a03-854d3e56ee34\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-79c47f-5dfb89f897-kwzwd"
Nov 03 19:55:56.243628 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:56.243609    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c38ce101-aa00-4700-8a03-854d3e56ee34-openshift-service-ca\") pod \"maas-api-e2e-inf-b-79c47f-5dfb89f897-kwzwd\" (UID: \"c38ce101-aa00-4700-8a03-854d3e56ee34\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-79c47f-5dfb89f897-kwzwd"
Nov 03 19:55:56.345587 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:56.345552    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7wnp\" (UniqueName: \"kubernetes.io/projected/c38ce101-aa00-4700-8a03-854d3e56ee34-kube-api-access-q7wnp\") pod \"maas-api-e2e-inf-b-79c47f-5dfb89f897-kwzwd\" (UID: \"c38ce101-aa00-4700-8a03-854d3e56ee34\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-79c47f-5dfb89f897-kwzwd"
Nov 03 19:55:56.345828 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:56.345628    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c38ce101-aa00-4700-8a03-854d3e56ee34-maas-api-tls\") pod \"maas-api-e2e-inf-b-79c47f-5dfb89f897-kwzwd\" (UID: \"c38ce101-aa00-4700-8a03-854d3e56ee34\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-79c47f-5dfb89f897-kwzwd"
Nov 03 19:55:56.345828 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:56.345815    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c38ce101-aa00-4700-8a03-854d3e56ee34-metrics-certs\") pod \"maas-api-e2e-inf-b-79c47f-5dfb89f897-kwzwd\" (UID: \"c38ce101-aa00-4700-8a03-854d3e56ee34\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-79c47f-5dfb89f897-kwzwd"
Nov 03 19:55:56.345967 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:56.345944    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c38ce101-aa00-4700-8a03-854d3e56ee34-openshift-service-ca\") pod \"maas-api-e2e-inf-b-79c47f-5dfb89f897-kwzwd\" (UID: \"c38ce101-aa00-4700-8a03-854d3e56ee34\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-79c47f-5dfb89f897-kwzwd"
Nov 03 19:55:56.346590 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:56.346552    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c38ce101-aa00-4700-8a03-854d3e56ee34-openshift-service-ca\") pod \"maas-api-e2e-inf-b-79c47f-5dfb89f897-kwzwd\" (UID: \"c38ce101-aa00-4700-8a03-854d3e56ee34\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-79c47f-5dfb89f897-kwzwd"
Nov 03 19:55:56.348536 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:56.348506    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c38ce101-aa00-4700-8a03-854d3e56ee34-maas-api-tls\") pod \"maas-api-e2e-inf-b-79c47f-5dfb89f897-kwzwd\" (UID: \"c38ce101-aa00-4700-8a03-854d3e56ee34\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-79c47f-5dfb89f897-kwzwd"
Nov 03 19:55:56.348844 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:56.348822    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c38ce101-aa00-4700-8a03-854d3e56ee34-metrics-certs\") pod \"maas-api-e2e-inf-b-79c47f-5dfb89f897-kwzwd\" (UID: \"c38ce101-aa00-4700-8a03-854d3e56ee34\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-79c47f-5dfb89f897-kwzwd"
Nov 03 19:55:56.378848 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:56.355239    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7wnp\" (UniqueName: \"kubernetes.io/projected/c38ce101-aa00-4700-8a03-854d3e56ee34-kube-api-access-q7wnp\") pod \"maas-api-e2e-inf-b-79c47f-5dfb89f897-kwzwd\" (UID: \"c38ce101-aa00-4700-8a03-854d3e56ee34\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-79c47f-5dfb89f897-kwzwd"
Nov 03 19:55:56.483326 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:56.483288    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-79c47f-5dfb89f897-kwzwd"
Nov 03 19:55:56.786244 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:56.785582    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-79c47f-5dfb89f897-kwzwd"]
Nov 03 19:55:56.786436 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:55:56.786352    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc38ce101_aa00_4700_8a03_854d3e56ee34.slice/crio-7b3336bd9477acfb2e71e6a87f002d7fec35b17148e8ff34ef936c5ee48271ee WatchSource:0}: Error finding container 7b3336bd9477acfb2e71e6a87f002d7fec35b17148e8ff34ef936c5ee48271ee: Status 404 returned error can't find the container with id 7b3336bd9477acfb2e71e6a87f002d7fec35b17148e8ff34ef936c5ee48271ee
Nov 03 19:55:56.852259 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:56.852225    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-79c47f-5dfb89f897-kwzwd" event={"ID":"c38ce101-aa00-4700-8a03-854d3e56ee34","Type":"ContainerStarted","Data":"7b3336bd9477acfb2e71e6a87f002d7fec35b17148e8ff34ef936c5ee48271ee"}
Nov 03 19:55:57.864309 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:57.864269    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-79c47f-5dfb89f897-kwzwd" event={"ID":"c38ce101-aa00-4700-8a03-854d3e56ee34","Type":"ContainerStarted","Data":"b7d633906df2c647305971a12c48549330b9649c5f4b5013cf4bb0e505a41ccb"}
Nov 03 19:55:57.864725 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:57.864390    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-79c47f-5dfb89f897-kwzwd"
Nov 03 19:55:57.880420 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:55:57.880369    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-79c47f-5dfb89f897-kwzwd" podStartSLOduration=1.429231851 podStartE2EDuration="1.880356141s" totalImagesPullingTime="451.12429ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:55:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:55:57.878800389 +0000 UTC m=+1471.122765855" watchObservedRunningTime="2026-11-03 19:55:57.880356141 +0000 UTC m=+1471.124321623"
Nov 03 19:56:00.005775 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:00.005464    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f58171-57c45cc64c-2xsx5"]
Nov 03 19:56:00.393587 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:00.373514    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f58171-57c45cc64c-2xsx5"
Nov 03 19:56:00.406768 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:00.401761    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-f58171\""
Nov 03 19:56:00.469790 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:00.469755    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ba7b96b-f2c0-4ddf-8117-c578d6ba1d69-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f58171-57c45cc64c-2xsx5\" (UID: \"5ba7b96b-f2c0-4ddf-8117-c578d6ba1d69\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f58171-57c45cc64c-2xsx5"
Nov 03 19:56:00.469974 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:00.469847    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmbgp\" (UniqueName: \"kubernetes.io/projected/5ba7b96b-f2c0-4ddf-8117-c578d6ba1d69-kube-api-access-hmbgp\") pod \"payload-pre-processing-e2e-inf-b-f58171-57c45cc64c-2xsx5\" (UID: \"5ba7b96b-f2c0-4ddf-8117-c578d6ba1d69\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f58171-57c45cc64c-2xsx5"
Nov 03 19:56:00.472789 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:00.472761    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f58171-57c45cc64c-2xsx5"]
Nov 03 19:56:00.571728 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:00.571678    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ba7b96b-f2c0-4ddf-8117-c578d6ba1d69-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f58171-57c45cc64c-2xsx5\" (UID: \"5ba7b96b-f2c0-4ddf-8117-c578d6ba1d69\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f58171-57c45cc64c-2xsx5"
Nov 03 19:56:00.571914 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:00.571809    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmbgp\" (UniqueName: \"kubernetes.io/projected/5ba7b96b-f2c0-4ddf-8117-c578d6ba1d69-kube-api-access-hmbgp\") pod \"payload-pre-processing-e2e-inf-b-f58171-57c45cc64c-2xsx5\" (UID: \"5ba7b96b-f2c0-4ddf-8117-c578d6ba1d69\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f58171-57c45cc64c-2xsx5"
Nov 03 19:56:00.572494 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:00.572474    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ba7b96b-f2c0-4ddf-8117-c578d6ba1d69-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f58171-57c45cc64c-2xsx5\" (UID: \"5ba7b96b-f2c0-4ddf-8117-c578d6ba1d69\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f58171-57c45cc64c-2xsx5"
Nov 03 19:56:00.579177 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:00.579151    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmbgp\" (UniqueName: \"kubernetes.io/projected/5ba7b96b-f2c0-4ddf-8117-c578d6ba1d69-kube-api-access-hmbgp\") pod \"payload-pre-processing-e2e-inf-b-f58171-57c45cc64c-2xsx5\" (UID: \"5ba7b96b-f2c0-4ddf-8117-c578d6ba1d69\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f58171-57c45cc64c-2xsx5"
Nov 03 19:56:00.773693 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:00.768950    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f58171-57c45cc64c-2xsx5"
Nov 03 19:56:01.322732 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:56:01.321730    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ba7b96b_f2c0_4ddf_8117_c578d6ba1d69.slice/crio-9b51fdd8c07e519d6918069a9c7468e527d4867b6b9bb58107f6637eb0a2c7c2 WatchSource:0}: Error finding container 9b51fdd8c07e519d6918069a9c7468e527d4867b6b9bb58107f6637eb0a2c7c2: Status 404 returned error can't find the container with id 9b51fdd8c07e519d6918069a9c7468e527d4867b6b9bb58107f6637eb0a2c7c2
Nov 03 19:56:01.323159 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:01.322892    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f58171-57c45cc64c-2xsx5"]
Nov 03 19:56:01.448174 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:01.448113    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f58171-57c45cc64c-2xsx5" event={"ID":"5ba7b96b-f2c0-4ddf-8117-c578d6ba1d69","Type":"ContainerStarted","Data":"a00db52098aab2cdf265bdb589432764d81f2d05a3f2cd437ec73b9964f3cf6d"}
Nov 03 19:56:01.448355 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:01.448196    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f58171-57c45cc64c-2xsx5" event={"ID":"5ba7b96b-f2c0-4ddf-8117-c578d6ba1d69","Type":"ContainerStarted","Data":"9b51fdd8c07e519d6918069a9c7468e527d4867b6b9bb58107f6637eb0a2c7c2"}
Nov 03 19:56:01.448355 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:01.448217    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f58171-57c45cc64c-2xsx5"
Nov 03 19:56:01.468761 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:01.468637    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f58171-57c45cc64c-2xsx5" podStartSLOduration=2.468617996 podStartE2EDuration="2.468617996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:55:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:01.461486671 +0000 UTC m=+1474.705452137" watchObservedRunningTime="2026-11-03 19:56:01.468617996 +0000 UTC m=+1474.712583461"
Nov 03 19:56:02.924300 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:56:02.924259    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 19:56:03.824840 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:03.824807    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-3cee6a-799d7c674c-fj9vk"
Nov 03 19:56:03.825021 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:03.824885    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3cee6a-544fd5876c-56hwf"
Nov 03 19:56:03.906555 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:03.906526    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-79c47f-5dfb89f897-kwzwd"
Nov 03 19:56:11.022283 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:11.022201    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1754953c-gkm4f"]
Nov 03 19:56:11.027633 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:11.027605    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1754953c-gkm4f"]
Nov 03 19:56:12.066223 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:12.066184    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0c8eec-fc8bcf68c-bnkz8"]
Nov 03 19:56:12.067035 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:12.066999    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c8eec-fc8bcf68c-bnkz8" podUID="1f4ba087-9ef9-45b1-adb0-337db91e1fed" containerName="maas-api" containerID="cri-o://8e5a9850d278e261ed914d643928f5e1d7dac0b1a0d4eab0933142bb0d4e9b59" gracePeriod=30
Nov 03 19:56:12.656691 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:12.654158    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f58171-57c45cc64c-2xsx5"
Nov 03 19:56:12.704716 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:12.704603    4921 generic.go:356] "Generic (PLEG): container finished" podID="1f4ba087-9ef9-45b1-adb0-337db91e1fed" containerID="8e5a9850d278e261ed914d643928f5e1d7dac0b1a0d4eab0933142bb0d4e9b59" exitCode=0
Nov 03 19:56:12.704926 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:12.704802    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c8eec-fc8bcf68c-bnkz8" event={"ID":"1f4ba087-9ef9-45b1-adb0-337db91e1fed","Type":"ContainerDied","Data":"8e5a9850d278e261ed914d643928f5e1d7dac0b1a0d4eab0933142bb0d4e9b59"}
Nov 03 19:56:12.795849 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:12.795824    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c8eec-fc8bcf68c-bnkz8"
Nov 03 19:56:12.892367 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:12.892334    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c1f13c8-7950-4374-9e7a-48ed6007dc77" path="/var/lib/kubelet/pods/6c1f13c8-7950-4374-9e7a-48ed6007dc77/volumes"
Nov 03 19:56:12.929750 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:12.923789    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f4ba087-9ef9-45b1-adb0-337db91e1fed-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f4ba087-9ef9-45b1-adb0-337db91e1fed-metrics-certs\") pod \"1f4ba087-9ef9-45b1-adb0-337db91e1fed\" (UID: \"1f4ba087-9ef9-45b1-adb0-337db91e1fed\") "
Nov 03 19:56:12.929750 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:12.923920    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f4ba087-9ef9-45b1-adb0-337db91e1fed-kube-api-access-gwsgb\" (UniqueName: \"kubernetes.io/projected/1f4ba087-9ef9-45b1-adb0-337db91e1fed-kube-api-access-gwsgb\") pod \"1f4ba087-9ef9-45b1-adb0-337db91e1fed\" (UID: \"1f4ba087-9ef9-45b1-adb0-337db91e1fed\") "
Nov 03 19:56:12.929750 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:12.923952    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f4ba087-9ef9-45b1-adb0-337db91e1fed-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f4ba087-9ef9-45b1-adb0-337db91e1fed-maas-api-tls\") pod \"1f4ba087-9ef9-45b1-adb0-337db91e1fed\" (UID: \"1f4ba087-9ef9-45b1-adb0-337db91e1fed\") "
Nov 03 19:56:12.929750 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:12.924200    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f4ba087-9ef9-45b1-adb0-337db91e1fed-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f4ba087-9ef9-45b1-adb0-337db91e1fed-openshift-service-ca\") pod \"1f4ba087-9ef9-45b1-adb0-337db91e1fed\" (UID: \"1f4ba087-9ef9-45b1-adb0-337db91e1fed\") "
Nov 03 19:56:12.943034 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:12.942980    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f4ba087-9ef9-45b1-adb0-337db91e1fed-openshift-service-ca" pod "1f4ba087-9ef9-45b1-adb0-337db91e1fed" (UID: "1f4ba087-9ef9-45b1-adb0-337db91e1fed"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:56:12.983179 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:12.983128    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f4ba087-9ef9-45b1-adb0-337db91e1fed-metrics-certs" pod "1f4ba087-9ef9-45b1-adb0-337db91e1fed" (UID: "1f4ba087-9ef9-45b1-adb0-337db91e1fed"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:56:12.995700 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:12.985994    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f4ba087-9ef9-45b1-adb0-337db91e1fed-maas-api-tls" pod "1f4ba087-9ef9-45b1-adb0-337db91e1fed" (UID: "1f4ba087-9ef9-45b1-adb0-337db91e1fed"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:56:13.003294 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:13.003252    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f4ba087-9ef9-45b1-adb0-337db91e1fed-kube-api-access-gwsgb" pod "1f4ba087-9ef9-45b1-adb0-337db91e1fed" (UID: "1f4ba087-9ef9-45b1-adb0-337db91e1fed"). InnerVolumeSpecName "kube-api-access-gwsgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:56:13.037744 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:13.031265    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwsgb\" (UniqueName: \"kubernetes.io/projected/1f4ba087-9ef9-45b1-adb0-337db91e1fed-kube-api-access-gwsgb\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:56:13.038088 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:13.038046    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f4ba087-9ef9-45b1-adb0-337db91e1fed-maas-api-tls\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:56:13.038230 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:13.038094    4921 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f4ba087-9ef9-45b1-adb0-337db91e1fed-openshift-service-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:56:13.038230 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:13.038111    4921 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f4ba087-9ef9-45b1-adb0-337db91e1fed-metrics-certs\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:56:13.716396 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:13.716363    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c8eec-fc8bcf68c-bnkz8"
Nov 03 19:56:13.716865 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:13.716394    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c8eec-fc8bcf68c-bnkz8" event={"ID":"1f4ba087-9ef9-45b1-adb0-337db91e1fed","Type":"ContainerDied","Data":"f340f623857e5c9cf4b52f2969b48eac148fd89dab01539bd87bbbffd009243f"}
Nov 03 19:56:13.716865 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:13.716470    4921 scope.go:122] "RemoveContainer" containerID="8e5a9850d278e261ed914d643928f5e1d7dac0b1a0d4eab0933142bb0d4e9b59"
Nov 03 19:56:13.744854 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:13.744817    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0c8eec-fc8bcf68c-bnkz8"]
Nov 03 19:56:13.804692 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:13.802619    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0c8eec-fc8bcf68c-bnkz8"]
Nov 03 19:56:14.434852 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:14.434818    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3cee6a-5f99576765-9s4g2"]
Nov 03 19:56:14.435615 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:14.435600    4921 cpu_manager.go:414] "containerMap: removing container" podUID="1f4ba087-9ef9-45b1-adb0-337db91e1fed" containerName="maas-api"
Nov 03 19:56:14.435663 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:14.435617    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f4ba087-9ef9-45b1-adb0-337db91e1fed" containerName="maas-api"
Nov 03 19:56:14.435791 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:14.435781    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f4ba087-9ef9-45b1-adb0-337db91e1fed" containerName="maas-api"
Nov 03 19:56:14.439627 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:14.439609    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3cee6a-5f99576765-9s4g2"
Nov 03 19:56:14.443543 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:14.443522    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-3cee6a-serving-cert\""
Nov 03 19:56:14.449116 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:14.449090    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3cee6a-5f99576765-9s4g2"]
Nov 03 19:56:14.498225 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:14.498189    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a2ce54f-1cb3-47b0-aa6e-e8b699ea8241-metrics-certs\") pod \"maas-api-e2e-inf-a-3cee6a-5f99576765-9s4g2\" (UID: \"9a2ce54f-1cb3-47b0-aa6e-e8b699ea8241\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3cee6a-5f99576765-9s4g2"
Nov 03 19:56:14.498397 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:14.498348    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a2ce54f-1cb3-47b0-aa6e-e8b699ea8241-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3cee6a-5f99576765-9s4g2\" (UID: \"9a2ce54f-1cb3-47b0-aa6e-e8b699ea8241\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3cee6a-5f99576765-9s4g2"
Nov 03 19:56:14.498551 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:14.498525    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a2ce54f-1cb3-47b0-aa6e-e8b699ea8241-maas-api-tls\") pod \"maas-api-e2e-inf-a-3cee6a-5f99576765-9s4g2\" (UID: \"9a2ce54f-1cb3-47b0-aa6e-e8b699ea8241\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3cee6a-5f99576765-9s4g2"
Nov 03 19:56:14.498637 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:14.498623    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx72n\" (UniqueName: \"kubernetes.io/projected/9a2ce54f-1cb3-47b0-aa6e-e8b699ea8241-kube-api-access-bx72n\") pod \"maas-api-e2e-inf-a-3cee6a-5f99576765-9s4g2\" (UID: \"9a2ce54f-1cb3-47b0-aa6e-e8b699ea8241\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3cee6a-5f99576765-9s4g2"
Nov 03 19:56:14.600124 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:14.600085    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a2ce54f-1cb3-47b0-aa6e-e8b699ea8241-maas-api-tls\") pod \"maas-api-e2e-inf-a-3cee6a-5f99576765-9s4g2\" (UID: \"9a2ce54f-1cb3-47b0-aa6e-e8b699ea8241\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3cee6a-5f99576765-9s4g2"
Nov 03 19:56:14.600309 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:14.600215    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx72n\" (UniqueName: \"kubernetes.io/projected/9a2ce54f-1cb3-47b0-aa6e-e8b699ea8241-kube-api-access-bx72n\") pod \"maas-api-e2e-inf-a-3cee6a-5f99576765-9s4g2\" (UID: \"9a2ce54f-1cb3-47b0-aa6e-e8b699ea8241\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3cee6a-5f99576765-9s4g2"
Nov 03 19:56:14.600384 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:14.600306    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a2ce54f-1cb3-47b0-aa6e-e8b699ea8241-metrics-certs\") pod \"maas-api-e2e-inf-a-3cee6a-5f99576765-9s4g2\" (UID: \"9a2ce54f-1cb3-47b0-aa6e-e8b699ea8241\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3cee6a-5f99576765-9s4g2"
Nov 03 19:56:14.600485 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:14.600422    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a2ce54f-1cb3-47b0-aa6e-e8b699ea8241-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3cee6a-5f99576765-9s4g2\" (UID: \"9a2ce54f-1cb3-47b0-aa6e-e8b699ea8241\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3cee6a-5f99576765-9s4g2"
Nov 03 19:56:14.620166 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:14.601470    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a2ce54f-1cb3-47b0-aa6e-e8b699ea8241-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3cee6a-5f99576765-9s4g2\" (UID: \"9a2ce54f-1cb3-47b0-aa6e-e8b699ea8241\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3cee6a-5f99576765-9s4g2"
Nov 03 19:56:14.620166 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:56:14.609798    4921 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-3cee6a-serving-cert: secret "maas-api-e2e-inf-a-3cee6a-serving-cert" not found
Nov 03 19:56:14.620166 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:56:14.609908    4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a2ce54f-1cb3-47b0-aa6e-e8b699ea8241-maas-api-tls podName:9a2ce54f-1cb3-47b0-aa6e-e8b699ea8241 nodeName:}" failed. No retries permitted until 2026-11-03 19:56:15.109885898 +0000 UTC m=+1488.353851355 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9a2ce54f-1cb3-47b0-aa6e-e8b699ea8241-maas-api-tls") pod "maas-api-e2e-inf-a-3cee6a-5f99576765-9s4g2" (UID: "9a2ce54f-1cb3-47b0-aa6e-e8b699ea8241") : secret "maas-api-e2e-inf-a-3cee6a-serving-cert" not found
Nov 03 19:56:14.620166 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:14.615947    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a2ce54f-1cb3-47b0-aa6e-e8b699ea8241-metrics-certs\") pod \"maas-api-e2e-inf-a-3cee6a-5f99576765-9s4g2\" (UID: \"9a2ce54f-1cb3-47b0-aa6e-e8b699ea8241\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3cee6a-5f99576765-9s4g2"
Nov 03 19:56:14.640720 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:14.640629    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx72n\" (UniqueName: \"kubernetes.io/projected/9a2ce54f-1cb3-47b0-aa6e-e8b699ea8241-kube-api-access-bx72n\") pod \"maas-api-e2e-inf-a-3cee6a-5f99576765-9s4g2\" (UID: \"9a2ce54f-1cb3-47b0-aa6e-e8b699ea8241\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3cee6a-5f99576765-9s4g2"
Nov 03 19:56:14.893432 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:14.893394    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f4ba087-9ef9-45b1-adb0-337db91e1fed" path="/var/lib/kubelet/pods/1f4ba087-9ef9-45b1-adb0-337db91e1fed/volumes"
Nov 03 19:56:15.148170 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:15.148079    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a2ce54f-1cb3-47b0-aa6e-e8b699ea8241-maas-api-tls\") pod \"maas-api-e2e-inf-a-3cee6a-5f99576765-9s4g2\" (UID: \"9a2ce54f-1cb3-47b0-aa6e-e8b699ea8241\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3cee6a-5f99576765-9s4g2"
Nov 03 19:56:15.188143 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:15.188112    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a2ce54f-1cb3-47b0-aa6e-e8b699ea8241-maas-api-tls\") pod \"maas-api-e2e-inf-a-3cee6a-5f99576765-9s4g2\" (UID: \"9a2ce54f-1cb3-47b0-aa6e-e8b699ea8241\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3cee6a-5f99576765-9s4g2"
Nov 03 19:56:15.356398 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:15.356349    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3cee6a-5f99576765-9s4g2"
Nov 03 19:56:15.924751 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:56:15.924652    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a2ce54f_1cb3_47b0_aa6e_e8b699ea8241.slice/crio-dbb222211d8b38cd14a1fff35b9a25e31d49507e5915ee0e9d4cec99dab3f062 WatchSource:0}: Error finding container dbb222211d8b38cd14a1fff35b9a25e31d49507e5915ee0e9d4cec99dab3f062: Status 404 returned error can't find the container with id dbb222211d8b38cd14a1fff35b9a25e31d49507e5915ee0e9d4cec99dab3f062
Nov 03 19:56:15.934065 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:15.934041    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3cee6a-5f99576765-9s4g2"]
Nov 03 19:56:16.944820 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:16.944788    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3cee6a-5f99576765-9s4g2" event={"ID":"9a2ce54f-1cb3-47b0-aa6e-e8b699ea8241","Type":"ContainerStarted","Data":"b10ad26b42709406642991c89e224e16e03f2e8e7be48ee478ca64bd25ed2d21"}
Nov 03 19:56:16.945213 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:16.944844    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3cee6a-5f99576765-9s4g2" event={"ID":"9a2ce54f-1cb3-47b0-aa6e-e8b699ea8241","Type":"ContainerStarted","Data":"dbb222211d8b38cd14a1fff35b9a25e31d49507e5915ee0e9d4cec99dab3f062"}
Nov 03 19:56:16.945213 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:16.944868    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3cee6a-5f99576765-9s4g2"
Nov 03 19:56:16.964728 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:16.964683    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3cee6a-5f99576765-9s4g2" podStartSLOduration=2.564841695 podStartE2EDuration="2.964655228s" totalImagesPullingTime="399.813533ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:56:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:16.958787089 +0000 UTC m=+1490.202752566" watchObservedRunningTime="2026-11-03 19:56:16.964655228 +0000 UTC m=+1490.208620692"
Nov 03 19:56:22.969624 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:22.969590    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3cee6a-5f99576765-9s4g2"
Nov 03 19:56:27.667693 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:27.665857    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv" podUID="01c749b9-d188-4673-9a3d-99b9c4ae0793" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Nov 03 19:56:27.710124 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:27.697779    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk" podUID="2d0a934e-4816-44d2-9bb5-e8b4e95dd73c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:56:27.729806 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:27.729755    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6" podUID="1aaee61d-b036-4c2b-bee7-a692b8bed612" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:56:27.947554 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:27.945209    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7" podUID="d4de0090-d775-4bd7-9ee2-058b5d380638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:56:29.840945 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:29.840911    4921 scope.go:122] "RemoveContainer" containerID="62c14dd321b2c5ee0052904551391b13ea36b106e71ca6765f867cbc3ce3a5e8"
Nov 03 19:56:31.676871 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:31.676847    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gzf8k_6f215b42-0f09-46a0-bc98-392c0fa909c9/console-operator/2.log"
Nov 03 19:56:31.694480 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:31.694456    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gzf8k_6f215b42-0f09-46a0-bc98-392c0fa909c9/console-operator/2.log"
Nov 03 19:56:32.929529 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:56:32.929459    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 19:56:38.254139 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:38.254105    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-79c47f/test-model-79c47f-kserve-665c6d6575-b5svq"]
Nov 03 19:56:38.262159 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:38.262130    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-79c47f/test-model-79c47f-kserve-665c6d6575-b5svq"
Nov 03 19:56:38.288131 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:38.288072    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-79c47f\"/\"openshift-service-ca.crt\""
Nov 03 19:56:38.288875 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:38.288850    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-79c47f\"/\"default-dockercfg-msr42\""
Nov 03 19:56:38.289035 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:38.288865    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-79c47f\"/\"test-model-79c47f-kserve-self-signed-certs\""
Nov 03 19:56:38.289134 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:38.288978    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-79c47f\"/\"kube-root-ca.crt\""
Nov 03 19:56:38.293998 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:38.293970    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-79c47f/test-model-79c47f-kserve-665c6d6575-b5svq"]
Nov 03 19:56:38.347629 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:38.347592    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9be42b54-1aa8-4751-9671-b83f78cf6d3a-dshm\") pod \"test-model-79c47f-kserve-665c6d6575-b5svq\" (UID: \"9be42b54-1aa8-4751-9671-b83f78cf6d3a\") " pod="ai-tenant-e2e-inf-b-79c47f/test-model-79c47f-kserve-665c6d6575-b5svq"
Nov 03 19:56:38.347950 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:38.347919    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9be42b54-1aa8-4751-9671-b83f78cf6d3a-tmp-dir\") pod \"test-model-79c47f-kserve-665c6d6575-b5svq\" (UID: \"9be42b54-1aa8-4751-9671-b83f78cf6d3a\") " pod="ai-tenant-e2e-inf-b-79c47f/test-model-79c47f-kserve-665c6d6575-b5svq"
Nov 03 19:56:38.348298 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:38.348276    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9be42b54-1aa8-4751-9671-b83f78cf6d3a-home\") pod \"test-model-79c47f-kserve-665c6d6575-b5svq\" (UID: \"9be42b54-1aa8-4751-9671-b83f78cf6d3a\") " pod="ai-tenant-e2e-inf-b-79c47f/test-model-79c47f-kserve-665c6d6575-b5svq"
Nov 03 19:56:38.348380 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:38.348315    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9be42b54-1aa8-4751-9671-b83f78cf6d3a-tls-certs\") pod \"test-model-79c47f-kserve-665c6d6575-b5svq\" (UID: \"9be42b54-1aa8-4751-9671-b83f78cf6d3a\") " pod="ai-tenant-e2e-inf-b-79c47f/test-model-79c47f-kserve-665c6d6575-b5svq"
Nov 03 19:56:38.348438 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:38.348400    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pzzc\" (UniqueName: \"kubernetes.io/projected/9be42b54-1aa8-4751-9671-b83f78cf6d3a-kube-api-access-2pzzc\") pod \"test-model-79c47f-kserve-665c6d6575-b5svq\" (UID: \"9be42b54-1aa8-4751-9671-b83f78cf6d3a\") " pod="ai-tenant-e2e-inf-b-79c47f/test-model-79c47f-kserve-665c6d6575-b5svq"
Nov 03 19:56:38.348500 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:38.348441    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9be42b54-1aa8-4751-9671-b83f78cf6d3a-model-cache\") pod \"test-model-79c47f-kserve-665c6d6575-b5svq\" (UID: \"9be42b54-1aa8-4751-9671-b83f78cf6d3a\") " pod="ai-tenant-e2e-inf-b-79c47f/test-model-79c47f-kserve-665c6d6575-b5svq"
Nov 03 19:56:38.459071 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:38.456985    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9be42b54-1aa8-4751-9671-b83f78cf6d3a-home\") pod \"test-model-79c47f-kserve-665c6d6575-b5svq\" (UID: \"9be42b54-1aa8-4751-9671-b83f78cf6d3a\") " pod="ai-tenant-e2e-inf-b-79c47f/test-model-79c47f-kserve-665c6d6575-b5svq"
Nov 03 19:56:38.459071 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:38.457049    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9be42b54-1aa8-4751-9671-b83f78cf6d3a-tls-certs\") pod \"test-model-79c47f-kserve-665c6d6575-b5svq\" (UID: \"9be42b54-1aa8-4751-9671-b83f78cf6d3a\") " pod="ai-tenant-e2e-inf-b-79c47f/test-model-79c47f-kserve-665c6d6575-b5svq"
Nov 03 19:56:38.459071 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:38.457158    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pzzc\" (UniqueName: \"kubernetes.io/projected/9be42b54-1aa8-4751-9671-b83f78cf6d3a-kube-api-access-2pzzc\") pod \"test-model-79c47f-kserve-665c6d6575-b5svq\" (UID: \"9be42b54-1aa8-4751-9671-b83f78cf6d3a\") " pod="ai-tenant-e2e-inf-b-79c47f/test-model-79c47f-kserve-665c6d6575-b5svq"
Nov 03 19:56:38.459071 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:38.457193    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9be42b54-1aa8-4751-9671-b83f78cf6d3a-model-cache\") pod \"test-model-79c47f-kserve-665c6d6575-b5svq\" (UID: \"9be42b54-1aa8-4751-9671-b83f78cf6d3a\") " pod="ai-tenant-e2e-inf-b-79c47f/test-model-79c47f-kserve-665c6d6575-b5svq"
Nov 03 19:56:38.459071 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:38.457311    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9be42b54-1aa8-4751-9671-b83f78cf6d3a-dshm\") pod \"test-model-79c47f-kserve-665c6d6575-b5svq\" (UID: \"9be42b54-1aa8-4751-9671-b83f78cf6d3a\") " pod="ai-tenant-e2e-inf-b-79c47f/test-model-79c47f-kserve-665c6d6575-b5svq"
Nov 03 19:56:38.459071 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:38.457416    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9be42b54-1aa8-4751-9671-b83f78cf6d3a-tmp-dir\") pod \"test-model-79c47f-kserve-665c6d6575-b5svq\" (UID: \"9be42b54-1aa8-4751-9671-b83f78cf6d3a\") " pod="ai-tenant-e2e-inf-b-79c47f/test-model-79c47f-kserve-665c6d6575-b5svq"
Nov 03 19:56:38.459071 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:38.458102    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9be42b54-1aa8-4751-9671-b83f78cf6d3a-tmp-dir\") pod \"test-model-79c47f-kserve-665c6d6575-b5svq\" (UID: \"9be42b54-1aa8-4751-9671-b83f78cf6d3a\") " pod="ai-tenant-e2e-inf-b-79c47f/test-model-79c47f-kserve-665c6d6575-b5svq"
Nov 03 19:56:38.459071 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:38.458362    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9be42b54-1aa8-4751-9671-b83f78cf6d3a-home\") pod \"test-model-79c47f-kserve-665c6d6575-b5svq\" (UID: \"9be42b54-1aa8-4751-9671-b83f78cf6d3a\") " pod="ai-tenant-e2e-inf-b-79c47f/test-model-79c47f-kserve-665c6d6575-b5svq"
Nov 03 19:56:38.459071 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:38.459073    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9be42b54-1aa8-4751-9671-b83f78cf6d3a-model-cache\") pod \"test-model-79c47f-kserve-665c6d6575-b5svq\" (UID: \"9be42b54-1aa8-4751-9671-b83f78cf6d3a\") " pod="ai-tenant-e2e-inf-b-79c47f/test-model-79c47f-kserve-665c6d6575-b5svq"
Nov 03 19:56:38.482447 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:38.482412    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pzzc\" (UniqueName: \"kubernetes.io/projected/9be42b54-1aa8-4751-9671-b83f78cf6d3a-kube-api-access-2pzzc\") pod \"test-model-79c47f-kserve-665c6d6575-b5svq\" (UID: \"9be42b54-1aa8-4751-9671-b83f78cf6d3a\") " pod="ai-tenant-e2e-inf-b-79c47f/test-model-79c47f-kserve-665c6d6575-b5svq"
Nov 03 19:56:38.488363 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:38.485445    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9be42b54-1aa8-4751-9671-b83f78cf6d3a-tls-certs\") pod \"test-model-79c47f-kserve-665c6d6575-b5svq\" (UID: \"9be42b54-1aa8-4751-9671-b83f78cf6d3a\") " pod="ai-tenant-e2e-inf-b-79c47f/test-model-79c47f-kserve-665c6d6575-b5svq"
Nov 03 19:56:38.488887 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:38.488831    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9be42b54-1aa8-4751-9671-b83f78cf6d3a-dshm\") pod \"test-model-79c47f-kserve-665c6d6575-b5svq\" (UID: \"9be42b54-1aa8-4751-9671-b83f78cf6d3a\") " pod="ai-tenant-e2e-inf-b-79c47f/test-model-79c47f-kserve-665c6d6575-b5svq"
Nov 03 19:56:38.612350 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:38.612264    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-79c47f/test-model-79c47f-kserve-665c6d6575-b5svq"
Nov 03 19:56:39.210907 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:39.210871    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-79c47f/test-model-79c47f-kserve-665c6d6575-b5svq"]
Nov 03 19:56:39.211660 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:56:39.211628    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9be42b54_1aa8_4751_9671_b83f78cf6d3a.slice/crio-d7612d822d411a06c07c879006015348a1c1f3544859657f817f2bd7f44947a7 WatchSource:0}: Error finding container d7612d822d411a06c07c879006015348a1c1f3544859657f817f2bd7f44947a7: Status 404 returned error can't find the container with id d7612d822d411a06c07c879006015348a1c1f3544859657f817f2bd7f44947a7
Nov 03 19:56:39.755812 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:39.755657    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-79c47f/test-model-79c47f-kserve-665c6d6575-b5svq" event={"ID":"9be42b54-1aa8-4751-9671-b83f78cf6d3a","Type":"ContainerStarted","Data":"ecb553a8fa1b1601d2028e72f2b6dfc13211238b4f6e4b5c34d01b955231609e"}
Nov 03 19:56:39.755812 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:39.755779    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-79c47f/test-model-79c47f-kserve-665c6d6575-b5svq" event={"ID":"9be42b54-1aa8-4751-9671-b83f78cf6d3a","Type":"ContainerStarted","Data":"d7612d822d411a06c07c879006015348a1c1f3544859657f817f2bd7f44947a7"}
Nov 03 19:56:39.774740 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:39.774627    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-79c47f/test-model-79c47f-kserve-665c6d6575-b5svq" podStartSLOduration=1.774613081 podStartE2EDuration="1.774613081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:56:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:39.770553181 +0000 UTC m=+1513.014518646" watchObservedRunningTime="2026-11-03 19:56:39.774613081 +0000 UTC m=+1513.018578607"
Nov 03 19:56:40.623038 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:40.623006    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-79c47f/test-model-79c47f-kserve-665c6d6575-b5svq"
Nov 03 19:56:43.022742 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:43.022702    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9850de4-89j2h"]
Nov 03 19:56:43.028220 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:43.028182    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9850de4-89j2h"]
Nov 03 19:56:44.969580 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:44.969538    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da6eb7e1-0dbb-41cb-8c32-d438b2f05ee7" path="/var/lib/kubelet/pods/da6eb7e1-0dbb-41cb-8c32-d438b2f05ee7/volumes"
Nov 03 19:56:48.618204 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:48.618160    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-79c47f/test-model-79c47f-kserve-665c6d6575-b5svq"
Nov 03 19:56:48.684006 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:48.683971    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-79c47f/test-model-79c47f-kserve-665c6d6575-b5svq"
Nov 03 19:56:48.963879 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:48.963798    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-79c47f/test-model-79c47f-kserve-665c6d6575-b5svq"
Nov 03 19:56:54.974127 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:54.974092    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-11c1a8/test-model-11c1a8-kserve-6c47c9d749-pvxrt"]
Nov 03 19:56:54.992408 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:54.992376    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-11c1a8/test-model-11c1a8-kserve-6c47c9d749-pvxrt"]
Nov 03 19:56:54.992755 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:54.992731    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-11c1a8/test-model-11c1a8-kserve-6c47c9d749-pvxrt"
Nov 03 19:56:54.997542 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:54.997517    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-11c1a8\"/\"test-model-11c1a8-kserve-self-signed-certs\""
Nov 03 19:56:54.997918 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:54.997884    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-11c1a8\"/\"default-dockercfg-ghq45\""
Nov 03 19:56:54.998396 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:54.998372    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-11c1a8\"/\"openshift-service-ca.crt\""
Nov 03 19:56:55.003401 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:55.003376    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-11c1a8\"/\"kube-root-ca.crt\""
Nov 03 19:56:55.094117 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:55.094079    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f1dd983-8a22-4bfb-8d24-37b1c67c8070-model-cache\") pod \"test-model-11c1a8-kserve-6c47c9d749-pvxrt\" (UID: \"8f1dd983-8a22-4bfb-8d24-37b1c67c8070\") " pod="ai-tenant-e2e-inf-a-11c1a8/test-model-11c1a8-kserve-6c47c9d749-pvxrt"
Nov 03 19:56:55.094117 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:55.094126    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lfpb\" (UniqueName: \"kubernetes.io/projected/8f1dd983-8a22-4bfb-8d24-37b1c67c8070-kube-api-access-7lfpb\") pod \"test-model-11c1a8-kserve-6c47c9d749-pvxrt\" (UID: \"8f1dd983-8a22-4bfb-8d24-37b1c67c8070\") " pod="ai-tenant-e2e-inf-a-11c1a8/test-model-11c1a8-kserve-6c47c9d749-pvxrt"
Nov 03 19:56:55.094345 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:55.094280    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f1dd983-8a22-4bfb-8d24-37b1c67c8070-tmp-dir\") pod \"test-model-11c1a8-kserve-6c47c9d749-pvxrt\" (UID: \"8f1dd983-8a22-4bfb-8d24-37b1c67c8070\") " pod="ai-tenant-e2e-inf-a-11c1a8/test-model-11c1a8-kserve-6c47c9d749-pvxrt"
Nov 03 19:56:55.094512 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:55.094482    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f1dd983-8a22-4bfb-8d24-37b1c67c8070-home\") pod \"test-model-11c1a8-kserve-6c47c9d749-pvxrt\" (UID: \"8f1dd983-8a22-4bfb-8d24-37b1c67c8070\") " pod="ai-tenant-e2e-inf-a-11c1a8/test-model-11c1a8-kserve-6c47c9d749-pvxrt"
Nov 03 19:56:55.094549 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:55.094528    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f1dd983-8a22-4bfb-8d24-37b1c67c8070-tls-certs\") pod \"test-model-11c1a8-kserve-6c47c9d749-pvxrt\" (UID: \"8f1dd983-8a22-4bfb-8d24-37b1c67c8070\") " pod="ai-tenant-e2e-inf-a-11c1a8/test-model-11c1a8-kserve-6c47c9d749-pvxrt"
Nov 03 19:56:55.094704 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:55.094661    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f1dd983-8a22-4bfb-8d24-37b1c67c8070-dshm\") pod \"test-model-11c1a8-kserve-6c47c9d749-pvxrt\" (UID: \"8f1dd983-8a22-4bfb-8d24-37b1c67c8070\") " pod="ai-tenant-e2e-inf-a-11c1a8/test-model-11c1a8-kserve-6c47c9d749-pvxrt"
Nov 03 19:56:55.196969 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:55.196924    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f1dd983-8a22-4bfb-8d24-37b1c67c8070-model-cache\") pod \"test-model-11c1a8-kserve-6c47c9d749-pvxrt\" (UID: \"8f1dd983-8a22-4bfb-8d24-37b1c67c8070\") " pod="ai-tenant-e2e-inf-a-11c1a8/test-model-11c1a8-kserve-6c47c9d749-pvxrt"
Nov 03 19:56:55.197248 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:55.197212    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lfpb\" (UniqueName: \"kubernetes.io/projected/8f1dd983-8a22-4bfb-8d24-37b1c67c8070-kube-api-access-7lfpb\") pod \"test-model-11c1a8-kserve-6c47c9d749-pvxrt\" (UID: \"8f1dd983-8a22-4bfb-8d24-37b1c67c8070\") " pod="ai-tenant-e2e-inf-a-11c1a8/test-model-11c1a8-kserve-6c47c9d749-pvxrt"
Nov 03 19:56:55.197500 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:55.197477    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f1dd983-8a22-4bfb-8d24-37b1c67c8070-tmp-dir\") pod \"test-model-11c1a8-kserve-6c47c9d749-pvxrt\" (UID: \"8f1dd983-8a22-4bfb-8d24-37b1c67c8070\") " pod="ai-tenant-e2e-inf-a-11c1a8/test-model-11c1a8-kserve-6c47c9d749-pvxrt"
Nov 03 19:56:55.197785 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:55.197765    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f1dd983-8a22-4bfb-8d24-37b1c67c8070-home\") pod \"test-model-11c1a8-kserve-6c47c9d749-pvxrt\" (UID: \"8f1dd983-8a22-4bfb-8d24-37b1c67c8070\") " pod="ai-tenant-e2e-inf-a-11c1a8/test-model-11c1a8-kserve-6c47c9d749-pvxrt"
Nov 03 19:56:55.197923 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:55.197909    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f1dd983-8a22-4bfb-8d24-37b1c67c8070-tls-certs\") pod \"test-model-11c1a8-kserve-6c47c9d749-pvxrt\" (UID: \"8f1dd983-8a22-4bfb-8d24-37b1c67c8070\") " pod="ai-tenant-e2e-inf-a-11c1a8/test-model-11c1a8-kserve-6c47c9d749-pvxrt"
Nov 03 19:56:55.198095 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:55.198080    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f1dd983-8a22-4bfb-8d24-37b1c67c8070-dshm\") pod \"test-model-11c1a8-kserve-6c47c9d749-pvxrt\" (UID: \"8f1dd983-8a22-4bfb-8d24-37b1c67c8070\") " pod="ai-tenant-e2e-inf-a-11c1a8/test-model-11c1a8-kserve-6c47c9d749-pvxrt"
Nov 03 19:56:55.215692 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:55.198138    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f1dd983-8a22-4bfb-8d24-37b1c67c8070-model-cache\") pod \"test-model-11c1a8-kserve-6c47c9d749-pvxrt\" (UID: \"8f1dd983-8a22-4bfb-8d24-37b1c67c8070\") " pod="ai-tenant-e2e-inf-a-11c1a8/test-model-11c1a8-kserve-6c47c9d749-pvxrt"
Nov 03 19:56:55.215927 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:55.198365    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f1dd983-8a22-4bfb-8d24-37b1c67c8070-tmp-dir\") pod \"test-model-11c1a8-kserve-6c47c9d749-pvxrt\" (UID: \"8f1dd983-8a22-4bfb-8d24-37b1c67c8070\") " pod="ai-tenant-e2e-inf-a-11c1a8/test-model-11c1a8-kserve-6c47c9d749-pvxrt"
Nov 03 19:56:55.216035 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:55.199320    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f1dd983-8a22-4bfb-8d24-37b1c67c8070-home\") pod \"test-model-11c1a8-kserve-6c47c9d749-pvxrt\" (UID: \"8f1dd983-8a22-4bfb-8d24-37b1c67c8070\") " pod="ai-tenant-e2e-inf-a-11c1a8/test-model-11c1a8-kserve-6c47c9d749-pvxrt"
Nov 03 19:56:55.216131 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:55.215155    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f1dd983-8a22-4bfb-8d24-37b1c67c8070-dshm\") pod \"test-model-11c1a8-kserve-6c47c9d749-pvxrt\" (UID: \"8f1dd983-8a22-4bfb-8d24-37b1c67c8070\") " pod="ai-tenant-e2e-inf-a-11c1a8/test-model-11c1a8-kserve-6c47c9d749-pvxrt"
Nov 03 19:56:55.221364 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:55.221334    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lfpb\" (UniqueName: \"kubernetes.io/projected/8f1dd983-8a22-4bfb-8d24-37b1c67c8070-kube-api-access-7lfpb\") pod \"test-model-11c1a8-kserve-6c47c9d749-pvxrt\" (UID: \"8f1dd983-8a22-4bfb-8d24-37b1c67c8070\") " pod="ai-tenant-e2e-inf-a-11c1a8/test-model-11c1a8-kserve-6c47c9d749-pvxrt"
Nov 03 19:56:55.231748 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:55.228868    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f1dd983-8a22-4bfb-8d24-37b1c67c8070-tls-certs\") pod \"test-model-11c1a8-kserve-6c47c9d749-pvxrt\" (UID: \"8f1dd983-8a22-4bfb-8d24-37b1c67c8070\") " pod="ai-tenant-e2e-inf-a-11c1a8/test-model-11c1a8-kserve-6c47c9d749-pvxrt"
Nov 03 19:56:55.312726 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:55.312649    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-11c1a8/test-model-11c1a8-kserve-6c47c9d749-pvxrt"
Nov 03 19:56:55.734610 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:56:55.734581    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8f1dd983_8a22_4bfb_8d24_37b1c67c8070.slice/crio-00994291297dcef58d86c0d7521ea5685cc5d4ee8775d9ca8ff2f709ae07475b WatchSource:0}: Error finding container 00994291297dcef58d86c0d7521ea5685cc5d4ee8775d9ca8ff2f709ae07475b: Status 404 returned error can't find the container with id 00994291297dcef58d86c0d7521ea5685cc5d4ee8775d9ca8ff2f709ae07475b
Nov 03 19:56:55.734961 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:55.734937    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-11c1a8/test-model-11c1a8-kserve-6c47c9d749-pvxrt"]
Nov 03 19:56:56.057265 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:56.057172    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-11c1a8/test-model-11c1a8-kserve-6c47c9d749-pvxrt" event={"ID":"8f1dd983-8a22-4bfb-8d24-37b1c67c8070","Type":"ContainerStarted","Data":"93b985490eaec58f8e32e127ffe404fdd832a68bd3cbdd4d9b2a8b5fc891f713"}
Nov 03 19:56:56.057835 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:56.057803    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-11c1a8/test-model-11c1a8-kserve-6c47c9d749-pvxrt" event={"ID":"8f1dd983-8a22-4bfb-8d24-37b1c67c8070","Type":"ContainerStarted","Data":"00994291297dcef58d86c0d7521ea5685cc5d4ee8775d9ca8ff2f709ae07475b"}
Nov 03 19:56:56.075907 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:56.075854    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-11c1a8/test-model-11c1a8-kserve-6c47c9d749-pvxrt" podStartSLOduration=2.075840045 podStartE2EDuration="2.075840045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:56:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:56.071896464 +0000 UTC m=+1529.315861930" watchObservedRunningTime="2026-11-03 19:56:56.075840045 +0000 UTC m=+1529.319805510"
Nov 03 19:56:56.318742 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:56:56.318570    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-11c1a8/test-model-11c1a8-kserve-6c47c9d749-pvxrt"
Nov 03 19:57:02.970531 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:57:02.969904    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 19:57:05.314260 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:05.314222    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-11c1a8/test-model-11c1a8-kserve-6c47c9d749-pvxrt"
Nov 03 19:57:05.328659 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:05.328629    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-11c1a8/test-model-11c1a8-kserve-6c47c9d749-pvxrt"
Nov 03 19:57:06.172751 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:06.172724    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-11c1a8/test-model-11c1a8-kserve-6c47c9d749-pvxrt"
Nov 03 19:57:07.195962 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:07.195927    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-f58171/test-model-f58171-kserve-5c596fcddc-vgqtt"]
Nov 03 19:57:07.215422 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:07.215392    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f58171/test-model-f58171-kserve-5c596fcddc-vgqtt"
Nov 03 19:57:07.222628 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:07.222591    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-f58171/test-model-f58171-kserve-5c596fcddc-vgqtt"]
Nov 03 19:57:07.223224 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:07.223193    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-f58171\"/\"test-model-f58171-kserve-self-signed-certs\""
Nov 03 19:57:07.223615 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:07.223574    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-f58171\"/\"kube-root-ca.crt\""
Nov 03 19:57:07.223911 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:07.223892    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-f58171\"/\"openshift-service-ca.crt\""
Nov 03 19:57:07.224001 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:07.223911    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-f58171\"/\"default-dockercfg-l8l6d\""
Nov 03 19:57:07.317206 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:07.317166    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e839d396-3136-41e8-b900-1194d9067c88-tmp-dir\") pod \"test-model-f58171-kserve-5c596fcddc-vgqtt\" (UID: \"e839d396-3136-41e8-b900-1194d9067c88\") " pod="ai-tenant-e2e-inf-b-f58171/test-model-f58171-kserve-5c596fcddc-vgqtt"
Nov 03 19:57:07.317395 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:07.317320    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e839d396-3136-41e8-b900-1194d9067c88-dshm\") pod \"test-model-f58171-kserve-5c596fcddc-vgqtt\" (UID: \"e839d396-3136-41e8-b900-1194d9067c88\") " pod="ai-tenant-e2e-inf-b-f58171/test-model-f58171-kserve-5c596fcddc-vgqtt"
Nov 03 19:57:07.317514 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:07.317493    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e839d396-3136-41e8-b900-1194d9067c88-tls-certs\") pod \"test-model-f58171-kserve-5c596fcddc-vgqtt\" (UID: \"e839d396-3136-41e8-b900-1194d9067c88\") " pod="ai-tenant-e2e-inf-b-f58171/test-model-f58171-kserve-5c596fcddc-vgqtt"
Nov 03 19:57:07.317607 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:07.317588    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lct7x\" (UniqueName: \"kubernetes.io/projected/e839d396-3136-41e8-b900-1194d9067c88-kube-api-access-lct7x\") pod \"test-model-f58171-kserve-5c596fcddc-vgqtt\" (UID: \"e839d396-3136-41e8-b900-1194d9067c88\") " pod="ai-tenant-e2e-inf-b-f58171/test-model-f58171-kserve-5c596fcddc-vgqtt"
Nov 03 19:57:07.317747 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:07.317729    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e839d396-3136-41e8-b900-1194d9067c88-model-cache\") pod \"test-model-f58171-kserve-5c596fcddc-vgqtt\" (UID: \"e839d396-3136-41e8-b900-1194d9067c88\") " pod="ai-tenant-e2e-inf-b-f58171/test-model-f58171-kserve-5c596fcddc-vgqtt"
Nov 03 19:57:07.317826 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:07.317808    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e839d396-3136-41e8-b900-1194d9067c88-home\") pod \"test-model-f58171-kserve-5c596fcddc-vgqtt\" (UID: \"e839d396-3136-41e8-b900-1194d9067c88\") " pod="ai-tenant-e2e-inf-b-f58171/test-model-f58171-kserve-5c596fcddc-vgqtt"
Nov 03 19:57:07.419514 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:07.419475    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e839d396-3136-41e8-b900-1194d9067c88-home\") pod \"test-model-f58171-kserve-5c596fcddc-vgqtt\" (UID: \"e839d396-3136-41e8-b900-1194d9067c88\") " pod="ai-tenant-e2e-inf-b-f58171/test-model-f58171-kserve-5c596fcddc-vgqtt"
Nov 03 19:57:07.419744 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:07.419725    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e839d396-3136-41e8-b900-1194d9067c88-tmp-dir\") pod \"test-model-f58171-kserve-5c596fcddc-vgqtt\" (UID: \"e839d396-3136-41e8-b900-1194d9067c88\") " pod="ai-tenant-e2e-inf-b-f58171/test-model-f58171-kserve-5c596fcddc-vgqtt"
Nov 03 19:57:07.419809 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:07.419771    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e839d396-3136-41e8-b900-1194d9067c88-dshm\") pod \"test-model-f58171-kserve-5c596fcddc-vgqtt\" (UID: \"e839d396-3136-41e8-b900-1194d9067c88\") " pod="ai-tenant-e2e-inf-b-f58171/test-model-f58171-kserve-5c596fcddc-vgqtt"
Nov 03 19:57:07.419867 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:07.419850    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e839d396-3136-41e8-b900-1194d9067c88-tls-certs\") pod \"test-model-f58171-kserve-5c596fcddc-vgqtt\" (UID: \"e839d396-3136-41e8-b900-1194d9067c88\") " pod="ai-tenant-e2e-inf-b-f58171/test-model-f58171-kserve-5c596fcddc-vgqtt"
Nov 03 19:57:07.419922 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:07.419884    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e839d396-3136-41e8-b900-1194d9067c88-home\") pod \"test-model-f58171-kserve-5c596fcddc-vgqtt\" (UID: \"e839d396-3136-41e8-b900-1194d9067c88\") " pod="ai-tenant-e2e-inf-b-f58171/test-model-f58171-kserve-5c596fcddc-vgqtt"
Nov 03 19:57:07.419984 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:07.419947    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lct7x\" (UniqueName: \"kubernetes.io/projected/e839d396-3136-41e8-b900-1194d9067c88-kube-api-access-lct7x\") pod \"test-model-f58171-kserve-5c596fcddc-vgqtt\" (UID: \"e839d396-3136-41e8-b900-1194d9067c88\") " pod="ai-tenant-e2e-inf-b-f58171/test-model-f58171-kserve-5c596fcddc-vgqtt"
Nov 03 19:57:07.420084 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:07.420067    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e839d396-3136-41e8-b900-1194d9067c88-model-cache\") pod \"test-model-f58171-kserve-5c596fcddc-vgqtt\" (UID: \"e839d396-3136-41e8-b900-1194d9067c88\") " pod="ai-tenant-e2e-inf-b-f58171/test-model-f58171-kserve-5c596fcddc-vgqtt"
Nov 03 19:57:07.420157 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:07.420120    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e839d396-3136-41e8-b900-1194d9067c88-tmp-dir\") pod \"test-model-f58171-kserve-5c596fcddc-vgqtt\" (UID: \"e839d396-3136-41e8-b900-1194d9067c88\") " pod="ai-tenant-e2e-inf-b-f58171/test-model-f58171-kserve-5c596fcddc-vgqtt"
Nov 03 19:57:07.420423 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:07.420403    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e839d396-3136-41e8-b900-1194d9067c88-model-cache\") pod \"test-model-f58171-kserve-5c596fcddc-vgqtt\" (UID: \"e839d396-3136-41e8-b900-1194d9067c88\") " pod="ai-tenant-e2e-inf-b-f58171/test-model-f58171-kserve-5c596fcddc-vgqtt"
Nov 03 19:57:07.422211 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:07.422180    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e839d396-3136-41e8-b900-1194d9067c88-dshm\") pod \"test-model-f58171-kserve-5c596fcddc-vgqtt\" (UID: \"e839d396-3136-41e8-b900-1194d9067c88\") " pod="ai-tenant-e2e-inf-b-f58171/test-model-f58171-kserve-5c596fcddc-vgqtt"
Nov 03 19:57:07.422643 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:07.422619    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e839d396-3136-41e8-b900-1194d9067c88-tls-certs\") pod \"test-model-f58171-kserve-5c596fcddc-vgqtt\" (UID: \"e839d396-3136-41e8-b900-1194d9067c88\") " pod="ai-tenant-e2e-inf-b-f58171/test-model-f58171-kserve-5c596fcddc-vgqtt"
Nov 03 19:57:07.427461 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:07.427413    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lct7x\" (UniqueName: \"kubernetes.io/projected/e839d396-3136-41e8-b900-1194d9067c88-kube-api-access-lct7x\") pod \"test-model-f58171-kserve-5c596fcddc-vgqtt\" (UID: \"e839d396-3136-41e8-b900-1194d9067c88\") " pod="ai-tenant-e2e-inf-b-f58171/test-model-f58171-kserve-5c596fcddc-vgqtt"
Nov 03 19:57:07.552620 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:07.552341    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f58171/test-model-f58171-kserve-5c596fcddc-vgqtt"
Nov 03 19:57:08.054450 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:57:08.054411    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode839d396_3136_41e8_b900_1194d9067c88.slice/crio-09dafb51f19ed55324915456a0e0499bdb2d4b78db8aa7e186009ad798711d61 WatchSource:0}: Error finding container 09dafb51f19ed55324915456a0e0499bdb2d4b78db8aa7e186009ad798711d61: Status 404 returned error can't find the container with id 09dafb51f19ed55324915456a0e0499bdb2d4b78db8aa7e186009ad798711d61
Nov 03 19:57:08.055071 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:08.055042    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-f58171/test-model-f58171-kserve-5c596fcddc-vgqtt"]
Nov 03 19:57:08.187014 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:08.186976    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f58171/test-model-f58171-kserve-5c596fcddc-vgqtt" event={"ID":"e839d396-3136-41e8-b900-1194d9067c88","Type":"ContainerStarted","Data":"626bf069fede1710a5d55ceaf3ef690c8fcf23559dbbc99bdebf661bc85dea19"}
Nov 03 19:57:08.187216 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:08.187026    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f58171/test-model-f58171-kserve-5c596fcddc-vgqtt" event={"ID":"e839d396-3136-41e8-b900-1194d9067c88","Type":"ContainerStarted","Data":"09dafb51f19ed55324915456a0e0499bdb2d4b78db8aa7e186009ad798711d61"}
Nov 03 19:57:08.209887 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:08.209825    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-f58171/test-model-f58171-kserve-5c596fcddc-vgqtt" podStartSLOduration=1.209806941 podStartE2EDuration="1.209806941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:57:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:57:08.204843637 +0000 UTC m=+1541.448809108" watchObservedRunningTime="2026-11-03 19:57:08.209806941 +0000 UTC m=+1541.453772410"
Nov 03 19:57:08.554794 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:08.554756    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-f58171/test-model-f58171-kserve-5c596fcddc-vgqtt"
Nov 03 19:57:12.489593 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:12.489554    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0d0987-6f7b54c8cb-8s9ds"]
Nov 03 19:57:12.490143 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:12.489838    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0d0987-6f7b54c8cb-8s9ds" podUID="4fcdf540-fd42-4f5e-bb3a-422d726c3281" containerName="maas-api" containerID="cri-o://5d6520597ecef5f2e98b68cb493d5cf092a2e9bdfa1da40bc65a3840582732dc" gracePeriod=30
Nov 03 19:57:13.237902 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:13.237873    4921 generic.go:356] "Generic (PLEG): container finished" podID="4fcdf540-fd42-4f5e-bb3a-422d726c3281" containerID="5d6520597ecef5f2e98b68cb493d5cf092a2e9bdfa1da40bc65a3840582732dc" exitCode=0
Nov 03 19:57:13.238048 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:13.237950    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0d0987-6f7b54c8cb-8s9ds" event={"ID":"4fcdf540-fd42-4f5e-bb3a-422d726c3281","Type":"ContainerDied","Data":"5d6520597ecef5f2e98b68cb493d5cf092a2e9bdfa1da40bc65a3840582732dc"}
Nov 03 19:57:13.238048 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:13.238013    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0d0987-6f7b54c8cb-8s9ds" event={"ID":"4fcdf540-fd42-4f5e-bb3a-422d726c3281","Type":"ContainerDied","Data":"471b9222e2be05c6afce2e16d3597579c2ccff649546d50184313eb560c7f88c"}
Nov 03 19:57:13.238048 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:13.238031    4921 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="471b9222e2be05c6afce2e16d3597579c2ccff649546d50184313eb560c7f88c"
Nov 03 19:57:13.242382 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:13.242362    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0d0987-6f7b54c8cb-8s9ds"
Nov 03 19:57:13.413182 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:13.413143    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fcdf540-fd42-4f5e-bb3a-422d726c3281-kube-api-access-jfvsx\" (UniqueName: \"kubernetes.io/projected/4fcdf540-fd42-4f5e-bb3a-422d726c3281-kube-api-access-jfvsx\") pod \"4fcdf540-fd42-4f5e-bb3a-422d726c3281\" (UID: \"4fcdf540-fd42-4f5e-bb3a-422d726c3281\") "
Nov 03 19:57:13.413432 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:13.413288    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fcdf540-fd42-4f5e-bb3a-422d726c3281-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fcdf540-fd42-4f5e-bb3a-422d726c3281-openshift-service-ca\") pod \"4fcdf540-fd42-4f5e-bb3a-422d726c3281\" (UID: \"4fcdf540-fd42-4f5e-bb3a-422d726c3281\") "
Nov 03 19:57:13.413432 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:13.413332    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4fcdf540-fd42-4f5e-bb3a-422d726c3281-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fcdf540-fd42-4f5e-bb3a-422d726c3281-maas-api-tls\") pod \"4fcdf540-fd42-4f5e-bb3a-422d726c3281\" (UID: \"4fcdf540-fd42-4f5e-bb3a-422d726c3281\") "
Nov 03 19:57:13.413606 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:13.413445    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4fcdf540-fd42-4f5e-bb3a-422d726c3281-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fcdf540-fd42-4f5e-bb3a-422d726c3281-metrics-certs\") pod \"4fcdf540-fd42-4f5e-bb3a-422d726c3281\" (UID: \"4fcdf540-fd42-4f5e-bb3a-422d726c3281\") "
Nov 03 19:57:13.434801 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:13.424259    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fcdf540-fd42-4f5e-bb3a-422d726c3281-openshift-service-ca" pod "4fcdf540-fd42-4f5e-bb3a-422d726c3281" (UID: "4fcdf540-fd42-4f5e-bb3a-422d726c3281"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:57:13.481394 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:13.480173    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fcdf540-fd42-4f5e-bb3a-422d726c3281-kube-api-access-jfvsx" pod "4fcdf540-fd42-4f5e-bb3a-422d726c3281" (UID: "4fcdf540-fd42-4f5e-bb3a-422d726c3281"). InnerVolumeSpecName "kube-api-access-jfvsx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:57:13.481394 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:13.480308    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fcdf540-fd42-4f5e-bb3a-422d726c3281-maas-api-tls" pod "4fcdf540-fd42-4f5e-bb3a-422d726c3281" (UID: "4fcdf540-fd42-4f5e-bb3a-422d726c3281"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:57:13.487407 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:13.487194    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fcdf540-fd42-4f5e-bb3a-422d726c3281-metrics-certs" pod "4fcdf540-fd42-4f5e-bb3a-422d726c3281" (UID: "4fcdf540-fd42-4f5e-bb3a-422d726c3281"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:57:13.516470 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:13.516442    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfvsx\" (UniqueName: \"kubernetes.io/projected/4fcdf540-fd42-4f5e-bb3a-422d726c3281-kube-api-access-jfvsx\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:57:13.516470 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:13.516470    4921 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fcdf540-fd42-4f5e-bb3a-422d726c3281-openshift-service-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:57:13.516832 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:13.516482    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fcdf540-fd42-4f5e-bb3a-422d726c3281-maas-api-tls\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:57:13.516832 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:13.516496    4921 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fcdf540-fd42-4f5e-bb3a-422d726c3281-metrics-certs\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:57:14.246528 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:14.246498    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0d0987-6f7b54c8cb-8s9ds"
Nov 03 19:57:14.272273 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:14.272245    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0d0987-6f7b54c8cb-8s9ds"]
Nov 03 19:57:14.276824 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:14.276798    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0d0987-6f7b54c8cb-8s9ds"]
Nov 03 19:57:14.890778 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:14.890748    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fcdf540-fd42-4f5e-bb3a-422d726c3281" path="/var/lib/kubelet/pods/4fcdf540-fd42-4f5e-bb3a-422d726c3281/volumes"
Nov 03 19:57:15.425756 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:15.425728    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c18bea-5b7994d5ff-s76l2"]
Nov 03 19:57:15.426594 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:15.426579    4921 cpu_manager.go:414] "containerMap: removing container" podUID="4fcdf540-fd42-4f5e-bb3a-422d726c3281" containerName="maas-api"
Nov 03 19:57:15.426656 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:15.426596    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fcdf540-fd42-4f5e-bb3a-422d726c3281" containerName="maas-api"
Nov 03 19:57:15.426812 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:15.426801    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fcdf540-fd42-4f5e-bb3a-422d726c3281" containerName="maas-api"
Nov 03 19:57:15.430931 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:15.430905    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c18bea-5b7994d5ff-s76l2"
Nov 03 19:57:15.443925 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:15.443892    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c18bea-5b7994d5ff-s76l2"]
Nov 03 19:57:15.462210 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:15.462175    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-c18bea-serving-cert\""
Nov 03 19:57:15.510021 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:15.509994    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f347075-1266-4c50-81ff-a8d5ab87b1b3-maas-api-tls\") pod \"maas-api-e2e-inf-b-c18bea-5b7994d5ff-s76l2\" (UID: \"3f347075-1266-4c50-81ff-a8d5ab87b1b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c18bea-5b7994d5ff-s76l2"
Nov 03 19:57:15.510168 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:15.510048    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f347075-1266-4c50-81ff-a8d5ab87b1b3-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c18bea-5b7994d5ff-s76l2\" (UID: \"3f347075-1266-4c50-81ff-a8d5ab87b1b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c18bea-5b7994d5ff-s76l2"
Nov 03 19:57:15.510289 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:15.510265    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj7mt\" (UniqueName: \"kubernetes.io/projected/3f347075-1266-4c50-81ff-a8d5ab87b1b3-kube-api-access-jj7mt\") pod \"maas-api-e2e-inf-b-c18bea-5b7994d5ff-s76l2\" (UID: \"3f347075-1266-4c50-81ff-a8d5ab87b1b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c18bea-5b7994d5ff-s76l2"
Nov 03 19:57:15.510390 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:15.510372    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f347075-1266-4c50-81ff-a8d5ab87b1b3-metrics-certs\") pod \"maas-api-e2e-inf-b-c18bea-5b7994d5ff-s76l2\" (UID: \"3f347075-1266-4c50-81ff-a8d5ab87b1b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c18bea-5b7994d5ff-s76l2"
Nov 03 19:57:15.616253 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:15.616210    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f347075-1266-4c50-81ff-a8d5ab87b1b3-metrics-certs\") pod \"maas-api-e2e-inf-b-c18bea-5b7994d5ff-s76l2\" (UID: \"3f347075-1266-4c50-81ff-a8d5ab87b1b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c18bea-5b7994d5ff-s76l2"
Nov 03 19:57:15.616635 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:15.616609    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f347075-1266-4c50-81ff-a8d5ab87b1b3-maas-api-tls\") pod \"maas-api-e2e-inf-b-c18bea-5b7994d5ff-s76l2\" (UID: \"3f347075-1266-4c50-81ff-a8d5ab87b1b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c18bea-5b7994d5ff-s76l2"
Nov 03 19:57:15.616839 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:15.616820    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f347075-1266-4c50-81ff-a8d5ab87b1b3-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c18bea-5b7994d5ff-s76l2\" (UID: \"3f347075-1266-4c50-81ff-a8d5ab87b1b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c18bea-5b7994d5ff-s76l2"
Nov 03 19:57:15.617202 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:15.617178    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj7mt\" (UniqueName: \"kubernetes.io/projected/3f347075-1266-4c50-81ff-a8d5ab87b1b3-kube-api-access-jj7mt\") pod \"maas-api-e2e-inf-b-c18bea-5b7994d5ff-s76l2\" (UID: \"3f347075-1266-4c50-81ff-a8d5ab87b1b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c18bea-5b7994d5ff-s76l2"
Nov 03 19:57:15.625539 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:57:15.625505    4921 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-c18bea-serving-cert: secret "maas-api-e2e-inf-b-c18bea-serving-cert" not found
Nov 03 19:57:15.625901 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:57:15.625881    4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f347075-1266-4c50-81ff-a8d5ab87b1b3-maas-api-tls podName:3f347075-1266-4c50-81ff-a8d5ab87b1b3 nodeName:}" failed. No retries permitted until 2026-11-03 19:57:16.125856639 +0000 UTC m=+1549.369822092 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3f347075-1266-4c50-81ff-a8d5ab87b1b3-maas-api-tls") pod "maas-api-e2e-inf-b-c18bea-5b7994d5ff-s76l2" (UID: "3f347075-1266-4c50-81ff-a8d5ab87b1b3") : secret "maas-api-e2e-inf-b-c18bea-serving-cert" not found
Nov 03 19:57:15.626579 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:15.626556    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f347075-1266-4c50-81ff-a8d5ab87b1b3-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c18bea-5b7994d5ff-s76l2\" (UID: \"3f347075-1266-4c50-81ff-a8d5ab87b1b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c18bea-5b7994d5ff-s76l2"
Nov 03 19:57:15.641689 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:15.638287    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f347075-1266-4c50-81ff-a8d5ab87b1b3-metrics-certs\") pod \"maas-api-e2e-inf-b-c18bea-5b7994d5ff-s76l2\" (UID: \"3f347075-1266-4c50-81ff-a8d5ab87b1b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c18bea-5b7994d5ff-s76l2"
Nov 03 19:57:15.656691 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:15.653819    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj7mt\" (UniqueName: \"kubernetes.io/projected/3f347075-1266-4c50-81ff-a8d5ab87b1b3-kube-api-access-jj7mt\") pod \"maas-api-e2e-inf-b-c18bea-5b7994d5ff-s76l2\" (UID: \"3f347075-1266-4c50-81ff-a8d5ab87b1b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c18bea-5b7994d5ff-s76l2"
Nov 03 19:57:16.135698 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:16.132016    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f347075-1266-4c50-81ff-a8d5ab87b1b3-maas-api-tls\") pod \"maas-api-e2e-inf-b-c18bea-5b7994d5ff-s76l2\" (UID: \"3f347075-1266-4c50-81ff-a8d5ab87b1b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c18bea-5b7994d5ff-s76l2"
Nov 03 19:57:16.161694 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:16.154892    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f347075-1266-4c50-81ff-a8d5ab87b1b3-maas-api-tls\") pod \"maas-api-e2e-inf-b-c18bea-5b7994d5ff-s76l2\" (UID: \"3f347075-1266-4c50-81ff-a8d5ab87b1b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c18bea-5b7994d5ff-s76l2"
Nov 03 19:57:16.368535 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:16.367591    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c18bea-5b7994d5ff-s76l2"
Nov 03 19:57:16.871479 ip-10-0-134-222 kubenswrapper[4921]: W1103 19:57:16.871425    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f347075_1266_4c50_81ff_a8d5ab87b1b3.slice/crio-148fbd7921d71c9c2cffab0d74eb560eabe33a596fa06f2c56cb9a706d17e667 WatchSource:0}: Error finding container 148fbd7921d71c9c2cffab0d74eb560eabe33a596fa06f2c56cb9a706d17e667: Status 404 returned error can't find the container with id 148fbd7921d71c9c2cffab0d74eb560eabe33a596fa06f2c56cb9a706d17e667
Nov 03 19:57:16.914720 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:16.896199    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c18bea-5b7994d5ff-s76l2"]
Nov 03 19:57:17.317322 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:17.315144    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c18bea-5b7994d5ff-s76l2" event={"ID":"3f347075-1266-4c50-81ff-a8d5ab87b1b3","Type":"ContainerStarted","Data":"148fbd7921d71c9c2cffab0d74eb560eabe33a596fa06f2c56cb9a706d17e667"}
Nov 03 19:57:17.553973 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:17.553943    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-f58171/test-model-f58171-kserve-5c596fcddc-vgqtt"
Nov 03 19:57:17.566132 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:17.566108    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-f58171/test-model-f58171-kserve-5c596fcddc-vgqtt"
Nov 03 19:57:18.325542 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:18.325502    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c18bea-5b7994d5ff-s76l2" event={"ID":"3f347075-1266-4c50-81ff-a8d5ab87b1b3","Type":"ContainerStarted","Data":"08ad2f541dbadaee372a2dd7116146b3cdbf66a389fe455ac536396c9e338c32"}
Nov 03 19:57:18.325773 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:18.325611    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c18bea-5b7994d5ff-s76l2"
Nov 03 19:57:18.337332 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:18.337306    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-f58171/test-model-f58171-kserve-5c596fcddc-vgqtt"
Nov 03 19:57:18.352498 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:18.352433    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c18bea-5b7994d5ff-s76l2" podStartSLOduration=2.81488039 podStartE2EDuration="3.352419323s" totalImagesPullingTime="537.538933ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:57:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:57:18.34961784 +0000 UTC m=+1551.593583305" watchObservedRunningTime="2026-11-03 19:57:18.352419323 +0000 UTC m=+1551.596384788"
Nov 03 19:57:19.625556 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:19.625519    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl"]
Nov 03 19:57:19.626096 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:19.625800    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl" podUID="55ff258b-9816-4fff-a74a-3c417bf0f0ee" containerName="istio-proxy" containerID="cri-o://e1d1f3573b618e8015e9ef6897a1d097c3e97de954faf99fb4f27db3bf4cdec6" gracePeriod=30
Nov 03 19:57:20.322639 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:20.322605    4921 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 19:57:20.322889 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:20.322661    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl" podUID="55ff258b-9816-4fff-a74a-3c417bf0f0ee" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 19:57:22.058814 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:22.058742    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db72c9bb-sdzcv"]
Nov 03 19:57:22.081699 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:22.079290    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db72c9bb-sdzcv"]
Nov 03 19:57:22.896509 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:22.896463    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="758e8ecf-b0c0-4133-889f-6aa948dc7a89" path="/var/lib/kubelet/pods/758e8ecf-b0c0-4133-889f-6aa948dc7a89/volumes"
Nov 03 19:57:24.343058 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:24.343029    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c18bea-5b7994d5ff-s76l2"
Nov 03 19:57:25.266404 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:25.266376    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl"
Nov 03 19:57:25.378697 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:25.378651    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55ff258b-9816-4fff-a74a-3c417bf0f0ee-kube-api-access-qwr7x\" (UniqueName: \"kubernetes.io/projected/55ff258b-9816-4fff-a74a-3c417bf0f0ee-kube-api-access-qwr7x\") pod \"55ff258b-9816-4fff-a74a-3c417bf0f0ee\" (UID: \"55ff258b-9816-4fff-a74a-3c417bf0f0ee\") "
Nov 03 19:57:25.379160 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:25.378773    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55ff258b-9816-4fff-a74a-3c417bf0f0ee-istio-token\" (UniqueName: \"kubernetes.io/projected/55ff258b-9816-4fff-a74a-3c417bf0f0ee-istio-token\") pod \"55ff258b-9816-4fff-a74a-3c417bf0f0ee\" (UID: \"55ff258b-9816-4fff-a74a-3c417bf0f0ee\") "
Nov 03 19:57:25.379160 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:25.378861    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55ff258b-9816-4fff-a74a-3c417bf0f0ee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55ff258b-9816-4fff-a74a-3c417bf0f0ee-workload-certs\") pod \"55ff258b-9816-4fff-a74a-3c417bf0f0ee\" (UID: \"55ff258b-9816-4fff-a74a-3c417bf0f0ee\") "
Nov 03 19:57:25.379160 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:25.378895    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/55ff258b-9816-4fff-a74a-3c417bf0f0ee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55ff258b-9816-4fff-a74a-3c417bf0f0ee-istio-podinfo\") pod \"55ff258b-9816-4fff-a74a-3c417bf0f0ee\" (UID: \"55ff258b-9816-4fff-a74a-3c417bf0f0ee\") "
Nov 03 19:57:25.379160 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:25.378962    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55ff258b-9816-4fff-a74a-3c417bf0f0ee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55ff258b-9816-4fff-a74a-3c417bf0f0ee-istio-envoy\") pod \"55ff258b-9816-4fff-a74a-3c417bf0f0ee\" (UID: \"55ff258b-9816-4fff-a74a-3c417bf0f0ee\") "
Nov 03 19:57:25.379160 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:25.379008    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55ff258b-9816-4fff-a74a-3c417bf0f0ee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55ff258b-9816-4fff-a74a-3c417bf0f0ee-credential-socket\") pod \"55ff258b-9816-4fff-a74a-3c417bf0f0ee\" (UID: \"55ff258b-9816-4fff-a74a-3c417bf0f0ee\") "
Nov 03 19:57:25.379160 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:25.379093    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55ff258b-9816-4fff-a74a-3c417bf0f0ee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55ff258b-9816-4fff-a74a-3c417bf0f0ee-istiod-ca-cert\") pod \"55ff258b-9816-4fff-a74a-3c417bf0f0ee\" (UID: \"55ff258b-9816-4fff-a74a-3c417bf0f0ee\") "
Nov 03 19:57:25.379160 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:25.379139    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55ff258b-9816-4fff-a74a-3c417bf0f0ee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55ff258b-9816-4fff-a74a-3c417bf0f0ee-istio-data\") pod \"55ff258b-9816-4fff-a74a-3c417bf0f0ee\" (UID: \"55ff258b-9816-4fff-a74a-3c417bf0f0ee\") "
Nov 03 19:57:25.379527 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:25.379199    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55ff258b-9816-4fff-a74a-3c417bf0f0ee-workload-certs" pod "55ff258b-9816-4fff-a74a-3c417bf0f0ee" (UID: "55ff258b-9816-4fff-a74a-3c417bf0f0ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:57:25.379527 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:25.379276    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55ff258b-9816-4fff-a74a-3c417bf0f0ee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55ff258b-9816-4fff-a74a-3c417bf0f0ee-workload-socket\") pod \"55ff258b-9816-4fff-a74a-3c417bf0f0ee\" (UID: \"55ff258b-9816-4fff-a74a-3c417bf0f0ee\") "
Nov 03 19:57:25.379527 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:25.379495    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55ff258b-9816-4fff-a74a-3c417bf0f0ee-istio-data" pod "55ff258b-9816-4fff-a74a-3c417bf0f0ee" (UID: "55ff258b-9816-4fff-a74a-3c417bf0f0ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:57:25.379723 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:25.379520    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55ff258b-9816-4fff-a74a-3c417bf0f0ee-istiod-ca-cert" pod "55ff258b-9816-4fff-a74a-3c417bf0f0ee" (UID: "55ff258b-9816-4fff-a74a-3c417bf0f0ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:57:25.379723 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:25.379606    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55ff258b-9816-4fff-a74a-3c417bf0f0ee-credential-socket" pod "55ff258b-9816-4fff-a74a-3c417bf0f0ee" (UID: "55ff258b-9816-4fff-a74a-3c417bf0f0ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:57:25.379857 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:25.379834    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55ff258b-9816-4fff-a74a-3c417bf0f0ee-workload-socket" pod "55ff258b-9816-4fff-a74a-3c417bf0f0ee" (UID: "55ff258b-9816-4fff-a74a-3c417bf0f0ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:57:25.380240 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:25.380220    4921 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55ff258b-9816-4fff-a74a-3c417bf0f0ee-workload-socket\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:57:25.380333 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:25.380247    4921 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55ff258b-9816-4fff-a74a-3c417bf0f0ee-workload-certs\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:57:25.380333 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:25.380265    4921 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55ff258b-9816-4fff-a74a-3c417bf0f0ee-credential-socket\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:57:25.380333 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:25.380279    4921 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55ff258b-9816-4fff-a74a-3c417bf0f0ee-istiod-ca-cert\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:57:25.380333 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:25.380293    4921 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55ff258b-9816-4fff-a74a-3c417bf0f0ee-istio-data\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:57:25.381326 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:25.381299    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55ff258b-9816-4fff-a74a-3c417bf0f0ee-kube-api-access-qwr7x" pod "55ff258b-9816-4fff-a74a-3c417bf0f0ee" (UID: "55ff258b-9816-4fff-a74a-3c417bf0f0ee"). InnerVolumeSpecName "kube-api-access-qwr7x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:57:25.381446 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:25.381420    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55ff258b-9816-4fff-a74a-3c417bf0f0ee-istio-token" pod "55ff258b-9816-4fff-a74a-3c417bf0f0ee" (UID: "55ff258b-9816-4fff-a74a-3c417bf0f0ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:57:25.381720 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:25.381655    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/55ff258b-9816-4fff-a74a-3c417bf0f0ee-istio-podinfo" pod "55ff258b-9816-4fff-a74a-3c417bf0f0ee" (UID: "55ff258b-9816-4fff-a74a-3c417bf0f0ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:57:25.381866 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:25.381844    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55ff258b-9816-4fff-a74a-3c417bf0f0ee-istio-envoy" pod "55ff258b-9816-4fff-a74a-3c417bf0f0ee" (UID: "55ff258b-9816-4fff-a74a-3c417bf0f0ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:57:25.485916 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:25.482835    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwr7x\" (UniqueName: \"kubernetes.io/projected/55ff258b-9816-4fff-a74a-3c417bf0f0ee-kube-api-access-qwr7x\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:57:25.485916 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:25.482874    4921 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55ff258b-9816-4fff-a74a-3c417bf0f0ee-istio-token\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:57:25.485916 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:25.482890    4921 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55ff258b-9816-4fff-a74a-3c417bf0f0ee-istio-podinfo\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:57:25.485916 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:25.482904    4921 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55ff258b-9816-4fff-a74a-3c417bf0f0ee-istio-envoy\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:57:25.491059 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:25.489538    4921 generic.go:356] "Generic (PLEG): container finished" podID="55ff258b-9816-4fff-a74a-3c417bf0f0ee" containerID="e1d1f3573b618e8015e9ef6897a1d097c3e97de954faf99fb4f27db3bf4cdec6" exitCode=0
Nov 03 19:57:25.491059 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:25.489765    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl" event={"ID":"55ff258b-9816-4fff-a74a-3c417bf0f0ee","Type":"ContainerDied","Data":"e1d1f3573b618e8015e9ef6897a1d097c3e97de954faf99fb4f27db3bf4cdec6"}
Nov 03 19:57:25.491059 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:25.489828    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl" event={"ID":"55ff258b-9816-4fff-a74a-3c417bf0f0ee","Type":"ContainerDied","Data":"3be859da63518ae95264a909a91e86ac85f7d614dc10619e91980518bdd87687"}
Nov 03 19:57:25.491059 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:25.489862    4921 scope.go:122] "RemoveContainer" containerID="e1d1f3573b618e8015e9ef6897a1d097c3e97de954faf99fb4f27db3bf4cdec6"
Nov 03 19:57:25.491059 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:25.490045    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl"
Nov 03 19:57:25.508789 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:25.508769    4921 scope.go:122] "RemoveContainer" containerID="e1d1f3573b618e8015e9ef6897a1d097c3e97de954faf99fb4f27db3bf4cdec6"
Nov 03 19:57:25.509129 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:57:25.509087    4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1d1f3573b618e8015e9ef6897a1d097c3e97de954faf99fb4f27db3bf4cdec6\": container with ID starting with e1d1f3573b618e8015e9ef6897a1d097c3e97de954faf99fb4f27db3bf4cdec6 not found: ID does not exist" containerID="e1d1f3573b618e8015e9ef6897a1d097c3e97de954faf99fb4f27db3bf4cdec6"
Nov 03 19:57:25.509289 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:25.509135    4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1d1f3573b618e8015e9ef6897a1d097c3e97de954faf99fb4f27db3bf4cdec6"} err="failed to get container status \"e1d1f3573b618e8015e9ef6897a1d097c3e97de954faf99fb4f27db3bf4cdec6\": rpc error: code = NotFound desc = could not find container \"e1d1f3573b618e8015e9ef6897a1d097c3e97de954faf99fb4f27db3bf4cdec6\": container with ID starting with e1d1f3573b618e8015e9ef6897a1d097c3e97de954faf99fb4f27db3bf4cdec6 not found: ID does not exist"
Nov 03 19:57:25.522697 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:25.522640    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl"]
Nov 03 19:57:25.527830 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:25.527802    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-25trl"]
Nov 03 19:57:26.928247 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:26.928199    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55ff258b-9816-4fff-a74a-3c417bf0f0ee" path="/var/lib/kubelet/pods/55ff258b-9816-4fff-a74a-3c417bf0f0ee/volumes"
Nov 03 19:57:29.830898 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:29.830863    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2747a4-647847c6c9-njbpx"]
Nov 03 19:57:29.831682 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:29.831642    4921 cpu_manager.go:414] "containerMap: removing container" podUID="55ff258b-9816-4fff-a74a-3c417bf0f0ee" containerName="istio-proxy"
Nov 03 19:57:29.831682 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:29.831662    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55ff258b-9816-4fff-a74a-3c417bf0f0ee" containerName="istio-proxy"
Nov 03 19:57:29.831898 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:29.831861    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="55ff258b-9816-4fff-a74a-3c417bf0f0ee" containerName="istio-proxy"
Nov 03 19:57:29.837219 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:29.837200    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2747a4-647847c6c9-njbpx"
Nov 03 19:57:29.846006 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:29.845976    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-2747a4-dockercfg-q8ppw\""
Nov 03 19:57:29.846995 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:29.846969    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2747a4-647847c6c9-njbpx"]
Nov 03 19:57:29.850050 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:29.849999    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-2747a4\""
Nov 03 19:57:29.877055 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:29.877017    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2747a4-67946747c9-4smgl"]
Nov 03 19:57:29.882801 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:29.882081    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2747a4-67946747c9-4smgl"
Nov 03 19:57:29.894084 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:29.894052    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2747a4-67946747c9-4smgl"]
Nov 03 19:57:29.958597 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:29.958557    4921 scope.go:122] "RemoveContainer" containerID="f2f5191060062a885cdd5037691537542252c87e4d60e861f7f2edf63f475e58"
Nov 03 19:57:30.080023 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:30.043395    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ed39371-700a-43b3-a66b-a500d13cfd77-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2747a4-67946747c9-4smgl\" (UID: \"0ed39371-700a-43b3-a66b-a500d13cfd77\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2747a4-67946747c9-4smgl"
Nov 03 19:57:30.095885 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:30.095801    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b642f55a-3767-464c-b9ff-280627b3e277-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2747a4-647847c6c9-njbpx\" (UID: \"b642f55a-3767-464c-b9ff-280627b3e277\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2747a4-647847c6c9-njbpx"
Nov 03 19:57:30.096448 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:30.096422    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw8jj\" (UniqueName: \"kubernetes.io/projected/0ed39371-700a-43b3-a66b-a500d13cfd77-kube-api-access-vw8jj\") pod \"payload-processing-e2e-inf-a-2747a4-67946747c9-4smgl\" (UID: \"0ed39371-700a-43b3-a66b-a500d13cfd77\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2747a4-67946747c9-4smgl"
Nov 03 19:57:30.096728 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:30.096708    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bnsn\" (UniqueName: \"kubernetes.io/projected/b642f55a-3767-464c-b9ff-280627b3e277-kube-api-access-5bnsn\") pod \"payload-pre-processing-e2e-inf-a-2747a4-647847c6c9-njbpx\" (UID: \"b642f55a-3767-464c-b9ff-280627b3e277\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2747a4-647847c6c9-njbpx"
Nov 03 19:57:30.191724 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:30.189056    4921 scope.go:122] "RemoveContainer" containerID="a5a2f71b5a27308b852fad607e41cc6e90e17a28ef3c7acdd91323a982db80d0"
Nov 03 19:57:30.210943 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:30.210905    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw8jj\" (UniqueName: \"kubernetes.io/projected/0ed39371-700a-43b3-a66b-a500d13cfd77-kube-api-access-vw8jj\") pod \"payload-processing-e2e-inf-a-2747a4-67946747c9-4smgl\" (UID: \"0ed39371-700a-43b3-a66b-a500d13cfd77\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2747a4-67946747c9-4smgl"
Nov 03 19:57:30.211313 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:30.211280    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bnsn\" (UniqueName: \"kubernetes.io/projected/b642f55a-3767-464c-b9ff-280627b3e277-kube-api-access-5bnsn\") pod \"payload-pre-processing-e2e-inf-a-2747a4-647847c6c9-njbpx\" (UID: \"b642f55a-3767-464c-b9ff-280627b3e277\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2747a4-647847c6c9-njbpx"
Nov 03 19:57:30.211522 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:30.211508    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ed39371-700a-43b3-a66b-a500d13cfd77-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2747a4-67946747c9-4smgl\" (UID: \"0ed39371-700a-43b3-a66b-a500d13cfd77\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2747a4-67946747c9-4smgl"
Nov 03 19:57:30.211804 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:30.211785    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b642f55a-3767-464c-b9ff-280627b3e277-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2747a4-647847c6c9-njbpx\" (UID: \"b642f55a-3767-464c-b9ff-280627b3e277\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2747a4-647847c6c9-njbpx"
Nov 03 19:57:30.217059 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:30.216322    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ed39371-700a-43b3-a66b-a500d13cfd77-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2747a4-67946747c9-4smgl\" (UID: \"0ed39371-700a-43b3-a66b-a500d13cfd77\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2747a4-67946747c9-4smgl"
Nov 03 19:57:30.217273 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:30.217079    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b642f55a-3767-464c-b9ff-280627b3e277-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2747a4-647847c6c9-njbpx\" (UID: \"b642f55a-3767-464c-b9ff-280627b3e277\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2747a4-647847c6c9-njbpx"
Nov 03 19:57:30.304691 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:30.300497    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw8jj\" (UniqueName: \"kubernetes.io/projected/0ed39371-700a-43b3-a66b-a500d13cfd77-kube-api-access-vw8jj\") pod \"payload-processing-e2e-inf-a-2747a4-67946747c9-4smgl\" (UID: \"0ed39371-700a-43b3-a66b-a500d13cfd77\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2747a4-67946747c9-4smgl"
Nov 03 19:57:30.321801 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:30.319413    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bnsn\" (UniqueName: \"kubernetes.io/projected/b642f55a-3767-464c-b9ff-280627b3e277-kube-api-access-5bnsn\") pod \"payload-pre-processing-e2e-inf-a-2747a4-647847c6c9-njbpx\" (UID: \"b642f55a-3767-464c-b9ff-280627b3e277\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2747a4-647847c6c9-njbpx"
Nov 03 19:57:30.490693 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:30.486132    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2747a4-647847c6c9-njbpx"
Nov 03 19:57:30.534713 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:30.530952    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2747a4-67946747c9-4smgl"
Nov 03 19:57:32.386856 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:32.386807    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2747a4-647847c6c9-njbpx"]
Nov 03 19:57:32.600806 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:32.600572    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2747a4-67946747c9-4smgl"]
Nov 03 19:57:32.827277 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:32.827231    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2747a4-647847c6c9-njbpx" event={"ID":"b642f55a-3767-464c-b9ff-280627b3e277","Type":"ContainerStarted","Data":"bb93252d5bb8e8021bb71d16b794c26a1397b3d0c9ce11660459476a4236c3b6"}
Nov 03 19:57:32.827474 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:32.827304    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2747a4-647847c6c9-njbpx"
Nov 03 19:57:32.827474 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:32.827328    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2747a4-647847c6c9-njbpx" event={"ID":"b642f55a-3767-464c-b9ff-280627b3e277","Type":"ContainerStarted","Data":"ef7279a809abc914d76746a4e13d5fc702515789b5eb312991f0ebe755e4ec47"}
Nov 03 19:57:32.829564 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:32.829534    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2747a4-67946747c9-4smgl" event={"ID":"0ed39371-700a-43b3-a66b-a500d13cfd77","Type":"ContainerStarted","Data":"dc7004d6f92d63728b8a7c9c79f11fb1040776c50ec2391b2e26b41350a8b9d2"}
Nov 03 19:57:32.829737 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:32.829585    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2747a4-67946747c9-4smgl" event={"ID":"0ed39371-700a-43b3-a66b-a500d13cfd77","Type":"ContainerStarted","Data":"d4a0f22ac40b005a8249c86804cd11da29b84bdd0da7c6cd72f3f0d53e17a5c1"}
Nov 03 19:57:32.829737 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:32.829617    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-2747a4-67946747c9-4smgl"
Nov 03 19:57:32.844918 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:32.844856    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2747a4-647847c6c9-njbpx" podStartSLOduration=3.844837616 podStartE2EDuration="3.844837616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:57:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:57:32.843817709 +0000 UTC m=+1566.087783177" watchObservedRunningTime="2026-11-03 19:57:32.844837616 +0000 UTC m=+1566.088803082"
Nov 03 19:57:32.865120 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:32.865055    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-2747a4-67946747c9-4smgl" podStartSLOduration=3.865036239 podStartE2EDuration="3.865036239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:57:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:57:32.858958261 +0000 UTC m=+1566.102923747" watchObservedRunningTime="2026-11-03 19:57:32.865036239 +0000 UTC m=+1566.109001705"
Nov 03 19:57:32.926901 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:57:32.926852    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 19:57:43.854512 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:43.854425    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-2747a4-67946747c9-4smgl"
Nov 03 19:57:43.855081 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:57:43.854863    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2747a4-647847c6c9-njbpx"
Nov 03 19:58:02.924467 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:58:02.924428    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 19:58:30.780058 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:58:30.780024    4921 scope.go:122] "RemoveContainer" containerID="f912e347d6458e55ab77c911f137bd6d0f4c88ab21d54c050f613e9845988436"
Nov 03 19:58:30.791817 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:58:30.791793    4921 scope.go:122] "RemoveContainer" containerID="1e4ab45d04a590f407ac437132822221f2c874b33043ec73c6fcb6de0f96490a"
Nov 03 19:58:30.802897 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:58:30.802867    4921 scope.go:122] "RemoveContainer" containerID="c2cc150bffaf5d2cf721dcb285e825f3a531d2381ac52e18af67a02af7c6eb2f"
Nov 03 19:58:30.832854 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:58:30.830338    4921 scope.go:122] "RemoveContainer" containerID="1e0fdf2b7b3fc4d1f41ced9bb3d946cc8a3d6a230eec30b852daa3607a02a10a"
Nov 03 19:58:32.949182 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:58:32.949132    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 19:59:02.972876 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:59:02.972835    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 19:59:30.951573 ip-10-0-134-222 kubenswrapper[4921]: I1103 19:59:30.951496    4921 scope.go:122] "RemoveContainer" containerID="1d5ff4531378f990e85ba09a2c4c5aee33634273659e00a6dad962a5fc178e02"
Nov 03 19:59:32.980063 ip-10-0-134-222 kubenswrapper[4921]: E1103 19:59:32.980024    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 20:00:00.139817 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:00.139780    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f4e123-29895600-jsc8z"]
Nov 03 20:00:00.144906 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:00.144884    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f4e123-29895600-jsc8z"
Nov 03 20:00:00.150961 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:00.150929    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 20:00:00.159658 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:00.158771    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f4e123-29895600-jsc8z"]
Nov 03 20:00:00.166961 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:00.166326    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d4127-29895600-fh6sm"]
Nov 03 20:00:00.182690 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:00.179449    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b794761-73bc-42a3-9efc-ba2eb7278181-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-f4e123-29895600-jsc8z\" (UID: \"1b794761-73bc-42a3-9efc-ba2eb7278181\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f4e123-29895600-jsc8z"
Nov 03 20:00:00.182690 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:00.179783    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfk6t\" (UniqueName: \"kubernetes.io/projected/1b794761-73bc-42a3-9efc-ba2eb7278181-kube-api-access-xfk6t\") pod \"maas-api-key-cleanup-e2e-worker-w4-f4e123-29895600-jsc8z\" (UID: \"1b794761-73bc-42a3-9efc-ba2eb7278181\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f4e123-29895600-jsc8z"
Nov 03 20:00:00.191153 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:00.191080    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3cee6a-29895600-rl2vr"]
Nov 03 20:00:00.194361 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:00.194330    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d4127-29895600-fh6sm"
Nov 03 20:00:00.211738 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:00.197917    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d4127-29895600-fh6sm"]
Nov 03 20:00:00.211738 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:00.197964    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3cee6a-29895600-rl2vr"]
Nov 03 20:00:00.211738 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:00.198059    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3cee6a-29895600-rl2vr"
Nov 03 20:00:00.211738 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:00.203717    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-82dbbf-29895600-9pjt9"]
Nov 03 20:00:00.213079 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:00.212492    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-82dbbf-29895600-9pjt9"
Nov 03 20:00:00.219757 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:00.219196    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9f3188-29895600-wtngr"]
Nov 03 20:00:00.227302 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:00.227273    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9f3188-29895600-wtngr"]
Nov 03 20:00:00.227469 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:00.227343    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9f3188-29895600-wtngr"
Nov 03 20:00:00.235336 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:00.235309    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-82dbbf-29895600-9pjt9"]
Nov 03 20:00:00.251356 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:00.251328    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-102379-29895600-l5pw6"]
Nov 03 20:00:00.258019 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:00.257442    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-102379-29895600-l5pw6"
Nov 03 20:00:00.258184 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:00.258148    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-79c47f-29895600-l6n9d"]
Nov 03 20:00:00.264463 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:00.264436    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-79c47f-29895600-l6n9d"
Nov 03 20:00:00.266288 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:00.266208    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-102379-29895600-l5pw6"]
Nov 03 20:00:00.273223 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:00.273181    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-79c47f-29895600-l6n9d"]
Nov 03 20:00:00.282058 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:00.282019    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfk6t\" (UniqueName: \"kubernetes.io/projected/1b794761-73bc-42a3-9efc-ba2eb7278181-kube-api-access-xfk6t\") pod \"maas-api-key-cleanup-e2e-worker-w4-f4e123-29895600-jsc8z\" (UID: \"1b794761-73bc-42a3-9efc-ba2eb7278181\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f4e123-29895600-jsc8z"
Nov 03 20:00:00.282267 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:00.282141    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b2bf43f-4806-48b4-bd9b-1d6092b082e8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-79c47f-29895600-l6n9d\" (UID: \"7b2bf43f-4806-48b4-bd9b-1d6092b082e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-79c47f-29895600-l6n9d"
Nov 03 20:00:00.282267 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:00.282184    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xxdj\" (UniqueName: \"kubernetes.io/projected/a0c0fc1f-186d-4c54-bd71-d36671cbb718-kube-api-access-9xxdj\") pod \"maas-api-key-cleanup-e2e-worker-w2-8d4127-29895600-fh6sm\" (UID: \"a0c0fc1f-186d-4c54-bd71-d36671cbb718\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d4127-29895600-fh6sm"
Nov 03 20:00:00.282267 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:00.282243    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-528q5\" (UniqueName: \"kubernetes.io/projected/9fe29ef3-6905-4213-bf69-63105ba9bc29-kube-api-access-528q5\") pod \"maas-api-key-cleanup-e2e-worker-w0-82dbbf-29895600-9pjt9\" (UID: \"9fe29ef3-6905-4213-bf69-63105ba9bc29\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-82dbbf-29895600-9pjt9"
Nov 03 20:00:00.282916 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:00.282704    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgjj8\" (UniqueName: \"kubernetes.io/projected/5769966d-308a-44a9-928a-984d95d95820-kube-api-access-lgjj8\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-9f3188-29895600-wtngr\" (UID: \"5769966d-308a-44a9-928a-984d95d95820\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9f3188-29895600-wtngr"
Nov 03 20:00:00.282916 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:00.282761    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tnbd\" (UniqueName: \"kubernetes.io/projected/7b2bf43f-4806-48b4-bd9b-1d6092b082e8-kube-api-access-6tnbd\") pod \"maas-api-key-cleanup-e2e-inf-b-79c47f-29895600-l6n9d\" (UID: \"7b2bf43f-4806-48b4-bd9b-1d6092b082e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-79c47f-29895600-l6n9d"
Nov 03 20:00:00.282916 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:00.282882    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/951f6658-11ce-4b14-b615-31aacfd6b26e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-102379-29895600-l5pw6\" (UID: \"951f6658-11ce-4b14-b615-31aacfd6b26e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-102379-29895600-l5pw6"
Nov 03 20:00:00.283300 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:00.282934    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b794761-73bc-42a3-9efc-ba2eb7278181-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-f4e123-29895600-jsc8z\" (UID: \"1b794761-73bc-42a3-9efc-ba2eb7278181\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f4e123-29895600-jsc8z"
Nov 03 20:00:00.283300 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:00.283010    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0c0fc1f-186d-4c54-bd71-d36671cbb718-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-8d4127-29895600-fh6sm\" (UID: \"a0c0fc1f-186d-4c54-bd71-d36671cbb718\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d4127-29895600-fh6sm"
Nov 03 20:00:00.283300 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:00.283047    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5769966d-308a-44a9-928a-984d95d95820-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-9f3188-29895600-wtngr\" (UID: \"5769966d-308a-44a9-928a-984d95d95820\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9f3188-29895600-wtngr"
Nov 03 20:00:00.283300 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:00.283181    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/957d5c20-d9b0-4d7e-a8df-544c55a1b456-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-3cee6a-29895600-rl2vr\" (UID: \"957d5c20-d9b0-4d7e-a8df-544c55a1b456\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3cee6a-29895600-rl2vr"
Nov 03 20:00:00.283889 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:00.283463    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fe29ef3-6905-4213-bf69-63105ba9bc29-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-82dbbf-29895600-9pjt9\" (UID: \"9fe29ef3-6905-4213-bf69-63105ba9bc29\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-82dbbf-29895600-9pjt9"
Nov 03 20:00:00.284003 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:00.283982    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5b6vz\" (UniqueName: \"kubernetes.io/projected/951f6658-11ce-4b14-b615-31aacfd6b26e-kube-api-access-5b6vz\") pod \"maas-api-key-cleanup-e2e-worker-w6-102379-29895600-l5pw6\" (UID: \"951f6658-11ce-4b14-b615-31aacfd6b26e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-102379-29895600-l5pw6"
Nov 03 20:00:00.284063 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:00.284015    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64k5c\" (UniqueName: \"kubernetes.io/projected/957d5c20-d9b0-4d7e-a8df-544c55a1b456-kube-api-access-64k5c\") pod \"maas-api-key-cleanup-e2e-inf-a-3cee6a-29895600-rl2vr\" (UID: \"957d5c20-d9b0-4d7e-a8df-544c55a1b456\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3cee6a-29895600-rl2vr"
Nov 03 20:00:00.284119 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:00.284067    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b794761-73bc-42a3-9efc-ba2eb7278181-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-f4e123-29895600-jsc8z\" (UID: \"1b794761-73bc-42a3-9efc-ba2eb7278181\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f4e123-29895600-jsc8z"
Nov 03 20:00:00.295508 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:00.295472    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfk6t\" (UniqueName: \"kubernetes.io/projected/1b794761-73bc-42a3-9efc-ba2eb7278181-kube-api-access-xfk6t\") pod \"maas-api-key-cleanup-e2e-worker-w4-f4e123-29895600-jsc8z\" (UID: \"1b794761-73bc-42a3-9efc-ba2eb7278181\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f4e123-29895600-jsc8z"
Nov 03 20:00:00.418780 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:00.414185    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b2bf43f-4806-48b4-bd9b-1d6092b082e8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-79c47f-29895600-l6n9d\" (UID: \"7b2bf43f-4806-48b4-bd9b-1d6092b082e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-79c47f-29895600-l6n9d"
Nov 03 20:00:00.418780 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:00.414255    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xxdj\" (UniqueName: \"kubernetes.io/projected/a0c0fc1f-186d-4c54-bd71-d36671cbb718-kube-api-access-9xxdj\") pod \"maas-api-key-cleanup-e2e-worker-w2-8d4127-29895600-fh6sm\" (UID: \"a0c0fc1f-186d-4c54-bd71-d36671cbb718\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d4127-29895600-fh6sm"
Nov 03 20:00:00.418780 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:00.414299    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-528q5\" (UniqueName: \"kubernetes.io/projected/9fe29ef3-6905-4213-bf69-63105ba9bc29-kube-api-access-528q5\") pod \"maas-api-key-cleanup-e2e-worker-w0-82dbbf-29895600-9pjt9\" (UID: \"9fe29ef3-6905-4213-bf69-63105ba9bc29\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-82dbbf-29895600-9pjt9"
Nov 03 20:00:00.418780 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:00.414387    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgjj8\" (UniqueName: \"kubernetes.io/projected/5769966d-308a-44a9-928a-984d95d95820-kube-api-access-lgjj8\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-9f3188-29895600-wtngr\" (UID: \"5769966d-308a-44a9-928a-984d95d95820\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9f3188-29895600-wtngr"
Nov 03 20:00:00.418780 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:00.414425    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tnbd\" (UniqueName: \"kubernetes.io/projected/7b2bf43f-4806-48b4-bd9b-1d6092b082e8-kube-api-access-6tnbd\") pod \"maas-api-key-cleanup-e2e-inf-b-79c47f-29895600-l6n9d\" (UID: \"7b2bf43f-4806-48b4-bd9b-1d6092b082e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-79c47f-29895600-l6n9d"
Nov 03 20:00:00.418780 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:00.414491    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/951f6658-11ce-4b14-b615-31aacfd6b26e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-102379-29895600-l5pw6\" (UID: \"951f6658-11ce-4b14-b615-31aacfd6b26e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-102379-29895600-l5pw6"
Nov 03 20:00:00.418780 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:00.414588    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0c0fc1f-186d-4c54-bd71-d36671cbb718-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-8d4127-29895600-fh6sm\" (UID: \"a0c0fc1f-186d-4c54-bd71-d36671cbb718\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d4127-29895600-fh6sm"
Nov 03 20:00:00.418780 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:00.414624    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5769966d-308a-44a9-928a-984d95d95820-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-9f3188-29895600-wtngr\" (UID: \"5769966d-308a-44a9-928a-984d95d95820\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9f3188-29895600-wtngr"
Nov 03 20:00:00.418780 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:00.414725    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/957d5c20-d9b0-4d7e-a8df-544c55a1b456-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-3cee6a-29895600-rl2vr\" (UID: \"957d5c20-d9b0-4d7e-a8df-544c55a1b456\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3cee6a-29895600-rl2vr"
Nov 03 20:00:00.418780 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:00.414787    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fe29ef3-6905-4213-bf69-63105ba9bc29-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-82dbbf-29895600-9pjt9\" (UID: \"9fe29ef3-6905-4213-bf69-63105ba9bc29\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-82dbbf-29895600-9pjt9"
Nov 03 20:00:00.418780 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:00.414886    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5b6vz\" (UniqueName: \"kubernetes.io/projected/951f6658-11ce-4b14-b615-31aacfd6b26e-kube-api-access-5b6vz\") pod \"maas-api-key-cleanup-e2e-worker-w6-102379-29895600-l5pw6\" (UID: \"951f6658-11ce-4b14-b615-31aacfd6b26e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-102379-29895600-l5pw6"
Nov 03 20:00:00.439709 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:00.425319    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b2bf43f-4806-48b4-bd9b-1d6092b082e8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-79c47f-29895600-l6n9d\" (UID: \"7b2bf43f-4806-48b4-bd9b-1d6092b082e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-79c47f-29895600-l6n9d"
Nov 03 20:00:00.452227 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:00.441881    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/951f6658-11ce-4b14-b615-31aacfd6b26e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-102379-29895600-l5pw6\" (UID: \"951f6658-11ce-4b14-b615-31aacfd6b26e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-102379-29895600-l5pw6"
Nov 03 20:00:00.452227 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:00.442562    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0c0fc1f-186d-4c54-bd71-d36671cbb718-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-8d4127-29895600-fh6sm\" (UID: \"a0c0fc1f-186d-4c54-bd71-d36671cbb718\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d4127-29895600-fh6sm"
Nov 03 20:00:00.452227 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:00.443320    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64k5c\" (UniqueName: \"kubernetes.io/projected/957d5c20-d9b0-4d7e-a8df-544c55a1b456-kube-api-access-64k5c\") pod \"maas-api-key-cleanup-e2e-inf-a-3cee6a-29895600-rl2vr\" (UID: \"957d5c20-d9b0-4d7e-a8df-544c55a1b456\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3cee6a-29895600-rl2vr"
Nov 03 20:00:00.452227 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:00.445541    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/957d5c20-d9b0-4d7e-a8df-544c55a1b456-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-3cee6a-29895600-rl2vr\" (UID: \"957d5c20-d9b0-4d7e-a8df-544c55a1b456\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3cee6a-29895600-rl2vr"
Nov 03 20:00:00.452227 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:00.446224    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fe29ef3-6905-4213-bf69-63105ba9bc29-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-82dbbf-29895600-9pjt9\" (UID: \"9fe29ef3-6905-4213-bf69-63105ba9bc29\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-82dbbf-29895600-9pjt9"
Nov 03 20:00:00.452227 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:00.447992    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tnbd\" (UniqueName: \"kubernetes.io/projected/7b2bf43f-4806-48b4-bd9b-1d6092b082e8-kube-api-access-6tnbd\") pod \"maas-api-key-cleanup-e2e-inf-b-79c47f-29895600-l6n9d\" (UID: \"7b2bf43f-4806-48b4-bd9b-1d6092b082e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-79c47f-29895600-l6n9d"
Nov 03 20:00:00.452227 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:00.451622    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64k5c\" (UniqueName: \"kubernetes.io/projected/957d5c20-d9b0-4d7e-a8df-544c55a1b456-kube-api-access-64k5c\") pod \"maas-api-key-cleanup-e2e-inf-a-3cee6a-29895600-rl2vr\" (UID: \"957d5c20-d9b0-4d7e-a8df-544c55a1b456\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3cee6a-29895600-rl2vr"
Nov 03 20:00:00.465781 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:00.457371    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5769966d-308a-44a9-928a-984d95d95820-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-9f3188-29895600-wtngr\" (UID: \"5769966d-308a-44a9-928a-984d95d95820\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9f3188-29895600-wtngr"
Nov 03 20:00:00.465781 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:00.460278    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5b6vz\" (UniqueName: \"kubernetes.io/projected/951f6658-11ce-4b14-b615-31aacfd6b26e-kube-api-access-5b6vz\") pod \"maas-api-key-cleanup-e2e-worker-w6-102379-29895600-l5pw6\" (UID: \"951f6658-11ce-4b14-b615-31aacfd6b26e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-102379-29895600-l5pw6"
Nov 03 20:00:00.465781 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:00.460742    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-528q5\" (UniqueName: \"kubernetes.io/projected/9fe29ef3-6905-4213-bf69-63105ba9bc29-kube-api-access-528q5\") pod \"maas-api-key-cleanup-e2e-worker-w0-82dbbf-29895600-9pjt9\" (UID: \"9fe29ef3-6905-4213-bf69-63105ba9bc29\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-82dbbf-29895600-9pjt9"
Nov 03 20:00:00.465781 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:00.462521    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xxdj\" (UniqueName: \"kubernetes.io/projected/a0c0fc1f-186d-4c54-bd71-d36671cbb718-kube-api-access-9xxdj\") pod \"maas-api-key-cleanup-e2e-worker-w2-8d4127-29895600-fh6sm\" (UID: \"a0c0fc1f-186d-4c54-bd71-d36671cbb718\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d4127-29895600-fh6sm"
Nov 03 20:00:00.465781 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:00.463516    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgjj8\" (UniqueName: \"kubernetes.io/projected/5769966d-308a-44a9-928a-984d95d95820-kube-api-access-lgjj8\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-9f3188-29895600-wtngr\" (UID: \"5769966d-308a-44a9-928a-984d95d95820\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9f3188-29895600-wtngr"
Nov 03 20:00:00.469944 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:00.469496    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f4e123-29895600-jsc8z"
Nov 03 20:00:00.523322 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:00.523286    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d4127-29895600-fh6sm"
Nov 03 20:00:00.536790 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:00.536759    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3cee6a-29895600-rl2vr"
Nov 03 20:00:00.563027 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:00.562993    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-82dbbf-29895600-9pjt9"
Nov 03 20:00:00.574707 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:00.574652    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9f3188-29895600-wtngr"
Nov 03 20:00:00.585972 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:00.585924    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-102379-29895600-l5pw6"
Nov 03 20:00:00.595815 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:00.595465    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-79c47f-29895600-l6n9d"
Nov 03 20:00:01.122895 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:01.122868    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-82dbbf-29895600-9pjt9"]
Nov 03 20:00:01.128780 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:01.128751    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f4e123-29895600-jsc8z"]
Nov 03 20:00:01.137050 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:01.137021    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d4127-29895600-fh6sm"]
Nov 03 20:00:01.137249 ip-10-0-134-222 kubenswrapper[4921]: W1103 20:00:01.133946    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fe29ef3_6905_4213_bf69_63105ba9bc29.slice/crio-d22d88372e44aa25feb89aa5544421412440dd2e308fbec773a9e81b3f3d29d0 WatchSource:0}: Error finding container d22d88372e44aa25feb89aa5544421412440dd2e308fbec773a9e81b3f3d29d0: Status 404 returned error can't find the container with id d22d88372e44aa25feb89aa5544421412440dd2e308fbec773a9e81b3f3d29d0
Nov 03 20:00:01.147986 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:01.147953    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3cee6a-29895600-rl2vr"]
Nov 03 20:00:01.161495 ip-10-0-134-222 kubenswrapper[4921]: W1103 20:00:01.161447    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod957d5c20_d9b0_4d7e_a8df_544c55a1b456.slice/crio-9eb49e21d278e6fabf0388b7a13c8895b6bead17b50a22fc9e3694279300ba40 WatchSource:0}: Error finding container 9eb49e21d278e6fabf0388b7a13c8895b6bead17b50a22fc9e3694279300ba40: Status 404 returned error can't find the container with id 9eb49e21d278e6fabf0388b7a13c8895b6bead17b50a22fc9e3694279300ba40
Nov 03 20:00:01.199916 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:01.199824    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-102379-29895600-l5pw6"]
Nov 03 20:00:01.202412 ip-10-0-134-222 kubenswrapper[4921]: W1103 20:00:01.202382    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5769966d_308a_44a9_928a_984d95d95820.slice/crio-459927442c4fa6fc36365e1ff61f5cf07a5d9f1b6e5478199603b2678b27f320 WatchSource:0}: Error finding container 459927442c4fa6fc36365e1ff61f5cf07a5d9f1b6e5478199603b2678b27f320: Status 404 returned error can't find the container with id 459927442c4fa6fc36365e1ff61f5cf07a5d9f1b6e5478199603b2678b27f320
Nov 03 20:00:01.207980 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:01.207949    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9f3188-29895600-wtngr"]
Nov 03 20:00:01.217421 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:01.216166    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-79c47f-29895600-l6n9d"]
Nov 03 20:00:02.081752 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:02.080411    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-79c47f-29895600-l6n9d" event={"ID":"7b2bf43f-4806-48b4-bd9b-1d6092b082e8","Type":"ContainerStarted","Data":"a979c1768f1d3b824c2a725956e2fa30fe84d7d5ea83876a84f4e4ebbe6283d9"}
Nov 03 20:00:02.081752 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:02.080491    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-79c47f-29895600-l6n9d" event={"ID":"7b2bf43f-4806-48b4-bd9b-1d6092b082e8","Type":"ContainerStarted","Data":"94700a460795a947122a52e19be7a914bebec1daa3efc44538d172c6d53b00ef"}
Nov 03 20:00:02.085547 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:02.085508    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f4e123-29895600-jsc8z" event={"ID":"1b794761-73bc-42a3-9efc-ba2eb7278181","Type":"ContainerStarted","Data":"9041198615cda1dc05983f5bd588d1b0a460fcdbef84f389ab2437170e70d9c1"}
Nov 03 20:00:02.092659 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:02.092620    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f4e123-29895600-jsc8z" event={"ID":"1b794761-73bc-42a3-9efc-ba2eb7278181","Type":"ContainerStarted","Data":"71864d53d341acc234b82a5a7268c253dd4b65290127fec5e18f2ecfdaf2fbeb"}
Nov 03 20:00:02.112258 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:02.094800    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-82dbbf-29895600-9pjt9" event={"ID":"9fe29ef3-6905-4213-bf69-63105ba9bc29","Type":"ContainerStarted","Data":"29ed24453693cba3e94abdfca92d6a1405c22f9b7028d44c62eeb4d5dffd0a94"}
Nov 03 20:00:02.112258 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:02.094862    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-82dbbf-29895600-9pjt9" event={"ID":"9fe29ef3-6905-4213-bf69-63105ba9bc29","Type":"ContainerStarted","Data":"d22d88372e44aa25feb89aa5544421412440dd2e308fbec773a9e81b3f3d29d0"}
Nov 03 20:00:02.112258 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:02.109391    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d4127-29895600-fh6sm" event={"ID":"a0c0fc1f-186d-4c54-bd71-d36671cbb718","Type":"ContainerStarted","Data":"d9021e5a3d2b46ff7ba520ba185ab9fc654c13879a37cf84e3cf4c87ad5f8a9b"}
Nov 03 20:00:02.112258 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:02.109454    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d4127-29895600-fh6sm" event={"ID":"a0c0fc1f-186d-4c54-bd71-d36671cbb718","Type":"ContainerStarted","Data":"b1033d3691830e042ffb7fd9a3baa20757badd2664794549c99d69f87d9ac2ac"}
Nov 03 20:00:02.126516 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:02.113726    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9f3188-29895600-wtngr" event={"ID":"5769966d-308a-44a9-928a-984d95d95820","Type":"ContainerStarted","Data":"c67f7cba113dec0a4489d1fcffb4fc3b366945148e16f59e8d75ce78d8ed8ef0"}
Nov 03 20:00:02.126516 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:02.113791    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9f3188-29895600-wtngr" event={"ID":"5769966d-308a-44a9-928a-984d95d95820","Type":"ContainerStarted","Data":"459927442c4fa6fc36365e1ff61f5cf07a5d9f1b6e5478199603b2678b27f320"}
Nov 03 20:00:02.126516 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:02.117642    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3cee6a-29895600-rl2vr" event={"ID":"957d5c20-d9b0-4d7e-a8df-544c55a1b456","Type":"ContainerStarted","Data":"8d40371362a025a9d3fad5f9a01a77b34097a62484f554b503baca66044bebbe"}
Nov 03 20:00:02.126516 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:02.117756    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3cee6a-29895600-rl2vr" event={"ID":"957d5c20-d9b0-4d7e-a8df-544c55a1b456","Type":"ContainerStarted","Data":"9eb49e21d278e6fabf0388b7a13c8895b6bead17b50a22fc9e3694279300ba40"}
Nov 03 20:00:02.126516 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:02.123767    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-102379-29895600-l5pw6" event={"ID":"951f6658-11ce-4b14-b615-31aacfd6b26e","Type":"ContainerStarted","Data":"6c487ccd2c48890be7b49b3ffeb4062cd1c6bcff410f26108d25a9a10c501c5d"}
Nov 03 20:00:02.126516 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:02.123878    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-102379-29895600-l5pw6" event={"ID":"951f6658-11ce-4b14-b615-31aacfd6b26e","Type":"ContainerStarted","Data":"f7394575044a0dabed8a69783d2e15d01dac8fbf91d13e42ca3d1edaec687490"}
Nov 03 20:00:02.143114 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:02.143043    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-79c47f-29895600-l6n9d" podStartSLOduration=2.14301895 podStartE2EDuration="2.14301895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:02.098144856 +0000 UTC m=+1715.342110321" watchObservedRunningTime="2026-11-03 20:00:02.14301895 +0000 UTC m=+1715.386984417"
Nov 03 20:00:02.153765 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:02.150761    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f4e123-29895600-jsc8z" podStartSLOduration=2.150738075 podStartE2EDuration="2.150738075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:02.119900971 +0000 UTC m=+1715.363866437" watchObservedRunningTime="2026-11-03 20:00:02.150738075 +0000 UTC m=+1715.394703541"
Nov 03 20:00:02.166113 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:02.166053    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-82dbbf-29895600-9pjt9" podStartSLOduration=2.166032437 podStartE2EDuration="2.166032437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:02.13510068 +0000 UTC m=+1715.379066145" watchObservedRunningTime="2026-11-03 20:00:02.166032437 +0000 UTC m=+1715.409997905"
Nov 03 20:00:02.174734 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:02.174663    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3cee6a-29895600-rl2vr" podStartSLOduration=2.174647343 podStartE2EDuration="2.174647343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:02.158780885 +0000 UTC m=+1715.402746351" watchObservedRunningTime="2026-11-03 20:00:02.174647343 +0000 UTC m=+1715.418612835"
Nov 03 20:00:02.180722 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:02.180619    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9f3188-29895600-wtngr" podStartSLOduration=2.180605183 podStartE2EDuration="2.180605183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:02.171564393 +0000 UTC m=+1715.415529858" watchObservedRunningTime="2026-11-03 20:00:02.180605183 +0000 UTC m=+1715.424570659"
Nov 03 20:00:02.186918 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:02.186864    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d4127-29895600-fh6sm" podStartSLOduration=2.186849607 podStartE2EDuration="2.186849607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:02.18410023 +0000 UTC m=+1715.428065695" watchObservedRunningTime="2026-11-03 20:00:02.186849607 +0000 UTC m=+1715.430815093"
Nov 03 20:00:02.200550 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:02.200491    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-102379-29895600-l5pw6" podStartSLOduration=2.200477132 podStartE2EDuration="2.200477132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:02.196479617 +0000 UTC m=+1715.440445083" watchObservedRunningTime="2026-11-03 20:00:02.200477132 +0000 UTC m=+1715.444442598"
Nov 03 20:00:02.941880 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:00:02.941843    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 20:00:03.135664 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:03.135574    4921 generic.go:356] "Generic (PLEG): container finished" podID="5769966d-308a-44a9-928a-984d95d95820" containerID="c67f7cba113dec0a4489d1fcffb4fc3b366945148e16f59e8d75ce78d8ed8ef0" exitCode=60
Nov 03 20:00:03.135664 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:03.135644    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9f3188-29895600-wtngr" event={"ID":"5769966d-308a-44a9-928a-984d95d95820","Type":"ContainerDied","Data":"c67f7cba113dec0a4489d1fcffb4fc3b366945148e16f59e8d75ce78d8ed8ef0"}
Nov 03 20:00:03.136093 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:03.136072    4921 scope.go:122] "RemoveContainer" containerID="c67f7cba113dec0a4489d1fcffb4fc3b366945148e16f59e8d75ce78d8ed8ef0"
Nov 03 20:00:03.137654 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:03.137633    4921 generic.go:356] "Generic (PLEG): container finished" podID="957d5c20-d9b0-4d7e-a8df-544c55a1b456" containerID="8d40371362a025a9d3fad5f9a01a77b34097a62484f554b503baca66044bebbe" exitCode=60
Nov 03 20:00:03.137748 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:03.137709    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3cee6a-29895600-rl2vr" event={"ID":"957d5c20-d9b0-4d7e-a8df-544c55a1b456","Type":"ContainerDied","Data":"8d40371362a025a9d3fad5f9a01a77b34097a62484f554b503baca66044bebbe"}
Nov 03 20:00:03.138000 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:03.137982    4921 scope.go:122] "RemoveContainer" containerID="8d40371362a025a9d3fad5f9a01a77b34097a62484f554b503baca66044bebbe"
Nov 03 20:00:03.139543 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:03.139518    4921 generic.go:356] "Generic (PLEG): container finished" podID="951f6658-11ce-4b14-b615-31aacfd6b26e" containerID="6c487ccd2c48890be7b49b3ffeb4062cd1c6bcff410f26108d25a9a10c501c5d" exitCode=60
Nov 03 20:00:03.140255 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:03.139602    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-102379-29895600-l5pw6" event={"ID":"951f6658-11ce-4b14-b615-31aacfd6b26e","Type":"ContainerDied","Data":"6c487ccd2c48890be7b49b3ffeb4062cd1c6bcff410f26108d25a9a10c501c5d"}
Nov 03 20:00:03.140255 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:03.139837    4921 scope.go:122] "RemoveContainer" containerID="6c487ccd2c48890be7b49b3ffeb4062cd1c6bcff410f26108d25a9a10c501c5d"
Nov 03 20:00:03.142364 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:03.142338    4921 generic.go:356] "Generic (PLEG): container finished" podID="7b2bf43f-4806-48b4-bd9b-1d6092b082e8" containerID="a979c1768f1d3b824c2a725956e2fa30fe84d7d5ea83876a84f4e4ebbe6283d9" exitCode=60
Nov 03 20:00:03.142723 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:03.142623    4921 scope.go:122] "RemoveContainer" containerID="a979c1768f1d3b824c2a725956e2fa30fe84d7d5ea83876a84f4e4ebbe6283d9"
Nov 03 20:00:03.142723 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:03.142648    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-79c47f-29895600-l6n9d" event={"ID":"7b2bf43f-4806-48b4-bd9b-1d6092b082e8","Type":"ContainerDied","Data":"a979c1768f1d3b824c2a725956e2fa30fe84d7d5ea83876a84f4e4ebbe6283d9"}
Nov 03 20:00:03.144770 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:03.144752    4921 generic.go:356] "Generic (PLEG): container finished" podID="1b794761-73bc-42a3-9efc-ba2eb7278181" containerID="9041198615cda1dc05983f5bd588d1b0a460fcdbef84f389ab2437170e70d9c1" exitCode=60
Nov 03 20:00:03.144865 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:03.144775    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f4e123-29895600-jsc8z" event={"ID":"1b794761-73bc-42a3-9efc-ba2eb7278181","Type":"ContainerDied","Data":"9041198615cda1dc05983f5bd588d1b0a460fcdbef84f389ab2437170e70d9c1"}
Nov 03 20:00:03.145039 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:03.145022    4921 scope.go:122] "RemoveContainer" containerID="9041198615cda1dc05983f5bd588d1b0a460fcdbef84f389ab2437170e70d9c1"
Nov 03 20:00:03.146488 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:03.146468    4921 generic.go:356] "Generic (PLEG): container finished" podID="9fe29ef3-6905-4213-bf69-63105ba9bc29" containerID="29ed24453693cba3e94abdfca92d6a1405c22f9b7028d44c62eeb4d5dffd0a94" exitCode=60
Nov 03 20:00:03.146593 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:03.146551    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-82dbbf-29895600-9pjt9" event={"ID":"9fe29ef3-6905-4213-bf69-63105ba9bc29","Type":"ContainerDied","Data":"29ed24453693cba3e94abdfca92d6a1405c22f9b7028d44c62eeb4d5dffd0a94"}
Nov 03 20:00:03.146792 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:03.146724    4921 scope.go:122] "RemoveContainer" containerID="29ed24453693cba3e94abdfca92d6a1405c22f9b7028d44c62eeb4d5dffd0a94"
Nov 03 20:00:03.148436 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:03.148408    4921 generic.go:356] "Generic (PLEG): container finished" podID="a0c0fc1f-186d-4c54-bd71-d36671cbb718" containerID="d9021e5a3d2b46ff7ba520ba185ab9fc654c13879a37cf84e3cf4c87ad5f8a9b" exitCode=60
Nov 03 20:00:03.148525 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:03.148489    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d4127-29895600-fh6sm" event={"ID":"a0c0fc1f-186d-4c54-bd71-d36671cbb718","Type":"ContainerDied","Data":"d9021e5a3d2b46ff7ba520ba185ab9fc654c13879a37cf84e3cf4c87ad5f8a9b"}
Nov 03 20:00:03.148861 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:03.148714    4921 scope.go:122] "RemoveContainer" containerID="d9021e5a3d2b46ff7ba520ba185ab9fc654c13879a37cf84e3cf4c87ad5f8a9b"
Nov 03 20:00:04.161431 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:04.161326    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9f3188-29895600-wtngr" event={"ID":"5769966d-308a-44a9-928a-984d95d95820","Type":"ContainerStarted","Data":"65ec268d2bd14218cac83c56db4f90ca8dcd5442a263fea47fa8a10e4dc05d1e"}
Nov 03 20:00:04.163597 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:04.163570    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3cee6a-29895600-rl2vr" event={"ID":"957d5c20-d9b0-4d7e-a8df-544c55a1b456","Type":"ContainerStarted","Data":"2b38cb23de71a2a3bc18392ccc818372a3304089d4151a516dfcf507be0ac173"}
Nov 03 20:00:04.165658 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:04.165633    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-102379-29895600-l5pw6" event={"ID":"951f6658-11ce-4b14-b615-31aacfd6b26e","Type":"ContainerStarted","Data":"9990788ec1e229864063a866f348d4173d548e0a0007340eeac8ad813f61e50c"}
Nov 03 20:00:04.167720 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:04.167694    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-79c47f-29895600-l6n9d" event={"ID":"7b2bf43f-4806-48b4-bd9b-1d6092b082e8","Type":"ContainerStarted","Data":"8541e612f32594f233f382e3385c18c6b76d79896639a98b5537be845724f0da"}
Nov 03 20:00:04.170054 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:04.170028    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f4e123-29895600-jsc8z" event={"ID":"1b794761-73bc-42a3-9efc-ba2eb7278181","Type":"ContainerStarted","Data":"8d10bb0d53b78a122d334ebeda367a170b5e59becf1203c3cb831e13ddc39098"}
Nov 03 20:00:04.172204 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:04.172181    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-82dbbf-29895600-9pjt9" event={"ID":"9fe29ef3-6905-4213-bf69-63105ba9bc29","Type":"ContainerStarted","Data":"dbb90708520ccf7d721d1574047187a8dd6cd46edf800b77d2fed90caf9a5c2f"}
Nov 03 20:00:04.175075 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:04.175051    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d4127-29895600-fh6sm" event={"ID":"a0c0fc1f-186d-4c54-bd71-d36671cbb718","Type":"ContainerStarted","Data":"2ad367a95f1ac8a327557c75d5c6fe40101d39b1091a31835898a57d86a604ef"}
Nov 03 20:00:05.186811 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:05.186768    4921 generic.go:356] "Generic (PLEG): container finished" podID="957d5c20-d9b0-4d7e-a8df-544c55a1b456" containerID="2b38cb23de71a2a3bc18392ccc818372a3304089d4151a516dfcf507be0ac173" exitCode=60
Nov 03 20:00:05.187299 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:05.186862    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3cee6a-29895600-rl2vr" event={"ID":"957d5c20-d9b0-4d7e-a8df-544c55a1b456","Type":"ContainerDied","Data":"2b38cb23de71a2a3bc18392ccc818372a3304089d4151a516dfcf507be0ac173"}
Nov 03 20:00:05.187299 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:05.186964    4921 scope.go:122] "RemoveContainer" containerID="8d40371362a025a9d3fad5f9a01a77b34097a62484f554b503baca66044bebbe"
Nov 03 20:00:05.187299 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:05.187251    4921 scope.go:122] "RemoveContainer" containerID="2b38cb23de71a2a3bc18392ccc818372a3304089d4151a516dfcf507be0ac173"
Nov 03 20:00:05.187777 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:00:05.187753    4921 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-3cee6a-29895600-rl2vr_odh-ai-gateway-infra(957d5c20-d9b0-4d7e-a8df-544c55a1b456)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3cee6a-29895600-rl2vr" podUID="957d5c20-d9b0-4d7e-a8df-544c55a1b456"
Nov 03 20:00:05.189208 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:05.189189    4921 generic.go:356] "Generic (PLEG): container finished" podID="951f6658-11ce-4b14-b615-31aacfd6b26e" containerID="9990788ec1e229864063a866f348d4173d548e0a0007340eeac8ad813f61e50c" exitCode=60
Nov 03 20:00:05.189328 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:05.189279    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-102379-29895600-l5pw6" event={"ID":"951f6658-11ce-4b14-b615-31aacfd6b26e","Type":"ContainerDied","Data":"9990788ec1e229864063a866f348d4173d548e0a0007340eeac8ad813f61e50c"}
Nov 03 20:00:05.189597 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:05.189580    4921 scope.go:122] "RemoveContainer" containerID="9990788ec1e229864063a866f348d4173d548e0a0007340eeac8ad813f61e50c"
Nov 03 20:00:05.190097 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:00:05.190070    4921 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-102379-29895600-l5pw6_odh-ai-gateway-infra(951f6658-11ce-4b14-b615-31aacfd6b26e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-102379-29895600-l5pw6" podUID="951f6658-11ce-4b14-b615-31aacfd6b26e"
Nov 03 20:00:05.192514 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:05.192486    4921 generic.go:356] "Generic (PLEG): container finished" podID="7b2bf43f-4806-48b4-bd9b-1d6092b082e8" containerID="8541e612f32594f233f382e3385c18c6b76d79896639a98b5537be845724f0da" exitCode=60
Nov 03 20:00:05.192640 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:05.192569    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-79c47f-29895600-l6n9d" event={"ID":"7b2bf43f-4806-48b4-bd9b-1d6092b082e8","Type":"ContainerDied","Data":"8541e612f32594f233f382e3385c18c6b76d79896639a98b5537be845724f0da"}
Nov 03 20:00:05.192902 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:05.192881    4921 scope.go:122] "RemoveContainer" containerID="8541e612f32594f233f382e3385c18c6b76d79896639a98b5537be845724f0da"
Nov 03 20:00:05.193257 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:00:05.193237    4921 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-79c47f-29895600-l6n9d_odh-ai-gateway-infra(7b2bf43f-4806-48b4-bd9b-1d6092b082e8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-79c47f-29895600-l6n9d" podUID="7b2bf43f-4806-48b4-bd9b-1d6092b082e8"
Nov 03 20:00:05.194928 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:05.194904    4921 generic.go:356] "Generic (PLEG): container finished" podID="1b794761-73bc-42a3-9efc-ba2eb7278181" containerID="8d10bb0d53b78a122d334ebeda367a170b5e59becf1203c3cb831e13ddc39098" exitCode=60
Nov 03 20:00:05.195010 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:05.194986    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f4e123-29895600-jsc8z" event={"ID":"1b794761-73bc-42a3-9efc-ba2eb7278181","Type":"ContainerDied","Data":"8d10bb0d53b78a122d334ebeda367a170b5e59becf1203c3cb831e13ddc39098"}
Nov 03 20:00:05.195207 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:05.195187    4921 scope.go:122] "RemoveContainer" containerID="8d10bb0d53b78a122d334ebeda367a170b5e59becf1203c3cb831e13ddc39098"
Nov 03 20:00:05.195720 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:00:05.195514    4921 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-f4e123-29895600-jsc8z_odh-ai-gateway-infra(1b794761-73bc-42a3-9efc-ba2eb7278181)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f4e123-29895600-jsc8z" podUID="1b794761-73bc-42a3-9efc-ba2eb7278181"
Nov 03 20:00:05.197195 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:05.197172    4921 generic.go:356] "Generic (PLEG): container finished" podID="9fe29ef3-6905-4213-bf69-63105ba9bc29" containerID="dbb90708520ccf7d721d1574047187a8dd6cd46edf800b77d2fed90caf9a5c2f" exitCode=60
Nov 03 20:00:05.197300 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:05.197250    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-82dbbf-29895600-9pjt9" event={"ID":"9fe29ef3-6905-4213-bf69-63105ba9bc29","Type":"ContainerDied","Data":"dbb90708520ccf7d721d1574047187a8dd6cd46edf800b77d2fed90caf9a5c2f"}
Nov 03 20:00:05.197503 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:05.197482    4921 scope.go:122] "RemoveContainer" containerID="dbb90708520ccf7d721d1574047187a8dd6cd46edf800b77d2fed90caf9a5c2f"
Nov 03 20:00:05.197803 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:00:05.197784    4921 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-82dbbf-29895600-9pjt9_odh-ai-gateway-infra(9fe29ef3-6905-4213-bf69-63105ba9bc29)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-82dbbf-29895600-9pjt9" podUID="9fe29ef3-6905-4213-bf69-63105ba9bc29"
Nov 03 20:00:05.199419 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:05.199398    4921 generic.go:356] "Generic (PLEG): container finished" podID="a0c0fc1f-186d-4c54-bd71-d36671cbb718" containerID="2ad367a95f1ac8a327557c75d5c6fe40101d39b1091a31835898a57d86a604ef" exitCode=60
Nov 03 20:00:05.199528 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:05.199469    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d4127-29895600-fh6sm" event={"ID":"a0c0fc1f-186d-4c54-bd71-d36671cbb718","Type":"ContainerDied","Data":"2ad367a95f1ac8a327557c75d5c6fe40101d39b1091a31835898a57d86a604ef"}
Nov 03 20:00:05.199698 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:05.199684    4921 scope.go:122] "RemoveContainer" containerID="2ad367a95f1ac8a327557c75d5c6fe40101d39b1091a31835898a57d86a604ef"
Nov 03 20:00:05.199960 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:00:05.199936    4921 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-8d4127-29895600-fh6sm_odh-ai-gateway-infra(a0c0fc1f-186d-4c54-bd71-d36671cbb718)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d4127-29895600-fh6sm" podUID="a0c0fc1f-186d-4c54-bd71-d36671cbb718"
Nov 03 20:00:05.202527 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:05.202509    4921 generic.go:356] "Generic (PLEG): container finished" podID="5769966d-308a-44a9-928a-984d95d95820" containerID="65ec268d2bd14218cac83c56db4f90ca8dcd5442a263fea47fa8a10e4dc05d1e" exitCode=60
Nov 03 20:00:05.202618 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:05.202552    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9f3188-29895600-wtngr" event={"ID":"5769966d-308a-44a9-928a-984d95d95820","Type":"ContainerDied","Data":"65ec268d2bd14218cac83c56db4f90ca8dcd5442a263fea47fa8a10e4dc05d1e"}
Nov 03 20:00:05.202885 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:05.202863    4921 scope.go:122] "RemoveContainer" containerID="65ec268d2bd14218cac83c56db4f90ca8dcd5442a263fea47fa8a10e4dc05d1e"
Nov 03 20:00:05.203119 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:00:05.203100    4921 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-9f3188-29895600-wtngr_odh-ai-gateway-infra(5769966d-308a-44a9-928a-984d95d95820)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9f3188-29895600-wtngr" podUID="5769966d-308a-44a9-928a-984d95d95820"
Nov 03 20:00:05.204528 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:05.204514    4921 scope.go:122] "RemoveContainer" containerID="6c487ccd2c48890be7b49b3ffeb4062cd1c6bcff410f26108d25a9a10c501c5d"
Nov 03 20:00:05.221172 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:05.221154    4921 scope.go:122] "RemoveContainer" containerID="a979c1768f1d3b824c2a725956e2fa30fe84d7d5ea83876a84f4e4ebbe6283d9"
Nov 03 20:00:05.240155 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:05.240136    4921 scope.go:122] "RemoveContainer" containerID="9041198615cda1dc05983f5bd588d1b0a460fcdbef84f389ab2437170e70d9c1"
Nov 03 20:00:05.256009 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:05.255986    4921 scope.go:122] "RemoveContainer" containerID="29ed24453693cba3e94abdfca92d6a1405c22f9b7028d44c62eeb4d5dffd0a94"
Nov 03 20:00:05.272624 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:05.272601    4921 scope.go:122] "RemoveContainer" containerID="d9021e5a3d2b46ff7ba520ba185ab9fc654c13879a37cf84e3cf4c87ad5f8a9b"
Nov 03 20:00:05.289592 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:05.289563    4921 scope.go:122] "RemoveContainer" containerID="c67f7cba113dec0a4489d1fcffb4fc3b366945148e16f59e8d75ce78d8ed8ef0"
Nov 03 20:00:06.214024 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:06.213991    4921 scope.go:122] "RemoveContainer" containerID="65ec268d2bd14218cac83c56db4f90ca8dcd5442a263fea47fa8a10e4dc05d1e"
Nov 03 20:00:06.214480 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:00:06.214401    4921 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-9f3188-29895600-wtngr_odh-ai-gateway-infra(5769966d-308a-44a9-928a-984d95d95820)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9f3188-29895600-wtngr" podUID="5769966d-308a-44a9-928a-984d95d95820"
Nov 03 20:00:06.216160 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:06.216135    4921 scope.go:122] "RemoveContainer" containerID="2b38cb23de71a2a3bc18392ccc818372a3304089d4151a516dfcf507be0ac173"
Nov 03 20:00:06.216534 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:00:06.216505    4921 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-3cee6a-29895600-rl2vr_odh-ai-gateway-infra(957d5c20-d9b0-4d7e-a8df-544c55a1b456)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3cee6a-29895600-rl2vr" podUID="957d5c20-d9b0-4d7e-a8df-544c55a1b456"
Nov 03 20:00:06.218038 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:06.218018    4921 scope.go:122] "RemoveContainer" containerID="9990788ec1e229864063a866f348d4173d548e0a0007340eeac8ad813f61e50c"
Nov 03 20:00:06.218367 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:00:06.218341    4921 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-102379-29895600-l5pw6_odh-ai-gateway-infra(951f6658-11ce-4b14-b615-31aacfd6b26e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-102379-29895600-l5pw6" podUID="951f6658-11ce-4b14-b615-31aacfd6b26e"
Nov 03 20:00:06.220062 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:06.220040    4921 scope.go:122] "RemoveContainer" containerID="8541e612f32594f233f382e3385c18c6b76d79896639a98b5537be845724f0da"
Nov 03 20:00:06.220379 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:00:06.220354    4921 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-79c47f-29895600-l6n9d_odh-ai-gateway-infra(7b2bf43f-4806-48b4-bd9b-1d6092b082e8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-79c47f-29895600-l6n9d" podUID="7b2bf43f-4806-48b4-bd9b-1d6092b082e8"
Nov 03 20:00:06.221714 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:06.221696    4921 scope.go:122] "RemoveContainer" containerID="8d10bb0d53b78a122d334ebeda367a170b5e59becf1203c3cb831e13ddc39098"
Nov 03 20:00:06.222014 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:00:06.221990    4921 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-f4e123-29895600-jsc8z_odh-ai-gateway-infra(1b794761-73bc-42a3-9efc-ba2eb7278181)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f4e123-29895600-jsc8z" podUID="1b794761-73bc-42a3-9efc-ba2eb7278181"
Nov 03 20:00:06.223525 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:06.223506    4921 scope.go:122] "RemoveContainer" containerID="dbb90708520ccf7d721d1574047187a8dd6cd46edf800b77d2fed90caf9a5c2f"
Nov 03 20:00:06.223830 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:00:06.223810    4921 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-82dbbf-29895600-9pjt9_odh-ai-gateway-infra(9fe29ef3-6905-4213-bf69-63105ba9bc29)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-82dbbf-29895600-9pjt9" podUID="9fe29ef3-6905-4213-bf69-63105ba9bc29"
Nov 03 20:00:06.225122 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:06.225104    4921 scope.go:122] "RemoveContainer" containerID="2ad367a95f1ac8a327557c75d5c6fe40101d39b1091a31835898a57d86a604ef"
Nov 03 20:00:06.225348 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:00:06.225331    4921 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-8d4127-29895600-fh6sm_odh-ai-gateway-infra(a0c0fc1f-186d-4c54-bd71-d36671cbb718)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d4127-29895600-fh6sm" podUID="a0c0fc1f-186d-4c54-bd71-d36671cbb718"
Nov 03 20:00:14.881733 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:14.881695    4921 scope.go:122] "RemoveContainer" containerID="dbb90708520ccf7d721d1574047187a8dd6cd46edf800b77d2fed90caf9a5c2f"
Nov 03 20:00:14.882209 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:14.881858    4921 scope.go:122] "RemoveContainer" containerID="8d10bb0d53b78a122d334ebeda367a170b5e59becf1203c3cb831e13ddc39098"
Nov 03 20:00:15.883966 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:15.881558    4921 scope.go:122] "RemoveContainer" containerID="2ad367a95f1ac8a327557c75d5c6fe40101d39b1091a31835898a57d86a604ef"
Nov 03 20:00:15.883966 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:15.881755    4921 scope.go:122] "RemoveContainer" containerID="2b38cb23de71a2a3bc18392ccc818372a3304089d4151a516dfcf507be0ac173"
Nov 03 20:00:16.386725 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:16.386608    4921 generic.go:356] "Generic (PLEG): container finished" podID="1b794761-73bc-42a3-9efc-ba2eb7278181" containerID="6135504854ad0ba64f3f886661e22e905266bf03f04ac4e5dadbc87310f7f216" exitCode=60
Nov 03 20:00:16.386898 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:16.386715    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f4e123-29895600-jsc8z" event={"ID":"1b794761-73bc-42a3-9efc-ba2eb7278181","Type":"ContainerDied","Data":"6135504854ad0ba64f3f886661e22e905266bf03f04ac4e5dadbc87310f7f216"}
Nov 03 20:00:16.386898 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:16.386798    4921 scope.go:122] "RemoveContainer" containerID="8d10bb0d53b78a122d334ebeda367a170b5e59becf1203c3cb831e13ddc39098"
Nov 03 20:00:16.387226 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:16.387206    4921 scope.go:122] "RemoveContainer" containerID="6135504854ad0ba64f3f886661e22e905266bf03f04ac4e5dadbc87310f7f216"
Nov 03 20:00:16.387605 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:00:16.387577    4921 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-f4e123-29895600-jsc8z_odh-ai-gateway-infra(1b794761-73bc-42a3-9efc-ba2eb7278181)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f4e123-29895600-jsc8z" podUID="1b794761-73bc-42a3-9efc-ba2eb7278181"
Nov 03 20:00:16.388992 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:16.388962    4921 generic.go:356] "Generic (PLEG): container finished" podID="9fe29ef3-6905-4213-bf69-63105ba9bc29" containerID="d82a7accdee2ff8ae44cdd365bd7b0a84680bdc76396d9a9ca57fbecb7f5804b" exitCode=60
Nov 03 20:00:16.389095 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:16.389010    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-82dbbf-29895600-9pjt9" event={"ID":"9fe29ef3-6905-4213-bf69-63105ba9bc29","Type":"ContainerDied","Data":"d82a7accdee2ff8ae44cdd365bd7b0a84680bdc76396d9a9ca57fbecb7f5804b"}
Nov 03 20:00:16.389369 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:16.389355    4921 scope.go:122] "RemoveContainer" containerID="d82a7accdee2ff8ae44cdd365bd7b0a84680bdc76396d9a9ca57fbecb7f5804b"
Nov 03 20:00:16.389752 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:00:16.389663    4921 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-82dbbf-29895600-9pjt9_odh-ai-gateway-infra(9fe29ef3-6905-4213-bf69-63105ba9bc29)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-82dbbf-29895600-9pjt9" podUID="9fe29ef3-6905-4213-bf69-63105ba9bc29"
Nov 03 20:00:16.403641 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:16.403616    4921 scope.go:122] "RemoveContainer" containerID="dbb90708520ccf7d721d1574047187a8dd6cd46edf800b77d2fed90caf9a5c2f"
Nov 03 20:00:16.887593 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:16.887555    4921 scope.go:122] "RemoveContainer" containerID="65ec268d2bd14218cac83c56db4f90ca8dcd5442a263fea47fa8a10e4dc05d1e"
Nov 03 20:00:16.888099 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:16.887611    4921 scope.go:122] "RemoveContainer" containerID="9990788ec1e229864063a866f348d4173d548e0a0007340eeac8ad813f61e50c"
Nov 03 20:00:17.440287 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:17.437278    4921 generic.go:356] "Generic (PLEG): container finished" podID="957d5c20-d9b0-4d7e-a8df-544c55a1b456" containerID="1f8ce1c26fcf51717c6f3cad8dff103b626de089757065f7d83f66941c5a02e4" exitCode=60
Nov 03 20:00:17.440287 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:17.437349    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3cee6a-29895600-rl2vr" event={"ID":"957d5c20-d9b0-4d7e-a8df-544c55a1b456","Type":"ContainerDied","Data":"1f8ce1c26fcf51717c6f3cad8dff103b626de089757065f7d83f66941c5a02e4"}
Nov 03 20:00:17.440287 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:17.437425    4921 scope.go:122] "RemoveContainer" containerID="2b38cb23de71a2a3bc18392ccc818372a3304089d4151a516dfcf507be0ac173"
Nov 03 20:00:17.440287 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:17.437947    4921 scope.go:122] "RemoveContainer" containerID="1f8ce1c26fcf51717c6f3cad8dff103b626de089757065f7d83f66941c5a02e4"
Nov 03 20:00:17.440287 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:00:17.438692    4921 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-3cee6a-29895600-rl2vr_odh-ai-gateway-infra(957d5c20-d9b0-4d7e-a8df-544c55a1b456)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3cee6a-29895600-rl2vr" podUID="957d5c20-d9b0-4d7e-a8df-544c55a1b456"
Nov 03 20:00:17.440287 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:17.439207    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f4e123-29895600-jsc8z"]
Nov 03 20:00:17.451821 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:17.451795    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-82dbbf-29895600-9pjt9"]
Nov 03 20:00:17.463458 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:17.463426    4921 generic.go:356] "Generic (PLEG): container finished" podID="a0c0fc1f-186d-4c54-bd71-d36671cbb718" containerID="f2b5abcffc23800b52d138d61d8bff1397088931e37ee85c40a28d22c2d298b8" exitCode=60
Nov 03 20:00:17.463645 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:17.463495    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d4127-29895600-fh6sm" event={"ID":"a0c0fc1f-186d-4c54-bd71-d36671cbb718","Type":"ContainerDied","Data":"f2b5abcffc23800b52d138d61d8bff1397088931e37ee85c40a28d22c2d298b8"}
Nov 03 20:00:17.463645 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:17.463559    4921 scope.go:122] "RemoveContainer" containerID="2ad367a95f1ac8a327557c75d5c6fe40101d39b1091a31835898a57d86a604ef"
Nov 03 20:00:17.464032 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:17.464016    4921 scope.go:122] "RemoveContainer" containerID="f2b5abcffc23800b52d138d61d8bff1397088931e37ee85c40a28d22c2d298b8"
Nov 03 20:00:17.464421 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:00:17.464400    4921 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-8d4127-29895600-fh6sm_odh-ai-gateway-infra(a0c0fc1f-186d-4c54-bd71-d36671cbb718)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d4127-29895600-fh6sm" podUID="a0c0fc1f-186d-4c54-bd71-d36671cbb718"
Nov 03 20:00:17.675074 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:17.675010    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f4e123-29895600-jsc8z"
Nov 03 20:00:17.707598 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:17.707565    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d12868-29895600-bqplj"]
Nov 03 20:00:17.708592 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:17.708569    4921 cpu_manager.go:414] "containerMap: removing container" podUID="1b794761-73bc-42a3-9efc-ba2eb7278181" containerName="cleanup"
Nov 03 20:00:17.708592 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:17.708589    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b794761-73bc-42a3-9efc-ba2eb7278181" containerName="cleanup"
Nov 03 20:00:17.708830 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:17.708602    4921 cpu_manager.go:414] "containerMap: removing container" podUID="1b794761-73bc-42a3-9efc-ba2eb7278181" containerName="cleanup"
Nov 03 20:00:17.708830 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:17.708608    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b794761-73bc-42a3-9efc-ba2eb7278181" containerName="cleanup"
Nov 03 20:00:17.708830 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:17.708623    4921 cpu_manager.go:414] "containerMap: removing container" podUID="1b794761-73bc-42a3-9efc-ba2eb7278181" containerName="cleanup"
Nov 03 20:00:17.708830 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:17.708629    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b794761-73bc-42a3-9efc-ba2eb7278181" containerName="cleanup"
Nov 03 20:00:17.708830 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:17.708791    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b794761-73bc-42a3-9efc-ba2eb7278181" containerName="cleanup"
Nov 03 20:00:17.708830 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:17.708803    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b794761-73bc-42a3-9efc-ba2eb7278181" containerName="cleanup"
Nov 03 20:00:17.709284 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:17.709262    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-82dbbf-29895600-9pjt9"
Nov 03 20:00:17.714389 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:17.714368    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d12868-29895600-bqplj"
Nov 03 20:00:17.721388 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:17.721352    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b794761-73bc-42a3-9efc-ba2eb7278181-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b794761-73bc-42a3-9efc-ba2eb7278181-maas-api-service-ca\") pod \"1b794761-73bc-42a3-9efc-ba2eb7278181\" (UID: \"1b794761-73bc-42a3-9efc-ba2eb7278181\") "
Nov 03 20:00:17.721518 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:17.721452    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b794761-73bc-42a3-9efc-ba2eb7278181-kube-api-access-xfk6t\" (UniqueName: \"kubernetes.io/projected/1b794761-73bc-42a3-9efc-ba2eb7278181-kube-api-access-xfk6t\") pod \"1b794761-73bc-42a3-9efc-ba2eb7278181\" (UID: \"1b794761-73bc-42a3-9efc-ba2eb7278181\") "
Nov 03 20:00:17.723040 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:17.721852    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b794761-73bc-42a3-9efc-ba2eb7278181-maas-api-service-ca" pod "1b794761-73bc-42a3-9efc-ba2eb7278181" (UID: "1b794761-73bc-42a3-9efc-ba2eb7278181"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:00:17.723040 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:17.722288    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b794761-73bc-42a3-9efc-ba2eb7278181-maas-api-service-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:17.724064 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:17.724043    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d12868-29895600-bqplj"]
Nov 03 20:00:17.738512 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:17.737254    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b794761-73bc-42a3-9efc-ba2eb7278181-kube-api-access-xfk6t" pod "1b794761-73bc-42a3-9efc-ba2eb7278181" (UID: "1b794761-73bc-42a3-9efc-ba2eb7278181"). InnerVolumeSpecName "kube-api-access-xfk6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:00:17.742319 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:17.741096    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-332694-29895600-s686j"]
Nov 03 20:00:17.742319 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:17.741976    4921 cpu_manager.go:414] "containerMap: removing container" podUID="9fe29ef3-6905-4213-bf69-63105ba9bc29" containerName="cleanup"
Nov 03 20:00:17.742319 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:17.741991    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fe29ef3-6905-4213-bf69-63105ba9bc29" containerName="cleanup"
Nov 03 20:00:17.742319 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:17.742023    4921 cpu_manager.go:414] "containerMap: removing container" podUID="9fe29ef3-6905-4213-bf69-63105ba9bc29" containerName="cleanup"
Nov 03 20:00:17.742319 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:17.742028    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fe29ef3-6905-4213-bf69-63105ba9bc29" containerName="cleanup"
Nov 03 20:00:17.742663 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:17.742403    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b794761-73bc-42a3-9efc-ba2eb7278181" containerName="cleanup"
Nov 03 20:00:17.742663 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:17.742430    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fe29ef3-6905-4213-bf69-63105ba9bc29" containerName="cleanup"
Nov 03 20:00:17.742663 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:17.742449    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fe29ef3-6905-4213-bf69-63105ba9bc29" containerName="cleanup"
Nov 03 20:00:17.747873 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:17.747479    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-332694-29895600-s686j"
Nov 03 20:00:17.759218 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:17.759183    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-332694-29895600-s686j"]
Nov 03 20:00:17.822977 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:17.822940    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fe29ef3-6905-4213-bf69-63105ba9bc29-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fe29ef3-6905-4213-bf69-63105ba9bc29-maas-api-service-ca\") pod \"9fe29ef3-6905-4213-bf69-63105ba9bc29\" (UID: \"9fe29ef3-6905-4213-bf69-63105ba9bc29\") "
Nov 03 20:00:17.823293 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:17.823277    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fe29ef3-6905-4213-bf69-63105ba9bc29-kube-api-access-528q5\" (UniqueName: \"kubernetes.io/projected/9fe29ef3-6905-4213-bf69-63105ba9bc29-kube-api-access-528q5\") pod \"9fe29ef3-6905-4213-bf69-63105ba9bc29\" (UID: \"9fe29ef3-6905-4213-bf69-63105ba9bc29\") "
Nov 03 20:00:17.823576 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:17.823553    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhnwh\" (UniqueName: \"kubernetes.io/projected/40c57d6c-5b96-4713-8c12-c41c5f435a9b-kube-api-access-nhnwh\") pod \"maas-api-key-cleanup-e2e-inf-b-332694-29895600-s686j\" (UID: \"40c57d6c-5b96-4713-8c12-c41c5f435a9b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-332694-29895600-s686j"
Nov 03 20:00:17.823725 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:17.823644    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7db24b6-7319-492c-bd17-05eab3aab9da-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-d12868-29895600-bqplj\" (UID: \"a7db24b6-7319-492c-bd17-05eab3aab9da\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d12868-29895600-bqplj"
Nov 03 20:00:17.823725 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:17.823710    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40c57d6c-5b96-4713-8c12-c41c5f435a9b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-332694-29895600-s686j\" (UID: \"40c57d6c-5b96-4713-8c12-c41c5f435a9b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-332694-29895600-s686j"
Nov 03 20:00:17.824071 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:17.824047    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmvn9\" (UniqueName: \"kubernetes.io/projected/a7db24b6-7319-492c-bd17-05eab3aab9da-kube-api-access-nmvn9\") pod \"maas-api-key-cleanup-e2e-inf-b-d12868-29895600-bqplj\" (UID: \"a7db24b6-7319-492c-bd17-05eab3aab9da\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d12868-29895600-bqplj"
Nov 03 20:00:17.824156 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:17.824089    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fe29ef3-6905-4213-bf69-63105ba9bc29-maas-api-service-ca" pod "9fe29ef3-6905-4213-bf69-63105ba9bc29" (UID: "9fe29ef3-6905-4213-bf69-63105ba9bc29"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:00:17.824258 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:17.824242    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fe29ef3-6905-4213-bf69-63105ba9bc29-maas-api-service-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:17.824311 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:17.824267    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfk6t\" (UniqueName: \"kubernetes.io/projected/1b794761-73bc-42a3-9efc-ba2eb7278181-kube-api-access-xfk6t\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:17.825866 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:17.825847    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fe29ef3-6905-4213-bf69-63105ba9bc29-kube-api-access-528q5" pod "9fe29ef3-6905-4213-bf69-63105ba9bc29" (UID: "9fe29ef3-6905-4213-bf69-63105ba9bc29"). InnerVolumeSpecName "kube-api-access-528q5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:00:17.881855 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:17.881468    4921 scope.go:122] "RemoveContainer" containerID="8541e612f32594f233f382e3385c18c6b76d79896639a98b5537be845724f0da"
Nov 03 20:00:17.926520 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:17.926484    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmvn9\" (UniqueName: \"kubernetes.io/projected/a7db24b6-7319-492c-bd17-05eab3aab9da-kube-api-access-nmvn9\") pod \"maas-api-key-cleanup-e2e-inf-b-d12868-29895600-bqplj\" (UID: \"a7db24b6-7319-492c-bd17-05eab3aab9da\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d12868-29895600-bqplj"
Nov 03 20:00:17.926977 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:17.926632    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhnwh\" (UniqueName: \"kubernetes.io/projected/40c57d6c-5b96-4713-8c12-c41c5f435a9b-kube-api-access-nhnwh\") pod \"maas-api-key-cleanup-e2e-inf-b-332694-29895600-s686j\" (UID: \"40c57d6c-5b96-4713-8c12-c41c5f435a9b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-332694-29895600-s686j"
Nov 03 20:00:17.926977 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:17.926784    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7db24b6-7319-492c-bd17-05eab3aab9da-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-d12868-29895600-bqplj\" (UID: \"a7db24b6-7319-492c-bd17-05eab3aab9da\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d12868-29895600-bqplj"
Nov 03 20:00:17.926977 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:17.926848    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40c57d6c-5b96-4713-8c12-c41c5f435a9b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-332694-29895600-s686j\" (UID: \"40c57d6c-5b96-4713-8c12-c41c5f435a9b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-332694-29895600-s686j"
Nov 03 20:00:17.927131 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:17.927106    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-528q5\" (UniqueName: \"kubernetes.io/projected/9fe29ef3-6905-4213-bf69-63105ba9bc29-kube-api-access-528q5\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:17.927462 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:17.927441    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40c57d6c-5b96-4713-8c12-c41c5f435a9b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-332694-29895600-s686j\" (UID: \"40c57d6c-5b96-4713-8c12-c41c5f435a9b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-332694-29895600-s686j"
Nov 03 20:00:17.927566 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:17.927546    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7db24b6-7319-492c-bd17-05eab3aab9da-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-d12868-29895600-bqplj\" (UID: \"a7db24b6-7319-492c-bd17-05eab3aab9da\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d12868-29895600-bqplj"
Nov 03 20:00:17.936967 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:17.936943    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmvn9\" (UniqueName: \"kubernetes.io/projected/a7db24b6-7319-492c-bd17-05eab3aab9da-kube-api-access-nmvn9\") pod \"maas-api-key-cleanup-e2e-inf-b-d12868-29895600-bqplj\" (UID: \"a7db24b6-7319-492c-bd17-05eab3aab9da\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d12868-29895600-bqplj"
Nov 03 20:00:17.937064 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:17.936985    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhnwh\" (UniqueName: \"kubernetes.io/projected/40c57d6c-5b96-4713-8c12-c41c5f435a9b-kube-api-access-nhnwh\") pod \"maas-api-key-cleanup-e2e-inf-b-332694-29895600-s686j\" (UID: \"40c57d6c-5b96-4713-8c12-c41c5f435a9b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-332694-29895600-s686j"
Nov 03 20:00:18.056305 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:18.056269    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d12868-29895600-bqplj"
Nov 03 20:00:18.075688 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:18.072717    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-332694-29895600-s686j"
Nov 03 20:00:18.378075 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:18.375780    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d12868-29895600-bqplj"]
Nov 03 20:00:18.402886 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:18.402828    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-332694-29895600-s686j"]
Nov 03 20:00:18.472322 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:18.472280    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3cee6a-29895600-rl2vr"]
Nov 03 20:00:18.481438 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:18.481394    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-79c47f-29895600-l6n9d" event={"ID":"7b2bf43f-4806-48b4-bd9b-1d6092b082e8","Type":"ContainerStarted","Data":"8568c308fad97b56ac0ace31846ad8fd635f4ca6ce139287811cb2179ef33fe9"}
Nov 03 20:00:18.482982 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:18.482952    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f4e123-29895600-jsc8z" event={"ID":"1b794761-73bc-42a3-9efc-ba2eb7278181","Type":"ContainerDied","Data":"71864d53d341acc234b82a5a7268c253dd4b65290127fec5e18f2ecfdaf2fbeb"}
Nov 03 20:00:18.483102 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:18.483012    4921 scope.go:122] "RemoveContainer" containerID="6135504854ad0ba64f3f886661e22e905266bf03f04ac4e5dadbc87310f7f216"
Nov 03 20:00:18.483102 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:18.482962    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f4e123-29895600-jsc8z"
Nov 03 20:00:18.484837 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:18.484812    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d12868-29895600-bqplj" event={"ID":"a7db24b6-7319-492c-bd17-05eab3aab9da","Type":"ContainerStarted","Data":"51cc7e85c11c781953801999869d645931f39d1f322859353880888c0404cdc7"}
Nov 03 20:00:18.486301 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:18.486273    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-332694-29895600-s686j" event={"ID":"40c57d6c-5b96-4713-8c12-c41c5f435a9b","Type":"ContainerStarted","Data":"38fe35eae881338578b5532ad3fdbf26695e40d2a80e0853823630fb3eb1e180"}
Nov 03 20:00:18.487925 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:18.487904    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-82dbbf-29895600-9pjt9"
Nov 03 20:00:18.488028 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:18.487907    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-82dbbf-29895600-9pjt9" event={"ID":"9fe29ef3-6905-4213-bf69-63105ba9bc29","Type":"ContainerDied","Data":"d22d88372e44aa25feb89aa5544421412440dd2e308fbec773a9e81b3f3d29d0"}
Nov 03 20:00:18.498910 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:18.498890    4921 scope.go:122] "RemoveContainer" containerID="d82a7accdee2ff8ae44cdd365bd7b0a84680bdc76396d9a9ca57fbecb7f5804b"
Nov 03 20:00:18.499462 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:18.499440    4921 generic.go:356] "Generic (PLEG): container finished" podID="5769966d-308a-44a9-928a-984d95d95820" containerID="a177a9b6772d66e8fa04fd4f1af72e043deeb2b212ce0453b7f65c83fdc5f78d" exitCode=60
Nov 03 20:00:18.499585 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:18.499522    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9f3188-29895600-wtngr" event={"ID":"5769966d-308a-44a9-928a-984d95d95820","Type":"ContainerDied","Data":"a177a9b6772d66e8fa04fd4f1af72e043deeb2b212ce0453b7f65c83fdc5f78d"}
Nov 03 20:00:18.500033 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:18.500013    4921 scope.go:122] "RemoveContainer" containerID="a177a9b6772d66e8fa04fd4f1af72e043deeb2b212ce0453b7f65c83fdc5f78d"
Nov 03 20:00:18.500588 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:00:18.500447    4921 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-9f3188-29895600-wtngr_odh-ai-gateway-infra(5769966d-308a-44a9-928a-984d95d95820)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9f3188-29895600-wtngr" podUID="5769966d-308a-44a9-928a-984d95d95820"
Nov 03 20:00:18.502698 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:18.502679    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d4127-29895600-fh6sm"]
Nov 03 20:00:18.505164 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:18.505145    4921 generic.go:356] "Generic (PLEG): container finished" podID="951f6658-11ce-4b14-b615-31aacfd6b26e" containerID="0deb97c9af9ad4cc25e2f163a6e5583ca8c094597dd3dcb70ad56e9feb7aa8d4" exitCode=60
Nov 03 20:00:18.505269 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:18.505233    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-102379-29895600-l5pw6" event={"ID":"951f6658-11ce-4b14-b615-31aacfd6b26e","Type":"ContainerDied","Data":"0deb97c9af9ad4cc25e2f163a6e5583ca8c094597dd3dcb70ad56e9feb7aa8d4"}
Nov 03 20:00:18.505638 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:18.505619    4921 scope.go:122] "RemoveContainer" containerID="0deb97c9af9ad4cc25e2f163a6e5583ca8c094597dd3dcb70ad56e9feb7aa8d4"
Nov 03 20:00:18.505985 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:00:18.505963    4921 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-102379-29895600-l5pw6_odh-ai-gateway-infra(951f6658-11ce-4b14-b615-31aacfd6b26e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-102379-29895600-l5pw6" podUID="951f6658-11ce-4b14-b615-31aacfd6b26e"
Nov 03 20:00:18.515380 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:18.515355    4921 scope.go:122] "RemoveContainer" containerID="65ec268d2bd14218cac83c56db4f90ca8dcd5442a263fea47fa8a10e4dc05d1e"
Nov 03 20:00:18.516208 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:18.516185    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f4e123-29895600-jsc8z"]
Nov 03 20:00:18.526929 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:18.526891    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f4e123-29895600-jsc8z"]
Nov 03 20:00:18.534871 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:18.534704    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-82dbbf-29895600-9pjt9"]
Nov 03 20:00:18.535305 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:18.535286    4921 scope.go:122] "RemoveContainer" containerID="9990788ec1e229864063a866f348d4173d548e0a0007340eeac8ad813f61e50c"
Nov 03 20:00:18.539728 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:18.539704    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-82dbbf-29895600-9pjt9"]
Nov 03 20:00:18.732899 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:18.732869    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3cee6a-29895600-rl2vr"
Nov 03 20:00:18.772763 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:18.772733    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a35658-29895600-29r5s"]
Nov 03 20:00:18.776713 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:18.774005    4921 cpu_manager.go:414] "containerMap: removing container" podUID="9fe29ef3-6905-4213-bf69-63105ba9bc29" containerName="cleanup"
Nov 03 20:00:18.776713 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:18.774031    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fe29ef3-6905-4213-bf69-63105ba9bc29" containerName="cleanup"
Nov 03 20:00:18.776713 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:18.774062    4921 cpu_manager.go:414] "containerMap: removing container" podUID="957d5c20-d9b0-4d7e-a8df-544c55a1b456" containerName="cleanup"
Nov 03 20:00:18.776713 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:18.774070    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="957d5c20-d9b0-4d7e-a8df-544c55a1b456" containerName="cleanup"
Nov 03 20:00:18.776713 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:18.774113    4921 cpu_manager.go:414] "containerMap: removing container" podUID="957d5c20-d9b0-4d7e-a8df-544c55a1b456" containerName="cleanup"
Nov 03 20:00:18.776713 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:18.774121    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="957d5c20-d9b0-4d7e-a8df-544c55a1b456" containerName="cleanup"
Nov 03 20:00:18.776713 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:18.774142    4921 cpu_manager.go:414] "containerMap: removing container" podUID="957d5c20-d9b0-4d7e-a8df-544c55a1b456" containerName="cleanup"
Nov 03 20:00:18.776713 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:18.774150    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="957d5c20-d9b0-4d7e-a8df-544c55a1b456" containerName="cleanup"
Nov 03 20:00:18.776713 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:18.774395    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fe29ef3-6905-4213-bf69-63105ba9bc29" containerName="cleanup"
Nov 03 20:00:18.776713 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:18.774417    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="957d5c20-d9b0-4d7e-a8df-544c55a1b456" containerName="cleanup"
Nov 03 20:00:18.776713 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:18.774431    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="957d5c20-d9b0-4d7e-a8df-544c55a1b456" containerName="cleanup"
Nov 03 20:00:18.776713 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:18.774442    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="957d5c20-d9b0-4d7e-a8df-544c55a1b456" containerName="cleanup"
Nov 03 20:00:18.779480 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:18.779446    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a35658-29895600-29r5s"]
Nov 03 20:00:18.779601 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:18.779560    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a35658-29895600-29r5s"
Nov 03 20:00:18.859333 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:18.859298    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/957d5c20-d9b0-4d7e-a8df-544c55a1b456-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/957d5c20-d9b0-4d7e-a8df-544c55a1b456-maas-api-service-ca\") pod \"957d5c20-d9b0-4d7e-a8df-544c55a1b456\" (UID: \"957d5c20-d9b0-4d7e-a8df-544c55a1b456\") "
Nov 03 20:00:18.859508 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:18.859461    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/957d5c20-d9b0-4d7e-a8df-544c55a1b456-kube-api-access-64k5c\" (UniqueName: \"kubernetes.io/projected/957d5c20-d9b0-4d7e-a8df-544c55a1b456-kube-api-access-64k5c\") pod \"957d5c20-d9b0-4d7e-a8df-544c55a1b456\" (UID: \"957d5c20-d9b0-4d7e-a8df-544c55a1b456\") "
Nov 03 20:00:18.859749 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:18.859728    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/957d5c20-d9b0-4d7e-a8df-544c55a1b456-maas-api-service-ca" pod "957d5c20-d9b0-4d7e-a8df-544c55a1b456" (UID: "957d5c20-d9b0-4d7e-a8df-544c55a1b456"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:00:18.859867 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:18.859830    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/794c564b-dfea-4347-b7b7-10befa5ca787-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-a35658-29895600-29r5s\" (UID: \"794c564b-dfea-4347-b7b7-10befa5ca787\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a35658-29895600-29r5s"
Nov 03 20:00:18.859922 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:18.859878    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d97jl\" (UniqueName: \"kubernetes.io/projected/794c564b-dfea-4347-b7b7-10befa5ca787-kube-api-access-d97jl\") pod \"maas-api-key-cleanup-e2e-worker-w1-a35658-29895600-29r5s\" (UID: \"794c564b-dfea-4347-b7b7-10befa5ca787\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a35658-29895600-29r5s"
Nov 03 20:00:18.860922 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:18.860861    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/957d5c20-d9b0-4d7e-a8df-544c55a1b456-maas-api-service-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:18.863092 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:18.863066    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/957d5c20-d9b0-4d7e-a8df-544c55a1b456-kube-api-access-64k5c" pod "957d5c20-d9b0-4d7e-a8df-544c55a1b456" (UID: "957d5c20-d9b0-4d7e-a8df-544c55a1b456"). InnerVolumeSpecName "kube-api-access-64k5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:00:18.865253 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:18.865228    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d4127-29895600-fh6sm"
Nov 03 20:00:18.896037 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:18.895955    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b794761-73bc-42a3-9efc-ba2eb7278181" path="/var/lib/kubelet/pods/1b794761-73bc-42a3-9efc-ba2eb7278181/volumes"
Nov 03 20:00:18.896786 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:18.896756    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fe29ef3-6905-4213-bf69-63105ba9bc29" path="/var/lib/kubelet/pods/9fe29ef3-6905-4213-bf69-63105ba9bc29/volumes"
Nov 03 20:00:18.903925 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:18.903897    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a99fad-29895600-dwn5b"]
Nov 03 20:00:18.905201 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:18.905175    4921 cpu_manager.go:414] "containerMap: removing container" podUID="a0c0fc1f-186d-4c54-bd71-d36671cbb718" containerName="cleanup"
Nov 03 20:00:18.905201 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:18.905202    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0c0fc1f-186d-4c54-bd71-d36671cbb718" containerName="cleanup"
Nov 03 20:00:18.905448 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:18.905256    4921 cpu_manager.go:414] "containerMap: removing container" podUID="a0c0fc1f-186d-4c54-bd71-d36671cbb718" containerName="cleanup"
Nov 03 20:00:18.905448 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:18.905265    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0c0fc1f-186d-4c54-bd71-d36671cbb718" containerName="cleanup"
Nov 03 20:00:18.905448 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:18.905279    4921 cpu_manager.go:414] "containerMap: removing container" podUID="a0c0fc1f-186d-4c54-bd71-d36671cbb718" containerName="cleanup"
Nov 03 20:00:18.905448 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:18.905287    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0c0fc1f-186d-4c54-bd71-d36671cbb718" containerName="cleanup"
Nov 03 20:00:18.905715 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:18.905509    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0c0fc1f-186d-4c54-bd71-d36671cbb718" containerName="cleanup"
Nov 03 20:00:18.905715 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:18.905530    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0c0fc1f-186d-4c54-bd71-d36671cbb718" containerName="cleanup"
Nov 03 20:00:18.905715 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:18.905541    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0c0fc1f-186d-4c54-bd71-d36671cbb718" containerName="cleanup"
Nov 03 20:00:18.928944 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:18.928912    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a99fad-29895600-dwn5b"]
Nov 03 20:00:18.928944 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:18.928945    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a99fad-29895600-dwn5b"
Nov 03 20:00:18.971692 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:18.969390    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0c0fc1f-186d-4c54-bd71-d36671cbb718-kube-api-access-9xxdj\" (UniqueName: \"kubernetes.io/projected/a0c0fc1f-186d-4c54-bd71-d36671cbb718-kube-api-access-9xxdj\") pod \"a0c0fc1f-186d-4c54-bd71-d36671cbb718\" (UID: \"a0c0fc1f-186d-4c54-bd71-d36671cbb718\") "
Nov 03 20:00:18.971692 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:18.969628    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0c0fc1f-186d-4c54-bd71-d36671cbb718-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0c0fc1f-186d-4c54-bd71-d36671cbb718-maas-api-service-ca\") pod \"a0c0fc1f-186d-4c54-bd71-d36671cbb718\" (UID: \"a0c0fc1f-186d-4c54-bd71-d36671cbb718\") "
Nov 03 20:00:19.038963 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:18.985436    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0c0fc1f-186d-4c54-bd71-d36671cbb718-kube-api-access-9xxdj" pod "a0c0fc1f-186d-4c54-bd71-d36671cbb718" (UID: "a0c0fc1f-186d-4c54-bd71-d36671cbb718"). InnerVolumeSpecName "kube-api-access-9xxdj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:00:19.038963 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:18.988552    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0c0fc1f-186d-4c54-bd71-d36671cbb718-maas-api-service-ca" pod "a0c0fc1f-186d-4c54-bd71-d36671cbb718" (UID: "a0c0fc1f-186d-4c54-bd71-d36671cbb718"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:00:19.038963 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:18.990179    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/794c564b-dfea-4347-b7b7-10befa5ca787-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-a35658-29895600-29r5s\" (UID: \"794c564b-dfea-4347-b7b7-10befa5ca787\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a35658-29895600-29r5s"
Nov 03 20:00:19.038963 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:18.990231    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d97jl\" (UniqueName: \"kubernetes.io/projected/794c564b-dfea-4347-b7b7-10befa5ca787-kube-api-access-d97jl\") pod \"maas-api-key-cleanup-e2e-worker-w1-a35658-29895600-29r5s\" (UID: \"794c564b-dfea-4347-b7b7-10befa5ca787\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a35658-29895600-29r5s"
Nov 03 20:00:19.038963 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:18.991749    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64k5c\" (UniqueName: \"kubernetes.io/projected/957d5c20-d9b0-4d7e-a8df-544c55a1b456-kube-api-access-64k5c\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:19.038963 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:18.991795    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xxdj\" (UniqueName: \"kubernetes.io/projected/a0c0fc1f-186d-4c54-bd71-d36671cbb718-kube-api-access-9xxdj\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:19.038963 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:18.991821    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0c0fc1f-186d-4c54-bd71-d36671cbb718-maas-api-service-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:19.038963 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:18.992527    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/794c564b-dfea-4347-b7b7-10befa5ca787-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-a35658-29895600-29r5s\" (UID: \"794c564b-dfea-4347-b7b7-10befa5ca787\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a35658-29895600-29r5s"
Nov 03 20:00:19.071278 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:19.071243    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d97jl\" (UniqueName: \"kubernetes.io/projected/794c564b-dfea-4347-b7b7-10befa5ca787-kube-api-access-d97jl\") pod \"maas-api-key-cleanup-e2e-worker-w1-a35658-29895600-29r5s\" (UID: \"794c564b-dfea-4347-b7b7-10befa5ca787\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a35658-29895600-29r5s"
Nov 03 20:00:19.093513 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:19.093391    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e8ef6e5-7bec-40e1-a363-b35cca7d4347-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-a99fad-29895600-dwn5b\" (UID: \"5e8ef6e5-7bec-40e1-a363-b35cca7d4347\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a99fad-29895600-dwn5b"
Nov 03 20:00:19.093513 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:19.093484    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjfs8\" (UniqueName: \"kubernetes.io/projected/5e8ef6e5-7bec-40e1-a363-b35cca7d4347-kube-api-access-rjfs8\") pod \"maas-api-key-cleanup-e2e-inf-b-a99fad-29895600-dwn5b\" (UID: \"5e8ef6e5-7bec-40e1-a363-b35cca7d4347\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a99fad-29895600-dwn5b"
Nov 03 20:00:19.099537 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:19.099261    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a35658-29895600-29r5s"
Nov 03 20:00:19.199423 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:19.199383    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e8ef6e5-7bec-40e1-a363-b35cca7d4347-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-a99fad-29895600-dwn5b\" (UID: \"5e8ef6e5-7bec-40e1-a363-b35cca7d4347\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a99fad-29895600-dwn5b"
Nov 03 20:00:19.199579 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:19.199486    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjfs8\" (UniqueName: \"kubernetes.io/projected/5e8ef6e5-7bec-40e1-a363-b35cca7d4347-kube-api-access-rjfs8\") pod \"maas-api-key-cleanup-e2e-inf-b-a99fad-29895600-dwn5b\" (UID: \"5e8ef6e5-7bec-40e1-a363-b35cca7d4347\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a99fad-29895600-dwn5b"
Nov 03 20:00:19.200135 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:19.200112    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e8ef6e5-7bec-40e1-a363-b35cca7d4347-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-a99fad-29895600-dwn5b\" (UID: \"5e8ef6e5-7bec-40e1-a363-b35cca7d4347\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a99fad-29895600-dwn5b"
Nov 03 20:00:19.208619 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:19.208579    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjfs8\" (UniqueName: \"kubernetes.io/projected/5e8ef6e5-7bec-40e1-a363-b35cca7d4347-kube-api-access-rjfs8\") pod \"maas-api-key-cleanup-e2e-inf-b-a99fad-29895600-dwn5b\" (UID: \"5e8ef6e5-7bec-40e1-a363-b35cca7d4347\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a99fad-29895600-dwn5b"
Nov 03 20:00:19.248235 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:19.248204    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a99fad-29895600-dwn5b"
Nov 03 20:00:19.401825 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:19.401794    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a35658-29895600-29r5s"]
Nov 03 20:00:19.404760 ip-10-0-134-222 kubenswrapper[4921]: W1103 20:00:19.404726    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod794c564b_dfea_4347_b7b7_10befa5ca787.slice/crio-0ebee325492bd27b0d362a9dab1c3e7c3345cee79c0ea9821601a0603b085b71 WatchSource:0}: Error finding container 0ebee325492bd27b0d362a9dab1c3e7c3345cee79c0ea9821601a0603b085b71: Status 404 returned error can't find the container with id 0ebee325492bd27b0d362a9dab1c3e7c3345cee79c0ea9821601a0603b085b71
Nov 03 20:00:19.511872 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:19.511837    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-79c47f-29895600-l6n9d"]
Nov 03 20:00:19.521087 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:19.520957    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a35658-29895600-29r5s" event={"ID":"794c564b-dfea-4347-b7b7-10befa5ca787","Type":"ContainerStarted","Data":"0ebee325492bd27b0d362a9dab1c3e7c3345cee79c0ea9821601a0603b085b71"}
Nov 03 20:00:19.523639 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:19.523600    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3cee6a-29895600-rl2vr" event={"ID":"957d5c20-d9b0-4d7e-a8df-544c55a1b456","Type":"ContainerDied","Data":"9eb49e21d278e6fabf0388b7a13c8895b6bead17b50a22fc9e3694279300ba40"}
Nov 03 20:00:19.524108 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:19.524084    4921 scope.go:122] "RemoveContainer" containerID="1f8ce1c26fcf51717c6f3cad8dff103b626de089757065f7d83f66941c5a02e4"
Nov 03 20:00:19.524304 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:19.524284    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3cee6a-29895600-rl2vr"
Nov 03 20:00:19.535745 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:19.535715    4921 generic.go:356] "Generic (PLEG): container finished" podID="7b2bf43f-4806-48b4-bd9b-1d6092b082e8" containerID="8568c308fad97b56ac0ace31846ad8fd635f4ca6ce139287811cb2179ef33fe9" exitCode=60
Nov 03 20:00:19.535989 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:19.535954    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-79c47f-29895600-l6n9d" event={"ID":"7b2bf43f-4806-48b4-bd9b-1d6092b082e8","Type":"ContainerDied","Data":"8568c308fad97b56ac0ace31846ad8fd635f4ca6ce139287811cb2179ef33fe9"}
Nov 03 20:00:19.546411 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:19.546367    4921 scope.go:122] "RemoveContainer" containerID="8541e612f32594f233f382e3385c18c6b76d79896639a98b5537be845724f0da"
Nov 03 20:00:19.548151 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:19.548120    4921 generic.go:356] "Generic (PLEG): container finished" podID="a7db24b6-7319-492c-bd17-05eab3aab9da" containerID="899364c48cdee389ea4bf194037bc68efd2b3acbcd48f421e27bbf25c13f40cc" exitCode=60
Nov 03 20:00:19.548262 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:19.548242    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d12868-29895600-bqplj" event={"ID":"a7db24b6-7319-492c-bd17-05eab3aab9da","Type":"ContainerDied","Data":"899364c48cdee389ea4bf194037bc68efd2b3acbcd48f421e27bbf25c13f40cc"}
Nov 03 20:00:19.549606 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:19.549583    4921 scope.go:122] "RemoveContainer" containerID="899364c48cdee389ea4bf194037bc68efd2b3acbcd48f421e27bbf25c13f40cc"
Nov 03 20:00:19.587496 ip-10-0-134-222 kubenswrapper[4921]: W1103 20:00:19.587457    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e8ef6e5_7bec_40e1_a363_b35cca7d4347.slice/crio-5839eabbc8d0f9ef7e32d7d7196d3181d52af690c9a53685cf2ea622833d2dce WatchSource:0}: Error finding container 5839eabbc8d0f9ef7e32d7d7196d3181d52af690c9a53685cf2ea622833d2dce: Status 404 returned error can't find the container with id 5839eabbc8d0f9ef7e32d7d7196d3181d52af690c9a53685cf2ea622833d2dce
Nov 03 20:00:19.593390 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:19.593356    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-332694-29895600-s686j" event={"ID":"40c57d6c-5b96-4713-8c12-c41c5f435a9b","Type":"ContainerStarted","Data":"0b422f7ebcad8cdb95c2664056016bf6e43badc6ed662959362b0b9de1d0cc88"}
Nov 03 20:00:19.610069 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:19.610019    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3cee6a-29895600-rl2vr"]
Nov 03 20:00:19.619174 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:19.619145    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d4127-29895600-fh6sm" event={"ID":"a0c0fc1f-186d-4c54-bd71-d36671cbb718","Type":"ContainerDied","Data":"b1033d3691830e042ffb7fd9a3baa20757badd2664794549c99d69f87d9ac2ac"}
Nov 03 20:00:19.619278 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:19.619226    4921 scope.go:122] "RemoveContainer" containerID="f2b5abcffc23800b52d138d61d8bff1397088931e37ee85c40a28d22c2d298b8"
Nov 03 20:00:19.619409 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:19.619393    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d4127-29895600-fh6sm"
Nov 03 20:00:19.627321 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:19.627292    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3cee6a-29895600-rl2vr"]
Nov 03 20:00:19.639053 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:19.639029    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a99fad-29895600-dwn5b"]
Nov 03 20:00:19.650409 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:19.649185    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9f3188-29895600-wtngr"]
Nov 03 20:00:19.682879 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:19.679098    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-102379-29895600-l5pw6"]
Nov 03 20:00:19.698328 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:19.698266    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-332694-29895600-s686j" podStartSLOduration=19.698247181 podStartE2EDuration="19.698247181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:19.628458056 +0000 UTC m=+1732.872423524" watchObservedRunningTime="2026-11-03 20:00:19.698247181 +0000 UTC m=+1732.942212647"
Nov 03 20:00:19.718439 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:19.718380    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d4127-29895600-fh6sm"]
Nov 03 20:00:19.739700 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:19.733235    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-8d4127-29895600-fh6sm"]
Nov 03 20:00:19.888615 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:19.888274    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-79c47f-29895600-l6n9d"
Nov 03 20:00:19.933477 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:19.933449    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9f3188-29895600-wtngr"
Nov 03 20:00:20.023167 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:20.023130    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5769966d-308a-44a9-928a-984d95d95820-kube-api-access-lgjj8\" (UniqueName: \"kubernetes.io/projected/5769966d-308a-44a9-928a-984d95d95820-kube-api-access-lgjj8\") pod \"5769966d-308a-44a9-928a-984d95d95820\" (UID: \"5769966d-308a-44a9-928a-984d95d95820\") "
Nov 03 20:00:20.023340 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:20.023318    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5769966d-308a-44a9-928a-984d95d95820-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5769966d-308a-44a9-928a-984d95d95820-maas-api-service-ca\") pod \"5769966d-308a-44a9-928a-984d95d95820\" (UID: \"5769966d-308a-44a9-928a-984d95d95820\") "
Nov 03 20:00:20.023418 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:20.023396    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b2bf43f-4806-48b4-bd9b-1d6092b082e8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b2bf43f-4806-48b4-bd9b-1d6092b082e8-maas-api-service-ca\") pod \"7b2bf43f-4806-48b4-bd9b-1d6092b082e8\" (UID: \"7b2bf43f-4806-48b4-bd9b-1d6092b082e8\") "
Nov 03 20:00:20.023497 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:20.023466    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b2bf43f-4806-48b4-bd9b-1d6092b082e8-kube-api-access-6tnbd\" (UniqueName: \"kubernetes.io/projected/7b2bf43f-4806-48b4-bd9b-1d6092b082e8-kube-api-access-6tnbd\") pod \"7b2bf43f-4806-48b4-bd9b-1d6092b082e8\" (UID: \"7b2bf43f-4806-48b4-bd9b-1d6092b082e8\") "
Nov 03 20:00:20.023877 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:20.023834    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b2bf43f-4806-48b4-bd9b-1d6092b082e8-maas-api-service-ca" pod "7b2bf43f-4806-48b4-bd9b-1d6092b082e8" (UID: "7b2bf43f-4806-48b4-bd9b-1d6092b082e8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:00:20.023877 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:20.023852    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5769966d-308a-44a9-928a-984d95d95820-maas-api-service-ca" pod "5769966d-308a-44a9-928a-984d95d95820" (UID: "5769966d-308a-44a9-928a-984d95d95820"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:00:20.025359 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:20.025333    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5769966d-308a-44a9-928a-984d95d95820-maas-api-service-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:20.025530 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:20.025372    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b2bf43f-4806-48b4-bd9b-1d6092b082e8-maas-api-service-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:20.049973 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:20.049930    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b2bf43f-4806-48b4-bd9b-1d6092b082e8-kube-api-access-6tnbd" pod "7b2bf43f-4806-48b4-bd9b-1d6092b082e8" (UID: "7b2bf43f-4806-48b4-bd9b-1d6092b082e8"). InnerVolumeSpecName "kube-api-access-6tnbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:00:20.050115 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:20.050049    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5769966d-308a-44a9-928a-984d95d95820-kube-api-access-lgjj8" pod "5769966d-308a-44a9-928a-984d95d95820" (UID: "5769966d-308a-44a9-928a-984d95d95820"). InnerVolumeSpecName "kube-api-access-lgjj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:00:20.134969 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:20.128237    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgjj8\" (UniqueName: \"kubernetes.io/projected/5769966d-308a-44a9-928a-984d95d95820-kube-api-access-lgjj8\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:20.134969 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:20.128273    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tnbd\" (UniqueName: \"kubernetes.io/projected/7b2bf43f-4806-48b4-bd9b-1d6092b082e8-kube-api-access-6tnbd\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:20.157313 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:20.156943    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-102379-29895600-l5pw6"
Nov 03 20:00:20.229852 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:20.229812    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/951f6658-11ce-4b14-b615-31aacfd6b26e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/951f6658-11ce-4b14-b615-31aacfd6b26e-maas-api-service-ca\") pod \"951f6658-11ce-4b14-b615-31aacfd6b26e\" (UID: \"951f6658-11ce-4b14-b615-31aacfd6b26e\") "
Nov 03 20:00:20.230118 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:20.230093    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/951f6658-11ce-4b14-b615-31aacfd6b26e-kube-api-access-5b6vz\" (UniqueName: \"kubernetes.io/projected/951f6658-11ce-4b14-b615-31aacfd6b26e-kube-api-access-5b6vz\") pod \"951f6658-11ce-4b14-b615-31aacfd6b26e\" (UID: \"951f6658-11ce-4b14-b615-31aacfd6b26e\") "
Nov 03 20:00:20.230241 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:20.230175    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/951f6658-11ce-4b14-b615-31aacfd6b26e-maas-api-service-ca" pod "951f6658-11ce-4b14-b615-31aacfd6b26e" (UID: "951f6658-11ce-4b14-b615-31aacfd6b26e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:00:20.231054 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:20.231032    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/951f6658-11ce-4b14-b615-31aacfd6b26e-maas-api-service-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:20.232289 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:20.232263    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/951f6658-11ce-4b14-b615-31aacfd6b26e-kube-api-access-5b6vz" pod "951f6658-11ce-4b14-b615-31aacfd6b26e" (UID: "951f6658-11ce-4b14-b615-31aacfd6b26e"). InnerVolumeSpecName "kube-api-access-5b6vz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:00:20.332738 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:20.332655    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5b6vz\" (UniqueName: \"kubernetes.io/projected/951f6658-11ce-4b14-b615-31aacfd6b26e-kube-api-access-5b6vz\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:20.629342 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:20.629251    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-79c47f-29895600-l6n9d" event={"ID":"7b2bf43f-4806-48b4-bd9b-1d6092b082e8","Type":"ContainerDied","Data":"94700a460795a947122a52e19be7a914bebec1daa3efc44538d172c6d53b00ef"}
Nov 03 20:00:20.629342 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:20.629281    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-79c47f-29895600-l6n9d"
Nov 03 20:00:20.629342 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:20.629315    4921 scope.go:122] "RemoveContainer" containerID="8568c308fad97b56ac0ace31846ad8fd635f4ca6ce139287811cb2179ef33fe9"
Nov 03 20:00:20.631543 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:20.631516    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d12868-29895600-bqplj" event={"ID":"a7db24b6-7319-492c-bd17-05eab3aab9da","Type":"ContainerStarted","Data":"27e9d34256367ffb539854487e0e91879e515c7e7007aa52e4579a4216ece8c2"}
Nov 03 20:00:20.633393 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:20.633372    4921 generic.go:356] "Generic (PLEG): container finished" podID="40c57d6c-5b96-4713-8c12-c41c5f435a9b" containerID="0b422f7ebcad8cdb95c2664056016bf6e43badc6ed662959362b0b9de1d0cc88" exitCode=60
Nov 03 20:00:20.633525 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:20.633455    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-332694-29895600-s686j" event={"ID":"40c57d6c-5b96-4713-8c12-c41c5f435a9b","Type":"ContainerDied","Data":"0b422f7ebcad8cdb95c2664056016bf6e43badc6ed662959362b0b9de1d0cc88"}
Nov 03 20:00:20.633840 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:20.633823    4921 scope.go:122] "RemoveContainer" containerID="0b422f7ebcad8cdb95c2664056016bf6e43badc6ed662959362b0b9de1d0cc88"
Nov 03 20:00:20.636381 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:20.636357    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9f3188-29895600-wtngr"
Nov 03 20:00:20.637001 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:20.636981    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9f3188-29895600-wtngr" event={"ID":"5769966d-308a-44a9-928a-984d95d95820","Type":"ContainerDied","Data":"459927442c4fa6fc36365e1ff61f5cf07a5d9f1b6e5478199603b2678b27f320"}
Nov 03 20:00:20.640663 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:20.640635    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a35658-29895600-29r5s" event={"ID":"794c564b-dfea-4347-b7b7-10befa5ca787","Type":"ContainerStarted","Data":"a66e981326d140dfd5a26763a6e1fcff85dbcc535c0612001da493c20bca341f"}
Nov 03 20:00:20.642522 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:20.642497    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a99fad-29895600-dwn5b" event={"ID":"5e8ef6e5-7bec-40e1-a363-b35cca7d4347","Type":"ContainerStarted","Data":"04e5f49dceeac669f683ce600356d6cb5f48720de07a65811b38fa7ab3b2e0fd"}
Nov 03 20:00:20.642615 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:20.642545    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a99fad-29895600-dwn5b" event={"ID":"5e8ef6e5-7bec-40e1-a363-b35cca7d4347","Type":"ContainerStarted","Data":"5839eabbc8d0f9ef7e32d7d7196d3181d52af690c9a53685cf2ea622833d2dce"}
Nov 03 20:00:20.643787 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:20.643769    4921 scope.go:122] "RemoveContainer" containerID="a177a9b6772d66e8fa04fd4f1af72e043deeb2b212ce0453b7f65c83fdc5f78d"
Nov 03 20:00:20.644855 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:20.644836    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-102379-29895600-l5pw6" event={"ID":"951f6658-11ce-4b14-b615-31aacfd6b26e","Type":"ContainerDied","Data":"f7394575044a0dabed8a69783d2e15d01dac8fbf91d13e42ca3d1edaec687490"}
Nov 03 20:00:20.644935 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:20.644872    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-102379-29895600-l5pw6"
Nov 03 20:00:20.651738 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:20.651688    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d12868-29895600-bqplj" podStartSLOduration=20.651648872 podStartE2EDuration="20.651648872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:20.645524271 +0000 UTC m=+1733.889489735" watchObservedRunningTime="2026-11-03 20:00:20.651648872 +0000 UTC m=+1733.895614339"
Nov 03 20:00:20.662632 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:20.662606    4921 scope.go:122] "RemoveContainer" containerID="0deb97c9af9ad4cc25e2f163a6e5583ca8c094597dd3dcb70ad56e9feb7aa8d4"
Nov 03 20:00:20.680841 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:20.680810    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-79c47f-29895600-l6n9d"]
Nov 03 20:00:20.688763 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:20.688735    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-79c47f-29895600-l6n9d"]
Nov 03 20:00:20.695151 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:20.695127    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9f3188-29895600-wtngr"]
Nov 03 20:00:20.700038 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:20.700017    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9f3188-29895600-wtngr"]
Nov 03 20:00:20.707047 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:20.707004    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a99fad-29895600-dwn5b" podStartSLOduration=20.706992009 podStartE2EDuration="20.706992009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:20.704494733 +0000 UTC m=+1733.948460210" watchObservedRunningTime="2026-11-03 20:00:20.706992009 +0000 UTC m=+1733.950957473"
Nov 03 20:00:20.721255 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:20.721200    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a35658-29895600-29r5s" podStartSLOduration=20.721187902 podStartE2EDuration="20.721187902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:20.716854356 +0000 UTC m=+1733.960819824" watchObservedRunningTime="2026-11-03 20:00:20.721187902 +0000 UTC m=+1733.965153367"
Nov 03 20:00:20.735380 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:20.735189    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-102379-29895600-l5pw6"]
Nov 03 20:00:20.741356 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:20.741323    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-102379-29895600-l5pw6"]
Nov 03 20:00:20.936565 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:20.936534    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5769966d-308a-44a9-928a-984d95d95820" path="/var/lib/kubelet/pods/5769966d-308a-44a9-928a-984d95d95820/volumes"
Nov 03 20:00:20.937113 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:20.937097    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b2bf43f-4806-48b4-bd9b-1d6092b082e8" path="/var/lib/kubelet/pods/7b2bf43f-4806-48b4-bd9b-1d6092b082e8/volumes"
Nov 03 20:00:20.937570 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:20.937555    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="951f6658-11ce-4b14-b615-31aacfd6b26e" path="/var/lib/kubelet/pods/951f6658-11ce-4b14-b615-31aacfd6b26e/volumes"
Nov 03 20:00:20.938307 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:20.938292    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="957d5c20-d9b0-4d7e-a8df-544c55a1b456" path="/var/lib/kubelet/pods/957d5c20-d9b0-4d7e-a8df-544c55a1b456/volumes"
Nov 03 20:00:20.938732 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:20.938716    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0c0fc1f-186d-4c54-bd71-d36671cbb718" path="/var/lib/kubelet/pods/a0c0fc1f-186d-4c54-bd71-d36671cbb718/volumes"
Nov 03 20:00:21.750071 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:21.750032    4921 generic.go:356] "Generic (PLEG): container finished" podID="a7db24b6-7319-492c-bd17-05eab3aab9da" containerID="27e9d34256367ffb539854487e0e91879e515c7e7007aa52e4579a4216ece8c2" exitCode=60
Nov 03 20:00:21.751169 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:21.750361    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d12868-29895600-bqplj" event={"ID":"a7db24b6-7319-492c-bd17-05eab3aab9da","Type":"ContainerDied","Data":"27e9d34256367ffb539854487e0e91879e515c7e7007aa52e4579a4216ece8c2"}
Nov 03 20:00:21.751501 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:21.751444    4921 scope.go:122] "RemoveContainer" containerID="899364c48cdee389ea4bf194037bc68efd2b3acbcd48f421e27bbf25c13f40cc"
Nov 03 20:00:21.754732 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:21.754706    4921 scope.go:122] "RemoveContainer" containerID="27e9d34256367ffb539854487e0e91879e515c7e7007aa52e4579a4216ece8c2"
Nov 03 20:00:21.755479 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:00:21.755449    4921 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-d12868-29895600-bqplj_odh-ai-gateway-infra(a7db24b6-7319-492c-bd17-05eab3aab9da)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d12868-29895600-bqplj" podUID="a7db24b6-7319-492c-bd17-05eab3aab9da"
Nov 03 20:00:21.755876 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:21.755848    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-332694-29895600-s686j" event={"ID":"40c57d6c-5b96-4713-8c12-c41c5f435a9b","Type":"ContainerStarted","Data":"4c5a52e161585399a79327422baf49e9fe76e451a45bc01dcddf869d895fba6b"}
Nov 03 20:00:21.759773 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:21.759748    4921 generic.go:356] "Generic (PLEG): container finished" podID="794c564b-dfea-4347-b7b7-10befa5ca787" containerID="a66e981326d140dfd5a26763a6e1fcff85dbcc535c0612001da493c20bca341f" exitCode=60
Nov 03 20:00:21.759869 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:21.759831    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a35658-29895600-29r5s" event={"ID":"794c564b-dfea-4347-b7b7-10befa5ca787","Type":"ContainerDied","Data":"a66e981326d140dfd5a26763a6e1fcff85dbcc535c0612001da493c20bca341f"}
Nov 03 20:00:21.760129 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:21.760104    4921 scope.go:122] "RemoveContainer" containerID="a66e981326d140dfd5a26763a6e1fcff85dbcc535c0612001da493c20bca341f"
Nov 03 20:00:21.761897 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:21.761872    4921 generic.go:356] "Generic (PLEG): container finished" podID="5e8ef6e5-7bec-40e1-a363-b35cca7d4347" containerID="04e5f49dceeac669f683ce600356d6cb5f48720de07a65811b38fa7ab3b2e0fd" exitCode=60
Nov 03 20:00:21.762012 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:21.761935    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a99fad-29895600-dwn5b" event={"ID":"5e8ef6e5-7bec-40e1-a363-b35cca7d4347","Type":"ContainerDied","Data":"04e5f49dceeac669f683ce600356d6cb5f48720de07a65811b38fa7ab3b2e0fd"}
Nov 03 20:00:21.762275 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:21.762233    4921 scope.go:122] "RemoveContainer" containerID="04e5f49dceeac669f683ce600356d6cb5f48720de07a65811b38fa7ab3b2e0fd"
Nov 03 20:00:22.775362 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:22.775332    4921 scope.go:122] "RemoveContainer" containerID="27e9d34256367ffb539854487e0e91879e515c7e7007aa52e4579a4216ece8c2"
Nov 03 20:00:22.775878 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:00:22.775844    4921 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-d12868-29895600-bqplj_odh-ai-gateway-infra(a7db24b6-7319-492c-bd17-05eab3aab9da)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d12868-29895600-bqplj" podUID="a7db24b6-7319-492c-bd17-05eab3aab9da"
Nov 03 20:00:22.777314 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:22.777287    4921 generic.go:356] "Generic (PLEG): container finished" podID="40c57d6c-5b96-4713-8c12-c41c5f435a9b" containerID="4c5a52e161585399a79327422baf49e9fe76e451a45bc01dcddf869d895fba6b" exitCode=60
Nov 03 20:00:22.777448 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:22.777326    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-332694-29895600-s686j" event={"ID":"40c57d6c-5b96-4713-8c12-c41c5f435a9b","Type":"ContainerDied","Data":"4c5a52e161585399a79327422baf49e9fe76e451a45bc01dcddf869d895fba6b"}
Nov 03 20:00:22.777448 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:22.777411    4921 scope.go:122] "RemoveContainer" containerID="0b422f7ebcad8cdb95c2664056016bf6e43badc6ed662959362b0b9de1d0cc88"
Nov 03 20:00:22.777846 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:22.777806    4921 scope.go:122] "RemoveContainer" containerID="4c5a52e161585399a79327422baf49e9fe76e451a45bc01dcddf869d895fba6b"
Nov 03 20:00:22.778176 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:00:22.778149    4921 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-332694-29895600-s686j_odh-ai-gateway-infra(40c57d6c-5b96-4713-8c12-c41c5f435a9b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-332694-29895600-s686j" podUID="40c57d6c-5b96-4713-8c12-c41c5f435a9b"
Nov 03 20:00:22.780442 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:22.780417    4921 generic.go:356] "Generic (PLEG): container finished" podID="794c564b-dfea-4347-b7b7-10befa5ca787" containerID="a389e2e726b8dd2fa22b9356f92a869c0a4335f19731aaf44ece9a5c302afbc8" exitCode=60
Nov 03 20:00:22.780543 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:22.780497    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a35658-29895600-29r5s" event={"ID":"794c564b-dfea-4347-b7b7-10befa5ca787","Type":"ContainerDied","Data":"a389e2e726b8dd2fa22b9356f92a869c0a4335f19731aaf44ece9a5c302afbc8"}
Nov 03 20:00:22.781032 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:22.781014    4921 scope.go:122] "RemoveContainer" containerID="a389e2e726b8dd2fa22b9356f92a869c0a4335f19731aaf44ece9a5c302afbc8"
Nov 03 20:00:22.781440 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:00:22.781418    4921 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-a35658-29895600-29r5s_odh-ai-gateway-infra(794c564b-dfea-4347-b7b7-10befa5ca787)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a35658-29895600-29r5s" podUID="794c564b-dfea-4347-b7b7-10befa5ca787"
Nov 03 20:00:22.783871 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:22.783850    4921 generic.go:356] "Generic (PLEG): container finished" podID="5e8ef6e5-7bec-40e1-a363-b35cca7d4347" containerID="14f52bb27426d5a93089ff9a76116d33834fc7b33e8e45f0d3d8fbbe311e7774" exitCode=60
Nov 03 20:00:22.783962 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:22.783936    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a99fad-29895600-dwn5b" event={"ID":"5e8ef6e5-7bec-40e1-a363-b35cca7d4347","Type":"ContainerDied","Data":"14f52bb27426d5a93089ff9a76116d33834fc7b33e8e45f0d3d8fbbe311e7774"}
Nov 03 20:00:22.784309 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:22.784292    4921 scope.go:122] "RemoveContainer" containerID="14f52bb27426d5a93089ff9a76116d33834fc7b33e8e45f0d3d8fbbe311e7774"
Nov 03 20:00:22.784729 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:00:22.784706    4921 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-a99fad-29895600-dwn5b_odh-ai-gateway-infra(5e8ef6e5-7bec-40e1-a363-b35cca7d4347)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a99fad-29895600-dwn5b" podUID="5e8ef6e5-7bec-40e1-a363-b35cca7d4347"
Nov 03 20:00:22.887197 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:22.887176    4921 scope.go:122] "RemoveContainer" containerID="a66e981326d140dfd5a26763a6e1fcff85dbcc535c0612001da493c20bca341f"
Nov 03 20:00:22.903010 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:22.902987    4921 scope.go:122] "RemoveContainer" containerID="04e5f49dceeac669f683ce600356d6cb5f48720de07a65811b38fa7ab3b2e0fd"
Nov 03 20:00:23.794705 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:23.794663    4921 scope.go:122] "RemoveContainer" containerID="a389e2e726b8dd2fa22b9356f92a869c0a4335f19731aaf44ece9a5c302afbc8"
Nov 03 20:00:23.795193 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:00:23.795095    4921 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-a35658-29895600-29r5s_odh-ai-gateway-infra(794c564b-dfea-4347-b7b7-10befa5ca787)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a35658-29895600-29r5s" podUID="794c564b-dfea-4347-b7b7-10befa5ca787"
Nov 03 20:00:23.796542 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:23.796517    4921 scope.go:122] "RemoveContainer" containerID="14f52bb27426d5a93089ff9a76116d33834fc7b33e8e45f0d3d8fbbe311e7774"
Nov 03 20:00:23.796921 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:00:23.796883    4921 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-a99fad-29895600-dwn5b_odh-ai-gateway-infra(5e8ef6e5-7bec-40e1-a363-b35cca7d4347)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a99fad-29895600-dwn5b" podUID="5e8ef6e5-7bec-40e1-a363-b35cca7d4347"
Nov 03 20:00:23.798547 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:23.798528    4921 scope.go:122] "RemoveContainer" containerID="4c5a52e161585399a79327422baf49e9fe76e451a45bc01dcddf869d895fba6b"
Nov 03 20:00:23.798870 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:00:23.798851    4921 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-332694-29895600-s686j_odh-ai-gateway-infra(40c57d6c-5b96-4713-8c12-c41c5f435a9b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-332694-29895600-s686j" podUID="40c57d6c-5b96-4713-8c12-c41c5f435a9b"
Nov 03 20:00:32.899692 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:32.891476    4921 scope.go:122] "RemoveContainer" containerID="4c5a52e161585399a79327422baf49e9fe76e451a45bc01dcddf869d895fba6b"
Nov 03 20:00:32.968727 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:00:32.967991    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 20:00:33.988216 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:33.988183    4921 generic.go:356] "Generic (PLEG): container finished" podID="40c57d6c-5b96-4713-8c12-c41c5f435a9b" containerID="be318a910202e7fbef2c46d4a06d5a8e49045f5c7c2300f5374e63fce5552efc" exitCode=60
Nov 03 20:00:33.988700 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:33.988258    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-332694-29895600-s686j" event={"ID":"40c57d6c-5b96-4713-8c12-c41c5f435a9b","Type":"ContainerDied","Data":"be318a910202e7fbef2c46d4a06d5a8e49045f5c7c2300f5374e63fce5552efc"}
Nov 03 20:00:33.988700 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:33.988348    4921 scope.go:122] "RemoveContainer" containerID="4c5a52e161585399a79327422baf49e9fe76e451a45bc01dcddf869d895fba6b"
Nov 03 20:00:33.988854 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:33.988838    4921 scope.go:122] "RemoveContainer" containerID="be318a910202e7fbef2c46d4a06d5a8e49045f5c7c2300f5374e63fce5552efc"
Nov 03 20:00:33.989203 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:00:33.989182    4921 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-332694-29895600-s686j_odh-ai-gateway-infra(40c57d6c-5b96-4713-8c12-c41c5f435a9b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-332694-29895600-s686j" podUID="40c57d6c-5b96-4713-8c12-c41c5f435a9b"
Nov 03 20:00:34.881113 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:34.881082    4921 scope.go:122] "RemoveContainer" containerID="27e9d34256367ffb539854487e0e91879e515c7e7007aa52e4579a4216ece8c2"
Nov 03 20:00:35.014651 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:35.014617    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-332694-29895600-s686j"]
Nov 03 20:00:35.332226 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:35.332197    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-332694-29895600-s686j"
Nov 03 20:00:35.415849 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:35.415820    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40c57d6c-5b96-4713-8c12-c41c5f435a9b-kube-api-access-nhnwh\" (UniqueName: \"kubernetes.io/projected/40c57d6c-5b96-4713-8c12-c41c5f435a9b-kube-api-access-nhnwh\") pod \"40c57d6c-5b96-4713-8c12-c41c5f435a9b\" (UID: \"40c57d6c-5b96-4713-8c12-c41c5f435a9b\") "
Nov 03 20:00:35.416079 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:35.416062    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40c57d6c-5b96-4713-8c12-c41c5f435a9b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40c57d6c-5b96-4713-8c12-c41c5f435a9b-maas-api-service-ca\") pod \"40c57d6c-5b96-4713-8c12-c41c5f435a9b\" (UID: \"40c57d6c-5b96-4713-8c12-c41c5f435a9b\") "
Nov 03 20:00:35.416492 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:35.416457    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40c57d6c-5b96-4713-8c12-c41c5f435a9b-maas-api-service-ca" pod "40c57d6c-5b96-4713-8c12-c41c5f435a9b" (UID: "40c57d6c-5b96-4713-8c12-c41c5f435a9b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:00:35.416957 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:35.416930    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40c57d6c-5b96-4713-8c12-c41c5f435a9b-maas-api-service-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:35.418926 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:35.418897    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40c57d6c-5b96-4713-8c12-c41c5f435a9b-kube-api-access-nhnwh" pod "40c57d6c-5b96-4713-8c12-c41c5f435a9b" (UID: "40c57d6c-5b96-4713-8c12-c41c5f435a9b"). InnerVolumeSpecName "kube-api-access-nhnwh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:00:35.519030 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:35.518999    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhnwh\" (UniqueName: \"kubernetes.io/projected/40c57d6c-5b96-4713-8c12-c41c5f435a9b-kube-api-access-nhnwh\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:35.880962 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:35.880877    4921 scope.go:122] "RemoveContainer" containerID="14f52bb27426d5a93089ff9a76116d33834fc7b33e8e45f0d3d8fbbe311e7774"
Nov 03 20:00:36.011002 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:36.010963    4921 generic.go:356] "Generic (PLEG): container finished" podID="a7db24b6-7319-492c-bd17-05eab3aab9da" containerID="fb21f3fefcf98c148cf0e0f3215446f6ee8b4642940635907dcdcfcdda03a2d3" exitCode=60
Nov 03 20:00:36.011180 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:36.011035    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d12868-29895600-bqplj" event={"ID":"a7db24b6-7319-492c-bd17-05eab3aab9da","Type":"ContainerDied","Data":"fb21f3fefcf98c148cf0e0f3215446f6ee8b4642940635907dcdcfcdda03a2d3"}
Nov 03 20:00:36.011180 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:36.011099    4921 scope.go:122] "RemoveContainer" containerID="27e9d34256367ffb539854487e0e91879e515c7e7007aa52e4579a4216ece8c2"
Nov 03 20:00:36.011638 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:36.011608    4921 scope.go:122] "RemoveContainer" containerID="fb21f3fefcf98c148cf0e0f3215446f6ee8b4642940635907dcdcfcdda03a2d3"
Nov 03 20:00:36.012233 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:00:36.012177    4921 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-d12868-29895600-bqplj_odh-ai-gateway-infra(a7db24b6-7319-492c-bd17-05eab3aab9da)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d12868-29895600-bqplj" podUID="a7db24b6-7319-492c-bd17-05eab3aab9da"
Nov 03 20:00:36.013070 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:36.013047    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-332694-29895600-s686j"
Nov 03 20:00:36.013189 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:36.013134    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-332694-29895600-s686j" event={"ID":"40c57d6c-5b96-4713-8c12-c41c5f435a9b","Type":"ContainerDied","Data":"38fe35eae881338578b5532ad3fdbf26695e40d2a80e0853823630fb3eb1e180"}
Nov 03 20:00:36.054860 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:36.054832    4921 scope.go:122] "RemoveContainer" containerID="be318a910202e7fbef2c46d4a06d5a8e49045f5c7c2300f5374e63fce5552efc"
Nov 03 20:00:36.067990 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:36.067951    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-332694-29895600-s686j"]
Nov 03 20:00:36.073517 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:36.073490    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-332694-29895600-s686j"]
Nov 03 20:00:36.887382 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:36.887357    4921 scope.go:122] "RemoveContainer" containerID="a389e2e726b8dd2fa22b9356f92a869c0a4335f19731aaf44ece9a5c302afbc8"
Nov 03 20:00:36.894178 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:36.894150    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40c57d6c-5b96-4713-8c12-c41c5f435a9b" path="/var/lib/kubelet/pods/40c57d6c-5b96-4713-8c12-c41c5f435a9b/volumes"
Nov 03 20:00:37.037425 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:37.037394    4921 generic.go:356] "Generic (PLEG): container finished" podID="5e8ef6e5-7bec-40e1-a363-b35cca7d4347" containerID="b14dfb580e899b56a40681c5fcf8b1477dd30c561207651c43a44d3a37a7bd1a" exitCode=60
Nov 03 20:00:37.037605 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:37.037466    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a99fad-29895600-dwn5b" event={"ID":"5e8ef6e5-7bec-40e1-a363-b35cca7d4347","Type":"ContainerDied","Data":"b14dfb580e899b56a40681c5fcf8b1477dd30c561207651c43a44d3a37a7bd1a"}
Nov 03 20:00:37.037605 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:37.037555    4921 scope.go:122] "RemoveContainer" containerID="14f52bb27426d5a93089ff9a76116d33834fc7b33e8e45f0d3d8fbbe311e7774"
Nov 03 20:00:37.038044 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:37.038023    4921 scope.go:122] "RemoveContainer" containerID="b14dfb580e899b56a40681c5fcf8b1477dd30c561207651c43a44d3a37a7bd1a"
Nov 03 20:00:37.038443 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:00:37.038418    4921 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-a99fad-29895600-dwn5b_odh-ai-gateway-infra(5e8ef6e5-7bec-40e1-a363-b35cca7d4347)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a99fad-29895600-dwn5b" podUID="5e8ef6e5-7bec-40e1-a363-b35cca7d4347"
Nov 03 20:00:37.049029 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:37.049004    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d12868-29895600-bqplj"]
Nov 03 20:00:37.496747 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:37.496717    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d12868-29895600-bqplj"
Nov 03 20:00:37.578986 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:37.578890    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7db24b6-7319-492c-bd17-05eab3aab9da-kube-api-access-nmvn9\" (UniqueName: \"kubernetes.io/projected/a7db24b6-7319-492c-bd17-05eab3aab9da-kube-api-access-nmvn9\") pod \"a7db24b6-7319-492c-bd17-05eab3aab9da\" (UID: \"a7db24b6-7319-492c-bd17-05eab3aab9da\") "
Nov 03 20:00:37.579175 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:37.579131    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7db24b6-7319-492c-bd17-05eab3aab9da-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7db24b6-7319-492c-bd17-05eab3aab9da-maas-api-service-ca\") pod \"a7db24b6-7319-492c-bd17-05eab3aab9da\" (UID: \"a7db24b6-7319-492c-bd17-05eab3aab9da\") "
Nov 03 20:00:37.580711 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:37.580249    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7db24b6-7319-492c-bd17-05eab3aab9da-maas-api-service-ca" pod "a7db24b6-7319-492c-bd17-05eab3aab9da" (UID: "a7db24b6-7319-492c-bd17-05eab3aab9da"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:00:37.585075 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:37.582452    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7db24b6-7319-492c-bd17-05eab3aab9da-kube-api-access-nmvn9" pod "a7db24b6-7319-492c-bd17-05eab3aab9da" (UID: "a7db24b6-7319-492c-bd17-05eab3aab9da"). InnerVolumeSpecName "kube-api-access-nmvn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:00:37.681131 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:37.681095    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmvn9\" (UniqueName: \"kubernetes.io/projected/a7db24b6-7319-492c-bd17-05eab3aab9da-kube-api-access-nmvn9\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:37.681131 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:37.681124    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7db24b6-7319-492c-bd17-05eab3aab9da-maas-api-service-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:38.050875 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:38.050847    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d12868-29895600-bqplj"
Nov 03 20:00:38.051053 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:38.050885    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d12868-29895600-bqplj" event={"ID":"a7db24b6-7319-492c-bd17-05eab3aab9da","Type":"ContainerDied","Data":"51cc7e85c11c781953801999869d645931f39d1f322859353880888c0404cdc7"}
Nov 03 20:00:38.051053 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:38.050948    4921 scope.go:122] "RemoveContainer" containerID="fb21f3fefcf98c148cf0e0f3215446f6ee8b4642940635907dcdcfcdda03a2d3"
Nov 03 20:00:38.053285 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:38.053259    4921 generic.go:356] "Generic (PLEG): container finished" podID="794c564b-dfea-4347-b7b7-10befa5ca787" containerID="3eb938d125c22393b6760d06ddb66915109a4f54ca5ebf641ac8cf36fe547361" exitCode=60
Nov 03 20:00:38.053426 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:38.053335    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a35658-29895600-29r5s" event={"ID":"794c564b-dfea-4347-b7b7-10befa5ca787","Type":"ContainerDied","Data":"3eb938d125c22393b6760d06ddb66915109a4f54ca5ebf641ac8cf36fe547361"}
Nov 03 20:00:38.053866 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:38.053824    4921 scope.go:122] "RemoveContainer" containerID="3eb938d125c22393b6760d06ddb66915109a4f54ca5ebf641ac8cf36fe547361"
Nov 03 20:00:38.054240 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:00:38.054215    4921 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-a35658-29895600-29r5s_odh-ai-gateway-infra(794c564b-dfea-4347-b7b7-10befa5ca787)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a35658-29895600-29r5s" podUID="794c564b-dfea-4347-b7b7-10befa5ca787"
Nov 03 20:00:38.066636 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:38.066332    4921 scope.go:122] "RemoveContainer" containerID="a389e2e726b8dd2fa22b9356f92a869c0a4335f19731aaf44ece9a5c302afbc8"
Nov 03 20:00:38.072118 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:38.072088    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a99fad-29895600-dwn5b"]
Nov 03 20:00:38.097831 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:38.097790    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d12868-29895600-bqplj"]
Nov 03 20:00:38.102978 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:38.102946    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d12868-29895600-bqplj"]
Nov 03 20:00:38.273097 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:38.273072    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a99fad-29895600-dwn5b"
Nov 03 20:00:38.396181 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:38.396145    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e8ef6e5-7bec-40e1-a363-b35cca7d4347-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e8ef6e5-7bec-40e1-a363-b35cca7d4347-maas-api-service-ca\") pod \"5e8ef6e5-7bec-40e1-a363-b35cca7d4347\" (UID: \"5e8ef6e5-7bec-40e1-a363-b35cca7d4347\") "
Nov 03 20:00:38.396353 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:38.396234    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e8ef6e5-7bec-40e1-a363-b35cca7d4347-kube-api-access-rjfs8\" (UniqueName: \"kubernetes.io/projected/5e8ef6e5-7bec-40e1-a363-b35cca7d4347-kube-api-access-rjfs8\") pod \"5e8ef6e5-7bec-40e1-a363-b35cca7d4347\" (UID: \"5e8ef6e5-7bec-40e1-a363-b35cca7d4347\") "
Nov 03 20:00:38.396569 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:38.396545    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e8ef6e5-7bec-40e1-a363-b35cca7d4347-maas-api-service-ca" pod "5e8ef6e5-7bec-40e1-a363-b35cca7d4347" (UID: "5e8ef6e5-7bec-40e1-a363-b35cca7d4347"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:00:38.397156 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:38.397033    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e8ef6e5-7bec-40e1-a363-b35cca7d4347-maas-api-service-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:38.398596 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:38.398575    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e8ef6e5-7bec-40e1-a363-b35cca7d4347-kube-api-access-rjfs8" pod "5e8ef6e5-7bec-40e1-a363-b35cca7d4347" (UID: "5e8ef6e5-7bec-40e1-a363-b35cca7d4347"). InnerVolumeSpecName "kube-api-access-rjfs8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:00:38.498644 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:38.498554    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjfs8\" (UniqueName: \"kubernetes.io/projected/5e8ef6e5-7bec-40e1-a363-b35cca7d4347-kube-api-access-rjfs8\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:38.929745 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:38.927436    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7db24b6-7319-492c-bd17-05eab3aab9da" path="/var/lib/kubelet/pods/a7db24b6-7319-492c-bd17-05eab3aab9da/volumes"
Nov 03 20:00:39.075546 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:39.075508    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a99fad-29895600-dwn5b" event={"ID":"5e8ef6e5-7bec-40e1-a363-b35cca7d4347","Type":"ContainerDied","Data":"5839eabbc8d0f9ef7e32d7d7196d3181d52af690c9a53685cf2ea622833d2dce"}
Nov 03 20:00:39.075757 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:39.075573    4921 scope.go:122] "RemoveContainer" containerID="b14dfb580e899b56a40681c5fcf8b1477dd30c561207651c43a44d3a37a7bd1a"
Nov 03 20:00:39.075757 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:39.075518    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a99fad-29895600-dwn5b"
Nov 03 20:00:39.098936 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:39.098901    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a35658-29895600-29r5s"]
Nov 03 20:00:39.107930 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:39.107900    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a99fad-29895600-dwn5b"]
Nov 03 20:00:39.116546 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:39.116508    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a99fad-29895600-dwn5b"]
Nov 03 20:00:39.289924 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:39.289896    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a35658-29895600-29r5s"
Nov 03 20:00:39.417448 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:39.417406    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/794c564b-dfea-4347-b7b7-10befa5ca787-kube-api-access-d97jl\" (UniqueName: \"kubernetes.io/projected/794c564b-dfea-4347-b7b7-10befa5ca787-kube-api-access-d97jl\") pod \"794c564b-dfea-4347-b7b7-10befa5ca787\" (UID: \"794c564b-dfea-4347-b7b7-10befa5ca787\") "
Nov 03 20:00:39.417618 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:39.417503    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/794c564b-dfea-4347-b7b7-10befa5ca787-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/794c564b-dfea-4347-b7b7-10befa5ca787-maas-api-service-ca\") pod \"794c564b-dfea-4347-b7b7-10befa5ca787\" (UID: \"794c564b-dfea-4347-b7b7-10befa5ca787\") "
Nov 03 20:00:39.417911 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:39.417886    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/794c564b-dfea-4347-b7b7-10befa5ca787-maas-api-service-ca" pod "794c564b-dfea-4347-b7b7-10befa5ca787" (UID: "794c564b-dfea-4347-b7b7-10befa5ca787"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:00:39.418141 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:39.418127    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/794c564b-dfea-4347-b7b7-10befa5ca787-maas-api-service-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:39.419653 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:39.419633    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/794c564b-dfea-4347-b7b7-10befa5ca787-kube-api-access-d97jl" pod "794c564b-dfea-4347-b7b7-10befa5ca787" (UID: "794c564b-dfea-4347-b7b7-10befa5ca787"). InnerVolumeSpecName "kube-api-access-d97jl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:00:39.520087 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:39.520002    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d97jl\" (UniqueName: \"kubernetes.io/projected/794c564b-dfea-4347-b7b7-10befa5ca787-kube-api-access-d97jl\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:40.087250 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:40.087215    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a35658-29895600-29r5s" event={"ID":"794c564b-dfea-4347-b7b7-10befa5ca787","Type":"ContainerDied","Data":"0ebee325492bd27b0d362a9dab1c3e7c3345cee79c0ea9821601a0603b085b71"}
Nov 03 20:00:40.087443 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:40.087280    4921 scope.go:122] "RemoveContainer" containerID="3eb938d125c22393b6760d06ddb66915109a4f54ca5ebf641ac8cf36fe547361"
Nov 03 20:00:40.087443 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:40.087222    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a35658-29895600-29r5s"
Nov 03 20:00:40.115234 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:40.115194    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a35658-29895600-29r5s"]
Nov 03 20:00:40.123700 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:40.123656    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a35658-29895600-29r5s"]
Nov 03 20:00:40.892034 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:40.892002    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e8ef6e5-7bec-40e1-a363-b35cca7d4347" path="/var/lib/kubelet/pods/5e8ef6e5-7bec-40e1-a363-b35cca7d4347/volumes"
Nov 03 20:00:40.892505 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:40.892490    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="794c564b-dfea-4347-b7b7-10befa5ca787" path="/var/lib/kubelet/pods/794c564b-dfea-4347-b7b7-10befa5ca787/volumes"
Nov 03 20:00:49.748169 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:49.748129    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6"]
Nov 03 20:00:49.749165 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:49.749129    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6" podUID="052a7a9f-ed4b-4cb9-911a-413079445a1c" containerName="istio-proxy" containerID="cri-o://fa3fdaac1525b4a72fe74d752a4613a5148615674024627d6ecf2e38ae7084e3" gracePeriod=30
Nov 03 20:00:55.065963 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.065892    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6"
Nov 03 20:00:55.097259 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.097212    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb"]
Nov 03 20:00:55.098113 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.098094    4921 cpu_manager.go:414] "containerMap: removing container" podUID="a7db24b6-7319-492c-bd17-05eab3aab9da" containerName="cleanup"
Nov 03 20:00:55.098182 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.098115    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7db24b6-7319-492c-bd17-05eab3aab9da" containerName="cleanup"
Nov 03 20:00:55.098182 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.098129    4921 cpu_manager.go:414] "containerMap: removing container" podUID="794c564b-dfea-4347-b7b7-10befa5ca787" containerName="cleanup"
Nov 03 20:00:55.098182 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.098135    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="794c564b-dfea-4347-b7b7-10befa5ca787" containerName="cleanup"
Nov 03 20:00:55.098182 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.098143    4921 cpu_manager.go:414] "containerMap: removing container" podUID="5769966d-308a-44a9-928a-984d95d95820" containerName="cleanup"
Nov 03 20:00:55.098182 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.098148    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5769966d-308a-44a9-928a-984d95d95820" containerName="cleanup"
Nov 03 20:00:55.098182 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.098157    4921 cpu_manager.go:414] "containerMap: removing container" podUID="951f6658-11ce-4b14-b615-31aacfd6b26e" containerName="cleanup"
Nov 03 20:00:55.098182 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.098163    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="951f6658-11ce-4b14-b615-31aacfd6b26e" containerName="cleanup"
Nov 03 20:00:55.098182 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.098172    4921 cpu_manager.go:414] "containerMap: removing container" podUID="7b2bf43f-4806-48b4-bd9b-1d6092b082e8" containerName="cleanup"
Nov 03 20:00:55.098182 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.098177    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b2bf43f-4806-48b4-bd9b-1d6092b082e8" containerName="cleanup"
Nov 03 20:00:55.098182 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.098182    4921 cpu_manager.go:414] "containerMap: removing container" podUID="a7db24b6-7319-492c-bd17-05eab3aab9da" containerName="cleanup"
Nov 03 20:00:55.098498 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.098187    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7db24b6-7319-492c-bd17-05eab3aab9da" containerName="cleanup"
Nov 03 20:00:55.098498 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.098200    4921 cpu_manager.go:414] "containerMap: removing container" podUID="5e8ef6e5-7bec-40e1-a363-b35cca7d4347" containerName="cleanup"
Nov 03 20:00:55.098498 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.098205    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e8ef6e5-7bec-40e1-a363-b35cca7d4347" containerName="cleanup"
Nov 03 20:00:55.098498 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.098224    4921 cpu_manager.go:414] "containerMap: removing container" podUID="5769966d-308a-44a9-928a-984d95d95820" containerName="cleanup"
Nov 03 20:00:55.098498 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.098229    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5769966d-308a-44a9-928a-984d95d95820" containerName="cleanup"
Nov 03 20:00:55.098498 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.098237    4921 cpu_manager.go:414] "containerMap: removing container" podUID="40c57d6c-5b96-4713-8c12-c41c5f435a9b" containerName="cleanup"
Nov 03 20:00:55.098498 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.098241    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40c57d6c-5b96-4713-8c12-c41c5f435a9b" containerName="cleanup"
Nov 03 20:00:55.098498 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.098249    4921 cpu_manager.go:414] "containerMap: removing container" podUID="794c564b-dfea-4347-b7b7-10befa5ca787" containerName="cleanup"
Nov 03 20:00:55.098498 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.098253    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="794c564b-dfea-4347-b7b7-10befa5ca787" containerName="cleanup"
Nov 03 20:00:55.098498 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.098260    4921 cpu_manager.go:414] "containerMap: removing container" podUID="40c57d6c-5b96-4713-8c12-c41c5f435a9b" containerName="cleanup"
Nov 03 20:00:55.098498 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.098264    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40c57d6c-5b96-4713-8c12-c41c5f435a9b" containerName="cleanup"
Nov 03 20:00:55.098498 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.098272    4921 cpu_manager.go:414] "containerMap: removing container" podUID="5e8ef6e5-7bec-40e1-a363-b35cca7d4347" containerName="cleanup"
Nov 03 20:00:55.098498 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.098276    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e8ef6e5-7bec-40e1-a363-b35cca7d4347" containerName="cleanup"
Nov 03 20:00:55.098498 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.098286    4921 cpu_manager.go:414] "containerMap: removing container" podUID="052a7a9f-ed4b-4cb9-911a-413079445a1c" containerName="istio-proxy"
Nov 03 20:00:55.098498 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.098290    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="052a7a9f-ed4b-4cb9-911a-413079445a1c" containerName="istio-proxy"
Nov 03 20:00:55.098498 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.098297    4921 cpu_manager.go:414] "containerMap: removing container" podUID="7b2bf43f-4806-48b4-bd9b-1d6092b082e8" containerName="cleanup"
Nov 03 20:00:55.098498 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.098302    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b2bf43f-4806-48b4-bd9b-1d6092b082e8" containerName="cleanup"
Nov 03 20:00:55.098498 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.098310    4921 cpu_manager.go:414] "containerMap: removing container" podUID="951f6658-11ce-4b14-b615-31aacfd6b26e" containerName="cleanup"
Nov 03 20:00:55.098498 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.098314    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="951f6658-11ce-4b14-b615-31aacfd6b26e" containerName="cleanup"
Nov 03 20:00:55.098498 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.098323    4921 cpu_manager.go:414] "containerMap: removing container" podUID="5e8ef6e5-7bec-40e1-a363-b35cca7d4347" containerName="cleanup"
Nov 03 20:00:55.098498 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.098327    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e8ef6e5-7bec-40e1-a363-b35cca7d4347" containerName="cleanup"
Nov 03 20:00:55.098498 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.098439    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b2bf43f-4806-48b4-bd9b-1d6092b082e8" containerName="cleanup"
Nov 03 20:00:55.098498 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.098446    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7db24b6-7319-492c-bd17-05eab3aab9da" containerName="cleanup"
Nov 03 20:00:55.098498 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.098454    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e8ef6e5-7bec-40e1-a363-b35cca7d4347" containerName="cleanup"
Nov 03 20:00:55.098498 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.098466    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="5769966d-308a-44a9-928a-984d95d95820" containerName="cleanup"
Nov 03 20:00:55.098498 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.098473    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="40c57d6c-5b96-4713-8c12-c41c5f435a9b" containerName="cleanup"
Nov 03 20:00:55.098498 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.098484    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="794c564b-dfea-4347-b7b7-10befa5ca787" containerName="cleanup"
Nov 03 20:00:55.098498 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.098490    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="40c57d6c-5b96-4713-8c12-c41c5f435a9b" containerName="cleanup"
Nov 03 20:00:55.098498 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.098497    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="951f6658-11ce-4b14-b615-31aacfd6b26e" containerName="cleanup"
Nov 03 20:00:55.098498 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.098503    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="5769966d-308a-44a9-928a-984d95d95820" containerName="cleanup"
Nov 03 20:00:55.098498 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.098509    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="40c57d6c-5b96-4713-8c12-c41c5f435a9b" containerName="cleanup"
Nov 03 20:00:55.098498 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.098514    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b2bf43f-4806-48b4-bd9b-1d6092b082e8" containerName="cleanup"
Nov 03 20:00:55.099502 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.098520    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="052a7a9f-ed4b-4cb9-911a-413079445a1c" containerName="istio-proxy"
Nov 03 20:00:55.099502 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.098526    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b2bf43f-4806-48b4-bd9b-1d6092b082e8" containerName="cleanup"
Nov 03 20:00:55.099502 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.098531    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="794c564b-dfea-4347-b7b7-10befa5ca787" containerName="cleanup"
Nov 03 20:00:55.099502 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.098537    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="951f6658-11ce-4b14-b615-31aacfd6b26e" containerName="cleanup"
Nov 03 20:00:55.099502 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.098542    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e8ef6e5-7bec-40e1-a363-b35cca7d4347" containerName="cleanup"
Nov 03 20:00:55.099502 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.098549    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7db24b6-7319-492c-bd17-05eab3aab9da" containerName="cleanup"
Nov 03 20:00:55.099502 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.098554    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="794c564b-dfea-4347-b7b7-10befa5ca787" containerName="cleanup"
Nov 03 20:00:55.099502 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.098561    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="951f6658-11ce-4b14-b615-31aacfd6b26e" containerName="cleanup"
Nov 03 20:00:55.102446 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.102420    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb"
Nov 03 20:00:55.115515 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.108956    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-a99fad-openshift-default-dockercfg-xgm4r\""
Nov 03 20:00:55.129024 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.128990    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb"]
Nov 03 20:00:55.186345 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.186304    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/052a7a9f-ed4b-4cb9-911a-413079445a1c-istio-token\" (UniqueName: \"kubernetes.io/projected/052a7a9f-ed4b-4cb9-911a-413079445a1c-istio-token\") pod \"052a7a9f-ed4b-4cb9-911a-413079445a1c\" (UID: \"052a7a9f-ed4b-4cb9-911a-413079445a1c\") "
Nov 03 20:00:55.186567 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.186377    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/052a7a9f-ed4b-4cb9-911a-413079445a1c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/052a7a9f-ed4b-4cb9-911a-413079445a1c-workload-certs\") pod \"052a7a9f-ed4b-4cb9-911a-413079445a1c\" (UID: \"052a7a9f-ed4b-4cb9-911a-413079445a1c\") "
Nov 03 20:00:55.186567 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.186456    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/052a7a9f-ed4b-4cb9-911a-413079445a1c-kube-api-access-9rndt\" (UniqueName: \"kubernetes.io/projected/052a7a9f-ed4b-4cb9-911a-413079445a1c-kube-api-access-9rndt\") pod \"052a7a9f-ed4b-4cb9-911a-413079445a1c\" (UID: \"052a7a9f-ed4b-4cb9-911a-413079445a1c\") "
Nov 03 20:00:55.186783 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.186741    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/052a7a9f-ed4b-4cb9-911a-413079445a1c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/052a7a9f-ed4b-4cb9-911a-413079445a1c-istio-podinfo\") pod \"052a7a9f-ed4b-4cb9-911a-413079445a1c\" (UID: \"052a7a9f-ed4b-4cb9-911a-413079445a1c\") "
Nov 03 20:00:55.186849 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.186834    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/052a7a9f-ed4b-4cb9-911a-413079445a1c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/052a7a9f-ed4b-4cb9-911a-413079445a1c-workload-socket\") pod \"052a7a9f-ed4b-4cb9-911a-413079445a1c\" (UID: \"052a7a9f-ed4b-4cb9-911a-413079445a1c\") "
Nov 03 20:00:55.186910 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.186868    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/052a7a9f-ed4b-4cb9-911a-413079445a1c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/052a7a9f-ed4b-4cb9-911a-413079445a1c-istio-data\") pod \"052a7a9f-ed4b-4cb9-911a-413079445a1c\" (UID: \"052a7a9f-ed4b-4cb9-911a-413079445a1c\") "
Nov 03 20:00:55.186966 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.186912    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/052a7a9f-ed4b-4cb9-911a-413079445a1c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/052a7a9f-ed4b-4cb9-911a-413079445a1c-credential-socket\") pod \"052a7a9f-ed4b-4cb9-911a-413079445a1c\" (UID: \"052a7a9f-ed4b-4cb9-911a-413079445a1c\") "
Nov 03 20:00:55.187025 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.186987    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/052a7a9f-ed4b-4cb9-911a-413079445a1c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/052a7a9f-ed4b-4cb9-911a-413079445a1c-istiod-ca-cert\") pod \"052a7a9f-ed4b-4cb9-911a-413079445a1c\" (UID: \"052a7a9f-ed4b-4cb9-911a-413079445a1c\") "
Nov 03 20:00:55.187079 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.187053    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/052a7a9f-ed4b-4cb9-911a-413079445a1c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/052a7a9f-ed4b-4cb9-911a-413079445a1c-istio-envoy\") pod \"052a7a9f-ed4b-4cb9-911a-413079445a1c\" (UID: \"052a7a9f-ed4b-4cb9-911a-413079445a1c\") "
Nov 03 20:00:55.187359 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.187325    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/052a7a9f-ed4b-4cb9-911a-413079445a1c-workload-socket" pod "052a7a9f-ed4b-4cb9-911a-413079445a1c" (UID: "052a7a9f-ed4b-4cb9-911a-413079445a1c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:00:55.187486 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.187429    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8423f401-70db-4f4a-9bf5-0597acb06be6-istio-data\") pod \"e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb\" (UID: \"8423f401-70db-4f4a-9bf5-0597acb06be6\") " pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb"
Nov 03 20:00:55.187486 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.187464    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfhjj\" (UniqueName: \"kubernetes.io/projected/8423f401-70db-4f4a-9bf5-0597acb06be6-kube-api-access-hfhjj\") pod \"e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb\" (UID: \"8423f401-70db-4f4a-9bf5-0597acb06be6\") " pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb"
Nov 03 20:00:55.187616 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.187569    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8423f401-70db-4f4a-9bf5-0597acb06be6-workload-socket\") pod \"e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb\" (UID: \"8423f401-70db-4f4a-9bf5-0597acb06be6\") " pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb"
Nov 03 20:00:55.187616 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.187580    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/052a7a9f-ed4b-4cb9-911a-413079445a1c-workload-certs" pod "052a7a9f-ed4b-4cb9-911a-413079445a1c" (UID: "052a7a9f-ed4b-4cb9-911a-413079445a1c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:00:55.187786 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.187722    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8423f401-70db-4f4a-9bf5-0597acb06be6-credential-socket\") pod \"e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb\" (UID: \"8423f401-70db-4f4a-9bf5-0597acb06be6\") " pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb"
Nov 03 20:00:55.187786 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.187780    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8423f401-70db-4f4a-9bf5-0597acb06be6-istio-token\") pod \"e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb\" (UID: \"8423f401-70db-4f4a-9bf5-0597acb06be6\") " pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb"
Nov 03 20:00:55.187902 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.187795    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/052a7a9f-ed4b-4cb9-911a-413079445a1c-credential-socket" pod "052a7a9f-ed4b-4cb9-911a-413079445a1c" (UID: "052a7a9f-ed4b-4cb9-911a-413079445a1c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:00:55.187902 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.187817    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8423f401-70db-4f4a-9bf5-0597acb06be6-workload-certs\") pod \"e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb\" (UID: \"8423f401-70db-4f4a-9bf5-0597acb06be6\") " pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb"
Nov 03 20:00:55.188011 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.187908    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8423f401-70db-4f4a-9bf5-0597acb06be6-istio-podinfo\") pod \"e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb\" (UID: \"8423f401-70db-4f4a-9bf5-0597acb06be6\") " pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb"
Nov 03 20:00:55.188011 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.187979    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/052a7a9f-ed4b-4cb9-911a-413079445a1c-istio-data" pod "052a7a9f-ed4b-4cb9-911a-413079445a1c" (UID: "052a7a9f-ed4b-4cb9-911a-413079445a1c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:00:55.188161 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.188104    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8423f401-70db-4f4a-9bf5-0597acb06be6-istiod-ca-cert\") pod \"e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb\" (UID: \"8423f401-70db-4f4a-9bf5-0597acb06be6\") " pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb"
Nov 03 20:00:55.188272 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.188254    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8423f401-70db-4f4a-9bf5-0597acb06be6-istio-envoy\") pod \"e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb\" (UID: \"8423f401-70db-4f4a-9bf5-0597acb06be6\") " pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb"
Nov 03 20:00:55.188343 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.188278    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/052a7a9f-ed4b-4cb9-911a-413079445a1c-istiod-ca-cert" pod "052a7a9f-ed4b-4cb9-911a-413079445a1c" (UID: "052a7a9f-ed4b-4cb9-911a-413079445a1c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:00:55.188539 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.188516    4921 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/052a7a9f-ed4b-4cb9-911a-413079445a1c-workload-socket\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:55.188633 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.188545    4921 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/052a7a9f-ed4b-4cb9-911a-413079445a1c-istio-data\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:55.188633 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.188561    4921 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/052a7a9f-ed4b-4cb9-911a-413079445a1c-credential-socket\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:55.188633 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.188576    4921 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/052a7a9f-ed4b-4cb9-911a-413079445a1c-istiod-ca-cert\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:55.188633 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.188592    4921 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/052a7a9f-ed4b-4cb9-911a-413079445a1c-workload-certs\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:55.189416 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.189390    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/052a7a9f-ed4b-4cb9-911a-413079445a1c-kube-api-access-9rndt" pod "052a7a9f-ed4b-4cb9-911a-413079445a1c" (UID: "052a7a9f-ed4b-4cb9-911a-413079445a1c"). InnerVolumeSpecName "kube-api-access-9rndt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:00:55.189876 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.189847    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/052a7a9f-ed4b-4cb9-911a-413079445a1c-istio-envoy" pod "052a7a9f-ed4b-4cb9-911a-413079445a1c" (UID: "052a7a9f-ed4b-4cb9-911a-413079445a1c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:00:55.190038 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.190018    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/052a7a9f-ed4b-4cb9-911a-413079445a1c-istio-token" pod "052a7a9f-ed4b-4cb9-911a-413079445a1c" (UID: "052a7a9f-ed4b-4cb9-911a-413079445a1c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:00:55.190369 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.190349    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/052a7a9f-ed4b-4cb9-911a-413079445a1c-istio-podinfo" pod "052a7a9f-ed4b-4cb9-911a-413079445a1c" (UID: "052a7a9f-ed4b-4cb9-911a-413079445a1c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:00:55.290481 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.290442    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8423f401-70db-4f4a-9bf5-0597acb06be6-istiod-ca-cert\") pod \"e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb\" (UID: \"8423f401-70db-4f4a-9bf5-0597acb06be6\") " pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb"
Nov 03 20:00:55.290650 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.290546    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8423f401-70db-4f4a-9bf5-0597acb06be6-istio-envoy\") pod \"e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb\" (UID: \"8423f401-70db-4f4a-9bf5-0597acb06be6\") " pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb"
Nov 03 20:00:55.290650 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.290619    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8423f401-70db-4f4a-9bf5-0597acb06be6-istio-data\") pod \"e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb\" (UID: \"8423f401-70db-4f4a-9bf5-0597acb06be6\") " pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb"
Nov 03 20:00:55.290650 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.290641    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfhjj\" (UniqueName: \"kubernetes.io/projected/8423f401-70db-4f4a-9bf5-0597acb06be6-kube-api-access-hfhjj\") pod \"e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb\" (UID: \"8423f401-70db-4f4a-9bf5-0597acb06be6\") " pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb"
Nov 03 20:00:55.290795 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.290692    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8423f401-70db-4f4a-9bf5-0597acb06be6-workload-socket\") pod \"e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb\" (UID: \"8423f401-70db-4f4a-9bf5-0597acb06be6\") " pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb"
Nov 03 20:00:55.290795 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.290735    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8423f401-70db-4f4a-9bf5-0597acb06be6-credential-socket\") pod \"e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb\" (UID: \"8423f401-70db-4f4a-9bf5-0597acb06be6\") " pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb"
Nov 03 20:00:55.290897 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.290798    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8423f401-70db-4f4a-9bf5-0597acb06be6-istio-token\") pod \"e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb\" (UID: \"8423f401-70db-4f4a-9bf5-0597acb06be6\") " pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb"
Nov 03 20:00:55.290897 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.290829    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8423f401-70db-4f4a-9bf5-0597acb06be6-workload-certs\") pod \"e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb\" (UID: \"8423f401-70db-4f4a-9bf5-0597acb06be6\") " pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb"
Nov 03 20:00:55.291006 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.290913    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8423f401-70db-4f4a-9bf5-0597acb06be6-istio-podinfo\") pod \"e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb\" (UID: \"8423f401-70db-4f4a-9bf5-0597acb06be6\") " pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb"
Nov 03 20:00:55.291058 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.291045    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8423f401-70db-4f4a-9bf5-0597acb06be6-istio-data\") pod \"e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb\" (UID: \"8423f401-70db-4f4a-9bf5-0597acb06be6\") " pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb"
Nov 03 20:00:55.291118 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.291072    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rndt\" (UniqueName: \"kubernetes.io/projected/052a7a9f-ed4b-4cb9-911a-413079445a1c-kube-api-access-9rndt\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:55.291118 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.291089    4921 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/052a7a9f-ed4b-4cb9-911a-413079445a1c-istio-podinfo\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:55.291118 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.291105    4921 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/052a7a9f-ed4b-4cb9-911a-413079445a1c-istio-envoy\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:55.291118 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.291117    4921 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/052a7a9f-ed4b-4cb9-911a-413079445a1c-istio-token\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:55.291331 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.291156    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8423f401-70db-4f4a-9bf5-0597acb06be6-workload-socket\") pod \"e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb\" (UID: \"8423f401-70db-4f4a-9bf5-0597acb06be6\") " pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb"
Nov 03 20:00:55.291331 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.291301    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8423f401-70db-4f4a-9bf5-0597acb06be6-istiod-ca-cert\") pod \"e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb\" (UID: \"8423f401-70db-4f4a-9bf5-0597acb06be6\") " pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb"
Nov 03 20:00:55.291442 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.291357    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8423f401-70db-4f4a-9bf5-0597acb06be6-workload-certs\") pod \"e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb\" (UID: \"8423f401-70db-4f4a-9bf5-0597acb06be6\") " pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb"
Nov 03 20:00:55.291648 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.291628    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8423f401-70db-4f4a-9bf5-0597acb06be6-credential-socket\") pod \"e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb\" (UID: \"8423f401-70db-4f4a-9bf5-0597acb06be6\") " pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb"
Nov 03 20:00:55.293036 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.293010    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8423f401-70db-4f4a-9bf5-0597acb06be6-istio-envoy\") pod \"e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb\" (UID: \"8423f401-70db-4f4a-9bf5-0597acb06be6\") " pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb"
Nov 03 20:00:55.293877 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.293860    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8423f401-70db-4f4a-9bf5-0597acb06be6-istio-podinfo\") pod \"e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb\" (UID: \"8423f401-70db-4f4a-9bf5-0597acb06be6\") " pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb"
Nov 03 20:00:55.299090 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.299072    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8423f401-70db-4f4a-9bf5-0597acb06be6-istio-token\") pod \"e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb\" (UID: \"8423f401-70db-4f4a-9bf5-0597acb06be6\") " pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb"
Nov 03 20:00:55.299278 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.299260    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfhjj\" (UniqueName: \"kubernetes.io/projected/8423f401-70db-4f4a-9bf5-0597acb06be6-kube-api-access-hfhjj\") pod \"e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb\" (UID: \"8423f401-70db-4f4a-9bf5-0597acb06be6\") " pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb"
Nov 03 20:00:55.333512 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.333417    4921 generic.go:356] "Generic (PLEG): container finished" podID="052a7a9f-ed4b-4cb9-911a-413079445a1c" containerID="fa3fdaac1525b4a72fe74d752a4613a5148615674024627d6ecf2e38ae7084e3" exitCode=0
Nov 03 20:00:55.333512 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.333506    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6"
Nov 03 20:00:55.333749 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.333504    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6" event={"ID":"052a7a9f-ed4b-4cb9-911a-413079445a1c","Type":"ContainerDied","Data":"fa3fdaac1525b4a72fe74d752a4613a5148615674024627d6ecf2e38ae7084e3"}
Nov 03 20:00:55.333749 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.333613    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6" event={"ID":"052a7a9f-ed4b-4cb9-911a-413079445a1c","Type":"ContainerDied","Data":"6ba7d9163a33a22610bd4b5bbc2bc1abe348b35b383e852f7db1f507b6e95bc1"}
Nov 03 20:00:55.333749 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.333646    4921 scope.go:122] "RemoveContainer" containerID="fa3fdaac1525b4a72fe74d752a4613a5148615674024627d6ecf2e38ae7084e3"
Nov 03 20:00:55.347786 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.347690    4921 scope.go:122] "RemoveContainer" containerID="fa3fdaac1525b4a72fe74d752a4613a5148615674024627d6ecf2e38ae7084e3"
Nov 03 20:00:55.348084 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:00:55.348057    4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa3fdaac1525b4a72fe74d752a4613a5148615674024627d6ecf2e38ae7084e3\": container with ID starting with fa3fdaac1525b4a72fe74d752a4613a5148615674024627d6ecf2e38ae7084e3 not found: ID does not exist" containerID="fa3fdaac1525b4a72fe74d752a4613a5148615674024627d6ecf2e38ae7084e3"
Nov 03 20:00:55.348151 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.348095    4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa3fdaac1525b4a72fe74d752a4613a5148615674024627d6ecf2e38ae7084e3"} err="failed to get container status \"fa3fdaac1525b4a72fe74d752a4613a5148615674024627d6ecf2e38ae7084e3\": rpc error: code = NotFound desc = could not find container \"fa3fdaac1525b4a72fe74d752a4613a5148615674024627d6ecf2e38ae7084e3\": container with ID starting with fa3fdaac1525b4a72fe74d752a4613a5148615674024627d6ecf2e38ae7084e3 not found: ID does not exist"
Nov 03 20:00:55.362327 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.362299    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6"]
Nov 03 20:00:55.367231 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.367200    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-fqxh6"]
Nov 03 20:00:55.427862 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.427826    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb"
Nov 03 20:00:55.721390 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.721357    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb"]
Nov 03 20:00:55.721647 ip-10-0-134-222 kubenswrapper[4921]: W1103 20:00:55.721626    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8423f401_70db_4f4a_9bf5_0597acb06be6.slice/crio-604d7ee62ad8c7e009724ec948831a9434eb1e273d1fdb5c367da56bc717ba51 WatchSource:0}: Error finding container 604d7ee62ad8c7e009724ec948831a9434eb1e273d1fdb5c367da56bc717ba51: Status 404 returned error can't find the container with id 604d7ee62ad8c7e009724ec948831a9434eb1e273d1fdb5c367da56bc717ba51
Nov 03 20:00:55.724894 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.724852    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:00:55.726087 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.726060    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:00:55.726169 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:55.726117    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:00:56.345891 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:56.345856    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb" event={"ID":"8423f401-70db-4f4a-9bf5-0597acb06be6","Type":"ContainerStarted","Data":"01974174fc1b417c37ea4625b302d901dba257d096d3d9455d313e3bff633938"}
Nov 03 20:00:56.346376 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:56.345920    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb" event={"ID":"8423f401-70db-4f4a-9bf5-0597acb06be6","Type":"ContainerStarted","Data":"604d7ee62ad8c7e009724ec948831a9434eb1e273d1fdb5c367da56bc717ba51"}
Nov 03 20:00:56.367892 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:56.367835    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb" podStartSLOduration=25.36781385 podStartE2EDuration="25.36781385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:56.362524964 +0000 UTC m=+1769.606490430" watchObservedRunningTime="2026-11-03 20:00:56.36781385 +0000 UTC m=+1769.611779317"
Nov 03 20:00:56.446199 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:56.444037    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb"
Nov 03 20:00:56.460713 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:56.452831    4921 patch_prober.go:29] interesting pod/e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body=
Nov 03 20:00:56.460713 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:56.452906    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb" podUID="8423f401-70db-4f4a-9bf5-0597acb06be6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused"
Nov 03 20:00:56.891864 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:56.891830    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="052a7a9f-ed4b-4cb9-911a-413079445a1c" path="/var/lib/kubelet/pods/052a7a9f-ed4b-4cb9-911a-413079445a1c/volumes"
Nov 03 20:00:57.428514 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:57.428479    4921 patch_prober.go:29] interesting pod/e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body=
Nov 03 20:00:57.429018 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:57.428539    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb" podUID="8423f401-70db-4f4a-9bf5-0597acb06be6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused"
Nov 03 20:00:58.428169 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:58.428134    4921 patch_prober.go:29] interesting pod/e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body=
Nov 03 20:00:58.428351 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:58.428185    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb" podUID="8423f401-70db-4f4a-9bf5-0597acb06be6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused"
Nov 03 20:00:59.557692 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:59.557280    4921 patch_prober.go:29] interesting pod/e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body=
Nov 03 20:00:59.557692 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:00:59.557339    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb" podUID="8423f401-70db-4f4a-9bf5-0597acb06be6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused"
Nov 03 20:01:00.449179 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:00.435330    4921 patch_prober.go:29] interesting pod/e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body=
Nov 03 20:01:00.449179 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:00.435394    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb" podUID="8423f401-70db-4f4a-9bf5-0597acb06be6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused"
Nov 03 20:01:01.428529 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:01.428492    4921 patch_prober.go:29] interesting pod/e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body=
Nov 03 20:01:01.428966 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:01.428561    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb" podUID="8423f401-70db-4f4a-9bf5-0597acb06be6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused"
Nov 03 20:01:02.428435 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:02.428399    4921 patch_prober.go:29] interesting pod/e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body=
Nov 03 20:01:02.428655 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:02.428476    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb" podUID="8423f401-70db-4f4a-9bf5-0597acb06be6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused"
Nov 03 20:01:02.994081 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:01:02.993936    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 20:01:03.433655 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:03.433571    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb"
Nov 03 20:01:03.434140 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:03.434033    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb"
Nov 03 20:01:03.434790 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:03.434773    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb"
Nov 03 20:01:15.550879 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:15.550842    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bdb694-57f5479559-fv5m8"]
Nov 03 20:01:15.551314 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:15.551060    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bdb694-57f5479559-fv5m8" podUID="e75a177a-f6a6-49c7-87a2-297154d3b7f3" containerName="maas-api" containerID="cri-o://98ee120bff9dc8970104391f9853b68b175605463c2d423f4f2bc857e416c072" gracePeriod=30
Nov 03 20:01:16.050690 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:16.050249    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bdb694-57f5479559-fv5m8" podUID="e75a177a-f6a6-49c7-87a2-297154d3b7f3" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.139:8443/health\": dial tcp 10.132.0.139:8443: connect: connection refused"
Nov 03 20:01:16.529413 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:16.529385    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bdb694-57f5479559-fv5m8"
Nov 03 20:01:16.561015 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:16.560931    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d"]
Nov 03 20:01:16.562279 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:16.562250    4921 cpu_manager.go:414] "containerMap: removing container" podUID="40c57d6c-5b96-4713-8c12-c41c5f435a9b" containerName="cleanup"
Nov 03 20:01:16.562279 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:16.562276    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40c57d6c-5b96-4713-8c12-c41c5f435a9b" containerName="cleanup"
Nov 03 20:01:16.562476 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:16.562328    4921 cpu_manager.go:414] "containerMap: removing container" podUID="a7db24b6-7319-492c-bd17-05eab3aab9da" containerName="cleanup"
Nov 03 20:01:16.562476 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:16.562336    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7db24b6-7319-492c-bd17-05eab3aab9da" containerName="cleanup"
Nov 03 20:01:16.562476 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:16.562349    4921 cpu_manager.go:414] "containerMap: removing container" podUID="e75a177a-f6a6-49c7-87a2-297154d3b7f3" containerName="maas-api"
Nov 03 20:01:16.562476 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:16.562356    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e75a177a-f6a6-49c7-87a2-297154d3b7f3" containerName="maas-api"
Nov 03 20:01:16.562476 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:16.562366    4921 cpu_manager.go:414] "containerMap: removing container" podUID="794c564b-dfea-4347-b7b7-10befa5ca787" containerName="cleanup"
Nov 03 20:01:16.562476 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:16.562372    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="794c564b-dfea-4347-b7b7-10befa5ca787" containerName="cleanup"
Nov 03 20:01:16.562476 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:16.562384    4921 cpu_manager.go:414] "containerMap: removing container" podUID="5769966d-308a-44a9-928a-984d95d95820" containerName="cleanup"
Nov 03 20:01:16.562476 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:16.562391    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5769966d-308a-44a9-928a-984d95d95820" containerName="cleanup"
Nov 03 20:01:16.562476 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:16.562404    4921 cpu_manager.go:414] "containerMap: removing container" podUID="951f6658-11ce-4b14-b615-31aacfd6b26e" containerName="cleanup"
Nov 03 20:01:16.562476 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:16.562411    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="951f6658-11ce-4b14-b615-31aacfd6b26e" containerName="cleanup"
Nov 03 20:01:16.562476 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:16.562426    4921 cpu_manager.go:414] "containerMap: removing container" podUID="7b2bf43f-4806-48b4-bd9b-1d6092b082e8" containerName="cleanup"
Nov 03 20:01:16.562476 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:16.562434    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b2bf43f-4806-48b4-bd9b-1d6092b082e8" containerName="cleanup"
Nov 03 20:01:16.563056 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:16.562680    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7db24b6-7319-492c-bd17-05eab3aab9da" containerName="cleanup"
Nov 03 20:01:16.563056 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:16.562700    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e8ef6e5-7bec-40e1-a363-b35cca7d4347" containerName="cleanup"
Nov 03 20:01:16.563056 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:16.562720    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="5769966d-308a-44a9-928a-984d95d95820" containerName="cleanup"
Nov 03 20:01:16.563056 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:16.562740    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="e75a177a-f6a6-49c7-87a2-297154d3b7f3" containerName="maas-api"
Nov 03 20:01:16.566956 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:16.566935    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d"
Nov 03 20:01:16.583437 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:16.583395    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d"]
Nov 03 20:01:16.583878 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:16.583850    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-f58171-openshift-default-dockercfg-kj42z\""
Nov 03 20:01:16.674340 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:16.672889    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e75a177a-f6a6-49c7-87a2-297154d3b7f3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e75a177a-f6a6-49c7-87a2-297154d3b7f3-maas-api-tls\") pod \"e75a177a-f6a6-49c7-87a2-297154d3b7f3\" (UID: \"e75a177a-f6a6-49c7-87a2-297154d3b7f3\") "
Nov 03 20:01:16.674340 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:16.673093    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e75a177a-f6a6-49c7-87a2-297154d3b7f3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e75a177a-f6a6-49c7-87a2-297154d3b7f3-openshift-service-ca\") pod \"e75a177a-f6a6-49c7-87a2-297154d3b7f3\" (UID: \"e75a177a-f6a6-49c7-87a2-297154d3b7f3\") "
Nov 03 20:01:16.674340 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:16.673254    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e75a177a-f6a6-49c7-87a2-297154d3b7f3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e75a177a-f6a6-49c7-87a2-297154d3b7f3-metrics-certs\") pod \"e75a177a-f6a6-49c7-87a2-297154d3b7f3\" (UID: \"e75a177a-f6a6-49c7-87a2-297154d3b7f3\") "
Nov 03 20:01:16.674340 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:16.673297    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e75a177a-f6a6-49c7-87a2-297154d3b7f3-kube-api-access-lh8k7\" (UniqueName: \"kubernetes.io/projected/e75a177a-f6a6-49c7-87a2-297154d3b7f3-kube-api-access-lh8k7\") pod \"e75a177a-f6a6-49c7-87a2-297154d3b7f3\" (UID: \"e75a177a-f6a6-49c7-87a2-297154d3b7f3\") "
Nov 03 20:01:16.674340 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:16.673524    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33d97e3e-9c11-4ca2-bddd-b2c057ab41e9-credential-socket\") pod \"e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d\" (UID: \"33d97e3e-9c11-4ca2-bddd-b2c057ab41e9\") " pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d"
Nov 03 20:01:16.674340 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:16.673563    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33d97e3e-9c11-4ca2-bddd-b2c057ab41e9-istiod-ca-cert\") pod \"e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d\" (UID: \"33d97e3e-9c11-4ca2-bddd-b2c057ab41e9\") " pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d"
Nov 03 20:01:16.674340 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:16.673618    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33d97e3e-9c11-4ca2-bddd-b2c057ab41e9-istio-envoy\") pod \"e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d\" (UID: \"33d97e3e-9c11-4ca2-bddd-b2c057ab41e9\") " pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d"
Nov 03 20:01:16.674340 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:16.673699    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33d97e3e-9c11-4ca2-bddd-b2c057ab41e9-istio-token\") pod \"e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d\" (UID: \"33d97e3e-9c11-4ca2-bddd-b2c057ab41e9\") " pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d"
Nov 03 20:01:16.674340 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:16.673836    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf6jt\" (UniqueName: \"kubernetes.io/projected/33d97e3e-9c11-4ca2-bddd-b2c057ab41e9-kube-api-access-tf6jt\") pod \"e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d\" (UID: \"33d97e3e-9c11-4ca2-bddd-b2c057ab41e9\") " pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d"
Nov 03 20:01:16.674340 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:16.673930    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33d97e3e-9c11-4ca2-bddd-b2c057ab41e9-istio-podinfo\") pod \"e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d\" (UID: \"33d97e3e-9c11-4ca2-bddd-b2c057ab41e9\") " pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d"
Nov 03 20:01:16.674340 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:16.674067    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33d97e3e-9c11-4ca2-bddd-b2c057ab41e9-workload-certs\") pod \"e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d\" (UID: \"33d97e3e-9c11-4ca2-bddd-b2c057ab41e9\") " pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d"
Nov 03 20:01:16.674340 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:16.674090    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33d97e3e-9c11-4ca2-bddd-b2c057ab41e9-istio-data\") pod \"e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d\" (UID: \"33d97e3e-9c11-4ca2-bddd-b2c057ab41e9\") " pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d"
Nov 03 20:01:16.674340 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:16.674211    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33d97e3e-9c11-4ca2-bddd-b2c057ab41e9-workload-socket\") pod \"e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d\" (UID: \"33d97e3e-9c11-4ca2-bddd-b2c057ab41e9\") " pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d"
Nov 03 20:01:16.675257 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:16.674991    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e75a177a-f6a6-49c7-87a2-297154d3b7f3-openshift-service-ca" pod "e75a177a-f6a6-49c7-87a2-297154d3b7f3" (UID: "e75a177a-f6a6-49c7-87a2-297154d3b7f3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:01:16.678564 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:16.678537    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e75a177a-f6a6-49c7-87a2-297154d3b7f3-metrics-certs" pod "e75a177a-f6a6-49c7-87a2-297154d3b7f3" (UID: "e75a177a-f6a6-49c7-87a2-297154d3b7f3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:01:16.679275 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:16.679244    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e75a177a-f6a6-49c7-87a2-297154d3b7f3-maas-api-tls" pod "e75a177a-f6a6-49c7-87a2-297154d3b7f3" (UID: "e75a177a-f6a6-49c7-87a2-297154d3b7f3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:01:16.680018 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:16.679985    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e75a177a-f6a6-49c7-87a2-297154d3b7f3-kube-api-access-lh8k7" pod "e75a177a-f6a6-49c7-87a2-297154d3b7f3" (UID: "e75a177a-f6a6-49c7-87a2-297154d3b7f3"). InnerVolumeSpecName "kube-api-access-lh8k7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:01:16.775846 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:16.775807    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33d97e3e-9c11-4ca2-bddd-b2c057ab41e9-workload-socket\") pod \"e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d\" (UID: \"33d97e3e-9c11-4ca2-bddd-b2c057ab41e9\") " pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d"
Nov 03 20:01:16.775846 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:16.775853    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33d97e3e-9c11-4ca2-bddd-b2c057ab41e9-credential-socket\") pod \"e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d\" (UID: \"33d97e3e-9c11-4ca2-bddd-b2c057ab41e9\") " pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d"
Nov 03 20:01:16.776104 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:16.775875    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33d97e3e-9c11-4ca2-bddd-b2c057ab41e9-istiod-ca-cert\") pod \"e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d\" (UID: \"33d97e3e-9c11-4ca2-bddd-b2c057ab41e9\") " pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d"
Nov 03 20:01:16.776104 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:16.776006    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33d97e3e-9c11-4ca2-bddd-b2c057ab41e9-istio-envoy\") pod \"e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d\" (UID: \"33d97e3e-9c11-4ca2-bddd-b2c057ab41e9\") " pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d"
Nov 03 20:01:16.776104 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:16.776078    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33d97e3e-9c11-4ca2-bddd-b2c057ab41e9-istio-token\") pod \"e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d\" (UID: \"33d97e3e-9c11-4ca2-bddd-b2c057ab41e9\") " pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d"
Nov 03 20:01:16.776266 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:16.776215    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf6jt\" (UniqueName: \"kubernetes.io/projected/33d97e3e-9c11-4ca2-bddd-b2c057ab41e9-kube-api-access-tf6jt\") pod \"e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d\" (UID: \"33d97e3e-9c11-4ca2-bddd-b2c057ab41e9\") " pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d"
Nov 03 20:01:16.776333 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:16.776259    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33d97e3e-9c11-4ca2-bddd-b2c057ab41e9-credential-socket\") pod \"e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d\" (UID: \"33d97e3e-9c11-4ca2-bddd-b2c057ab41e9\") " pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d"
Nov 03 20:01:16.776333 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:16.776324    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33d97e3e-9c11-4ca2-bddd-b2c057ab41e9-istio-podinfo\") pod \"e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d\" (UID: \"33d97e3e-9c11-4ca2-bddd-b2c057ab41e9\") " pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d"
Nov 03 20:01:16.776445 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:16.776354    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33d97e3e-9c11-4ca2-bddd-b2c057ab41e9-workload-socket\") pod \"e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d\" (UID: \"33d97e3e-9c11-4ca2-bddd-b2c057ab41e9\") " pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d"
Nov 03 20:01:16.776514 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:16.776493    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33d97e3e-9c11-4ca2-bddd-b2c057ab41e9-workload-certs\") pod \"e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d\" (UID: \"33d97e3e-9c11-4ca2-bddd-b2c057ab41e9\") " pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d"
Nov 03 20:01:16.776573 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:16.776528    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33d97e3e-9c11-4ca2-bddd-b2c057ab41e9-istio-data\") pod \"e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d\" (UID: \"33d97e3e-9c11-4ca2-bddd-b2c057ab41e9\") " pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d"
Nov 03 20:01:16.776727 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:16.776704    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e75a177a-f6a6-49c7-87a2-297154d3b7f3-maas-api-tls\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:16.776813 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:16.776727    4921 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e75a177a-f6a6-49c7-87a2-297154d3b7f3-openshift-service-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:16.776813 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:16.776742    4921 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e75a177a-f6a6-49c7-87a2-297154d3b7f3-metrics-certs\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:16.776813 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:16.776755    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lh8k7\" (UniqueName: \"kubernetes.io/projected/e75a177a-f6a6-49c7-87a2-297154d3b7f3-kube-api-access-lh8k7\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:16.776813 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:16.776753    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33d97e3e-9c11-4ca2-bddd-b2c057ab41e9-istiod-ca-cert\") pod \"e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d\" (UID: \"33d97e3e-9c11-4ca2-bddd-b2c057ab41e9\") " pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d"
Nov 03 20:01:16.776957 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:16.776851    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33d97e3e-9c11-4ca2-bddd-b2c057ab41e9-workload-certs\") pod \"e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d\" (UID: \"33d97e3e-9c11-4ca2-bddd-b2c057ab41e9\") " pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d"
Nov 03 20:01:16.776957 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:16.776925    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33d97e3e-9c11-4ca2-bddd-b2c057ab41e9-istio-data\") pod \"e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d\" (UID: \"33d97e3e-9c11-4ca2-bddd-b2c057ab41e9\") " pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d"
Nov 03 20:01:16.778482 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:16.778458    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33d97e3e-9c11-4ca2-bddd-b2c057ab41e9-istio-envoy\") pod \"e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d\" (UID: \"33d97e3e-9c11-4ca2-bddd-b2c057ab41e9\") " pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d"
Nov 03 20:01:16.778922 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:16.778904    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33d97e3e-9c11-4ca2-bddd-b2c057ab41e9-istio-podinfo\") pod \"e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d\" (UID: \"33d97e3e-9c11-4ca2-bddd-b2c057ab41e9\") " pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d"
Nov 03 20:01:16.783180 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:16.783153    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33d97e3e-9c11-4ca2-bddd-b2c057ab41e9-istio-token\") pod \"e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d\" (UID: \"33d97e3e-9c11-4ca2-bddd-b2c057ab41e9\") " pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d"
Nov 03 20:01:16.783331 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:16.783310    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf6jt\" (UniqueName: \"kubernetes.io/projected/33d97e3e-9c11-4ca2-bddd-b2c057ab41e9-kube-api-access-tf6jt\") pod \"e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d\" (UID: \"33d97e3e-9c11-4ca2-bddd-b2c057ab41e9\") " pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d"
Nov 03 20:01:16.911286 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:16.911256    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d"
Nov 03 20:01:16.961473 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:16.961431    4921 generic.go:356] "Generic (PLEG): container finished" podID="e75a177a-f6a6-49c7-87a2-297154d3b7f3" containerID="98ee120bff9dc8970104391f9853b68b175605463c2d423f4f2bc857e416c072" exitCode=0
Nov 03 20:01:16.961827 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:16.961802    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bdb694-57f5479559-fv5m8" event={"ID":"e75a177a-f6a6-49c7-87a2-297154d3b7f3","Type":"ContainerDied","Data":"98ee120bff9dc8970104391f9853b68b175605463c2d423f4f2bc857e416c072"}
Nov 03 20:01:16.961899 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:16.961863    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bdb694-57f5479559-fv5m8" event={"ID":"e75a177a-f6a6-49c7-87a2-297154d3b7f3","Type":"ContainerDied","Data":"16112ef31c12ef818658d1e813c51e43b958e35b95fe0fe12fc32c8b0403da9d"}
Nov 03 20:01:16.961948 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:16.961896    4921 scope.go:122] "RemoveContainer" containerID="98ee120bff9dc8970104391f9853b68b175605463c2d423f4f2bc857e416c072"
Nov 03 20:01:16.962103 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:16.962087    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bdb694-57f5479559-fv5m8"
Nov 03 20:01:17.003169 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:17.003139    4921 scope.go:122] "RemoveContainer" containerID="98ee120bff9dc8970104391f9853b68b175605463c2d423f4f2bc857e416c072"
Nov 03 20:01:17.005957 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:01:17.005894    4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98ee120bff9dc8970104391f9853b68b175605463c2d423f4f2bc857e416c072\": container with ID starting with 98ee120bff9dc8970104391f9853b68b175605463c2d423f4f2bc857e416c072 not found: ID does not exist" containerID="98ee120bff9dc8970104391f9853b68b175605463c2d423f4f2bc857e416c072"
Nov 03 20:01:17.006089 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:17.005962    4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98ee120bff9dc8970104391f9853b68b175605463c2d423f4f2bc857e416c072"} err="failed to get container status \"98ee120bff9dc8970104391f9853b68b175605463c2d423f4f2bc857e416c072\": rpc error: code = NotFound desc = could not find container \"98ee120bff9dc8970104391f9853b68b175605463c2d423f4f2bc857e416c072\": container with ID starting with 98ee120bff9dc8970104391f9853b68b175605463c2d423f4f2bc857e416c072 not found: ID does not exist"
Nov 03 20:01:17.011462 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:17.011435    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bdb694-57f5479559-fv5m8"]
Nov 03 20:01:17.026382 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:17.026345    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bdb694-57f5479559-fv5m8"]
Nov 03 20:01:17.216427 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:17.208499    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d"]
Nov 03 20:01:17.262403 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:17.262358    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:01:17.262572 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:17.262455    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:01:17.262572 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:17.262504    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:01:18.038203 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:18.038155    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d" event={"ID":"33d97e3e-9c11-4ca2-bddd-b2c057ab41e9","Type":"ContainerStarted","Data":"30b912486eeaf1f1f3fb846ff6d8148564788180791de064cb5a9585d7217c28"}
Nov 03 20:01:18.040533 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:18.040437    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d" event={"ID":"33d97e3e-9c11-4ca2-bddd-b2c057ab41e9","Type":"ContainerStarted","Data":"ea16b2d9ea06d43399d0411fd14dcd3c345e154357fbb5838e4db96c688b982c"}
Nov 03 20:01:18.890843 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:18.890807    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e75a177a-f6a6-49c7-87a2-297154d3b7f3" path="/var/lib/kubelet/pods/e75a177a-f6a6-49c7-87a2-297154d3b7f3/volumes"
Nov 03 20:01:18.911939 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:18.911911    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d"
Nov 03 20:01:18.913809 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:18.913785    4921 patch_prober.go:29] interesting pod/e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Nov 03 20:01:18.913938 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:18.913847    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d" podUID="33d97e3e-9c11-4ca2-bddd-b2c057ab41e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Nov 03 20:01:19.918026 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:19.917989    4921 patch_prober.go:29] interesting pod/e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Nov 03 20:01:19.918609 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:19.918574    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d" podUID="33d97e3e-9c11-4ca2-bddd-b2c057ab41e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Nov 03 20:01:20.912058 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:20.912020    4921 patch_prober.go:29] interesting pod/e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Nov 03 20:01:20.912241 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:20.912077    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d" podUID="33d97e3e-9c11-4ca2-bddd-b2c057ab41e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Nov 03 20:01:21.912112 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:21.912075    4921 patch_prober.go:29] interesting pod/e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Nov 03 20:01:21.912611 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:21.912147    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d" podUID="33d97e3e-9c11-4ca2-bddd-b2c057ab41e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Nov 03 20:01:22.922689 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:22.919649    4921 patch_prober.go:29] interesting pod/e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Nov 03 20:01:22.922689 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:22.919743    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d" podUID="33d97e3e-9c11-4ca2-bddd-b2c057ab41e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Nov 03 20:01:23.701530 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:23.701442    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d" podStartSLOduration=136.701418676 podStartE2EDuration="2m16.701418676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:59:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:18.076125455 +0000 UTC m=+1791.320090922" watchObservedRunningTime="2026-11-03 20:01:23.701418676 +0000 UTC m=+1796.945384142"
Nov 03 20:01:23.709128 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:23.709073    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-332694-8c85s"]
Nov 03 20:01:23.723435 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:23.723403    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-332694-8c85s"
Nov 03 20:01:23.740796 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:23.740761    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-332694-8c85s"]
Nov 03 20:01:23.741426 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:23.741402    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:01:23.741867 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:23.741848    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 20:01:23.873908 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:23.873869    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtdwt\" (UniqueName: \"kubernetes.io/projected/6e610d24-87d3-4b3a-936d-16bce9c36b94-kube-api-access-vtdwt\") pod \"maas-api-revoke-keys-e2e-inf-b-332694-8c85s\" (UID: \"6e610d24-87d3-4b3a-936d-16bce9c36b94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-332694-8c85s"
Nov 03 20:01:23.874102 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:23.874078    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e610d24-87d3-4b3a-936d-16bce9c36b94-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-332694-8c85s\" (UID: \"6e610d24-87d3-4b3a-936d-16bce9c36b94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-332694-8c85s"
Nov 03 20:01:23.874232 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:23.874218    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e610d24-87d3-4b3a-936d-16bce9c36b94-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-332694-8c85s\" (UID: \"6e610d24-87d3-4b3a-936d-16bce9c36b94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-332694-8c85s"
Nov 03 20:01:23.912011 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:23.911973    4921 patch_prober.go:29] interesting pod/e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Nov 03 20:01:23.912181 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:23.912037    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d" podUID="33d97e3e-9c11-4ca2-bddd-b2c057ab41e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Nov 03 20:01:23.976610 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:23.975939    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e610d24-87d3-4b3a-936d-16bce9c36b94-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-332694-8c85s\" (UID: \"6e610d24-87d3-4b3a-936d-16bce9c36b94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-332694-8c85s"
Nov 03 20:01:23.976610 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:23.976135    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e610d24-87d3-4b3a-936d-16bce9c36b94-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-332694-8c85s\" (UID: \"6e610d24-87d3-4b3a-936d-16bce9c36b94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-332694-8c85s"
Nov 03 20:01:23.976610 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:23.976292    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtdwt\" (UniqueName: \"kubernetes.io/projected/6e610d24-87d3-4b3a-936d-16bce9c36b94-kube-api-access-vtdwt\") pod \"maas-api-revoke-keys-e2e-inf-b-332694-8c85s\" (UID: \"6e610d24-87d3-4b3a-936d-16bce9c36b94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-332694-8c85s"
Nov 03 20:01:23.977222 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:23.976846    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e610d24-87d3-4b3a-936d-16bce9c36b94-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-332694-8c85s\" (UID: \"6e610d24-87d3-4b3a-936d-16bce9c36b94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-332694-8c85s"
Nov 03 20:01:23.978976 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:23.978949    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e610d24-87d3-4b3a-936d-16bce9c36b94-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-332694-8c85s\" (UID: \"6e610d24-87d3-4b3a-936d-16bce9c36b94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-332694-8c85s"
Nov 03 20:01:23.988951 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:23.988919    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtdwt\" (UniqueName: \"kubernetes.io/projected/6e610d24-87d3-4b3a-936d-16bce9c36b94-kube-api-access-vtdwt\") pod \"maas-api-revoke-keys-e2e-inf-b-332694-8c85s\" (UID: \"6e610d24-87d3-4b3a-936d-16bce9c36b94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-332694-8c85s"
Nov 03 20:01:24.105186 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:24.105147    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-332694-8c85s"
Nov 03 20:01:24.516231 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:24.516204    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-332694-8c85s"]
Nov 03 20:01:24.917810 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:24.917775    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d"
Nov 03 20:01:24.918259 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:24.918234    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d"
Nov 03 20:01:24.919062 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:24.919040    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d"
Nov 03 20:01:25.162813 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:25.162715    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-332694-8c85s" event={"ID":"6e610d24-87d3-4b3a-936d-16bce9c36b94","Type":"ContainerStarted","Data":"8464e2562ac52ee9e1b79c576735ff42e63ff6e81a1fda86a5bb26c1888b30c7"}
Nov 03 20:01:25.162813 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:25.162779    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-332694-8c85s" event={"ID":"6e610d24-87d3-4b3a-936d-16bce9c36b94","Type":"ContainerStarted","Data":"8c67354455cf53b1a8118aa6ecaae7c94344d2e03fbfc365fd22ae991ab80d18"}
Nov 03 20:01:25.178961 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:25.178900    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-332694-8c85s" podStartSLOduration=2.17888533 podStartE2EDuration="2.17888533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:01:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:25.177892947 +0000 UTC m=+1798.421858411" watchObservedRunningTime="2026-11-03 20:01:25.17888533 +0000 UTC m=+1798.422850817"
Nov 03 20:01:26.173972 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:26.173933    4921 generic.go:356] "Generic (PLEG): container finished" podID="6e610d24-87d3-4b3a-936d-16bce9c36b94" containerID="8464e2562ac52ee9e1b79c576735ff42e63ff6e81a1fda86a5bb26c1888b30c7" exitCode=0
Nov 03 20:01:26.174499 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:26.174009    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-332694-8c85s" event={"ID":"6e610d24-87d3-4b3a-936d-16bce9c36b94","Type":"ContainerDied","Data":"8464e2562ac52ee9e1b79c576735ff42e63ff6e81a1fda86a5bb26c1888b30c7"}
Nov 03 20:01:27.465049 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:27.465017    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-332694-8c85s"
Nov 03 20:01:27.593831 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:27.593799    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e610d24-87d3-4b3a-936d-16bce9c36b94-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e610d24-87d3-4b3a-936d-16bce9c36b94-maas-api-openshift-ca\") pod \"6e610d24-87d3-4b3a-936d-16bce9c36b94\" (UID: \"6e610d24-87d3-4b3a-936d-16bce9c36b94\") "
Nov 03 20:01:27.594005 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:27.593857    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e610d24-87d3-4b3a-936d-16bce9c36b94-kube-api-access-vtdwt\" (UniqueName: \"kubernetes.io/projected/6e610d24-87d3-4b3a-936d-16bce9c36b94-kube-api-access-vtdwt\") pod \"6e610d24-87d3-4b3a-936d-16bce9c36b94\" (UID: \"6e610d24-87d3-4b3a-936d-16bce9c36b94\") "
Nov 03 20:01:27.594005 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:27.593915    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e610d24-87d3-4b3a-936d-16bce9c36b94-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e610d24-87d3-4b3a-936d-16bce9c36b94-maas-api-xks-ca\") pod \"6e610d24-87d3-4b3a-936d-16bce9c36b94\" (UID: \"6e610d24-87d3-4b3a-936d-16bce9c36b94\") "
Nov 03 20:01:27.594249 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:27.594226    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e610d24-87d3-4b3a-936d-16bce9c36b94-maas-api-openshift-ca" pod "6e610d24-87d3-4b3a-936d-16bce9c36b94" (UID: "6e610d24-87d3-4b3a-936d-16bce9c36b94"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:01:27.594607 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:27.594584    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e610d24-87d3-4b3a-936d-16bce9c36b94-maas-api-openshift-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:27.596095 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:27.596068    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e610d24-87d3-4b3a-936d-16bce9c36b94-maas-api-xks-ca" pod "6e610d24-87d3-4b3a-936d-16bce9c36b94" (UID: "6e610d24-87d3-4b3a-936d-16bce9c36b94"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:01:27.596228 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:27.596214    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e610d24-87d3-4b3a-936d-16bce9c36b94-kube-api-access-vtdwt" pod "6e610d24-87d3-4b3a-936d-16bce9c36b94" (UID: "6e610d24-87d3-4b3a-936d-16bce9c36b94"). InnerVolumeSpecName "kube-api-access-vtdwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:01:27.696445 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:27.696355    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtdwt\" (UniqueName: \"kubernetes.io/projected/6e610d24-87d3-4b3a-936d-16bce9c36b94-kube-api-access-vtdwt\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:27.696445 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:27.696394    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e610d24-87d3-4b3a-936d-16bce9c36b94-maas-api-xks-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:28.093357 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:28.093316    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5f6acf3-45g45"]
Nov 03 20:01:28.094198 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:28.094180    4921 cpu_manager.go:414] "containerMap: removing container" podUID="6e610d24-87d3-4b3a-936d-16bce9c36b94" containerName="revoke-keys"
Nov 03 20:01:28.094262 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:28.094198    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e610d24-87d3-4b3a-936d-16bce9c36b94" containerName="revoke-keys"
Nov 03 20:01:28.094333 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:28.094323    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e610d24-87d3-4b3a-936d-16bce9c36b94" containerName="revoke-keys"
Nov 03 20:01:28.098548 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:28.098523    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5f6acf3-45g45"
Nov 03 20:01:28.107955 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:28.107922    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5f6acf3-45g45"]
Nov 03 20:01:28.128215 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:28.128178    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97249a07-bd8b-404c-a0e8-d416fa948542-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5f6acf3-45g45\" (UID: \"97249a07-bd8b-404c-a0e8-d416fa948542\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5f6acf3-45g45"
Nov 03 20:01:28.128480 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:28.128447    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7xtr\" (UniqueName: \"kubernetes.io/projected/97249a07-bd8b-404c-a0e8-d416fa948542-kube-api-access-v7xtr\") pod \"maas-api-revoke-sub-f5f6acf3-45g45\" (UID: \"97249a07-bd8b-404c-a0e8-d416fa948542\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5f6acf3-45g45"
Nov 03 20:01:28.128591 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:28.128566    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97249a07-bd8b-404c-a0e8-d416fa948542-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5f6acf3-45g45\" (UID: \"97249a07-bd8b-404c-a0e8-d416fa948542\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5f6acf3-45g45"
Nov 03 20:01:28.193423 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:28.193388    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-332694-8c85s" event={"ID":"6e610d24-87d3-4b3a-936d-16bce9c36b94","Type":"ContainerDied","Data":"8c67354455cf53b1a8118aa6ecaae7c94344d2e03fbfc365fd22ae991ab80d18"}
Nov 03 20:01:28.193423 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:28.193413    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-332694-8c85s"
Nov 03 20:01:28.193645 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:28.193439    4921 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c67354455cf53b1a8118aa6ecaae7c94344d2e03fbfc365fd22ae991ab80d18"
Nov 03 20:01:28.229915 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:28.229871    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97249a07-bd8b-404c-a0e8-d416fa948542-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5f6acf3-45g45\" (UID: \"97249a07-bd8b-404c-a0e8-d416fa948542\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5f6acf3-45g45"
Nov 03 20:01:28.230129 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:28.230022    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7xtr\" (UniqueName: \"kubernetes.io/projected/97249a07-bd8b-404c-a0e8-d416fa948542-kube-api-access-v7xtr\") pod \"maas-api-revoke-sub-f5f6acf3-45g45\" (UID: \"97249a07-bd8b-404c-a0e8-d416fa948542\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5f6acf3-45g45"
Nov 03 20:01:28.230129 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:28.230069    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97249a07-bd8b-404c-a0e8-d416fa948542-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5f6acf3-45g45\" (UID: \"97249a07-bd8b-404c-a0e8-d416fa948542\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5f6acf3-45g45"
Nov 03 20:01:28.230793 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:28.230772    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97249a07-bd8b-404c-a0e8-d416fa948542-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5f6acf3-45g45\" (UID: \"97249a07-bd8b-404c-a0e8-d416fa948542\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5f6acf3-45g45"
Nov 03 20:01:28.232375 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:28.232352    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97249a07-bd8b-404c-a0e8-d416fa948542-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5f6acf3-45g45\" (UID: \"97249a07-bd8b-404c-a0e8-d416fa948542\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5f6acf3-45g45"
Nov 03 20:01:28.236895 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:28.236868    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7xtr\" (UniqueName: \"kubernetes.io/projected/97249a07-bd8b-404c-a0e8-d416fa948542-kube-api-access-v7xtr\") pod \"maas-api-revoke-sub-f5f6acf3-45g45\" (UID: \"97249a07-bd8b-404c-a0e8-d416fa948542\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5f6acf3-45g45"
Nov 03 20:01:28.417253 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:28.417146    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5f6acf3-45g45"
Nov 03 20:01:28.722787 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:28.722753    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5f6acf3-45g45"]
Nov 03 20:01:28.723439 ip-10-0-134-222 kubenswrapper[4921]: W1103 20:01:28.723413    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97249a07_bd8b_404c_a0e8_d416fa948542.slice/crio-be93bb13bef404ef64b7fc075ddedf8c8bec506dfebb52e19b9e485c37c737ed WatchSource:0}: Error finding container be93bb13bef404ef64b7fc075ddedf8c8bec506dfebb52e19b9e485c37c737ed: Status 404 returned error can't find the container with id be93bb13bef404ef64b7fc075ddedf8c8bec506dfebb52e19b9e485c37c737ed
Nov 03 20:01:29.204600 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:29.204543    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5f6acf3-45g45" event={"ID":"97249a07-bd8b-404c-a0e8-d416fa948542","Type":"ContainerStarted","Data":"09364d1871f49d52685b16b5c35681673f68553431f9b98300c0323f5aa599ff"}
Nov 03 20:01:29.204818 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:29.204611    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5f6acf3-45g45" event={"ID":"97249a07-bd8b-404c-a0e8-d416fa948542","Type":"ContainerStarted","Data":"be93bb13bef404ef64b7fc075ddedf8c8bec506dfebb52e19b9e485c37c737ed"}
Nov 03 20:01:29.220759 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:29.220697    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5f6acf3-45g45" podStartSLOduration=1.220662031 podStartE2EDuration="1.220662031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:01:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:29.220167862 +0000 UTC m=+1802.464133330" watchObservedRunningTime="2026-11-03 20:01:29.220662031 +0000 UTC m=+1802.464627545"
Nov 03 20:01:30.216158 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:30.216064    4921 generic.go:356] "Generic (PLEG): container finished" podID="97249a07-bd8b-404c-a0e8-d416fa948542" containerID="09364d1871f49d52685b16b5c35681673f68553431f9b98300c0323f5aa599ff" exitCode=0
Nov 03 20:01:30.216574 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:30.216174    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5f6acf3-45g45" event={"ID":"97249a07-bd8b-404c-a0e8-d416fa948542","Type":"ContainerDied","Data":"09364d1871f49d52685b16b5c35681673f68553431f9b98300c0323f5aa599ff"}
Nov 03 20:01:31.045521 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:31.045489    4921 scope.go:122] "RemoveContainer" containerID="5d6520597ecef5f2e98b68cb493d5cf092a2e9bdfa1da40bc65a3840582732dc"
Nov 03 20:01:31.491447 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:31.491399    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5f6acf3-45g45"
Nov 03 20:01:31.601542 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:31.601454    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97249a07-bd8b-404c-a0e8-d416fa948542-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97249a07-bd8b-404c-a0e8-d416fa948542-maas-api-xks-ca\") pod \"97249a07-bd8b-404c-a0e8-d416fa948542\" (UID: \"97249a07-bd8b-404c-a0e8-d416fa948542\") "
Nov 03 20:01:31.601743 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:31.601548    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97249a07-bd8b-404c-a0e8-d416fa948542-kube-api-access-v7xtr\" (UniqueName: \"kubernetes.io/projected/97249a07-bd8b-404c-a0e8-d416fa948542-kube-api-access-v7xtr\") pod \"97249a07-bd8b-404c-a0e8-d416fa948542\" (UID: \"97249a07-bd8b-404c-a0e8-d416fa948542\") "
Nov 03 20:01:31.601879 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:31.601857    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97249a07-bd8b-404c-a0e8-d416fa948542-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97249a07-bd8b-404c-a0e8-d416fa948542-maas-api-openshift-ca\") pod \"97249a07-bd8b-404c-a0e8-d416fa948542\" (UID: \"97249a07-bd8b-404c-a0e8-d416fa948542\") "
Nov 03 20:01:31.602271 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:31.602233    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97249a07-bd8b-404c-a0e8-d416fa948542-maas-api-openshift-ca" pod "97249a07-bd8b-404c-a0e8-d416fa948542" (UID: "97249a07-bd8b-404c-a0e8-d416fa948542"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:01:31.602759 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:31.602736    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97249a07-bd8b-404c-a0e8-d416fa948542-maas-api-openshift-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:31.604057 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:31.604034    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97249a07-bd8b-404c-a0e8-d416fa948542-maas-api-xks-ca" pod "97249a07-bd8b-404c-a0e8-d416fa948542" (UID: "97249a07-bd8b-404c-a0e8-d416fa948542"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:01:31.604281 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:31.604267    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97249a07-bd8b-404c-a0e8-d416fa948542-kube-api-access-v7xtr" pod "97249a07-bd8b-404c-a0e8-d416fa948542" (UID: "97249a07-bd8b-404c-a0e8-d416fa948542"). InnerVolumeSpecName "kube-api-access-v7xtr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:01:31.712439 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:31.709191    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97249a07-bd8b-404c-a0e8-d416fa948542-maas-api-xks-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:31.712439 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:31.709241    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7xtr\" (UniqueName: \"kubernetes.io/projected/97249a07-bd8b-404c-a0e8-d416fa948542-kube-api-access-v7xtr\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:31.760554 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:31.760524    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gzf8k_6f215b42-0f09-46a0-bc98-392c0fa909c9/console-operator/2.log"
Nov 03 20:01:31.764566 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:31.764538    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gzf8k_6f215b42-0f09-46a0-bc98-392c0fa909c9/console-operator/2.log"
Nov 03 20:01:32.279267 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:32.279224    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5f6acf3-45g45" event={"ID":"97249a07-bd8b-404c-a0e8-d416fa948542","Type":"ContainerDied","Data":"be93bb13bef404ef64b7fc075ddedf8c8bec506dfebb52e19b9e485c37c737ed"}
Nov 03 20:01:32.279493 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:32.279297    4921 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be93bb13bef404ef64b7fc075ddedf8c8bec506dfebb52e19b9e485c37c737ed"
Nov 03 20:01:32.279493 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:32.279374    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5f6acf3-45g45"
Nov 03 20:01:32.959188 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:01:32.959144    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 20:01:37.691275 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:37.691239    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2747a4-gjqwl"]
Nov 03 20:01:37.692156 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:37.692135    4921 cpu_manager.go:414] "containerMap: removing container" podUID="97249a07-bd8b-404c-a0e8-d416fa948542" containerName="revoke-keys"
Nov 03 20:01:37.692156 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:37.692157    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97249a07-bd8b-404c-a0e8-d416fa948542" containerName="revoke-keys"
Nov 03 20:01:37.692342 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:37.692294    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="97249a07-bd8b-404c-a0e8-d416fa948542" containerName="revoke-keys"
Nov 03 20:01:37.696550 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:37.696527    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2747a4-gjqwl"
Nov 03 20:01:37.702882 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:37.701869    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:01:37.702882 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:37.701911    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 20:01:37.706368 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:37.706340    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2747a4-gjqwl"]
Nov 03 20:01:37.799373 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:37.799336    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7grq9\" (UniqueName: \"kubernetes.io/projected/8e813b6c-95bc-462b-bb02-0425f1b408ed-kube-api-access-7grq9\") pod \"maas-api-revoke-keys-e2e-inf-a-2747a4-gjqwl\" (UID: \"8e813b6c-95bc-462b-bb02-0425f1b408ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2747a4-gjqwl"
Nov 03 20:01:37.799542 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:37.799516    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e813b6c-95bc-462b-bb02-0425f1b408ed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2747a4-gjqwl\" (UID: \"8e813b6c-95bc-462b-bb02-0425f1b408ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2747a4-gjqwl"
Nov 03 20:01:37.799642 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:37.799627    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e813b6c-95bc-462b-bb02-0425f1b408ed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2747a4-gjqwl\" (UID: \"8e813b6c-95bc-462b-bb02-0425f1b408ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2747a4-gjqwl"
Nov 03 20:01:37.901316 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:37.901275    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7grq9\" (UniqueName: \"kubernetes.io/projected/8e813b6c-95bc-462b-bb02-0425f1b408ed-kube-api-access-7grq9\") pod \"maas-api-revoke-keys-e2e-inf-a-2747a4-gjqwl\" (UID: \"8e813b6c-95bc-462b-bb02-0425f1b408ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2747a4-gjqwl"
Nov 03 20:01:37.901471 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:37.901444    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e813b6c-95bc-462b-bb02-0425f1b408ed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2747a4-gjqwl\" (UID: \"8e813b6c-95bc-462b-bb02-0425f1b408ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2747a4-gjqwl"
Nov 03 20:01:37.901580 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:37.901565    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e813b6c-95bc-462b-bb02-0425f1b408ed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2747a4-gjqwl\" (UID: \"8e813b6c-95bc-462b-bb02-0425f1b408ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2747a4-gjqwl"
Nov 03 20:01:37.902321 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:37.902296    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e813b6c-95bc-462b-bb02-0425f1b408ed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2747a4-gjqwl\" (UID: \"8e813b6c-95bc-462b-bb02-0425f1b408ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2747a4-gjqwl"
Nov 03 20:01:37.904224 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:37.904198    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e813b6c-95bc-462b-bb02-0425f1b408ed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2747a4-gjqwl\" (UID: \"8e813b6c-95bc-462b-bb02-0425f1b408ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2747a4-gjqwl"
Nov 03 20:01:37.909815 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:37.909790    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7grq9\" (UniqueName: \"kubernetes.io/projected/8e813b6c-95bc-462b-bb02-0425f1b408ed-kube-api-access-7grq9\") pod \"maas-api-revoke-keys-e2e-inf-a-2747a4-gjqwl\" (UID: \"8e813b6c-95bc-462b-bb02-0425f1b408ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2747a4-gjqwl"
Nov 03 20:01:38.015271 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:38.015233    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2747a4-gjqwl"
Nov 03 20:01:38.311979 ip-10-0-134-222 kubenswrapper[4921]: W1103 20:01:38.311943    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e813b6c_95bc_462b_bb02_0425f1b408ed.slice/crio-cd5f232f3ff44331e9b671739ab446d8420193537fd737289ebd7d18a84b0cba WatchSource:0}: Error finding container cd5f232f3ff44331e9b671739ab446d8420193537fd737289ebd7d18a84b0cba: Status 404 returned error can't find the container with id cd5f232f3ff44331e9b671739ab446d8420193537fd737289ebd7d18a84b0cba
Nov 03 20:01:38.312630 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:38.312601    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2747a4-gjqwl"]
Nov 03 20:01:38.468394 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:38.468357    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2747a4-gjqwl" event={"ID":"8e813b6c-95bc-462b-bb02-0425f1b408ed","Type":"ContainerStarted","Data":"cd5f232f3ff44331e9b671739ab446d8420193537fd737289ebd7d18a84b0cba"}
Nov 03 20:01:39.479644 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:39.479603    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2747a4-gjqwl" event={"ID":"8e813b6c-95bc-462b-bb02-0425f1b408ed","Type":"ContainerStarted","Data":"8419e40ba0382609488be7d78f20a38389d1869cf12496d568ce054898a5873a"}
Nov 03 20:01:39.494482 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:39.494427    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2747a4-gjqwl" podStartSLOduration=2.494411529 podStartE2EDuration="2.494411529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:01:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:39.492712196 +0000 UTC m=+1812.736677663" watchObservedRunningTime="2026-11-03 20:01:39.494411529 +0000 UTC m=+1812.738376994"
Nov 03 20:01:39.761077 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:01:39.760986    4921 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e813b6c_95bc_462b_bb02_0425f1b408ed.slice/crio-conmon-8419e40ba0382609488be7d78f20a38389d1869cf12496d568ce054898a5873a.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 20:01:40.490903 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:40.490867    4921 generic.go:356] "Generic (PLEG): container finished" podID="8e813b6c-95bc-462b-bb02-0425f1b408ed" containerID="8419e40ba0382609488be7d78f20a38389d1869cf12496d568ce054898a5873a" exitCode=0
Nov 03 20:01:40.491365 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:40.490949    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2747a4-gjqwl" event={"ID":"8e813b6c-95bc-462b-bb02-0425f1b408ed","Type":"ContainerDied","Data":"8419e40ba0382609488be7d78f20a38389d1869cf12496d568ce054898a5873a"}
Nov 03 20:01:41.821264 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:41.821176    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2747a4-gjqwl"
Nov 03 20:01:41.926495 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:41.926460    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e813b6c-95bc-462b-bb02-0425f1b408ed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e813b6c-95bc-462b-bb02-0425f1b408ed-maas-api-xks-ca\") pod \"8e813b6c-95bc-462b-bb02-0425f1b408ed\" (UID: \"8e813b6c-95bc-462b-bb02-0425f1b408ed\") "
Nov 03 20:01:41.926754 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:41.926663    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e813b6c-95bc-462b-bb02-0425f1b408ed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e813b6c-95bc-462b-bb02-0425f1b408ed-maas-api-openshift-ca\") pod \"8e813b6c-95bc-462b-bb02-0425f1b408ed\" (UID: \"8e813b6c-95bc-462b-bb02-0425f1b408ed\") "
Nov 03 20:01:41.926948 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:41.926927    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e813b6c-95bc-462b-bb02-0425f1b408ed-kube-api-access-7grq9\" (UniqueName: \"kubernetes.io/projected/8e813b6c-95bc-462b-bb02-0425f1b408ed-kube-api-access-7grq9\") pod \"8e813b6c-95bc-462b-bb02-0425f1b408ed\" (UID: \"8e813b6c-95bc-462b-bb02-0425f1b408ed\") "
Nov 03 20:01:41.927117 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:41.927086    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e813b6c-95bc-462b-bb02-0425f1b408ed-maas-api-openshift-ca" pod "8e813b6c-95bc-462b-bb02-0425f1b408ed" (UID: "8e813b6c-95bc-462b-bb02-0425f1b408ed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:01:41.927592 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:41.927568    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e813b6c-95bc-462b-bb02-0425f1b408ed-maas-api-openshift-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:41.928940 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:41.928909    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e813b6c-95bc-462b-bb02-0425f1b408ed-maas-api-xks-ca" pod "8e813b6c-95bc-462b-bb02-0425f1b408ed" (UID: "8e813b6c-95bc-462b-bb02-0425f1b408ed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:01:41.929463 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:41.929439    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e813b6c-95bc-462b-bb02-0425f1b408ed-kube-api-access-7grq9" pod "8e813b6c-95bc-462b-bb02-0425f1b408ed" (UID: "8e813b6c-95bc-462b-bb02-0425f1b408ed"). InnerVolumeSpecName "kube-api-access-7grq9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:01:42.029210 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:42.029168    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7grq9\" (UniqueName: \"kubernetes.io/projected/8e813b6c-95bc-462b-bb02-0425f1b408ed-kube-api-access-7grq9\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:42.029210 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:42.029209    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e813b6c-95bc-462b-bb02-0425f1b408ed-maas-api-xks-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:42.669418 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:42.669373    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2747a4-gjqwl" event={"ID":"8e813b6c-95bc-462b-bb02-0425f1b408ed","Type":"ContainerDied","Data":"cd5f232f3ff44331e9b671739ab446d8420193537fd737289ebd7d18a84b0cba"}
Nov 03 20:01:42.669607 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:42.669492    4921 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd5f232f3ff44331e9b671739ab446d8420193537fd737289ebd7d18a84b0cba"
Nov 03 20:01:42.669607 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:42.669605    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2747a4-gjqwl"
Nov 03 20:01:49.743927 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:49.743891    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf"]
Nov 03 20:01:49.744349 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:49.744116    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf" podUID="9e747102-94ba-49d0-a2b5-5ff6040c1556" containerName="istio-proxy" containerID="cri-o://61502eec9e6f9e0bc34eb6c038be35268c145fda46530d24978534d7916ae539" gracePeriod=30
Nov 03 20:01:50.669317 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:50.666729    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ffe132-ngjqb"]
Nov 03 20:01:50.669317 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:50.668047    4921 cpu_manager.go:414] "containerMap: removing container" podUID="8e813b6c-95bc-462b-bb02-0425f1b408ed" containerName="revoke-keys"
Nov 03 20:01:50.669317 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:50.668069    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e813b6c-95bc-462b-bb02-0425f1b408ed" containerName="revoke-keys"
Nov 03 20:01:50.669317 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:50.668261    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e813b6c-95bc-462b-bb02-0425f1b408ed" containerName="revoke-keys"
Nov 03 20:01:50.690241 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:50.690209    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ffe132-ngjqb"
Nov 03 20:01:50.692194 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:50.692158    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ffe132-ngjqb"]
Nov 03 20:01:50.696865 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:50.696825    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:01:50.700326 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:50.699774    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 20:01:50.788139 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:50.788107    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d47483d-40b0-47ae-9478-0531cfb8dc77-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ffe132-ngjqb\" (UID: \"1d47483d-40b0-47ae-9478-0531cfb8dc77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ffe132-ngjqb"
Nov 03 20:01:50.788656 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:50.788411    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d47483d-40b0-47ae-9478-0531cfb8dc77-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ffe132-ngjqb\" (UID: \"1d47483d-40b0-47ae-9478-0531cfb8dc77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ffe132-ngjqb"
Nov 03 20:01:50.788656 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:50.788448    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxk6q\" (UniqueName: \"kubernetes.io/projected/1d47483d-40b0-47ae-9478-0531cfb8dc77-kube-api-access-kxk6q\") pod \"maas-api-revoke-keys-e2e-inf-a-ffe132-ngjqb\" (UID: \"1d47483d-40b0-47ae-9478-0531cfb8dc77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ffe132-ngjqb"
Nov 03 20:01:50.890194 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:50.890151    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d47483d-40b0-47ae-9478-0531cfb8dc77-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ffe132-ngjqb\" (UID: \"1d47483d-40b0-47ae-9478-0531cfb8dc77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ffe132-ngjqb"
Nov 03 20:01:50.890375 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:50.890202    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxk6q\" (UniqueName: \"kubernetes.io/projected/1d47483d-40b0-47ae-9478-0531cfb8dc77-kube-api-access-kxk6q\") pod \"maas-api-revoke-keys-e2e-inf-a-ffe132-ngjqb\" (UID: \"1d47483d-40b0-47ae-9478-0531cfb8dc77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ffe132-ngjqb"
Nov 03 20:01:50.890375 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:50.890318    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d47483d-40b0-47ae-9478-0531cfb8dc77-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ffe132-ngjqb\" (UID: \"1d47483d-40b0-47ae-9478-0531cfb8dc77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ffe132-ngjqb"
Nov 03 20:01:50.890894 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:50.890874    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d47483d-40b0-47ae-9478-0531cfb8dc77-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ffe132-ngjqb\" (UID: \"1d47483d-40b0-47ae-9478-0531cfb8dc77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ffe132-ngjqb"
Nov 03 20:01:50.892804 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:50.892786    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d47483d-40b0-47ae-9478-0531cfb8dc77-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ffe132-ngjqb\" (UID: \"1d47483d-40b0-47ae-9478-0531cfb8dc77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ffe132-ngjqb"
Nov 03 20:01:50.896952 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:50.896931    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxk6q\" (UniqueName: \"kubernetes.io/projected/1d47483d-40b0-47ae-9478-0531cfb8dc77-kube-api-access-kxk6q\") pod \"maas-api-revoke-keys-e2e-inf-a-ffe132-ngjqb\" (UID: \"1d47483d-40b0-47ae-9478-0531cfb8dc77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ffe132-ngjqb"
Nov 03 20:01:51.030311 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:51.029987    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ffe132-ngjqb"
Nov 03 20:01:51.354278 ip-10-0-134-222 kubenswrapper[4921]: W1103 20:01:51.354241    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d47483d_40b0_47ae_9478_0531cfb8dc77.slice/crio-5f3d5a44f81ecb45dddcfaaf073b27703e46f2e15dca662fcaa9f0fd1bc6b626 WatchSource:0}: Error finding container 5f3d5a44f81ecb45dddcfaaf073b27703e46f2e15dca662fcaa9f0fd1bc6b626: Status 404 returned error can't find the container with id 5f3d5a44f81ecb45dddcfaaf073b27703e46f2e15dca662fcaa9f0fd1bc6b626
Nov 03 20:01:51.354932 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:51.354904    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ffe132-ngjqb"]
Nov 03 20:01:51.364752 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:51.364726    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2747a4-67946747c9-4smgl"]
Nov 03 20:01:51.383385 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:51.382156    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2747a4-647847c6c9-njbpx"]
Nov 03 20:01:51.804740 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:51.804688    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ffe132-ngjqb" event={"ID":"1d47483d-40b0-47ae-9478-0531cfb8dc77","Type":"ContainerStarted","Data":"5f3d5a44f81ecb45dddcfaaf073b27703e46f2e15dca662fcaa9f0fd1bc6b626"}
Nov 03 20:01:52.479284 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:52.479232    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-332694-8c85s"]
Nov 03 20:01:52.484266 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:52.484235    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-332694-8c85s"]
Nov 03 20:01:52.815296 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:52.815214    4921 generic.go:356] "Generic (PLEG): container finished" podID="1d47483d-40b0-47ae-9478-0531cfb8dc77" containerID="961f0731683dd96ac13f5d500e8b444ec3daf371310bdebfd246fb2d73050e86" exitCode=0
Nov 03 20:01:52.815723 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:52.815294    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ffe132-ngjqb" event={"ID":"1d47483d-40b0-47ae-9478-0531cfb8dc77","Type":"ContainerDied","Data":"961f0731683dd96ac13f5d500e8b444ec3daf371310bdebfd246fb2d73050e86"}
Nov 03 20:01:52.892042 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:52.892005    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e610d24-87d3-4b3a-936d-16bce9c36b94" path="/var/lib/kubelet/pods/6e610d24-87d3-4b3a-936d-16bce9c36b94/volumes"
Nov 03 20:01:54.017734 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:54.017708    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ffe132-ngjqb"
Nov 03 20:01:54.097461 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:54.097367    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d47483d-40b0-47ae-9478-0531cfb8dc77-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d47483d-40b0-47ae-9478-0531cfb8dc77-maas-api-openshift-ca\") pod \"1d47483d-40b0-47ae-9478-0531cfb8dc77\" (UID: \"1d47483d-40b0-47ae-9478-0531cfb8dc77\") "
Nov 03 20:01:54.097461 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:54.097443    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d47483d-40b0-47ae-9478-0531cfb8dc77-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d47483d-40b0-47ae-9478-0531cfb8dc77-maas-api-xks-ca\") pod \"1d47483d-40b0-47ae-9478-0531cfb8dc77\" (UID: \"1d47483d-40b0-47ae-9478-0531cfb8dc77\") "
Nov 03 20:01:54.097717 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:54.097570    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d47483d-40b0-47ae-9478-0531cfb8dc77-kube-api-access-kxk6q\" (UniqueName: \"kubernetes.io/projected/1d47483d-40b0-47ae-9478-0531cfb8dc77-kube-api-access-kxk6q\") pod \"1d47483d-40b0-47ae-9478-0531cfb8dc77\" (UID: \"1d47483d-40b0-47ae-9478-0531cfb8dc77\") "
Nov 03 20:01:54.098699 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:54.097927    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d47483d-40b0-47ae-9478-0531cfb8dc77-maas-api-openshift-ca" pod "1d47483d-40b0-47ae-9478-0531cfb8dc77" (UID: "1d47483d-40b0-47ae-9478-0531cfb8dc77"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:01:54.098699 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:54.098606    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d47483d-40b0-47ae-9478-0531cfb8dc77-maas-api-openshift-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:54.100011 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:54.099983    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d47483d-40b0-47ae-9478-0531cfb8dc77-kube-api-access-kxk6q" pod "1d47483d-40b0-47ae-9478-0531cfb8dc77" (UID: "1d47483d-40b0-47ae-9478-0531cfb8dc77"). InnerVolumeSpecName "kube-api-access-kxk6q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:01:54.100321 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:54.100302    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d47483d-40b0-47ae-9478-0531cfb8dc77-maas-api-xks-ca" pod "1d47483d-40b0-47ae-9478-0531cfb8dc77" (UID: "1d47483d-40b0-47ae-9478-0531cfb8dc77"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:01:54.200107 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:54.200073    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d47483d-40b0-47ae-9478-0531cfb8dc77-maas-api-xks-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:54.200107 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:54.200103    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxk6q\" (UniqueName: \"kubernetes.io/projected/1d47483d-40b0-47ae-9478-0531cfb8dc77-kube-api-access-kxk6q\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:54.838266 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:54.838219    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ffe132-ngjqb"
Nov 03 20:01:54.838266 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:54.838237    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ffe132-ngjqb" event={"ID":"1d47483d-40b0-47ae-9478-0531cfb8dc77","Type":"ContainerDied","Data":"5f3d5a44f81ecb45dddcfaaf073b27703e46f2e15dca662fcaa9f0fd1bc6b626"}
Nov 03 20:01:54.838516 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:54.838335    4921 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f3d5a44f81ecb45dddcfaaf073b27703e46f2e15dca662fcaa9f0fd1bc6b626"
Nov 03 20:01:55.070897 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:55.070868    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf"
Nov 03 20:01:55.102517 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:55.102478    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx"]
Nov 03 20:01:55.103366 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:55.103346    4921 cpu_manager.go:414] "containerMap: removing container" podUID="1d47483d-40b0-47ae-9478-0531cfb8dc77" containerName="revoke-keys"
Nov 03 20:01:55.103455 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:55.103369    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d47483d-40b0-47ae-9478-0531cfb8dc77" containerName="revoke-keys"
Nov 03 20:01:55.103455 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:55.103401    4921 cpu_manager.go:414] "containerMap: removing container" podUID="9e747102-94ba-49d0-a2b5-5ff6040c1556" containerName="istio-proxy"
Nov 03 20:01:55.103455 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:55.103407    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e747102-94ba-49d0-a2b5-5ff6040c1556" containerName="istio-proxy"
Nov 03 20:01:55.103558 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:55.103548    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e747102-94ba-49d0-a2b5-5ff6040c1556" containerName="istio-proxy"
Nov 03 20:01:55.103594 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:55.103569    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d47483d-40b0-47ae-9478-0531cfb8dc77" containerName="revoke-keys"
Nov 03 20:01:55.107757 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:55.107728    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx"
Nov 03 20:01:55.112537 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:55.112503    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-11c1a8-openshift-default-dockercfg-7l2db\""
Nov 03 20:01:55.112775 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:55.112544    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e747102-94ba-49d0-a2b5-5ff6040c1556-kube-api-access-98klw\" (UniqueName: \"kubernetes.io/projected/9e747102-94ba-49d0-a2b5-5ff6040c1556-kube-api-access-98klw\") pod \"9e747102-94ba-49d0-a2b5-5ff6040c1556\" (UID: \"9e747102-94ba-49d0-a2b5-5ff6040c1556\") "
Nov 03 20:01:55.112917 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:55.112782    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9e747102-94ba-49d0-a2b5-5ff6040c1556-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e747102-94ba-49d0-a2b5-5ff6040c1556-istio-podinfo\") pod \"9e747102-94ba-49d0-a2b5-5ff6040c1556\" (UID: \"9e747102-94ba-49d0-a2b5-5ff6040c1556\") "
Nov 03 20:01:55.112917 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:55.112847    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e747102-94ba-49d0-a2b5-5ff6040c1556-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e747102-94ba-49d0-a2b5-5ff6040c1556-credential-socket\") pod \"9e747102-94ba-49d0-a2b5-5ff6040c1556\" (UID: \"9e747102-94ba-49d0-a2b5-5ff6040c1556\") "
Nov 03 20:01:55.113070 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:55.112985    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e747102-94ba-49d0-a2b5-5ff6040c1556-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e747102-94ba-49d0-a2b5-5ff6040c1556-istio-envoy\") pod \"9e747102-94ba-49d0-a2b5-5ff6040c1556\" (UID: \"9e747102-94ba-49d0-a2b5-5ff6040c1556\") "
Nov 03 20:01:55.113070 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:55.113046    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e747102-94ba-49d0-a2b5-5ff6040c1556-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e747102-94ba-49d0-a2b5-5ff6040c1556-workload-socket\") pod \"9e747102-94ba-49d0-a2b5-5ff6040c1556\" (UID: \"9e747102-94ba-49d0-a2b5-5ff6040c1556\") "
Nov 03 20:01:55.113197 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:55.113087    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e747102-94ba-49d0-a2b5-5ff6040c1556-istio-token\" (UniqueName: \"kubernetes.io/projected/9e747102-94ba-49d0-a2b5-5ff6040c1556-istio-token\") pod \"9e747102-94ba-49d0-a2b5-5ff6040c1556\" (UID: \"9e747102-94ba-49d0-a2b5-5ff6040c1556\") "
Nov 03 20:01:55.113197 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:55.113130    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e747102-94ba-49d0-a2b5-5ff6040c1556-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e747102-94ba-49d0-a2b5-5ff6040c1556-workload-certs\") pod \"9e747102-94ba-49d0-a2b5-5ff6040c1556\" (UID: \"9e747102-94ba-49d0-a2b5-5ff6040c1556\") "
Nov 03 20:01:55.113345 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:55.113201    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e747102-94ba-49d0-a2b5-5ff6040c1556-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e747102-94ba-49d0-a2b5-5ff6040c1556-istio-data\") pod \"9e747102-94ba-49d0-a2b5-5ff6040c1556\" (UID: \"9e747102-94ba-49d0-a2b5-5ff6040c1556\") "
Nov 03 20:01:55.113345 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:55.113273    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e747102-94ba-49d0-a2b5-5ff6040c1556-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e747102-94ba-49d0-a2b5-5ff6040c1556-istiod-ca-cert\") pod \"9e747102-94ba-49d0-a2b5-5ff6040c1556\" (UID: \"9e747102-94ba-49d0-a2b5-5ff6040c1556\") "
Nov 03 20:01:55.114184 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:55.114154    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e747102-94ba-49d0-a2b5-5ff6040c1556-workload-socket" pod "9e747102-94ba-49d0-a2b5-5ff6040c1556" (UID: "9e747102-94ba-49d0-a2b5-5ff6040c1556"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:01:55.114390 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:55.114360    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e747102-94ba-49d0-a2b5-5ff6040c1556-credential-socket" pod "9e747102-94ba-49d0-a2b5-5ff6040c1556" (UID: "9e747102-94ba-49d0-a2b5-5ff6040c1556"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:01:55.114474 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:55.114457    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e747102-94ba-49d0-a2b5-5ff6040c1556-istiod-ca-cert" pod "9e747102-94ba-49d0-a2b5-5ff6040c1556" (UID: "9e747102-94ba-49d0-a2b5-5ff6040c1556"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:01:55.119751 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:55.119460    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx"]
Nov 03 20:01:55.120081 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:55.120060    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e747102-94ba-49d0-a2b5-5ff6040c1556-workload-certs" pod "9e747102-94ba-49d0-a2b5-5ff6040c1556" (UID: "9e747102-94ba-49d0-a2b5-5ff6040c1556"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:01:55.120508 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:55.120251    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e747102-94ba-49d0-a2b5-5ff6040c1556-istio-data" pod "9e747102-94ba-49d0-a2b5-5ff6040c1556" (UID: "9e747102-94ba-49d0-a2b5-5ff6040c1556"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:01:55.123353 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:55.121922    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e747102-94ba-49d0-a2b5-5ff6040c1556-istio-envoy" pod "9e747102-94ba-49d0-a2b5-5ff6040c1556" (UID: "9e747102-94ba-49d0-a2b5-5ff6040c1556"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:01:55.123353 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:55.122201    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e747102-94ba-49d0-a2b5-5ff6040c1556-istio-token" pod "9e747102-94ba-49d0-a2b5-5ff6040c1556" (UID: "9e747102-94ba-49d0-a2b5-5ff6040c1556"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:01:55.123353 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:55.122451    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e747102-94ba-49d0-a2b5-5ff6040c1556-kube-api-access-98klw" pod "9e747102-94ba-49d0-a2b5-5ff6040c1556" (UID: "9e747102-94ba-49d0-a2b5-5ff6040c1556"). InnerVolumeSpecName "kube-api-access-98klw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:01:55.123353 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:55.124187    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e747102-94ba-49d0-a2b5-5ff6040c1556-istio-podinfo" pod "9e747102-94ba-49d0-a2b5-5ff6040c1556" (UID: "9e747102-94ba-49d0-a2b5-5ff6040c1556"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:01:55.215932 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:55.215895    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/204760c8-b3c3-4b1d-8b67-5bb6cf918407-istio-data\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx\" (UID: \"204760c8-b3c3-4b1d-8b67-5bb6cf918407\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx"
Nov 03 20:01:55.216131 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:55.215937    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/204760c8-b3c3-4b1d-8b67-5bb6cf918407-istio-podinfo\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx\" (UID: \"204760c8-b3c3-4b1d-8b67-5bb6cf918407\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx"
Nov 03 20:01:55.216131 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:55.216055    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/204760c8-b3c3-4b1d-8b67-5bb6cf918407-istio-envoy\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx\" (UID: \"204760c8-b3c3-4b1d-8b67-5bb6cf918407\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx"
Nov 03 20:01:55.216131 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:55.216097    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/204760c8-b3c3-4b1d-8b67-5bb6cf918407-workload-socket\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx\" (UID: \"204760c8-b3c3-4b1d-8b67-5bb6cf918407\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx"
Nov 03 20:01:55.216266 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:55.216249    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/204760c8-b3c3-4b1d-8b67-5bb6cf918407-workload-certs\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx\" (UID: \"204760c8-b3c3-4b1d-8b67-5bb6cf918407\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx"
Nov 03 20:01:55.216311 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:55.216284    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/204760c8-b3c3-4b1d-8b67-5bb6cf918407-istiod-ca-cert\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx\" (UID: \"204760c8-b3c3-4b1d-8b67-5bb6cf918407\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx"
Nov 03 20:01:55.216387 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:55.216371    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/204760c8-b3c3-4b1d-8b67-5bb6cf918407-credential-socket\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx\" (UID: \"204760c8-b3c3-4b1d-8b67-5bb6cf918407\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx"
Nov 03 20:01:55.216435 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:55.216423    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdv2r\" (UniqueName: \"kubernetes.io/projected/204760c8-b3c3-4b1d-8b67-5bb6cf918407-kube-api-access-hdv2r\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx\" (UID: \"204760c8-b3c3-4b1d-8b67-5bb6cf918407\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx"
Nov 03 20:01:55.216506 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:55.216494    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/204760c8-b3c3-4b1d-8b67-5bb6cf918407-istio-token\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx\" (UID: \"204760c8-b3c3-4b1d-8b67-5bb6cf918407\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx"
Nov 03 20:01:55.216749 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:55.216685    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98klw\" (UniqueName: \"kubernetes.io/projected/9e747102-94ba-49d0-a2b5-5ff6040c1556-kube-api-access-98klw\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:55.216749 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:55.216703    4921 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e747102-94ba-49d0-a2b5-5ff6040c1556-istio-podinfo\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:55.216749 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:55.216712    4921 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e747102-94ba-49d0-a2b5-5ff6040c1556-credential-socket\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:55.216749 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:55.216722    4921 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e747102-94ba-49d0-a2b5-5ff6040c1556-istio-envoy\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:55.216749 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:55.216731    4921 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e747102-94ba-49d0-a2b5-5ff6040c1556-workload-socket\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:55.216749 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:55.216740    4921 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e747102-94ba-49d0-a2b5-5ff6040c1556-istio-token\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:55.216749 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:55.216748    4921 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e747102-94ba-49d0-a2b5-5ff6040c1556-workload-certs\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:55.217008 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:55.216757    4921 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e747102-94ba-49d0-a2b5-5ff6040c1556-istio-data\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:55.217008 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:55.216766    4921 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e747102-94ba-49d0-a2b5-5ff6040c1556-istiod-ca-cert\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:55.318527 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:55.318438    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/204760c8-b3c3-4b1d-8b67-5bb6cf918407-istio-data\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx\" (UID: \"204760c8-b3c3-4b1d-8b67-5bb6cf918407\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx"
Nov 03 20:01:55.318527 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:55.318485    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/204760c8-b3c3-4b1d-8b67-5bb6cf918407-istio-podinfo\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx\" (UID: \"204760c8-b3c3-4b1d-8b67-5bb6cf918407\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx"
Nov 03 20:01:55.318826 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:55.318533    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/204760c8-b3c3-4b1d-8b67-5bb6cf918407-istio-envoy\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx\" (UID: \"204760c8-b3c3-4b1d-8b67-5bb6cf918407\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx"
Nov 03 20:01:55.318826 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:55.318567    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/204760c8-b3c3-4b1d-8b67-5bb6cf918407-workload-socket\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx\" (UID: \"204760c8-b3c3-4b1d-8b67-5bb6cf918407\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx"
Nov 03 20:01:55.318826 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:55.318748    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/204760c8-b3c3-4b1d-8b67-5bb6cf918407-workload-certs\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx\" (UID: \"204760c8-b3c3-4b1d-8b67-5bb6cf918407\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx"
Nov 03 20:01:55.318826 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:55.318783    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/204760c8-b3c3-4b1d-8b67-5bb6cf918407-istiod-ca-cert\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx\" (UID: \"204760c8-b3c3-4b1d-8b67-5bb6cf918407\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx"
Nov 03 20:01:55.319116 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:55.318901    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/204760c8-b3c3-4b1d-8b67-5bb6cf918407-istio-data\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx\" (UID: \"204760c8-b3c3-4b1d-8b67-5bb6cf918407\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx"
Nov 03 20:01:55.319116 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:55.319017    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/204760c8-b3c3-4b1d-8b67-5bb6cf918407-workload-socket\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx\" (UID: \"204760c8-b3c3-4b1d-8b67-5bb6cf918407\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx"
Nov 03 20:01:55.319116 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:55.319071    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/204760c8-b3c3-4b1d-8b67-5bb6cf918407-workload-certs\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx\" (UID: \"204760c8-b3c3-4b1d-8b67-5bb6cf918407\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx"
Nov 03 20:01:55.319271 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:55.319222    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/204760c8-b3c3-4b1d-8b67-5bb6cf918407-credential-socket\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx\" (UID: \"204760c8-b3c3-4b1d-8b67-5bb6cf918407\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx"
Nov 03 20:01:55.319588 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:55.319565    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/204760c8-b3c3-4b1d-8b67-5bb6cf918407-istiod-ca-cert\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx\" (UID: \"204760c8-b3c3-4b1d-8b67-5bb6cf918407\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx"
Nov 03 20:01:55.319716 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:55.318872    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/204760c8-b3c3-4b1d-8b67-5bb6cf918407-credential-socket\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx\" (UID: \"204760c8-b3c3-4b1d-8b67-5bb6cf918407\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx"
Nov 03 20:01:55.319716 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:55.319694    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdv2r\" (UniqueName: \"kubernetes.io/projected/204760c8-b3c3-4b1d-8b67-5bb6cf918407-kube-api-access-hdv2r\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx\" (UID: \"204760c8-b3c3-4b1d-8b67-5bb6cf918407\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx"
Nov 03 20:01:55.319854 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:55.319768    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/204760c8-b3c3-4b1d-8b67-5bb6cf918407-istio-token\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx\" (UID: \"204760c8-b3c3-4b1d-8b67-5bb6cf918407\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx"
Nov 03 20:01:55.321007 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:55.320984    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/204760c8-b3c3-4b1d-8b67-5bb6cf918407-istio-envoy\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx\" (UID: \"204760c8-b3c3-4b1d-8b67-5bb6cf918407\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx"
Nov 03 20:01:55.321229 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:55.321202    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/204760c8-b3c3-4b1d-8b67-5bb6cf918407-istio-podinfo\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx\" (UID: \"204760c8-b3c3-4b1d-8b67-5bb6cf918407\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx"
Nov 03 20:01:55.333840 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:55.333817    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdv2r\" (UniqueName: \"kubernetes.io/projected/204760c8-b3c3-4b1d-8b67-5bb6cf918407-kube-api-access-hdv2r\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx\" (UID: \"204760c8-b3c3-4b1d-8b67-5bb6cf918407\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx"
Nov 03 20:01:55.336267 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:55.336235    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/204760c8-b3c3-4b1d-8b67-5bb6cf918407-istio-token\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx\" (UID: \"204760c8-b3c3-4b1d-8b67-5bb6cf918407\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx"
Nov 03 20:01:55.468462 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:55.468163    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx"
Nov 03 20:01:55.797496 ip-10-0-134-222 kubenswrapper[4921]: W1103 20:01:55.797459    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod204760c8_b3c3_4b1d_8b67_5bb6cf918407.slice/crio-10537d31d73f4d93ec48cb12a59a4c7be02b35c86ec19c236a28eebece834464 WatchSource:0}: Error finding container 10537d31d73f4d93ec48cb12a59a4c7be02b35c86ec19c236a28eebece834464: Status 404 returned error can't find the container with id 10537d31d73f4d93ec48cb12a59a4c7be02b35c86ec19c236a28eebece834464
Nov 03 20:01:55.798065 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:55.798041    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx"]
Nov 03 20:01:55.804740 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:55.804701    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:01:55.804852 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:55.804778    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:01:55.804852 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:55.804822    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:01:55.848826 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:55.848796    4921 generic.go:356] "Generic (PLEG): container finished" podID="9e747102-94ba-49d0-a2b5-5ff6040c1556" containerID="61502eec9e6f9e0bc34eb6c038be35268c145fda46530d24978534d7916ae539" exitCode=0
Nov 03 20:01:55.848971 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:55.848842    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf"
Nov 03 20:01:55.848971 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:55.848872    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf" event={"ID":"9e747102-94ba-49d0-a2b5-5ff6040c1556","Type":"ContainerDied","Data":"61502eec9e6f9e0bc34eb6c038be35268c145fda46530d24978534d7916ae539"}
Nov 03 20:01:55.848971 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:55.848931    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf" event={"ID":"9e747102-94ba-49d0-a2b5-5ff6040c1556","Type":"ContainerDied","Data":"f3048fee65ff651a366a75845e2d80c3a9e8953a9f191d33d30846be315215a6"}
Nov 03 20:01:55.848971 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:55.848959    4921 scope.go:122] "RemoveContainer" containerID="61502eec9e6f9e0bc34eb6c038be35268c145fda46530d24978534d7916ae539"
Nov 03 20:01:55.850915 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:55.850884    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx" event={"ID":"204760c8-b3c3-4b1d-8b67-5bb6cf918407","Type":"ContainerStarted","Data":"10537d31d73f4d93ec48cb12a59a4c7be02b35c86ec19c236a28eebece834464"}
Nov 03 20:01:55.862840 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:55.862813    4921 scope.go:122] "RemoveContainer" containerID="61502eec9e6f9e0bc34eb6c038be35268c145fda46530d24978534d7916ae539"
Nov 03 20:01:55.863215 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:01:55.863182    4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61502eec9e6f9e0bc34eb6c038be35268c145fda46530d24978534d7916ae539\": container with ID starting with 61502eec9e6f9e0bc34eb6c038be35268c145fda46530d24978534d7916ae539 not found: ID does not exist" containerID="61502eec9e6f9e0bc34eb6c038be35268c145fda46530d24978534d7916ae539"
Nov 03 20:01:55.863318 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:55.863225    4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61502eec9e6f9e0bc34eb6c038be35268c145fda46530d24978534d7916ae539"} err="failed to get container status \"61502eec9e6f9e0bc34eb6c038be35268c145fda46530d24978534d7916ae539\": rpc error: code = NotFound desc = could not find container \"61502eec9e6f9e0bc34eb6c038be35268c145fda46530d24978534d7916ae539\": container with ID starting with 61502eec9e6f9e0bc34eb6c038be35268c145fda46530d24978534d7916ae539 not found: ID does not exist"
Nov 03 20:01:55.899505 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:55.899471    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf"]
Nov 03 20:01:55.904986 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:55.904949    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-8h8mf"]
Nov 03 20:01:56.365602 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:56.365556    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-2747a4-67946747c9-4smgl" podUID="0ed39371-700a-43b3-a66b-a500d13cfd77" containerName="payload-processing" containerID="cri-o://dc7004d6f92d63728b8a7c9c79f11fb1040776c50ec2391b2e26b41350a8b9d2" gracePeriod=25
Nov 03 20:01:56.382737 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:56.382698    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2747a4-647847c6c9-njbpx" podUID="b642f55a-3767-464c-b9ff-280627b3e277" containerName="payload-pre-processing" containerID="cri-o://bb93252d5bb8e8021bb71d16b794c26a1397b3d0c9ce11660459476a4236c3b6" gracePeriod=25
Nov 03 20:01:56.712405 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:56.712375    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2747a4-67946747c9-4smgl"
Nov 03 20:01:56.758515 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:56.758478    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ed39371-700a-43b3-a66b-a500d13cfd77-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ed39371-700a-43b3-a66b-a500d13cfd77-plugins-config-volume\") pod \"0ed39371-700a-43b3-a66b-a500d13cfd77\" (UID: \"0ed39371-700a-43b3-a66b-a500d13cfd77\") "
Nov 03 20:01:56.758748 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:56.758609    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ed39371-700a-43b3-a66b-a500d13cfd77-kube-api-access-vw8jj\" (UniqueName: \"kubernetes.io/projected/0ed39371-700a-43b3-a66b-a500d13cfd77-kube-api-access-vw8jj\") pod \"0ed39371-700a-43b3-a66b-a500d13cfd77\" (UID: \"0ed39371-700a-43b3-a66b-a500d13cfd77\") "
Nov 03 20:01:56.758913 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:56.758891    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ed39371-700a-43b3-a66b-a500d13cfd77-plugins-config-volume" pod "0ed39371-700a-43b3-a66b-a500d13cfd77" (UID: "0ed39371-700a-43b3-a66b-a500d13cfd77"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:01:56.759232 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:56.759209    4921 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ed39371-700a-43b3-a66b-a500d13cfd77-plugins-config-volume\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:56.761076 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:56.761053    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ed39371-700a-43b3-a66b-a500d13cfd77-kube-api-access-vw8jj" pod "0ed39371-700a-43b3-a66b-a500d13cfd77" (UID: "0ed39371-700a-43b3-a66b-a500d13cfd77"). InnerVolumeSpecName "kube-api-access-vw8jj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:01:56.794826 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:56.794798    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2747a4-647847c6c9-njbpx"
Nov 03 20:01:56.860394 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:56.860362    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b642f55a-3767-464c-b9ff-280627b3e277-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b642f55a-3767-464c-b9ff-280627b3e277-plugins-config-volume\") pod \"b642f55a-3767-464c-b9ff-280627b3e277\" (UID: \"b642f55a-3767-464c-b9ff-280627b3e277\") "
Nov 03 20:01:56.860563 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:56.860494    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b642f55a-3767-464c-b9ff-280627b3e277-kube-api-access-5bnsn\" (UniqueName: \"kubernetes.io/projected/b642f55a-3767-464c-b9ff-280627b3e277-kube-api-access-5bnsn\") pod \"b642f55a-3767-464c-b9ff-280627b3e277\" (UID: \"b642f55a-3767-464c-b9ff-280627b3e277\") "
Nov 03 20:01:56.860823 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:56.860798    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b642f55a-3767-464c-b9ff-280627b3e277-plugins-config-volume" pod "b642f55a-3767-464c-b9ff-280627b3e277" (UID: "b642f55a-3767-464c-b9ff-280627b3e277"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:01:56.861188 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:56.861167    4921 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b642f55a-3767-464c-b9ff-280627b3e277-plugins-config-volume\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:56.861188 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:56.861187    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vw8jj\" (UniqueName: \"kubernetes.io/projected/0ed39371-700a-43b3-a66b-a500d13cfd77-kube-api-access-vw8jj\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:56.863087 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:56.863062    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b642f55a-3767-464c-b9ff-280627b3e277-kube-api-access-5bnsn" pod "b642f55a-3767-464c-b9ff-280627b3e277" (UID: "b642f55a-3767-464c-b9ff-280627b3e277"). InnerVolumeSpecName "kube-api-access-5bnsn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:01:56.863882 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:56.863849    4921 generic.go:356] "Generic (PLEG): container finished" podID="0ed39371-700a-43b3-a66b-a500d13cfd77" containerID="dc7004d6f92d63728b8a7c9c79f11fb1040776c50ec2391b2e26b41350a8b9d2" exitCode=0
Nov 03 20:01:56.863997 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:56.863890    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2747a4-67946747c9-4smgl" event={"ID":"0ed39371-700a-43b3-a66b-a500d13cfd77","Type":"ContainerDied","Data":"dc7004d6f92d63728b8a7c9c79f11fb1040776c50ec2391b2e26b41350a8b9d2"}
Nov 03 20:01:56.863997 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:56.863965    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2747a4-67946747c9-4smgl" event={"ID":"0ed39371-700a-43b3-a66b-a500d13cfd77","Type":"ContainerDied","Data":"d4a0f22ac40b005a8249c86804cd11da29b84bdd0da7c6cd72f3f0d53e17a5c1"}
Nov 03 20:01:56.863997 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:56.863899    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2747a4-67946747c9-4smgl"
Nov 03 20:01:56.863997 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:56.863995    4921 scope.go:122] "RemoveContainer" containerID="dc7004d6f92d63728b8a7c9c79f11fb1040776c50ec2391b2e26b41350a8b9d2"
Nov 03 20:01:56.866325 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:56.866302    4921 generic.go:356] "Generic (PLEG): container finished" podID="b642f55a-3767-464c-b9ff-280627b3e277" containerID="bb93252d5bb8e8021bb71d16b794c26a1397b3d0c9ce11660459476a4236c3b6" exitCode=0
Nov 03 20:01:56.866477 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:56.866429    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2747a4-647847c6c9-njbpx"
Nov 03 20:01:56.867580 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:56.867553    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2747a4-647847c6c9-njbpx" event={"ID":"b642f55a-3767-464c-b9ff-280627b3e277","Type":"ContainerDied","Data":"bb93252d5bb8e8021bb71d16b794c26a1397b3d0c9ce11660459476a4236c3b6"}
Nov 03 20:01:56.867747 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:56.867612    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2747a4-647847c6c9-njbpx" event={"ID":"b642f55a-3767-464c-b9ff-280627b3e277","Type":"ContainerDied","Data":"ef7279a809abc914d76746a4e13d5fc702515789b5eb312991f0ebe755e4ec47"}
Nov 03 20:01:56.869330 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:56.869299    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx" event={"ID":"204760c8-b3c3-4b1d-8b67-5bb6cf918407","Type":"ContainerStarted","Data":"d8db868e7d3754b44916f08023a06e1f3503ee59fc6156bd976cfc873dfa26ee"}
Nov 03 20:01:56.885724 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:56.883301    4921 scope.go:122] "RemoveContainer" containerID="dc7004d6f92d63728b8a7c9c79f11fb1040776c50ec2391b2e26b41350a8b9d2"
Nov 03 20:01:56.885724 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:01:56.883974    4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc7004d6f92d63728b8a7c9c79f11fb1040776c50ec2391b2e26b41350a8b9d2\": container with ID starting with dc7004d6f92d63728b8a7c9c79f11fb1040776c50ec2391b2e26b41350a8b9d2 not found: ID does not exist" containerID="dc7004d6f92d63728b8a7c9c79f11fb1040776c50ec2391b2e26b41350a8b9d2"
Nov 03 20:01:56.885724 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:56.884008    4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc7004d6f92d63728b8a7c9c79f11fb1040776c50ec2391b2e26b41350a8b9d2"} err="failed to get container status \"dc7004d6f92d63728b8a7c9c79f11fb1040776c50ec2391b2e26b41350a8b9d2\": rpc error: code = NotFound desc = could not find container \"dc7004d6f92d63728b8a7c9c79f11fb1040776c50ec2391b2e26b41350a8b9d2\": container with ID starting with dc7004d6f92d63728b8a7c9c79f11fb1040776c50ec2391b2e26b41350a8b9d2 not found: ID does not exist"
Nov 03 20:01:56.885724 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:56.884033    4921 scope.go:122] "RemoveContainer" containerID="bb93252d5bb8e8021bb71d16b794c26a1397b3d0c9ce11660459476a4236c3b6"
Nov 03 20:01:56.918863 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:56.918447    4921 scope.go:122] "RemoveContainer" containerID="bb93252d5bb8e8021bb71d16b794c26a1397b3d0c9ce11660459476a4236c3b6"
Nov 03 20:01:56.923741 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:01:56.923631    4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb93252d5bb8e8021bb71d16b794c26a1397b3d0c9ce11660459476a4236c3b6\": container with ID starting with bb93252d5bb8e8021bb71d16b794c26a1397b3d0c9ce11660459476a4236c3b6 not found: ID does not exist" containerID="bb93252d5bb8e8021bb71d16b794c26a1397b3d0c9ce11660459476a4236c3b6"
Nov 03 20:01:56.923928 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:56.923747    4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb93252d5bb8e8021bb71d16b794c26a1397b3d0c9ce11660459476a4236c3b6"} err="failed to get container status \"bb93252d5bb8e8021bb71d16b794c26a1397b3d0c9ce11660459476a4236c3b6\": rpc error: code = NotFound desc = could not find container \"bb93252d5bb8e8021bb71d16b794c26a1397b3d0c9ce11660459476a4236c3b6\": container with ID starting with bb93252d5bb8e8021bb71d16b794c26a1397b3d0c9ce11660459476a4236c3b6 not found: ID does not exist"
Nov 03 20:01:56.924060 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:56.924031    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e747102-94ba-49d0-a2b5-5ff6040c1556" path="/var/lib/kubelet/pods/9e747102-94ba-49d0-a2b5-5ff6040c1556/volumes"
Nov 03 20:01:56.940756 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:56.940685    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx" podStartSLOduration=211.940647492 podStartE2EDuration="3m31.940647492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:58:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:56.914698812 +0000 UTC m=+1830.158664278" watchObservedRunningTime="2026-11-03 20:01:56.940647492 +0000 UTC m=+1830.184612958"
Nov 03 20:01:56.960758 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:56.960721    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2747a4-647847c6c9-njbpx"]
Nov 03 20:01:56.960943 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:56.960782    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2747a4-647847c6c9-njbpx"]
Nov 03 20:01:56.967989 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:56.967890    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2747a4-67946747c9-4smgl"]
Nov 03 20:01:56.974421 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:56.974378    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bnsn\" (UniqueName: \"kubernetes.io/projected/b642f55a-3767-464c-b9ff-280627b3e277-kube-api-access-5bnsn\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:56.990729 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:56.988794    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2747a4-67946747c9-4smgl"]
Nov 03 20:01:57.474832 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:57.474646    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx"
Nov 03 20:01:57.476780 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:57.476750    4921 patch_prober.go:29] interesting pod/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body=
Nov 03 20:01:57.476978 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:57.476944    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx" podUID="204760c8-b3c3-4b1d-8b67-5bb6cf918407" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused"
Nov 03 20:01:58.470693 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:58.469400    4921 patch_prober.go:29] interesting pod/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body=
Nov 03 20:01:58.470693 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:58.469462    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx" podUID="204760c8-b3c3-4b1d-8b67-5bb6cf918407" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused"
Nov 03 20:01:58.893166 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:58.893130    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ed39371-700a-43b3-a66b-a500d13cfd77" path="/var/lib/kubelet/pods/0ed39371-700a-43b3-a66b-a500d13cfd77/volumes"
Nov 03 20:01:58.896042 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:58.896017    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b642f55a-3767-464c-b9ff-280627b3e277" path="/var/lib/kubelet/pods/b642f55a-3767-464c-b9ff-280627b3e277/volumes"
Nov 03 20:01:59.484717 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:59.478279    4921 patch_prober.go:29] interesting pod/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body=
Nov 03 20:01:59.484717 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:59.478363    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx" podUID="204760c8-b3c3-4b1d-8b67-5bb6cf918407" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused"
Nov 03 20:01:59.604693 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:59.603642    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-546fr"]
Nov 03 20:01:59.604693 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:59.603938    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-546fr" podUID="ca28dda5-c003-4070-b331-43c5a51d9f87" containerName="istio-proxy" containerID="cri-o://490d991e7f3e1faabfeb7d453d9ef637234a4215fc3d2a4d35d8f352c181338e" gracePeriod=30
Nov 03 20:01:59.806971 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:59.793939    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-x4frm"]
Nov 03 20:01:59.806971 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:01:59.794213    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-x4frm" podUID="6dec4941-e24a-4b8b-852c-a9d0128c533c" containerName="istio-proxy" containerID="cri-o://fad4cd1770166848ac8e89407e7eefd32fda359488ceff800282bebf483caea8" gracePeriod=30
Nov 03 20:02:00.477608 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:00.477573    4921 patch_prober.go:29] interesting pod/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body=
Nov 03 20:02:00.479329 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:00.479267    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx" podUID="204760c8-b3c3-4b1d-8b67-5bb6cf918407" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused"
Nov 03 20:02:01.480198 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:01.473119    4921 patch_prober.go:29] interesting pod/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body=
Nov 03 20:02:01.480198 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:01.473190    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx" podUID="204760c8-b3c3-4b1d-8b67-5bb6cf918407" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused"
Nov 03 20:02:02.472704 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:02.472123    4921 patch_prober.go:29] interesting pod/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body=
Nov 03 20:02:02.472704 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:02.472195    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx" podUID="204760c8-b3c3-4b1d-8b67-5bb6cf918407" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused"
Nov 03 20:02:02.926201 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:02:02.926161    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 20:02:03.478321 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:03.478286    4921 patch_prober.go:29] interesting pod/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body=
Nov 03 20:02:03.483883 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:03.478532    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx" podUID="204760c8-b3c3-4b1d-8b67-5bb6cf918407" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused"
Nov 03 20:02:04.474301 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:04.474268    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx"
Nov 03 20:02:04.474837 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:04.474817    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx"
Nov 03 20:02:04.475499 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:04.475477    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx"
Nov 03 20:02:05.063404 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:05.063373    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-546fr"
Nov 03 20:02:05.154599 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:05.154559    4921 generic.go:356] "Generic (PLEG): container finished" podID="ca28dda5-c003-4070-b331-43c5a51d9f87" containerID="490d991e7f3e1faabfeb7d453d9ef637234a4215fc3d2a4d35d8f352c181338e" exitCode=0
Nov 03 20:02:05.154972 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:05.154941    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-546fr" event={"ID":"ca28dda5-c003-4070-b331-43c5a51d9f87","Type":"ContainerDied","Data":"490d991e7f3e1faabfeb7d453d9ef637234a4215fc3d2a4d35d8f352c181338e"}
Nov 03 20:02:05.155106 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:05.154999    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-546fr" event={"ID":"ca28dda5-c003-4070-b331-43c5a51d9f87","Type":"ContainerDied","Data":"0bf90900eeaeaaba4911349b3e8a79144fec40569e74fb732fdb5d9b1c09c7ec"}
Nov 03 20:02:05.155106 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:05.155021    4921 scope.go:122] "RemoveContainer" containerID="490d991e7f3e1faabfeb7d453d9ef637234a4215fc3d2a4d35d8f352c181338e"
Nov 03 20:02:05.155208 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:05.155135    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-546fr"
Nov 03 20:02:05.158025 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:05.157984    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca28dda5-c003-4070-b331-43c5a51d9f87-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca28dda5-c003-4070-b331-43c5a51d9f87-workload-certs\") pod \"ca28dda5-c003-4070-b331-43c5a51d9f87\" (UID: \"ca28dda5-c003-4070-b331-43c5a51d9f87\") "
Nov 03 20:02:05.158158 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:05.158068    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca28dda5-c003-4070-b331-43c5a51d9f87-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca28dda5-c003-4070-b331-43c5a51d9f87-istiod-ca-cert\") pod \"ca28dda5-c003-4070-b331-43c5a51d9f87\" (UID: \"ca28dda5-c003-4070-b331-43c5a51d9f87\") "
Nov 03 20:02:05.158232 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:05.158206    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca28dda5-c003-4070-b331-43c5a51d9f87-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca28dda5-c003-4070-b331-43c5a51d9f87-istio-envoy\") pod \"ca28dda5-c003-4070-b331-43c5a51d9f87\" (UID: \"ca28dda5-c003-4070-b331-43c5a51d9f87\") "
Nov 03 20:02:05.158288 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:05.158268    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca28dda5-c003-4070-b331-43c5a51d9f87-kube-api-access-7pr7s\" (UniqueName: \"kubernetes.io/projected/ca28dda5-c003-4070-b331-43c5a51d9f87-kube-api-access-7pr7s\") pod \"ca28dda5-c003-4070-b331-43c5a51d9f87\" (UID: \"ca28dda5-c003-4070-b331-43c5a51d9f87\") "
Nov 03 20:02:05.158347 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:05.158326    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca28dda5-c003-4070-b331-43c5a51d9f87-istio-token\" (UniqueName: \"kubernetes.io/projected/ca28dda5-c003-4070-b331-43c5a51d9f87-istio-token\") pod \"ca28dda5-c003-4070-b331-43c5a51d9f87\" (UID: \"ca28dda5-c003-4070-b331-43c5a51d9f87\") "
Nov 03 20:02:05.158402 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:05.158370    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ca28dda5-c003-4070-b331-43c5a51d9f87-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca28dda5-c003-4070-b331-43c5a51d9f87-istio-podinfo\") pod \"ca28dda5-c003-4070-b331-43c5a51d9f87\" (UID: \"ca28dda5-c003-4070-b331-43c5a51d9f87\") "
Nov 03 20:02:05.158461 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:05.158447    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca28dda5-c003-4070-b331-43c5a51d9f87-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca28dda5-c003-4070-b331-43c5a51d9f87-credential-socket\") pod \"ca28dda5-c003-4070-b331-43c5a51d9f87\" (UID: \"ca28dda5-c003-4070-b331-43c5a51d9f87\") "
Nov 03 20:02:05.159015 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:05.158650    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca28dda5-c003-4070-b331-43c5a51d9f87-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca28dda5-c003-4070-b331-43c5a51d9f87-workload-socket\") pod \"ca28dda5-c003-4070-b331-43c5a51d9f87\" (UID: \"ca28dda5-c003-4070-b331-43c5a51d9f87\") "
Nov 03 20:02:05.159015 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:05.158797    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca28dda5-c003-4070-b331-43c5a51d9f87-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca28dda5-c003-4070-b331-43c5a51d9f87-istio-data\") pod \"ca28dda5-c003-4070-b331-43c5a51d9f87\" (UID: \"ca28dda5-c003-4070-b331-43c5a51d9f87\") "
Nov 03 20:02:05.159800 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:05.159585    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca28dda5-c003-4070-b331-43c5a51d9f87-credential-socket" pod "ca28dda5-c003-4070-b331-43c5a51d9f87" (UID: "ca28dda5-c003-4070-b331-43c5a51d9f87"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:05.159800 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:05.159791    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca28dda5-c003-4070-b331-43c5a51d9f87-istio-data" pod "ca28dda5-c003-4070-b331-43c5a51d9f87" (UID: "ca28dda5-c003-4070-b331-43c5a51d9f87"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:05.160061 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:05.159969    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca28dda5-c003-4070-b331-43c5a51d9f87-workload-socket" pod "ca28dda5-c003-4070-b331-43c5a51d9f87" (UID: "ca28dda5-c003-4070-b331-43c5a51d9f87"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:05.160061 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:05.160018    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca28dda5-c003-4070-b331-43c5a51d9f87-workload-certs" pod "ca28dda5-c003-4070-b331-43c5a51d9f87" (UID: "ca28dda5-c003-4070-b331-43c5a51d9f87"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:05.160529 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:05.160495    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca28dda5-c003-4070-b331-43c5a51d9f87-istiod-ca-cert" pod "ca28dda5-c003-4070-b331-43c5a51d9f87" (UID: "ca28dda5-c003-4070-b331-43c5a51d9f87"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:02:05.162752 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:05.162725    4921 generic.go:356] "Generic (PLEG): container finished" podID="6dec4941-e24a-4b8b-852c-a9d0128c533c" containerID="fad4cd1770166848ac8e89407e7eefd32fda359488ceff800282bebf483caea8" exitCode=0
Nov 03 20:02:05.163616 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:05.163584    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-x4frm" event={"ID":"6dec4941-e24a-4b8b-852c-a9d0128c533c","Type":"ContainerDied","Data":"fad4cd1770166848ac8e89407e7eefd32fda359488ceff800282bebf483caea8"}
Nov 03 20:02:05.164441 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:05.164418    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca28dda5-c003-4070-b331-43c5a51d9f87-istio-envoy" pod "ca28dda5-c003-4070-b331-43c5a51d9f87" (UID: "ca28dda5-c003-4070-b331-43c5a51d9f87"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:05.166485 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:05.166456    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ca28dda5-c003-4070-b331-43c5a51d9f87-istio-podinfo" pod "ca28dda5-c003-4070-b331-43c5a51d9f87" (UID: "ca28dda5-c003-4070-b331-43c5a51d9f87"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:02:05.166632 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:05.166612    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca28dda5-c003-4070-b331-43c5a51d9f87-kube-api-access-7pr7s" pod "ca28dda5-c003-4070-b331-43c5a51d9f87" (UID: "ca28dda5-c003-4070-b331-43c5a51d9f87"). InnerVolumeSpecName "kube-api-access-7pr7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:02:05.166772 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:05.166751    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca28dda5-c003-4070-b331-43c5a51d9f87-istio-token" pod "ca28dda5-c003-4070-b331-43c5a51d9f87" (UID: "ca28dda5-c003-4070-b331-43c5a51d9f87"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:02:05.192792 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:05.192763    4921 scope.go:122] "RemoveContainer" containerID="490d991e7f3e1faabfeb7d453d9ef637234a4215fc3d2a4d35d8f352c181338e"
Nov 03 20:02:05.193091 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:02:05.193060    4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"490d991e7f3e1faabfeb7d453d9ef637234a4215fc3d2a4d35d8f352c181338e\": container with ID starting with 490d991e7f3e1faabfeb7d453d9ef637234a4215fc3d2a4d35d8f352c181338e not found: ID does not exist" containerID="490d991e7f3e1faabfeb7d453d9ef637234a4215fc3d2a4d35d8f352c181338e"
Nov 03 20:02:05.193197 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:05.193099    4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"490d991e7f3e1faabfeb7d453d9ef637234a4215fc3d2a4d35d8f352c181338e"} err="failed to get container status \"490d991e7f3e1faabfeb7d453d9ef637234a4215fc3d2a4d35d8f352c181338e\": rpc error: code = NotFound desc = could not find container \"490d991e7f3e1faabfeb7d453d9ef637234a4215fc3d2a4d35d8f352c181338e\": container with ID starting with 490d991e7f3e1faabfeb7d453d9ef637234a4215fc3d2a4d35d8f352c181338e not found: ID does not exist"
Nov 03 20:02:05.238020 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:05.237991    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-x4frm"
Nov 03 20:02:05.260840 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:05.260688    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6dec4941-e24a-4b8b-852c-a9d0128c533c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dec4941-e24a-4b8b-852c-a9d0128c533c-istio-podinfo\") pod \"6dec4941-e24a-4b8b-852c-a9d0128c533c\" (UID: \"6dec4941-e24a-4b8b-852c-a9d0128c533c\") "
Nov 03 20:02:05.260840 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:05.260762    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dec4941-e24a-4b8b-852c-a9d0128c533c-istio-token\" (UniqueName: \"kubernetes.io/projected/6dec4941-e24a-4b8b-852c-a9d0128c533c-istio-token\") pod \"6dec4941-e24a-4b8b-852c-a9d0128c533c\" (UID: \"6dec4941-e24a-4b8b-852c-a9d0128c533c\") "
Nov 03 20:02:05.261091 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:05.260877    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dec4941-e24a-4b8b-852c-a9d0128c533c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dec4941-e24a-4b8b-852c-a9d0128c533c-workload-certs\") pod \"6dec4941-e24a-4b8b-852c-a9d0128c533c\" (UID: \"6dec4941-e24a-4b8b-852c-a9d0128c533c\") "
Nov 03 20:02:05.261091 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:05.260919    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6dec4941-e24a-4b8b-852c-a9d0128c533c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dec4941-e24a-4b8b-852c-a9d0128c533c-istiod-ca-cert\") pod \"6dec4941-e24a-4b8b-852c-a9d0128c533c\" (UID: \"6dec4941-e24a-4b8b-852c-a9d0128c533c\") "
Nov 03 20:02:05.261091 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:05.260971    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dec4941-e24a-4b8b-852c-a9d0128c533c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dec4941-e24a-4b8b-852c-a9d0128c533c-istio-envoy\") pod \"6dec4941-e24a-4b8b-852c-a9d0128c533c\" (UID: \"6dec4941-e24a-4b8b-852c-a9d0128c533c\") "
Nov 03 20:02:05.261091 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:05.261050    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dec4941-e24a-4b8b-852c-a9d0128c533c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dec4941-e24a-4b8b-852c-a9d0128c533c-istio-data\") pod \"6dec4941-e24a-4b8b-852c-a9d0128c533c\" (UID: \"6dec4941-e24a-4b8b-852c-a9d0128c533c\") "
Nov 03 20:02:05.261312 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:05.261151    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dec4941-e24a-4b8b-852c-a9d0128c533c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dec4941-e24a-4b8b-852c-a9d0128c533c-credential-socket\") pod \"6dec4941-e24a-4b8b-852c-a9d0128c533c\" (UID: \"6dec4941-e24a-4b8b-852c-a9d0128c533c\") "
Nov 03 20:02:05.261312 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:05.261193    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dec4941-e24a-4b8b-852c-a9d0128c533c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dec4941-e24a-4b8b-852c-a9d0128c533c-workload-socket\") pod \"6dec4941-e24a-4b8b-852c-a9d0128c533c\" (UID: \"6dec4941-e24a-4b8b-852c-a9d0128c533c\") "
Nov 03 20:02:05.261312 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:05.261246    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dec4941-e24a-4b8b-852c-a9d0128c533c-kube-api-access-2rzzc\" (UniqueName: \"kubernetes.io/projected/6dec4941-e24a-4b8b-852c-a9d0128c533c-kube-api-access-2rzzc\") pod \"6dec4941-e24a-4b8b-852c-a9d0128c533c\" (UID: \"6dec4941-e24a-4b8b-852c-a9d0128c533c\") "
Nov 03 20:02:05.262793 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:05.261550    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dec4941-e24a-4b8b-852c-a9d0128c533c-istiod-ca-cert" pod "6dec4941-e24a-4b8b-852c-a9d0128c533c" (UID: "6dec4941-e24a-4b8b-852c-a9d0128c533c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:02:05.262793 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:05.262150    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dec4941-e24a-4b8b-852c-a9d0128c533c-workload-certs" pod "6dec4941-e24a-4b8b-852c-a9d0128c533c" (UID: "6dec4941-e24a-4b8b-852c-a9d0128c533c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:05.262793 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:05.262386    4921 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca28dda5-c003-4070-b331-43c5a51d9f87-workload-socket\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:05.262793 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:05.262407    4921 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca28dda5-c003-4070-b331-43c5a51d9f87-istio-data\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:05.262793 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:05.262422    4921 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dec4941-e24a-4b8b-852c-a9d0128c533c-workload-certs\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:05.262793 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:05.262435    4921 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dec4941-e24a-4b8b-852c-a9d0128c533c-istiod-ca-cert\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:05.262793 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:05.262448    4921 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca28dda5-c003-4070-b331-43c5a51d9f87-workload-certs\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:05.262793 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:05.262462    4921 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca28dda5-c003-4070-b331-43c5a51d9f87-istiod-ca-cert\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:05.262793 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:05.262477    4921 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca28dda5-c003-4070-b331-43c5a51d9f87-istio-envoy\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:05.262793 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:05.262491    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pr7s\" (UniqueName: \"kubernetes.io/projected/ca28dda5-c003-4070-b331-43c5a51d9f87-kube-api-access-7pr7s\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:05.262793 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:05.262507    4921 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca28dda5-c003-4070-b331-43c5a51d9f87-istio-token\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:05.262793 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:05.262522    4921 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca28dda5-c003-4070-b331-43c5a51d9f87-istio-podinfo\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:05.262793 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:05.262536    4921 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca28dda5-c003-4070-b331-43c5a51d9f87-credential-socket\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:05.262793 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:05.262599    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dec4941-e24a-4b8b-852c-a9d0128c533c-workload-socket" pod "6dec4941-e24a-4b8b-852c-a9d0128c533c" (UID: "6dec4941-e24a-4b8b-852c-a9d0128c533c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:05.263605 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:05.263163    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dec4941-e24a-4b8b-852c-a9d0128c533c-istio-data" pod "6dec4941-e24a-4b8b-852c-a9d0128c533c" (UID: "6dec4941-e24a-4b8b-852c-a9d0128c533c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:05.263605 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:05.263165    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dec4941-e24a-4b8b-852c-a9d0128c533c-credential-socket" pod "6dec4941-e24a-4b8b-852c-a9d0128c533c" (UID: "6dec4941-e24a-4b8b-852c-a9d0128c533c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:05.264995 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:05.264969    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6dec4941-e24a-4b8b-852c-a9d0128c533c-istio-podinfo" pod "6dec4941-e24a-4b8b-852c-a9d0128c533c" (UID: "6dec4941-e24a-4b8b-852c-a9d0128c533c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:02:05.265513 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:05.265492    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dec4941-e24a-4b8b-852c-a9d0128c533c-istio-token" pod "6dec4941-e24a-4b8b-852c-a9d0128c533c" (UID: "6dec4941-e24a-4b8b-852c-a9d0128c533c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:02:05.266237 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:05.266212    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dec4941-e24a-4b8b-852c-a9d0128c533c-istio-envoy" pod "6dec4941-e24a-4b8b-852c-a9d0128c533c" (UID: "6dec4941-e24a-4b8b-852c-a9d0128c533c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:05.267731 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:05.267658    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dec4941-e24a-4b8b-852c-a9d0128c533c-kube-api-access-2rzzc" pod "6dec4941-e24a-4b8b-852c-a9d0128c533c" (UID: "6dec4941-e24a-4b8b-852c-a9d0128c533c"). InnerVolumeSpecName "kube-api-access-2rzzc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:02:05.364027 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:05.363864    4921 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dec4941-e24a-4b8b-852c-a9d0128c533c-istio-envoy\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:05.364027 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:05.363898    4921 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dec4941-e24a-4b8b-852c-a9d0128c533c-istio-data\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:05.364027 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:05.363914    4921 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dec4941-e24a-4b8b-852c-a9d0128c533c-credential-socket\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:05.364027 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:05.363923    4921 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dec4941-e24a-4b8b-852c-a9d0128c533c-workload-socket\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:05.364027 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:05.363932    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rzzc\" (UniqueName: \"kubernetes.io/projected/6dec4941-e24a-4b8b-852c-a9d0128c533c-kube-api-access-2rzzc\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:05.364027 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:05.363947    4921 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dec4941-e24a-4b8b-852c-a9d0128c533c-istio-podinfo\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:05.364027 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:05.363960    4921 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dec4941-e24a-4b8b-852c-a9d0128c533c-istio-token\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:05.493694 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:05.492961    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-546fr"]
Nov 03 20:02:05.500987 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:05.500929    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-546fr"]
Nov 03 20:02:06.183307 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:06.183266    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-x4frm" event={"ID":"6dec4941-e24a-4b8b-852c-a9d0128c533c","Type":"ContainerDied","Data":"03aef56d98b20a9a730fe5e4c4fbf2509b9deaf08d9677a79ab25168faeb1ac2"}
Nov 03 20:02:06.183307 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:06.183301    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-x4frm"
Nov 03 20:02:06.183564 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:06.183404    4921 scope.go:122] "RemoveContainer" containerID="fad4cd1770166848ac8e89407e7eefd32fda359488ceff800282bebf483caea8"
Nov 03 20:02:06.215611 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:06.215576    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-x4frm"]
Nov 03 20:02:06.221375 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:06.221344    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-332694-openshift-default-5c5687c879-x4frm"]
Nov 03 20:02:06.960175 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:06.949687    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6dec4941-e24a-4b8b-852c-a9d0128c533c" path="/var/lib/kubelet/pods/6dec4941-e24a-4b8b-852c-a9d0128c533c/volumes"
Nov 03 20:02:06.982691 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:06.973915    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca28dda5-c003-4070-b331-43c5a51d9f87" path="/var/lib/kubelet/pods/ca28dda5-c003-4070-b331-43c5a51d9f87/volumes"
Nov 03 20:02:07.855710 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:07.855658    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2747a4-gjqwl"]
Nov 03 20:02:07.860612 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:07.860583    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2747a4-gjqwl"]
Nov 03 20:02:08.893337 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:08.893306    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e813b6c-95bc-462b-bb02-0425f1b408ed" path="/var/lib/kubelet/pods/8e813b6c-95bc-462b-bb02-0425f1b408ed/volumes"
Nov 03 20:02:21.006642 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:21.006601    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ffe132-ngjqb"]
Nov 03 20:02:21.017745 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:21.014582    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ffe132-ngjqb"]
Nov 03 20:02:21.323939 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:21.322283    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9"]
Nov 03 20:02:22.890762 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:22.890725    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d47483d-40b0-47ae-9478-0531cfb8dc77" path="/var/lib/kubelet/pods/1d47483d-40b0-47ae-9478-0531cfb8dc77/volumes"
Nov 03 20:02:26.620583 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:26.620550    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-102379-5c958cc4ff-t94ls"]
Nov 03 20:02:26.641502 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:26.641464    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-102379-64bd4947ff-7qsss"]
Nov 03 20:02:31.621069 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:31.621019    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-102379-5c958cc4ff-t94ls" podUID="2fd7fd41-37ac-4e6c-8042-2c7b4a1f0674" containerName="payload-processing" containerID="cri-o://fe8d07760a751fe0ac4ce6791008bcc8e054dd89fad7cae8d2698c419ba3d90a" gracePeriod=25
Nov 03 20:02:31.642781 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:31.642717    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-102379-64bd4947ff-7qsss" podUID="06971289-0da8-4921-9df7-140ff477056c" containerName="payload-pre-processing" containerID="cri-o://6fbecabd718fedb595dc30d1e25f95eb5572e0ea3f98a6c73d1f0ad159af4614" gracePeriod=25
Nov 03 20:02:32.027718 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:32.027643    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-79c47f/test-model-79c47f-kserve-665c6d6575-b5svq"]
Nov 03 20:02:32.962019 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:02:32.961972    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 20:02:35.533703 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:35.533624    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-102379-5c958cc4ff-t94ls" podUID="2fd7fd41-37ac-4e6c-8042-2c7b4a1f0674" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:02:35.534124 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:35.533770    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-102379-64bd4947ff-7qsss" podUID="06971289-0da8-4921-9df7-140ff477056c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:02:36.564422 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:36.564379    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9" podUID="e6a71aa3-33b3-42bd-b270-64ce3260cc2a" containerName="main" containerID="cri-o://0c4f0308f1137c1f5452829c6718700c7aada360b1c60edb67f1d52798b368d9" gracePeriod=45
Nov 03 20:02:36.888310 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:36.888283    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9"
Nov 03 20:02:36.941999 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:36.941963    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6a71aa3-33b3-42bd-b270-64ce3260cc2a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6a71aa3-33b3-42bd-b270-64ce3260cc2a-tmp-dir\") pod \"e6a71aa3-33b3-42bd-b270-64ce3260cc2a\" (UID: \"e6a71aa3-33b3-42bd-b270-64ce3260cc2a\") "
Nov 03 20:02:36.942223 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:36.942089    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6a71aa3-33b3-42bd-b270-64ce3260cc2a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6a71aa3-33b3-42bd-b270-64ce3260cc2a-model-cache\") pod \"e6a71aa3-33b3-42bd-b270-64ce3260cc2a\" (UID: \"e6a71aa3-33b3-42bd-b270-64ce3260cc2a\") "
Nov 03 20:02:36.942223 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:36.942161    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6a71aa3-33b3-42bd-b270-64ce3260cc2a-home\" (UniqueName: \"kubernetes.io/empty-dir/e6a71aa3-33b3-42bd-b270-64ce3260cc2a-home\") pod \"e6a71aa3-33b3-42bd-b270-64ce3260cc2a\" (UID: \"e6a71aa3-33b3-42bd-b270-64ce3260cc2a\") "
Nov 03 20:02:36.942361 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:36.942302    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6a71aa3-33b3-42bd-b270-64ce3260cc2a-tmp-dir" pod "e6a71aa3-33b3-42bd-b270-64ce3260cc2a" (UID: "e6a71aa3-33b3-42bd-b270-64ce3260cc2a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:36.942361 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:36.942313    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6a71aa3-33b3-42bd-b270-64ce3260cc2a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6a71aa3-33b3-42bd-b270-64ce3260cc2a-dshm\") pod \"e6a71aa3-33b3-42bd-b270-64ce3260cc2a\" (UID: \"e6a71aa3-33b3-42bd-b270-64ce3260cc2a\") "
Nov 03 20:02:36.942480 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:36.942415    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6a71aa3-33b3-42bd-b270-64ce3260cc2a-tls-certs\" (UniqueName: \"kubernetes.io/secret/e6a71aa3-33b3-42bd-b270-64ce3260cc2a-tls-certs\") pod \"e6a71aa3-33b3-42bd-b270-64ce3260cc2a\" (UID: \"e6a71aa3-33b3-42bd-b270-64ce3260cc2a\") "
Nov 03 20:02:36.942561 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:36.942538    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6a71aa3-33b3-42bd-b270-64ce3260cc2a-kube-api-access-4fkrz\" (UniqueName: \"kubernetes.io/projected/e6a71aa3-33b3-42bd-b270-64ce3260cc2a-kube-api-access-4fkrz\") pod \"e6a71aa3-33b3-42bd-b270-64ce3260cc2a\" (UID: \"e6a71aa3-33b3-42bd-b270-64ce3260cc2a\") "
Nov 03 20:02:36.942942 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:36.942457    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6a71aa3-33b3-42bd-b270-64ce3260cc2a-model-cache" pod "e6a71aa3-33b3-42bd-b270-64ce3260cc2a" (UID: "e6a71aa3-33b3-42bd-b270-64ce3260cc2a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:36.943288 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:36.943265    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6a71aa3-33b3-42bd-b270-64ce3260cc2a-home" pod "e6a71aa3-33b3-42bd-b270-64ce3260cc2a" (UID: "e6a71aa3-33b3-42bd-b270-64ce3260cc2a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:36.943745 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:36.943720    4921 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6a71aa3-33b3-42bd-b270-64ce3260cc2a-model-cache\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:36.943865 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:36.943759    4921 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6a71aa3-33b3-42bd-b270-64ce3260cc2a-home\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:36.943865 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:36.943774    4921 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6a71aa3-33b3-42bd-b270-64ce3260cc2a-tmp-dir\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:36.946111 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:36.946070    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6a71aa3-33b3-42bd-b270-64ce3260cc2a-tls-certs" pod "e6a71aa3-33b3-42bd-b270-64ce3260cc2a" (UID: "e6a71aa3-33b3-42bd-b270-64ce3260cc2a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:02:36.946242 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:36.946219    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6a71aa3-33b3-42bd-b270-64ce3260cc2a-kube-api-access-4fkrz" pod "e6a71aa3-33b3-42bd-b270-64ce3260cc2a" (UID: "e6a71aa3-33b3-42bd-b270-64ce3260cc2a"). InnerVolumeSpecName "kube-api-access-4fkrz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:02:36.948861 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:36.948000    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6a71aa3-33b3-42bd-b270-64ce3260cc2a-dshm" pod "e6a71aa3-33b3-42bd-b270-64ce3260cc2a" (UID: "e6a71aa3-33b3-42bd-b270-64ce3260cc2a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:37.045905 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:37.045874    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fkrz\" (UniqueName: \"kubernetes.io/projected/e6a71aa3-33b3-42bd-b270-64ce3260cc2a-kube-api-access-4fkrz\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:37.045905 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:37.045901    4921 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6a71aa3-33b3-42bd-b270-64ce3260cc2a-dshm\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:37.046106 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:37.045916    4921 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6a71aa3-33b3-42bd-b270-64ce3260cc2a-tls-certs\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:37.645282 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:37.645243    4921 generic.go:356] "Generic (PLEG): container finished" podID="e6a71aa3-33b3-42bd-b270-64ce3260cc2a" containerID="0c4f0308f1137c1f5452829c6718700c7aada360b1c60edb67f1d52798b368d9" exitCode=0
Nov 03 20:02:37.645775 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:37.645309    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9"
Nov 03 20:02:37.645775 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:37.645428    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9" event={"ID":"e6a71aa3-33b3-42bd-b270-64ce3260cc2a","Type":"ContainerDied","Data":"0c4f0308f1137c1f5452829c6718700c7aada360b1c60edb67f1d52798b368d9"}
Nov 03 20:02:37.645775 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:37.645495    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9" event={"ID":"e6a71aa3-33b3-42bd-b270-64ce3260cc2a","Type":"ContainerDied","Data":"92bd9705094ebe80f92542898488a34d7082ff55ccb75dc6e8940ea794b04cf8"}
Nov 03 20:02:37.645775 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:37.645515    4921 scope.go:122] "RemoveContainer" containerID="0c4f0308f1137c1f5452829c6718700c7aada360b1c60edb67f1d52798b368d9"
Nov 03 20:02:37.666085 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:37.666054    4921 scope.go:122] "RemoveContainer" containerID="0c4f0308f1137c1f5452829c6718700c7aada360b1c60edb67f1d52798b368d9"
Nov 03 20:02:37.666420 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:02:37.666394    4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c4f0308f1137c1f5452829c6718700c7aada360b1c60edb67f1d52798b368d9\": container with ID starting with 0c4f0308f1137c1f5452829c6718700c7aada360b1c60edb67f1d52798b368d9 not found: ID does not exist" containerID="0c4f0308f1137c1f5452829c6718700c7aada360b1c60edb67f1d52798b368d9"
Nov 03 20:02:37.666487 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:37.666429    4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c4f0308f1137c1f5452829c6718700c7aada360b1c60edb67f1d52798b368d9"} err="failed to get container status \"0c4f0308f1137c1f5452829c6718700c7aada360b1c60edb67f1d52798b368d9\": rpc error: code = NotFound desc = could not find container \"0c4f0308f1137c1f5452829c6718700c7aada360b1c60edb67f1d52798b368d9\": container with ID starting with 0c4f0308f1137c1f5452829c6718700c7aada360b1c60edb67f1d52798b368d9 not found: ID does not exist"
Nov 03 20:02:37.673769 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:37.673743    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9"]
Nov 03 20:02:37.678594 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:37.678572    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-102379/distinct-2-w6-102379-kserve-5595c84fbb-qtcb9"]
Nov 03 20:02:38.894744 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:38.894701    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6a71aa3-33b3-42bd-b270-64ce3260cc2a" path="/var/lib/kubelet/pods/e6a71aa3-33b3-42bd-b270-64ce3260cc2a/volumes"
Nov 03 20:02:40.422910 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:40.422870    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rl9tr"]
Nov 03 20:02:40.423398 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:40.423079    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rl9tr" podUID="cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd" containerName="istio-proxy" containerID="cri-o://76fb700ed57858ed7215c3174791b2cfba78745c09e53fedd4ea62b8e92c7075" gracePeriod=30
Nov 03 20:02:42.467688 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:42.467640    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-225668-65675d6655-qdr5l"]
Nov 03 20:02:42.482997 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:42.482965    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-225668-6d7866c8dc-29bbm"]
Nov 03 20:02:45.533570 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:45.533530    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-102379-64bd4947ff-7qsss" podUID="06971289-0da8-4921-9df7-140ff477056c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:02:45.534570 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:45.534538    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-102379-5c958cc4ff-t94ls" podUID="2fd7fd41-37ac-4e6c-8042-2c7b4a1f0674" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:02:45.795365 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:45.795050    4921 generic.go:356] "Generic (PLEG): container finished" podID="cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd" containerID="76fb700ed57858ed7215c3174791b2cfba78745c09e53fedd4ea62b8e92c7075" exitCode=0
Nov 03 20:02:45.795365 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:45.795197    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rl9tr" event={"ID":"cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd","Type":"ContainerDied","Data":"76fb700ed57858ed7215c3174791b2cfba78745c09e53fedd4ea62b8e92c7075"}
Nov 03 20:02:45.795365 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:45.795290    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rl9tr" event={"ID":"cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd","Type":"ContainerDied","Data":"b29660423e049ddb04ef59ad053fa3fc392a5a275046f2c26df3cd5c92f1ac35"}
Nov 03 20:02:45.795365 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:45.795315    4921 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b29660423e049ddb04ef59ad053fa3fc392a5a275046f2c26df3cd5c92f1ac35"
Nov 03 20:02:45.810547 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:45.810520    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rl9tr"
Nov 03 20:02:45.893250 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:45.893203    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd-istiod-ca-cert\") pod \"cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd\" (UID: \"cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd\") "
Nov 03 20:02:45.893440 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:45.893309    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd-workload-socket\") pod \"cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd\" (UID: \"cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd\") "
Nov 03 20:02:45.893440 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:45.893348    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd-istio-envoy\") pod \"cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd\" (UID: \"cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd\") "
Nov 03 20:02:45.893440 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:45.893434    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd-istio-podinfo\") pod \"cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd\" (UID: \"cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd\") "
Nov 03 20:02:45.893627 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:45.893547    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd-workload-certs\") pod \"cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd\" (UID: \"cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd\") "
Nov 03 20:02:45.893720 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:45.893612    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd-istio-token\" (UniqueName: \"kubernetes.io/projected/cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd-istio-token\") pod \"cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd\" (UID: \"cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd\") "
Nov 03 20:02:45.893720 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:45.893692    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd-istio-data\") pod \"cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd\" (UID: \"cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd\") "
Nov 03 20:02:45.893840 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:45.893742    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd-workload-socket" pod "cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd" (UID: "cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:45.893840 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:45.893776    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd-kube-api-access-spz2z\" (UniqueName: \"kubernetes.io/projected/cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd-kube-api-access-spz2z\") pod \"cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd\" (UID: \"cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd\") "
Nov 03 20:02:45.893840 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:45.893813    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd-credential-socket\") pod \"cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd\" (UID: \"cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd\") "
Nov 03 20:02:45.894064 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:45.893931    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd-workload-certs" pod "cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd" (UID: "cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:45.894629 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:45.894605    4921 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd-workload-certs\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:45.894758 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:45.894632    4921 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd-workload-socket\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:45.894758 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:45.894651    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd-istio-data" pod "cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd" (UID: "cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:45.894955 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:45.894902    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd-credential-socket" pod "cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd" (UID: "cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:45.895859 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:45.895832    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd-istiod-ca-cert" pod "cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd" (UID: "cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:02:45.896934 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:45.896909    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd-kube-api-access-spz2z" pod "cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd" (UID: "cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd"). InnerVolumeSpecName "kube-api-access-spz2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:02:45.898811 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:45.898772    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd-istio-envoy" pod "cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd" (UID: "cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:45.899557 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:45.899238    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd-istio-podinfo" pod "cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd" (UID: "cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:02:45.905148 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:45.905112    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd-istio-token" pod "cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd" (UID: "cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:02:45.996645 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:45.996607    4921 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd-istio-envoy\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:45.996645 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:45.996640    4921 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd-istio-podinfo\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:45.996874 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:45.996653    4921 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd-istio-token\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:45.996874 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:45.996682    4921 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd-istio-data\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:45.996874 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:45.996698    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spz2z\" (UniqueName: \"kubernetes.io/projected/cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd-kube-api-access-spz2z\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:45.996874 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:45.996710    4921 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd-credential-socket\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:45.996874 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:45.996723    4921 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd-istiod-ca-cert\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:46.804606 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:46.804577    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rl9tr"
Nov 03 20:02:46.831655 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:46.831605    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rl9tr"]
Nov 03 20:02:46.837327 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:46.837301    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rl9tr"]
Nov 03 20:02:46.892761 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:46.892718    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd" path="/var/lib/kubelet/pods/cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd/volumes"
Nov 03 20:02:46.979595 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:46.979557    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-225668-758c694f89-tsbbm"]
Nov 03 20:02:46.979872 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:46.979814    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-225668-758c694f89-tsbbm" podUID="6f79e6cd-82c7-4e8b-b40b-67f5c8948173" containerName="maas-api" containerID="cri-o://ecc1defb828c6022397a390a95c73e043debe3d0adc2740336f071cce34439f7" gracePeriod=30
Nov 03 20:02:47.053187 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:47.053142    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-79c47f/test-model-79c47f-kserve-665c6d6575-b5svq" podUID="9be42b54-1aa8-4751-9671-b83f78cf6d3a" containerName="main" containerID="cri-o://ecb553a8fa1b1601d2028e72f2b6dfc13211238b4f6e4b5c34d01b955231609e" gracePeriod=45
Nov 03 20:02:47.500695 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:47.499603    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-225668-65675d6655-qdr5l" podUID="2105ca77-5b9c-45b2-9256-782b998a0041" containerName="payload-processing" containerID="cri-o://925ce0d76832c7f770773101fd88e6ee410b0ce690e8577656f6fb1702ed5cb3" gracePeriod=25
Nov 03 20:02:47.500695 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:47.499845    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-225668-6d7866c8dc-29bbm" podUID="e2c9f88f-de8e-406e-ad7d-2038e0faba39" containerName="payload-pre-processing" containerID="cri-o://3b0c6a0e30cb1fcb599d3a582876cbaaa9dcd837ed8e58ed55f26a4e6d29ec73" gracePeriod=25
Nov 03 20:02:47.776136 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:47.776108    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-225668-758c694f89-tsbbm"
Nov 03 20:02:47.797536 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:47.797508    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-79c47f/test-model-79c47f-kserve-665c6d6575-b5svq"
Nov 03 20:02:47.818264 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:47.818218    4921 generic.go:356] "Generic (PLEG): container finished" podID="6f79e6cd-82c7-4e8b-b40b-67f5c8948173" containerID="ecc1defb828c6022397a390a95c73e043debe3d0adc2740336f071cce34439f7" exitCode=0
Nov 03 20:02:47.818765 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:47.818544    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-225668-758c694f89-tsbbm" event={"ID":"6f79e6cd-82c7-4e8b-b40b-67f5c8948173","Type":"ContainerDied","Data":"ecc1defb828c6022397a390a95c73e043debe3d0adc2740336f071cce34439f7"}
Nov 03 20:02:47.818765 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:47.818598    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-225668-758c694f89-tsbbm" event={"ID":"6f79e6cd-82c7-4e8b-b40b-67f5c8948173","Type":"ContainerDied","Data":"a52bceac1e55c6f2e878419ddecb1834e2d5011d80659ca1cc8c98adc84f4d85"}
Nov 03 20:02:47.818765 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:47.818632    4921 scope.go:122] "RemoveContainer" containerID="ecc1defb828c6022397a390a95c73e043debe3d0adc2740336f071cce34439f7"
Nov 03 20:02:47.818966 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:47.818803    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-225668-758c694f89-tsbbm"
Nov 03 20:02:47.823198 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:47.823171    4921 generic.go:356] "Generic (PLEG): container finished" podID="9be42b54-1aa8-4751-9671-b83f78cf6d3a" containerID="ecb553a8fa1b1601d2028e72f2b6dfc13211238b4f6e4b5c34d01b955231609e" exitCode=0
Nov 03 20:02:47.823328 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:47.823225    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-79c47f/test-model-79c47f-kserve-665c6d6575-b5svq" event={"ID":"9be42b54-1aa8-4751-9671-b83f78cf6d3a","Type":"ContainerDied","Data":"ecb553a8fa1b1601d2028e72f2b6dfc13211238b4f6e4b5c34d01b955231609e"}
Nov 03 20:02:47.823328 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:47.823260    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-79c47f/test-model-79c47f-kserve-665c6d6575-b5svq" event={"ID":"9be42b54-1aa8-4751-9671-b83f78cf6d3a","Type":"ContainerDied","Data":"d7612d822d411a06c07c879006015348a1c1f3544859657f817f2bd7f44947a7"}
Nov 03 20:02:47.823328 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:47.823312    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-79c47f/test-model-79c47f-kserve-665c6d6575-b5svq"
Nov 03 20:02:47.842296 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:47.842277    4921 scope.go:122] "RemoveContainer" containerID="ecc1defb828c6022397a390a95c73e043debe3d0adc2740336f071cce34439f7"
Nov 03 20:02:47.842627 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:02:47.842607    4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecc1defb828c6022397a390a95c73e043debe3d0adc2740336f071cce34439f7\": container with ID starting with ecc1defb828c6022397a390a95c73e043debe3d0adc2740336f071cce34439f7 not found: ID does not exist" containerID="ecc1defb828c6022397a390a95c73e043debe3d0adc2740336f071cce34439f7"
Nov 03 20:02:47.842706 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:47.842634    4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecc1defb828c6022397a390a95c73e043debe3d0adc2740336f071cce34439f7"} err="failed to get container status \"ecc1defb828c6022397a390a95c73e043debe3d0adc2740336f071cce34439f7\": rpc error: code = NotFound desc = could not find container \"ecc1defb828c6022397a390a95c73e043debe3d0adc2740336f071cce34439f7\": container with ID starting with ecc1defb828c6022397a390a95c73e043debe3d0adc2740336f071cce34439f7 not found: ID does not exist"
Nov 03 20:02:47.842706 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:47.842651    4921 scope.go:122] "RemoveContainer" containerID="ecb553a8fa1b1601d2028e72f2b6dfc13211238b4f6e4b5c34d01b955231609e"
Nov 03 20:02:47.843173 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:47.843148    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f79e6cd-82c7-4e8b-b40b-67f5c8948173-kube-api-access-zlngx\" (UniqueName: \"kubernetes.io/projected/6f79e6cd-82c7-4e8b-b40b-67f5c8948173-kube-api-access-zlngx\") pod \"6f79e6cd-82c7-4e8b-b40b-67f5c8948173\" (UID: \"6f79e6cd-82c7-4e8b-b40b-67f5c8948173\") "
Nov 03 20:02:47.843256 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:47.843195    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f79e6cd-82c7-4e8b-b40b-67f5c8948173-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f79e6cd-82c7-4e8b-b40b-67f5c8948173-openshift-service-ca\") pod \"6f79e6cd-82c7-4e8b-b40b-67f5c8948173\" (UID: \"6f79e6cd-82c7-4e8b-b40b-67f5c8948173\") "
Nov 03 20:02:47.843297 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:47.843260    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9be42b54-1aa8-4751-9671-b83f78cf6d3a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9be42b54-1aa8-4751-9671-b83f78cf6d3a-tmp-dir\") pod \"9be42b54-1aa8-4751-9671-b83f78cf6d3a\" (UID: \"9be42b54-1aa8-4751-9671-b83f78cf6d3a\") "
Nov 03 20:02:47.843339 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:47.843321    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9be42b54-1aa8-4751-9671-b83f78cf6d3a-kube-api-access-2pzzc\" (UniqueName: \"kubernetes.io/projected/9be42b54-1aa8-4751-9671-b83f78cf6d3a-kube-api-access-2pzzc\") pod \"9be42b54-1aa8-4751-9671-b83f78cf6d3a\" (UID: \"9be42b54-1aa8-4751-9671-b83f78cf6d3a\") "
Nov 03 20:02:47.843438 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:47.843417    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9be42b54-1aa8-4751-9671-b83f78cf6d3a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9be42b54-1aa8-4751-9671-b83f78cf6d3a-model-cache\") pod \"9be42b54-1aa8-4751-9671-b83f78cf6d3a\" (UID: \"9be42b54-1aa8-4751-9671-b83f78cf6d3a\") "
Nov 03 20:02:47.843555 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:47.843463    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f79e6cd-82c7-4e8b-b40b-67f5c8948173-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f79e6cd-82c7-4e8b-b40b-67f5c8948173-metrics-certs\") pod \"6f79e6cd-82c7-4e8b-b40b-67f5c8948173\" (UID: \"6f79e6cd-82c7-4e8b-b40b-67f5c8948173\") "
Nov 03 20:02:47.843555 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:47.843525    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9be42b54-1aa8-4751-9671-b83f78cf6d3a-tls-certs\" (UniqueName: \"kubernetes.io/secret/9be42b54-1aa8-4751-9671-b83f78cf6d3a-tls-certs\") pod \"9be42b54-1aa8-4751-9671-b83f78cf6d3a\" (UID: \"9be42b54-1aa8-4751-9671-b83f78cf6d3a\") "
Nov 03 20:02:47.843718 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:47.843633    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9be42b54-1aa8-4751-9671-b83f78cf6d3a-tmp-dir" pod "9be42b54-1aa8-4751-9671-b83f78cf6d3a" (UID: "9be42b54-1aa8-4751-9671-b83f78cf6d3a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:47.843718 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:47.843658    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f79e6cd-82c7-4e8b-b40b-67f5c8948173-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f79e6cd-82c7-4e8b-b40b-67f5c8948173-maas-api-tls\") pod \"6f79e6cd-82c7-4e8b-b40b-67f5c8948173\" (UID: \"6f79e6cd-82c7-4e8b-b40b-67f5c8948173\") "
Nov 03 20:02:47.843845 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:47.843750    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9be42b54-1aa8-4751-9671-b83f78cf6d3a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9be42b54-1aa8-4751-9671-b83f78cf6d3a-dshm\") pod \"9be42b54-1aa8-4751-9671-b83f78cf6d3a\" (UID: \"9be42b54-1aa8-4751-9671-b83f78cf6d3a\") "
Nov 03 20:02:47.843845 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:47.843803    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9be42b54-1aa8-4751-9671-b83f78cf6d3a-home\" (UniqueName: \"kubernetes.io/empty-dir/9be42b54-1aa8-4751-9671-b83f78cf6d3a-home\") pod \"9be42b54-1aa8-4751-9671-b83f78cf6d3a\" (UID: \"9be42b54-1aa8-4751-9671-b83f78cf6d3a\") "
Nov 03 20:02:47.844375 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:47.843747    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f79e6cd-82c7-4e8b-b40b-67f5c8948173-openshift-service-ca" pod "6f79e6cd-82c7-4e8b-b40b-67f5c8948173" (UID: "6f79e6cd-82c7-4e8b-b40b-67f5c8948173"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:02:47.844375 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:47.844106    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9be42b54-1aa8-4751-9671-b83f78cf6d3a-model-cache" pod "9be42b54-1aa8-4751-9671-b83f78cf6d3a" (UID: "9be42b54-1aa8-4751-9671-b83f78cf6d3a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:47.844375 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:47.844358    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9be42b54-1aa8-4751-9671-b83f78cf6d3a-home" pod "9be42b54-1aa8-4751-9671-b83f78cf6d3a" (UID: "9be42b54-1aa8-4751-9671-b83f78cf6d3a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:47.844744 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:47.844711    4921 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9be42b54-1aa8-4751-9671-b83f78cf6d3a-home\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:47.844814 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:47.844751    4921 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f79e6cd-82c7-4e8b-b40b-67f5c8948173-openshift-service-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:47.844814 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:47.844768    4921 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9be42b54-1aa8-4751-9671-b83f78cf6d3a-tmp-dir\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:47.844814 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:47.844782    4921 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9be42b54-1aa8-4751-9671-b83f78cf6d3a-model-cache\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:47.846281 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:47.846248    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f79e6cd-82c7-4e8b-b40b-67f5c8948173-kube-api-access-zlngx" pod "6f79e6cd-82c7-4e8b-b40b-67f5c8948173" (UID: "6f79e6cd-82c7-4e8b-b40b-67f5c8948173"). InnerVolumeSpecName "kube-api-access-zlngx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:02:47.846388 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:47.846364    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9be42b54-1aa8-4751-9671-b83f78cf6d3a-dshm" pod "9be42b54-1aa8-4751-9671-b83f78cf6d3a" (UID: "9be42b54-1aa8-4751-9671-b83f78cf6d3a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:47.846550 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:47.846517    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9be42b54-1aa8-4751-9671-b83f78cf6d3a-kube-api-access-2pzzc" pod "9be42b54-1aa8-4751-9671-b83f78cf6d3a" (UID: "9be42b54-1aa8-4751-9671-b83f78cf6d3a"). InnerVolumeSpecName "kube-api-access-2pzzc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:02:47.847079 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:47.847060    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f79e6cd-82c7-4e8b-b40b-67f5c8948173-metrics-certs" pod "6f79e6cd-82c7-4e8b-b40b-67f5c8948173" (UID: "6f79e6cd-82c7-4e8b-b40b-67f5c8948173"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:02:47.847515 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:47.847484    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f79e6cd-82c7-4e8b-b40b-67f5c8948173-maas-api-tls" pod "6f79e6cd-82c7-4e8b-b40b-67f5c8948173" (UID: "6f79e6cd-82c7-4e8b-b40b-67f5c8948173"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:02:47.847515 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:47.847499    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9be42b54-1aa8-4751-9671-b83f78cf6d3a-tls-certs" pod "9be42b54-1aa8-4751-9671-b83f78cf6d3a" (UID: "9be42b54-1aa8-4751-9671-b83f78cf6d3a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:02:47.856140 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:47.856120    4921 scope.go:122] "RemoveContainer" containerID="ecb553a8fa1b1601d2028e72f2b6dfc13211238b4f6e4b5c34d01b955231609e"
Nov 03 20:02:47.856430 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:02:47.856407    4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecb553a8fa1b1601d2028e72f2b6dfc13211238b4f6e4b5c34d01b955231609e\": container with ID starting with ecb553a8fa1b1601d2028e72f2b6dfc13211238b4f6e4b5c34d01b955231609e not found: ID does not exist" containerID="ecb553a8fa1b1601d2028e72f2b6dfc13211238b4f6e4b5c34d01b955231609e"
Nov 03 20:02:47.856525 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:47.856435    4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecb553a8fa1b1601d2028e72f2b6dfc13211238b4f6e4b5c34d01b955231609e"} err="failed to get container status \"ecb553a8fa1b1601d2028e72f2b6dfc13211238b4f6e4b5c34d01b955231609e\": rpc error: code = NotFound desc = could not find container \"ecb553a8fa1b1601d2028e72f2b6dfc13211238b4f6e4b5c34d01b955231609e\": container with ID starting with ecb553a8fa1b1601d2028e72f2b6dfc13211238b4f6e4b5c34d01b955231609e not found: ID does not exist"
Nov 03 20:02:47.946879 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:47.946846    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlngx\" (UniqueName: \"kubernetes.io/projected/6f79e6cd-82c7-4e8b-b40b-67f5c8948173-kube-api-access-zlngx\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:47.946879 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:47.946878    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pzzc\" (UniqueName: \"kubernetes.io/projected/9be42b54-1aa8-4751-9671-b83f78cf6d3a-kube-api-access-2pzzc\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:47.947081 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:47.946889    4921 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f79e6cd-82c7-4e8b-b40b-67f5c8948173-metrics-certs\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:47.947081 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:47.946900    4921 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9be42b54-1aa8-4751-9671-b83f78cf6d3a-tls-certs\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:47.947081 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:47.946910    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f79e6cd-82c7-4e8b-b40b-67f5c8948173-maas-api-tls\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:47.947081 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:47.946920    4921 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9be42b54-1aa8-4751-9671-b83f78cf6d3a-dshm\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:48.324746 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:48.323967    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-79c47f/test-model-79c47f-kserve-665c6d6575-b5svq"]
Nov 03 20:02:48.339689 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:48.337781    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-79c47f/test-model-79c47f-kserve-665c6d6575-b5svq"]
Nov 03 20:02:48.364693 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:48.355922    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-225668-758c694f89-tsbbm"]
Nov 03 20:02:48.374903 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:48.367540    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-225668-758c694f89-tsbbm"]
Nov 03 20:02:48.891812 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:48.891779    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f79e6cd-82c7-4e8b-b40b-67f5c8948173" path="/var/lib/kubelet/pods/6f79e6cd-82c7-4e8b-b40b-67f5c8948173/volumes"
Nov 03 20:02:48.892457 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:48.892439    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9be42b54-1aa8-4751-9671-b83f78cf6d3a" path="/var/lib/kubelet/pods/9be42b54-1aa8-4751-9671-b83f78cf6d3a/volumes"
Nov 03 20:02:49.366966 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:49.366916    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p"]
Nov 03 20:02:49.367195 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:49.367166    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p" podUID="97dc926b-00fc-4a42-9283-938cac033482" containerName="istio-proxy" containerID="cri-o://635c43f53926eb55869cf6e08091a17be68c98632072693b101aa881a14c2117" gracePeriod=30
Nov 03 20:02:49.384144 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:49.384111    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf"]
Nov 03 20:02:49.384387 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:49.384356    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf" podUID="49c66530-de54-4ec9-83cb-61cafce44e7b" containerName="istio-proxy" containerID="cri-o://b1392ce25e2c9ca2cae30c03bd2245b22ef79386e97066c48a8160da89465b41" gracePeriod=30
Nov 03 20:02:49.406760 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:49.406123    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6"]
Nov 03 20:02:49.406760 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:49.406368    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6" podUID="7e96267a-d738-4325-90e7-d1c9ce87eeb5" containerName="istio-proxy" containerID="cri-o://a25a192dcf94a7c7e77f0c45f80b61c63cdd7ff4c624c61f20f6a0ce301e5368" gracePeriod=30
Nov 03 20:02:49.418230 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:49.417786    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz"]
Nov 03 20:02:49.418230 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:49.418017    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz" podUID="52048b65-d34b-412d-a473-36c9c72ed8e9" containerName="istio-proxy" containerID="cri-o://bca7846d26837d58bf7508e56fd1531752fd28989919c1e93698fb045526dc39" gracePeriod=30
Nov 03 20:02:49.675305 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:49.674928    4921 patch_prober.go:29] interesting pod/e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:02:49.675305 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:49.674990    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf" podUID="49c66530-de54-4ec9-83cb-61cafce44e7b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:02:50.005776 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:50.005730    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m"]
Nov 03 20:02:50.006295 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:50.006053    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m" podUID="ee6baa84-41ee-4bee-98c9-a9d0f078d4b9" containerName="istio-proxy" containerID="cri-o://a8f40df2f93cafd7ce8790e309e58a23dad94a4e563c39af9ee3fe65cc9dc7c3" gracePeriod=30
Nov 03 20:02:50.079504 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:50.078523    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj"]
Nov 03 20:02:50.079504 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:50.078770    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj" podUID="30ae46dd-e9ca-4cd8-b78c-1a5463c98d94" containerName="istio-proxy" containerID="cri-o://7260a03f31c099a7e261da6eb837d29cb48f6c8aafe972f88f5557465d2ca813" gracePeriod=30
Nov 03 20:02:50.880694 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:50.879428    4921 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:02:50.880694 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:50.879488    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj" podUID="30ae46dd-e9ca-4cd8-b78c-1a5463c98d94" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:02:52.278475 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:52.246192    4921 generic.go:356] "Generic (PLEG): container finished" podID="06971289-0da8-4921-9df7-140ff477056c" containerID="6fbecabd718fedb595dc30d1e25f95eb5572e0ea3f98a6c73d1f0ad159af4614" exitCode=0
Nov 03 20:02:52.278475 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:52.246561    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-102379-64bd4947ff-7qsss" event={"ID":"06971289-0da8-4921-9df7-140ff477056c","Type":"ContainerDied","Data":"6fbecabd718fedb595dc30d1e25f95eb5572e0ea3f98a6c73d1f0ad159af4614"}
Nov 03 20:02:52.278475 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:52.250165    4921 generic.go:356] "Generic (PLEG): container finished" podID="2fd7fd41-37ac-4e6c-8042-2c7b4a1f0674" containerID="fe8d07760a751fe0ac4ce6791008bcc8e054dd89fad7cae8d2698c419ba3d90a" exitCode=0
Nov 03 20:02:52.278475 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:52.250391    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-102379-5c958cc4ff-t94ls" event={"ID":"2fd7fd41-37ac-4e6c-8042-2c7b4a1f0674","Type":"ContainerDied","Data":"fe8d07760a751fe0ac4ce6791008bcc8e054dd89fad7cae8d2698c419ba3d90a"}
Nov 03 20:02:52.673694 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:52.673175    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-102379-64bd4947ff-7qsss"
Nov 03 20:02:52.740692 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:52.740514    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06971289-0da8-4921-9df7-140ff477056c-kube-api-access-6bdcj\" (UniqueName: \"kubernetes.io/projected/06971289-0da8-4921-9df7-140ff477056c-kube-api-access-6bdcj\") pod \"06971289-0da8-4921-9df7-140ff477056c\" (UID: \"06971289-0da8-4921-9df7-140ff477056c\") "
Nov 03 20:02:52.740692 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:52.740625    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06971289-0da8-4921-9df7-140ff477056c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06971289-0da8-4921-9df7-140ff477056c-plugins-config-volume\") pod \"06971289-0da8-4921-9df7-140ff477056c\" (UID: \"06971289-0da8-4921-9df7-140ff477056c\") "
Nov 03 20:02:52.741846 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:52.741770    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06971289-0da8-4921-9df7-140ff477056c-plugins-config-volume" pod "06971289-0da8-4921-9df7-140ff477056c" (UID: "06971289-0da8-4921-9df7-140ff477056c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:02:52.845844 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:52.844033    4921 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06971289-0da8-4921-9df7-140ff477056c-plugins-config-volume\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:52.907722 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:52.878854    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06971289-0da8-4921-9df7-140ff477056c-kube-api-access-6bdcj" pod "06971289-0da8-4921-9df7-140ff477056c" (UID: "06971289-0da8-4921-9df7-140ff477056c"). InnerVolumeSpecName "kube-api-access-6bdcj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:02:52.947027 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:52.946956    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bdcj\" (UniqueName: \"kubernetes.io/projected/06971289-0da8-4921-9df7-140ff477056c-kube-api-access-6bdcj\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:53.200742 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:53.200429    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-102379-5c958cc4ff-t94ls"
Nov 03 20:02:53.300300 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:53.300245    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fd7fd41-37ac-4e6c-8042-2c7b4a1f0674-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fd7fd41-37ac-4e6c-8042-2c7b4a1f0674-plugins-config-volume\") pod \"2fd7fd41-37ac-4e6c-8042-2c7b4a1f0674\" (UID: \"2fd7fd41-37ac-4e6c-8042-2c7b4a1f0674\") "
Nov 03 20:02:53.300912 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:53.300476    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fd7fd41-37ac-4e6c-8042-2c7b4a1f0674-kube-api-access-bfk9f\" (UniqueName: \"kubernetes.io/projected/2fd7fd41-37ac-4e6c-8042-2c7b4a1f0674-kube-api-access-bfk9f\") pod \"2fd7fd41-37ac-4e6c-8042-2c7b4a1f0674\" (UID: \"2fd7fd41-37ac-4e6c-8042-2c7b4a1f0674\") "
Nov 03 20:02:53.302205 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:53.301428    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fd7fd41-37ac-4e6c-8042-2c7b4a1f0674-plugins-config-volume" pod "2fd7fd41-37ac-4e6c-8042-2c7b4a1f0674" (UID: "2fd7fd41-37ac-4e6c-8042-2c7b4a1f0674"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:02:53.311952 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:53.305563    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fd7fd41-37ac-4e6c-8042-2c7b4a1f0674-kube-api-access-bfk9f" pod "2fd7fd41-37ac-4e6c-8042-2c7b4a1f0674" (UID: "2fd7fd41-37ac-4e6c-8042-2c7b4a1f0674"). InnerVolumeSpecName "kube-api-access-bfk9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:02:53.335154 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:53.335113    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-102379-5c958cc4ff-t94ls" event={"ID":"2fd7fd41-37ac-4e6c-8042-2c7b4a1f0674","Type":"ContainerDied","Data":"d4180486282705bab8df29a8da4aa045c244b2e3bebd3139b2c462b1105509d4"}
Nov 03 20:02:53.335334 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:53.335211    4921 scope.go:122] "RemoveContainer" containerID="fe8d07760a751fe0ac4ce6791008bcc8e054dd89fad7cae8d2698c419ba3d90a"
Nov 03 20:02:53.336542 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:53.336518    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-102379-5c958cc4ff-t94ls"
Nov 03 20:02:53.344123 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:53.343630    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-102379-64bd4947ff-7qsss" event={"ID":"06971289-0da8-4921-9df7-140ff477056c","Type":"ContainerDied","Data":"88abdc46cfaab1695d8e2e6b89e0a202f6580dbc1aad407097c1f07be0aee1e6"}
Nov 03 20:02:53.344123 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:53.343753    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-102379-64bd4947ff-7qsss"
Nov 03 20:02:53.356858 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:53.356305    4921 scope.go:122] "RemoveContainer" containerID="6fbecabd718fedb595dc30d1e25f95eb5572e0ea3f98a6c73d1f0ad159af4614"
Nov 03 20:02:53.367914 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:53.367890    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-102379-5c958cc4ff-t94ls"]
Nov 03 20:02:53.374123 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:53.373800    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-102379-5c958cc4ff-t94ls"]
Nov 03 20:02:53.384066 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:53.384041    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-102379-64bd4947ff-7qsss"]
Nov 03 20:02:53.384172 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:53.384083    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-102379-64bd4947ff-7qsss"]
Nov 03 20:02:53.402761 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:53.402681    4921 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fd7fd41-37ac-4e6c-8042-2c7b4a1f0674-plugins-config-volume\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:53.402761 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:53.402713    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfk9f\" (UniqueName: \"kubernetes.io/projected/2fd7fd41-37ac-4e6c-8042-2c7b4a1f0674-kube-api-access-bfk9f\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:54.676929 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:54.674883    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-225668-65675d6655-qdr5l" podUID="2105ca77-5b9c-45b2-9256-782b998a0041" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:02:54.784858 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:54.766727    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-225668-6d7866c8dc-29bbm" podUID="e2c9f88f-de8e-406e-ad7d-2038e0faba39" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:02:54.920889 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:54.920848    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06971289-0da8-4921-9df7-140ff477056c" path="/var/lib/kubelet/pods/06971289-0da8-4921-9df7-140ff477056c/volumes"
Nov 03 20:02:54.922963 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:54.922932    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fd7fd41-37ac-4e6c-8042-2c7b4a1f0674" path="/var/lib/kubelet/pods/2fd7fd41-37ac-4e6c-8042-2c7b4a1f0674/volumes"
Nov 03 20:02:55.062491 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.062463    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6"
Nov 03 20:02:55.161996 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.160768    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7e96267a-d738-4325-90e7-d1c9ce87eeb5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e96267a-d738-4325-90e7-d1c9ce87eeb5-istio-podinfo\") pod \"7e96267a-d738-4325-90e7-d1c9ce87eeb5\" (UID: \"7e96267a-d738-4325-90e7-d1c9ce87eeb5\") "
Nov 03 20:02:55.161996 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.160835    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e96267a-d738-4325-90e7-d1c9ce87eeb5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e96267a-d738-4325-90e7-d1c9ce87eeb5-credential-socket\") pod \"7e96267a-d738-4325-90e7-d1c9ce87eeb5\" (UID: \"7e96267a-d738-4325-90e7-d1c9ce87eeb5\") "
Nov 03 20:02:55.161996 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.160948    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e96267a-d738-4325-90e7-d1c9ce87eeb5-kube-api-access-frd24\" (UniqueName: \"kubernetes.io/projected/7e96267a-d738-4325-90e7-d1c9ce87eeb5-kube-api-access-frd24\") pod \"7e96267a-d738-4325-90e7-d1c9ce87eeb5\" (UID: \"7e96267a-d738-4325-90e7-d1c9ce87eeb5\") "
Nov 03 20:02:55.161996 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.161054    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e96267a-d738-4325-90e7-d1c9ce87eeb5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e96267a-d738-4325-90e7-d1c9ce87eeb5-istio-envoy\") pod \"7e96267a-d738-4325-90e7-d1c9ce87eeb5\" (UID: \"7e96267a-d738-4325-90e7-d1c9ce87eeb5\") "
Nov 03 20:02:55.161996 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.161129    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e96267a-d738-4325-90e7-d1c9ce87eeb5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e96267a-d738-4325-90e7-d1c9ce87eeb5-istiod-ca-cert\") pod \"7e96267a-d738-4325-90e7-d1c9ce87eeb5\" (UID: \"7e96267a-d738-4325-90e7-d1c9ce87eeb5\") "
Nov 03 20:02:55.161996 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.161183    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e96267a-d738-4325-90e7-d1c9ce87eeb5-credential-socket" pod "7e96267a-d738-4325-90e7-d1c9ce87eeb5" (UID: "7e96267a-d738-4325-90e7-d1c9ce87eeb5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:55.161996 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.161386    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e96267a-d738-4325-90e7-d1c9ce87eeb5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e96267a-d738-4325-90e7-d1c9ce87eeb5-istio-data\") pod \"7e96267a-d738-4325-90e7-d1c9ce87eeb5\" (UID: \"7e96267a-d738-4325-90e7-d1c9ce87eeb5\") "
Nov 03 20:02:55.161996 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.161427    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e96267a-d738-4325-90e7-d1c9ce87eeb5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e96267a-d738-4325-90e7-d1c9ce87eeb5-workload-certs\") pod \"7e96267a-d738-4325-90e7-d1c9ce87eeb5\" (UID: \"7e96267a-d738-4325-90e7-d1c9ce87eeb5\") "
Nov 03 20:02:55.161996 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.161476    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e96267a-d738-4325-90e7-d1c9ce87eeb5-istio-token\" (UniqueName: \"kubernetes.io/projected/7e96267a-d738-4325-90e7-d1c9ce87eeb5-istio-token\") pod \"7e96267a-d738-4325-90e7-d1c9ce87eeb5\" (UID: \"7e96267a-d738-4325-90e7-d1c9ce87eeb5\") "
Nov 03 20:02:55.161996 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.161542    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e96267a-d738-4325-90e7-d1c9ce87eeb5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e96267a-d738-4325-90e7-d1c9ce87eeb5-workload-socket\") pod \"7e96267a-d738-4325-90e7-d1c9ce87eeb5\" (UID: \"7e96267a-d738-4325-90e7-d1c9ce87eeb5\") "
Nov 03 20:02:55.161996 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.161917    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e96267a-d738-4325-90e7-d1c9ce87eeb5-workload-certs" pod "7e96267a-d738-4325-90e7-d1c9ce87eeb5" (UID: "7e96267a-d738-4325-90e7-d1c9ce87eeb5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:55.162742 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.162131    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e96267a-d738-4325-90e7-d1c9ce87eeb5-workload-socket" pod "7e96267a-d738-4325-90e7-d1c9ce87eeb5" (UID: "7e96267a-d738-4325-90e7-d1c9ce87eeb5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:55.162742 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.162375    4921 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e96267a-d738-4325-90e7-d1c9ce87eeb5-credential-socket\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:55.162742 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.162393    4921 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e96267a-d738-4325-90e7-d1c9ce87eeb5-workload-certs\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:55.162742 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.162407    4921 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e96267a-d738-4325-90e7-d1c9ce87eeb5-workload-socket\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:55.162742 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.162703    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e96267a-d738-4325-90e7-d1c9ce87eeb5-istio-data" pod "7e96267a-d738-4325-90e7-d1c9ce87eeb5" (UID: "7e96267a-d738-4325-90e7-d1c9ce87eeb5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:55.163767 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.163027    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e96267a-d738-4325-90e7-d1c9ce87eeb5-istiod-ca-cert" pod "7e96267a-d738-4325-90e7-d1c9ce87eeb5" (UID: "7e96267a-d738-4325-90e7-d1c9ce87eeb5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:02:55.164895 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.164871    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e96267a-d738-4325-90e7-d1c9ce87eeb5-istio-podinfo" pod "7e96267a-d738-4325-90e7-d1c9ce87eeb5" (UID: "7e96267a-d738-4325-90e7-d1c9ce87eeb5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:02:55.165041 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.164904    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e96267a-d738-4325-90e7-d1c9ce87eeb5-kube-api-access-frd24" pod "7e96267a-d738-4325-90e7-d1c9ce87eeb5" (UID: "7e96267a-d738-4325-90e7-d1c9ce87eeb5"). InnerVolumeSpecName "kube-api-access-frd24". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:02:55.167128 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.166309    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e96267a-d738-4325-90e7-d1c9ce87eeb5-istio-token" pod "7e96267a-d738-4325-90e7-d1c9ce87eeb5" (UID: "7e96267a-d738-4325-90e7-d1c9ce87eeb5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:02:55.172812 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.172788    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e96267a-d738-4325-90e7-d1c9ce87eeb5-istio-envoy" pod "7e96267a-d738-4325-90e7-d1c9ce87eeb5" (UID: "7e96267a-d738-4325-90e7-d1c9ce87eeb5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:55.216066 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.214364    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf"
Nov 03 20:02:55.218580 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.218401    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p"
Nov 03 20:02:55.244569 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.244437    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz"
Nov 03 20:02:55.264129 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.264088    4921 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e96267a-d738-4325-90e7-d1c9ce87eeb5-istio-envoy\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:55.264298 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.264145    4921 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e96267a-d738-4325-90e7-d1c9ce87eeb5-istiod-ca-cert\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:55.264298 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.264164    4921 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e96267a-d738-4325-90e7-d1c9ce87eeb5-istio-data\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:55.264298 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.264177    4921 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e96267a-d738-4325-90e7-d1c9ce87eeb5-istio-token\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:55.264298 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.264191    4921 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e96267a-d738-4325-90e7-d1c9ce87eeb5-istio-podinfo\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:55.264298 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.264205    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frd24\" (UniqueName: \"kubernetes.io/projected/7e96267a-d738-4325-90e7-d1c9ce87eeb5-kube-api-access-frd24\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:55.384990 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.384960    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97dc926b-00fc-4a42-9283-938cac033482-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97dc926b-00fc-4a42-9283-938cac033482-credential-socket\") pod \"97dc926b-00fc-4a42-9283-938cac033482\" (UID: \"97dc926b-00fc-4a42-9283-938cac033482\") "
Nov 03 20:02:55.386356 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.386295    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97dc926b-00fc-4a42-9283-938cac033482-kube-api-access-ggxw8\" (UniqueName: \"kubernetes.io/projected/97dc926b-00fc-4a42-9283-938cac033482-kube-api-access-ggxw8\") pod \"97dc926b-00fc-4a42-9283-938cac033482\" (UID: \"97dc926b-00fc-4a42-9283-938cac033482\") "
Nov 03 20:02:55.386563 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.385288    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97dc926b-00fc-4a42-9283-938cac033482-credential-socket" pod "97dc926b-00fc-4a42-9283-938cac033482" (UID: "97dc926b-00fc-4a42-9283-938cac033482"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:55.386974 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.386554    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52048b65-d34b-412d-a473-36c9c72ed8e9-istio-token\" (UniqueName: \"kubernetes.io/projected/52048b65-d34b-412d-a473-36c9c72ed8e9-istio-token\") pod \"52048b65-d34b-412d-a473-36c9c72ed8e9\" (UID: \"52048b65-d34b-412d-a473-36c9c72ed8e9\") "
Nov 03 20:02:55.387245 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.387223    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97dc926b-00fc-4a42-9283-938cac033482-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97dc926b-00fc-4a42-9283-938cac033482-istio-data\") pod \"97dc926b-00fc-4a42-9283-938cac033482\" (UID: \"97dc926b-00fc-4a42-9283-938cac033482\") "
Nov 03 20:02:55.388932 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.388900    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52048b65-d34b-412d-a473-36c9c72ed8e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52048b65-d34b-412d-a473-36c9c72ed8e9-workload-certs\") pod \"52048b65-d34b-412d-a473-36c9c72ed8e9\" (UID: \"52048b65-d34b-412d-a473-36c9c72ed8e9\") "
Nov 03 20:02:55.389070 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.388324    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97dc926b-00fc-4a42-9283-938cac033482-istio-data" pod "97dc926b-00fc-4a42-9283-938cac033482" (UID: "97dc926b-00fc-4a42-9283-938cac033482"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:55.389424 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.389389    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52048b65-d34b-412d-a473-36c9c72ed8e9-workload-certs" pod "52048b65-d34b-412d-a473-36c9c72ed8e9" (UID: "52048b65-d34b-412d-a473-36c9c72ed8e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:55.389586 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.389568    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49c66530-de54-4ec9-83cb-61cafce44e7b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49c66530-de54-4ec9-83cb-61cafce44e7b-istio-data\") pod \"49c66530-de54-4ec9-83cb-61cafce44e7b\" (UID: \"49c66530-de54-4ec9-83cb-61cafce44e7b\") "
Nov 03 20:02:55.389993 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.389958    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49c66530-de54-4ec9-83cb-61cafce44e7b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49c66530-de54-4ec9-83cb-61cafce44e7b-workload-certs\") pod \"49c66530-de54-4ec9-83cb-61cafce44e7b\" (UID: \"49c66530-de54-4ec9-83cb-61cafce44e7b\") "
Nov 03 20:02:55.390156 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.390138    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52048b65-d34b-412d-a473-36c9c72ed8e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52048b65-d34b-412d-a473-36c9c72ed8e9-istio-envoy\") pod \"52048b65-d34b-412d-a473-36c9c72ed8e9\" (UID: \"52048b65-d34b-412d-a473-36c9c72ed8e9\") "
Nov 03 20:02:55.390253 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.390233    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj" event={"ID":"30ae46dd-e9ca-4cd8-b78c-1a5463c98d94","Type":"ContainerDied","Data":"7260a03f31c099a7e261da6eb837d29cb48f6c8aafe972f88f5557465d2ca813"}
Nov 03 20:02:55.390351 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.390335    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52048b65-d34b-412d-a473-36c9c72ed8e9-kube-api-access-s28hv\" (UniqueName: \"kubernetes.io/projected/52048b65-d34b-412d-a473-36c9c72ed8e9-kube-api-access-s28hv\") pod \"52048b65-d34b-412d-a473-36c9c72ed8e9\" (UID: \"52048b65-d34b-412d-a473-36c9c72ed8e9\") "
Nov 03 20:02:55.390571 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.389845    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49c66530-de54-4ec9-83cb-61cafce44e7b-istio-data" pod "49c66530-de54-4ec9-83cb-61cafce44e7b" (UID: "49c66530-de54-4ec9-83cb-61cafce44e7b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:55.390655 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.390546    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49c66530-de54-4ec9-83cb-61cafce44e7b-workload-certs" pod "49c66530-de54-4ec9-83cb-61cafce44e7b" (UID: "49c66530-de54-4ec9-83cb-61cafce44e7b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:55.390655 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.390197    4921 generic.go:356] "Generic (PLEG): container finished" podID="30ae46dd-e9ca-4cd8-b78c-1a5463c98d94" containerID="7260a03f31c099a7e261da6eb837d29cb48f6c8aafe972f88f5557465d2ca813" exitCode=0
Nov 03 20:02:55.390808 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.390565    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52048b65-d34b-412d-a473-36c9c72ed8e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52048b65-d34b-412d-a473-36c9c72ed8e9-istio-data\") pod \"52048b65-d34b-412d-a473-36c9c72ed8e9\" (UID: \"52048b65-d34b-412d-a473-36c9c72ed8e9\") "
Nov 03 20:02:55.390940 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.390922    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/49c66530-de54-4ec9-83cb-61cafce44e7b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49c66530-de54-4ec9-83cb-61cafce44e7b-istio-podinfo\") pod \"49c66530-de54-4ec9-83cb-61cafce44e7b\" (UID: \"49c66530-de54-4ec9-83cb-61cafce44e7b\") "
Nov 03 20:02:55.391067 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.391050    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49c66530-de54-4ec9-83cb-61cafce44e7b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49c66530-de54-4ec9-83cb-61cafce44e7b-credential-socket\") pod \"49c66530-de54-4ec9-83cb-61cafce44e7b\" (UID: \"49c66530-de54-4ec9-83cb-61cafce44e7b\") "
Nov 03 20:02:55.391203 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.391173    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49c66530-de54-4ec9-83cb-61cafce44e7b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49c66530-de54-4ec9-83cb-61cafce44e7b-istio-envoy\") pod \"49c66530-de54-4ec9-83cb-61cafce44e7b\" (UID: \"49c66530-de54-4ec9-83cb-61cafce44e7b\") "
Nov 03 20:02:55.391333 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.391316    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52048b65-d34b-412d-a473-36c9c72ed8e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52048b65-d34b-412d-a473-36c9c72ed8e9-istiod-ca-cert\") pod \"52048b65-d34b-412d-a473-36c9c72ed8e9\" (UID: \"52048b65-d34b-412d-a473-36c9c72ed8e9\") "
Nov 03 20:02:55.391472 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.391458    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97dc926b-00fc-4a42-9283-938cac033482-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97dc926b-00fc-4a42-9283-938cac033482-istio-envoy\") pod \"97dc926b-00fc-4a42-9283-938cac033482\" (UID: \"97dc926b-00fc-4a42-9283-938cac033482\") "
Nov 03 20:02:55.391587 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.391570    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49c66530-de54-4ec9-83cb-61cafce44e7b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49c66530-de54-4ec9-83cb-61cafce44e7b-workload-socket\") pod \"49c66530-de54-4ec9-83cb-61cafce44e7b\" (UID: \"49c66530-de54-4ec9-83cb-61cafce44e7b\") "
Nov 03 20:02:55.391762 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.391745    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97dc926b-00fc-4a42-9283-938cac033482-istio-token\" (UniqueName: \"kubernetes.io/projected/97dc926b-00fc-4a42-9283-938cac033482-istio-token\") pod \"97dc926b-00fc-4a42-9283-938cac033482\" (UID: \"97dc926b-00fc-4a42-9283-938cac033482\") "
Nov 03 20:02:55.391895 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.391878    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/97dc926b-00fc-4a42-9283-938cac033482-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97dc926b-00fc-4a42-9283-938cac033482-istio-podinfo\") pod \"97dc926b-00fc-4a42-9283-938cac033482\" (UID: \"97dc926b-00fc-4a42-9283-938cac033482\") "
Nov 03 20:02:55.392032 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.392018    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52048b65-d34b-412d-a473-36c9c72ed8e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52048b65-d34b-412d-a473-36c9c72ed8e9-workload-socket\") pod \"52048b65-d34b-412d-a473-36c9c72ed8e9\" (UID: \"52048b65-d34b-412d-a473-36c9c72ed8e9\") "
Nov 03 20:02:55.392175 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.392159    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49c66530-de54-4ec9-83cb-61cafce44e7b-kube-api-access-gftmd\" (UniqueName: \"kubernetes.io/projected/49c66530-de54-4ec9-83cb-61cafce44e7b-kube-api-access-gftmd\") pod \"49c66530-de54-4ec9-83cb-61cafce44e7b\" (UID: \"49c66530-de54-4ec9-83cb-61cafce44e7b\") "
Nov 03 20:02:55.392377 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.392360    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49c66530-de54-4ec9-83cb-61cafce44e7b-istio-token\" (UniqueName: \"kubernetes.io/projected/49c66530-de54-4ec9-83cb-61cafce44e7b-istio-token\") pod \"49c66530-de54-4ec9-83cb-61cafce44e7b\" (UID: \"49c66530-de54-4ec9-83cb-61cafce44e7b\") "
Nov 03 20:02:55.392604 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.392585    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52048b65-d34b-412d-a473-36c9c72ed8e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52048b65-d34b-412d-a473-36c9c72ed8e9-credential-socket\") pod \"52048b65-d34b-412d-a473-36c9c72ed8e9\" (UID: \"52048b65-d34b-412d-a473-36c9c72ed8e9\") "
Nov 03 20:02:55.393300 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.393279    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97dc926b-00fc-4a42-9283-938cac033482-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97dc926b-00fc-4a42-9283-938cac033482-workload-certs\") pod \"97dc926b-00fc-4a42-9283-938cac033482\" (UID: \"97dc926b-00fc-4a42-9283-938cac033482\") "
Nov 03 20:02:55.393451 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.393432    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97dc926b-00fc-4a42-9283-938cac033482-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97dc926b-00fc-4a42-9283-938cac033482-workload-socket\") pod \"97dc926b-00fc-4a42-9283-938cac033482\" (UID: \"97dc926b-00fc-4a42-9283-938cac033482\") "
Nov 03 20:02:55.393527 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.393492    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97dc926b-00fc-4a42-9283-938cac033482-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97dc926b-00fc-4a42-9283-938cac033482-istiod-ca-cert\") pod \"97dc926b-00fc-4a42-9283-938cac033482\" (UID: \"97dc926b-00fc-4a42-9283-938cac033482\") "
Nov 03 20:02:55.393589 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.393540    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/52048b65-d34b-412d-a473-36c9c72ed8e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52048b65-d34b-412d-a473-36c9c72ed8e9-istio-podinfo\") pod \"52048b65-d34b-412d-a473-36c9c72ed8e9\" (UID: \"52048b65-d34b-412d-a473-36c9c72ed8e9\") "
Nov 03 20:02:55.393658 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.393640    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49c66530-de54-4ec9-83cb-61cafce44e7b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49c66530-de54-4ec9-83cb-61cafce44e7b-istiod-ca-cert\") pod \"49c66530-de54-4ec9-83cb-61cafce44e7b\" (UID: \"49c66530-de54-4ec9-83cb-61cafce44e7b\") "
Nov 03 20:02:55.394464 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.394438    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52048b65-d34b-412d-a473-36c9c72ed8e9-istiod-ca-cert" pod "52048b65-d34b-412d-a473-36c9c72ed8e9" (UID: "52048b65-d34b-412d-a473-36c9c72ed8e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:02:55.394728 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.394705    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49c66530-de54-4ec9-83cb-61cafce44e7b-credential-socket" pod "49c66530-de54-4ec9-83cb-61cafce44e7b" (UID: "49c66530-de54-4ec9-83cb-61cafce44e7b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:55.395858 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.395834    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49c66530-de54-4ec9-83cb-61cafce44e7b-istiod-ca-cert" pod "49c66530-de54-4ec9-83cb-61cafce44e7b" (UID: "49c66530-de54-4ec9-83cb-61cafce44e7b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:02:55.397206 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.397182    4921 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97dc926b-00fc-4a42-9283-938cac033482-credential-socket\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:55.397313 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.397218    4921 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97dc926b-00fc-4a42-9283-938cac033482-istio-data\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:55.397313 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.397233    4921 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52048b65-d34b-412d-a473-36c9c72ed8e9-workload-certs\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:55.397313 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.397254    4921 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49c66530-de54-4ec9-83cb-61cafce44e7b-istio-data\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:55.397313 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.397267    4921 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49c66530-de54-4ec9-83cb-61cafce44e7b-workload-certs\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:55.397313 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.397283    4921 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49c66530-de54-4ec9-83cb-61cafce44e7b-credential-socket\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:55.397313 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.397300    4921 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52048b65-d34b-412d-a473-36c9c72ed8e9-istiod-ca-cert\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:55.401536 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.401503    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52048b65-d34b-412d-a473-36c9c72ed8e9-kube-api-access-s28hv" pod "52048b65-d34b-412d-a473-36c9c72ed8e9" (UID: "52048b65-d34b-412d-a473-36c9c72ed8e9"). InnerVolumeSpecName "kube-api-access-s28hv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:02:55.403459 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.403436    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52048b65-d34b-412d-a473-36c9c72ed8e9-istio-data" pod "52048b65-d34b-412d-a473-36c9c72ed8e9" (UID: "52048b65-d34b-412d-a473-36c9c72ed8e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:55.405144 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.405112    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/49c66530-de54-4ec9-83cb-61cafce44e7b-istio-podinfo" pod "49c66530-de54-4ec9-83cb-61cafce44e7b" (UID: "49c66530-de54-4ec9-83cb-61cafce44e7b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:02:55.406569 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.405342    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52048b65-d34b-412d-a473-36c9c72ed8e9-workload-socket" pod "52048b65-d34b-412d-a473-36c9c72ed8e9" (UID: "52048b65-d34b-412d-a473-36c9c72ed8e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:55.406569 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.405562    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49c66530-de54-4ec9-83cb-61cafce44e7b-workload-socket" pod "49c66530-de54-4ec9-83cb-61cafce44e7b" (UID: "49c66530-de54-4ec9-83cb-61cafce44e7b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:55.406569 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.406265    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97dc926b-00fc-4a42-9283-938cac033482-workload-socket" pod "97dc926b-00fc-4a42-9283-938cac033482" (UID: "97dc926b-00fc-4a42-9283-938cac033482"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:55.406569 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.406458    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52048b65-d34b-412d-a473-36c9c72ed8e9-credential-socket" pod "52048b65-d34b-412d-a473-36c9c72ed8e9" (UID: "52048b65-d34b-412d-a473-36c9c72ed8e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:55.406888 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.406646    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97dc926b-00fc-4a42-9283-938cac033482-workload-certs" pod "97dc926b-00fc-4a42-9283-938cac033482" (UID: "97dc926b-00fc-4a42-9283-938cac033482"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:55.406969 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.406950    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97dc926b-00fc-4a42-9283-938cac033482-istiod-ca-cert" pod "97dc926b-00fc-4a42-9283-938cac033482" (UID: "97dc926b-00fc-4a42-9283-938cac033482"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:02:55.410010 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.408737    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/97dc926b-00fc-4a42-9283-938cac033482-istio-podinfo" pod "97dc926b-00fc-4a42-9283-938cac033482" (UID: "97dc926b-00fc-4a42-9283-938cac033482"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:02:55.410405 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.410379    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52048b65-d34b-412d-a473-36c9c72ed8e9-istio-envoy" pod "52048b65-d34b-412d-a473-36c9c72ed8e9" (UID: "52048b65-d34b-412d-a473-36c9c72ed8e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:55.410836 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.410637    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97dc926b-00fc-4a42-9283-938cac033482-kube-api-access-ggxw8" pod "97dc926b-00fc-4a42-9283-938cac033482" (UID: "97dc926b-00fc-4a42-9283-938cac033482"). InnerVolumeSpecName "kube-api-access-ggxw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:02:55.410836 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.410776    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52048b65-d34b-412d-a473-36c9c72ed8e9-istio-token" pod "52048b65-d34b-412d-a473-36c9c72ed8e9" (UID: "52048b65-d34b-412d-a473-36c9c72ed8e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:02:55.418196 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.418155    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97dc926b-00fc-4a42-9283-938cac033482-istio-envoy" pod "97dc926b-00fc-4a42-9283-938cac033482" (UID: "97dc926b-00fc-4a42-9283-938cac033482"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:55.418416 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.418392    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49c66530-de54-4ec9-83cb-61cafce44e7b-istio-envoy" pod "49c66530-de54-4ec9-83cb-61cafce44e7b" (UID: "49c66530-de54-4ec9-83cb-61cafce44e7b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:55.419265 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.419231    4921 generic.go:356] "Generic (PLEG): container finished" podID="ee6baa84-41ee-4bee-98c9-a9d0f078d4b9" containerID="a8f40df2f93cafd7ce8790e309e58a23dad94a4e563c39af9ee3fe65cc9dc7c3" exitCode=0
Nov 03 20:02:55.419499 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.419466    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m" event={"ID":"ee6baa84-41ee-4bee-98c9-a9d0f078d4b9","Type":"ContainerDied","Data":"a8f40df2f93cafd7ce8790e309e58a23dad94a4e563c39af9ee3fe65cc9dc7c3"}
Nov 03 20:02:55.420444 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.420394    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49c66530-de54-4ec9-83cb-61cafce44e7b-istio-token" pod "49c66530-de54-4ec9-83cb-61cafce44e7b" (UID: "49c66530-de54-4ec9-83cb-61cafce44e7b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:02:55.421573 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.421525    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97dc926b-00fc-4a42-9283-938cac033482-istio-token" pod "97dc926b-00fc-4a42-9283-938cac033482" (UID: "97dc926b-00fc-4a42-9283-938cac033482"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:02:55.422087 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.422066    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/52048b65-d34b-412d-a473-36c9c72ed8e9-istio-podinfo" pod "52048b65-d34b-412d-a473-36c9c72ed8e9" (UID: "52048b65-d34b-412d-a473-36c9c72ed8e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:02:55.431032 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.424067    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49c66530-de54-4ec9-83cb-61cafce44e7b-kube-api-access-gftmd" pod "49c66530-de54-4ec9-83cb-61cafce44e7b" (UID: "49c66530-de54-4ec9-83cb-61cafce44e7b"). InnerVolumeSpecName "kube-api-access-gftmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:02:55.431032 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.425506    4921 generic.go:356] "Generic (PLEG): container finished" podID="52048b65-d34b-412d-a473-36c9c72ed8e9" containerID="bca7846d26837d58bf7508e56fd1531752fd28989919c1e93698fb045526dc39" exitCode=0
Nov 03 20:02:55.431032 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.425623    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz" event={"ID":"52048b65-d34b-412d-a473-36c9c72ed8e9","Type":"ContainerDied","Data":"bca7846d26837d58bf7508e56fd1531752fd28989919c1e93698fb045526dc39"}
Nov 03 20:02:55.431032 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.425694    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz" event={"ID":"52048b65-d34b-412d-a473-36c9c72ed8e9","Type":"ContainerDied","Data":"a4654e12f76e6f82399c1cbb8204a0ae2ea8992bb92389c03cef2ccf0b8dd2aa"}
Nov 03 20:02:55.431032 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.425724    4921 scope.go:122] "RemoveContainer" containerID="bca7846d26837d58bf7508e56fd1531752fd28989919c1e93698fb045526dc39"
Nov 03 20:02:55.431032 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.425891    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz"
Nov 03 20:02:55.435798 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.434486    4921 generic.go:356] "Generic (PLEG): container finished" podID="7e96267a-d738-4325-90e7-d1c9ce87eeb5" containerID="a25a192dcf94a7c7e77f0c45f80b61c63cdd7ff4c624c61f20f6a0ce301e5368" exitCode=0
Nov 03 20:02:55.435798 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.434587    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6" event={"ID":"7e96267a-d738-4325-90e7-d1c9ce87eeb5","Type":"ContainerDied","Data":"a25a192dcf94a7c7e77f0c45f80b61c63cdd7ff4c624c61f20f6a0ce301e5368"}
Nov 03 20:02:55.435798 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.434639    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6" event={"ID":"7e96267a-d738-4325-90e7-d1c9ce87eeb5","Type":"ContainerDied","Data":"8ec0123073af5a11dda5f140782a76e73a7677ff5d557943d6a8213cfd5ebd4c"}
Nov 03 20:02:55.435798 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.434650    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6"
Nov 03 20:02:55.439708 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.438350    4921 generic.go:356] "Generic (PLEG): container finished" podID="49c66530-de54-4ec9-83cb-61cafce44e7b" containerID="b1392ce25e2c9ca2cae30c03bd2245b22ef79386e97066c48a8160da89465b41" exitCode=0
Nov 03 20:02:55.439708 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.438458    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf" event={"ID":"49c66530-de54-4ec9-83cb-61cafce44e7b","Type":"ContainerDied","Data":"b1392ce25e2c9ca2cae30c03bd2245b22ef79386e97066c48a8160da89465b41"}
Nov 03 20:02:55.439708 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.438503    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf" event={"ID":"49c66530-de54-4ec9-83cb-61cafce44e7b","Type":"ContainerDied","Data":"95dfb0f9de24a473dae99f88dd2775aca114f8eddbf26df8f7ac41474ae22476"}
Nov 03 20:02:55.439708 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.438591    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf"
Nov 03 20:02:55.449131 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.449100    4921 generic.go:356] "Generic (PLEG): container finished" podID="97dc926b-00fc-4a42-9283-938cac033482" containerID="635c43f53926eb55869cf6e08091a17be68c98632072693b101aa881a14c2117" exitCode=0
Nov 03 20:02:55.449309 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.449184    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p" event={"ID":"97dc926b-00fc-4a42-9283-938cac033482","Type":"ContainerDied","Data":"635c43f53926eb55869cf6e08091a17be68c98632072693b101aa881a14c2117"}
Nov 03 20:02:55.449309 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.449240    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p" event={"ID":"97dc926b-00fc-4a42-9283-938cac033482","Type":"ContainerDied","Data":"6aebaccd3b21b5aaf45ac44e7baa831f469b288316f4849df0ef61a4d240b64b"}
Nov 03 20:02:55.449428 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.449331    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p"
Nov 03 20:02:55.489128 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.487959    4921 scope.go:122] "RemoveContainer" containerID="bca7846d26837d58bf7508e56fd1531752fd28989919c1e93698fb045526dc39"
Nov 03 20:02:55.491584 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:02:55.489344    4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bca7846d26837d58bf7508e56fd1531752fd28989919c1e93698fb045526dc39\": container with ID starting with bca7846d26837d58bf7508e56fd1531752fd28989919c1e93698fb045526dc39 not found: ID does not exist" containerID="bca7846d26837d58bf7508e56fd1531752fd28989919c1e93698fb045526dc39"
Nov 03 20:02:55.491584 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.489383    4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bca7846d26837d58bf7508e56fd1531752fd28989919c1e93698fb045526dc39"} err="failed to get container status \"bca7846d26837d58bf7508e56fd1531752fd28989919c1e93698fb045526dc39\": rpc error: code = NotFound desc = could not find container \"bca7846d26837d58bf7508e56fd1531752fd28989919c1e93698fb045526dc39\": container with ID starting with bca7846d26837d58bf7508e56fd1531752fd28989919c1e93698fb045526dc39 not found: ID does not exist"
Nov 03 20:02:55.491584 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.489408    4921 scope.go:122] "RemoveContainer" containerID="a25a192dcf94a7c7e77f0c45f80b61c63cdd7ff4c624c61f20f6a0ce301e5368"
Nov 03 20:02:55.502286 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.500983    4921 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52048b65-d34b-412d-a473-36c9c72ed8e9-istio-envoy\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:55.502286 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.501017    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s28hv\" (UniqueName: \"kubernetes.io/projected/52048b65-d34b-412d-a473-36c9c72ed8e9-kube-api-access-s28hv\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:55.502286 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.501037    4921 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52048b65-d34b-412d-a473-36c9c72ed8e9-istio-data\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:55.502286 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.501053    4921 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49c66530-de54-4ec9-83cb-61cafce44e7b-istio-podinfo\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:55.502286 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.501068    4921 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49c66530-de54-4ec9-83cb-61cafce44e7b-istio-envoy\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:55.502286 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.501082    4921 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97dc926b-00fc-4a42-9283-938cac033482-istio-envoy\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:55.502286 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.501096    4921 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49c66530-de54-4ec9-83cb-61cafce44e7b-workload-socket\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:55.502286 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.501111    4921 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97dc926b-00fc-4a42-9283-938cac033482-istio-token\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:55.502286 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.501126    4921 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97dc926b-00fc-4a42-9283-938cac033482-istio-podinfo\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:55.502286 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.501140    4921 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52048b65-d34b-412d-a473-36c9c72ed8e9-workload-socket\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:55.502286 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.501155    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gftmd\" (UniqueName: \"kubernetes.io/projected/49c66530-de54-4ec9-83cb-61cafce44e7b-kube-api-access-gftmd\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:55.502286 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.501169    4921 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49c66530-de54-4ec9-83cb-61cafce44e7b-istio-token\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:55.502286 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.501182    4921 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52048b65-d34b-412d-a473-36c9c72ed8e9-credential-socket\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:55.502286 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.501196    4921 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97dc926b-00fc-4a42-9283-938cac033482-workload-certs\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:55.502286 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.501210    4921 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97dc926b-00fc-4a42-9283-938cac033482-workload-socket\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:55.502286 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.501232    4921 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97dc926b-00fc-4a42-9283-938cac033482-istiod-ca-cert\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:55.502286 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.501247    4921 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52048b65-d34b-412d-a473-36c9c72ed8e9-istio-podinfo\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:55.502286 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.501261    4921 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49c66530-de54-4ec9-83cb-61cafce44e7b-istiod-ca-cert\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:55.502286 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.501275    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggxw8\" (UniqueName: \"kubernetes.io/projected/97dc926b-00fc-4a42-9283-938cac033482-kube-api-access-ggxw8\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:55.502286 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.501290    4921 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52048b65-d34b-412d-a473-36c9c72ed8e9-istio-token\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:55.518859 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.518829    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf"]
Nov 03 20:02:55.522381 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.521954    4921 scope.go:122] "RemoveContainer" containerID="a25a192dcf94a7c7e77f0c45f80b61c63cdd7ff4c624c61f20f6a0ce301e5368"
Nov 03 20:02:55.522548 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:02:55.522519    4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a25a192dcf94a7c7e77f0c45f80b61c63cdd7ff4c624c61f20f6a0ce301e5368\": container with ID starting with a25a192dcf94a7c7e77f0c45f80b61c63cdd7ff4c624c61f20f6a0ce301e5368 not found: ID does not exist" containerID="a25a192dcf94a7c7e77f0c45f80b61c63cdd7ff4c624c61f20f6a0ce301e5368"
Nov 03 20:02:55.522618 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.522569    4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a25a192dcf94a7c7e77f0c45f80b61c63cdd7ff4c624c61f20f6a0ce301e5368"} err="failed to get container status \"a25a192dcf94a7c7e77f0c45f80b61c63cdd7ff4c624c61f20f6a0ce301e5368\": rpc error: code = NotFound desc = could not find container \"a25a192dcf94a7c7e77f0c45f80b61c63cdd7ff4c624c61f20f6a0ce301e5368\": container with ID starting with a25a192dcf94a7c7e77f0c45f80b61c63cdd7ff4c624c61f20f6a0ce301e5368 not found: ID does not exist"
Nov 03 20:02:55.522618 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.522594    4921 scope.go:122] "RemoveContainer" containerID="b1392ce25e2c9ca2cae30c03bd2245b22ef79386e97066c48a8160da89465b41"
Nov 03 20:02:55.533370 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.532133    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-zhnpf"]
Nov 03 20:02:55.552435 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.551835    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz"]
Nov 03 20:02:55.556221 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.556178    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-kxgcz"]
Nov 03 20:02:55.559415 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.559394    4921 scope.go:122] "RemoveContainer" containerID="b1392ce25e2c9ca2cae30c03bd2245b22ef79386e97066c48a8160da89465b41"
Nov 03 20:02:55.559908 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:02:55.559810    4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1392ce25e2c9ca2cae30c03bd2245b22ef79386e97066c48a8160da89465b41\": container with ID starting with b1392ce25e2c9ca2cae30c03bd2245b22ef79386e97066c48a8160da89465b41 not found: ID does not exist" containerID="b1392ce25e2c9ca2cae30c03bd2245b22ef79386e97066c48a8160da89465b41"
Nov 03 20:02:55.559908 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.559847    4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1392ce25e2c9ca2cae30c03bd2245b22ef79386e97066c48a8160da89465b41"} err="failed to get container status \"b1392ce25e2c9ca2cae30c03bd2245b22ef79386e97066c48a8160da89465b41\": rpc error: code = NotFound desc = could not find container \"b1392ce25e2c9ca2cae30c03bd2245b22ef79386e97066c48a8160da89465b41\": container with ID starting with b1392ce25e2c9ca2cae30c03bd2245b22ef79386e97066c48a8160da89465b41 not found: ID does not exist"
Nov 03 20:02:55.559908 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.559871    4921 scope.go:122] "RemoveContainer" containerID="635c43f53926eb55869cf6e08091a17be68c98632072693b101aa881a14c2117"
Nov 03 20:02:55.564720 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.563961    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6"]
Nov 03 20:02:55.568354 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.568332    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-7dnl6"]
Nov 03 20:02:55.572213 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.572192    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p"]
Nov 03 20:02:55.627690 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.624180    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-39043d-openshift-default-644f6b79bf-4bg6p"]
Nov 03 20:02:55.674754 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.674727    4921 scope.go:122] "RemoveContainer" containerID="635c43f53926eb55869cf6e08091a17be68c98632072693b101aa881a14c2117"
Nov 03 20:02:55.675275 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:02:55.675252    4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"635c43f53926eb55869cf6e08091a17be68c98632072693b101aa881a14c2117\": container with ID starting with 635c43f53926eb55869cf6e08091a17be68c98632072693b101aa881a14c2117 not found: ID does not exist" containerID="635c43f53926eb55869cf6e08091a17be68c98632072693b101aa881a14c2117"
Nov 03 20:02:55.675412 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.675387    4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"635c43f53926eb55869cf6e08091a17be68c98632072693b101aa881a14c2117"} err="failed to get container status \"635c43f53926eb55869cf6e08091a17be68c98632072693b101aa881a14c2117\": rpc error: code = NotFound desc = could not find container \"635c43f53926eb55869cf6e08091a17be68c98632072693b101aa881a14c2117\": container with ID starting with 635c43f53926eb55869cf6e08091a17be68c98632072693b101aa881a14c2117 not found: ID does not exist"
Nov 03 20:02:55.680735 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.677889    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj"
Nov 03 20:02:55.727286 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.726429    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30ae46dd-e9ca-4cd8-b78c-1a5463c98d94-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30ae46dd-e9ca-4cd8-b78c-1a5463c98d94-workload-socket\") pod \"30ae46dd-e9ca-4cd8-b78c-1a5463c98d94\" (UID: \"30ae46dd-e9ca-4cd8-b78c-1a5463c98d94\") "
Nov 03 20:02:55.727286 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.726564    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30ae46dd-e9ca-4cd8-b78c-1a5463c98d94-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30ae46dd-e9ca-4cd8-b78c-1a5463c98d94-istio-envoy\") pod \"30ae46dd-e9ca-4cd8-b78c-1a5463c98d94\" (UID: \"30ae46dd-e9ca-4cd8-b78c-1a5463c98d94\") "
Nov 03 20:02:55.727286 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.726607    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30ae46dd-e9ca-4cd8-b78c-1a5463c98d94-istio-token\" (UniqueName: \"kubernetes.io/projected/30ae46dd-e9ca-4cd8-b78c-1a5463c98d94-istio-token\") pod \"30ae46dd-e9ca-4cd8-b78c-1a5463c98d94\" (UID: \"30ae46dd-e9ca-4cd8-b78c-1a5463c98d94\") "
Nov 03 20:02:55.727286 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.726753    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30ae46dd-e9ca-4cd8-b78c-1a5463c98d94-kube-api-access-vwwq2\" (UniqueName: \"kubernetes.io/projected/30ae46dd-e9ca-4cd8-b78c-1a5463c98d94-kube-api-access-vwwq2\") pod \"30ae46dd-e9ca-4cd8-b78c-1a5463c98d94\" (UID: \"30ae46dd-e9ca-4cd8-b78c-1a5463c98d94\") "
Nov 03 20:02:55.727286 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.726806    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30ae46dd-e9ca-4cd8-b78c-1a5463c98d94-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30ae46dd-e9ca-4cd8-b78c-1a5463c98d94-istio-data\") pod \"30ae46dd-e9ca-4cd8-b78c-1a5463c98d94\" (UID: \"30ae46dd-e9ca-4cd8-b78c-1a5463c98d94\") "
Nov 03 20:02:55.727286 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.726840    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30ae46dd-e9ca-4cd8-b78c-1a5463c98d94-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30ae46dd-e9ca-4cd8-b78c-1a5463c98d94-workload-certs\") pod \"30ae46dd-e9ca-4cd8-b78c-1a5463c98d94\" (UID: \"30ae46dd-e9ca-4cd8-b78c-1a5463c98d94\") "
Nov 03 20:02:55.727286 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.726918    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30ae46dd-e9ca-4cd8-b78c-1a5463c98d94-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30ae46dd-e9ca-4cd8-b78c-1a5463c98d94-istiod-ca-cert\") pod \"30ae46dd-e9ca-4cd8-b78c-1a5463c98d94\" (UID: \"30ae46dd-e9ca-4cd8-b78c-1a5463c98d94\") "
Nov 03 20:02:55.727286 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.726966    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30ae46dd-e9ca-4cd8-b78c-1a5463c98d94-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30ae46dd-e9ca-4cd8-b78c-1a5463c98d94-credential-socket\") pod \"30ae46dd-e9ca-4cd8-b78c-1a5463c98d94\" (UID: \"30ae46dd-e9ca-4cd8-b78c-1a5463c98d94\") "
Nov 03 20:02:55.727286 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.727010    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/30ae46dd-e9ca-4cd8-b78c-1a5463c98d94-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30ae46dd-e9ca-4cd8-b78c-1a5463c98d94-istio-podinfo\") pod \"30ae46dd-e9ca-4cd8-b78c-1a5463c98d94\" (UID: \"30ae46dd-e9ca-4cd8-b78c-1a5463c98d94\") "
Nov 03 20:02:55.730223 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.729409    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30ae46dd-e9ca-4cd8-b78c-1a5463c98d94-istiod-ca-cert" pod "30ae46dd-e9ca-4cd8-b78c-1a5463c98d94" (UID: "30ae46dd-e9ca-4cd8-b78c-1a5463c98d94"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:02:55.730223 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.730096    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30ae46dd-e9ca-4cd8-b78c-1a5463c98d94-workload-certs" pod "30ae46dd-e9ca-4cd8-b78c-1a5463c98d94" (UID: "30ae46dd-e9ca-4cd8-b78c-1a5463c98d94"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:55.730223 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.730159    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30ae46dd-e9ca-4cd8-b78c-1a5463c98d94-credential-socket" pod "30ae46dd-e9ca-4cd8-b78c-1a5463c98d94" (UID: "30ae46dd-e9ca-4cd8-b78c-1a5463c98d94"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:55.730223 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.730203    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30ae46dd-e9ca-4cd8-b78c-1a5463c98d94-istio-data" pod "30ae46dd-e9ca-4cd8-b78c-1a5463c98d94" (UID: "30ae46dd-e9ca-4cd8-b78c-1a5463c98d94"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:55.730703 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.730648    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30ae46dd-e9ca-4cd8-b78c-1a5463c98d94-workload-socket" pod "30ae46dd-e9ca-4cd8-b78c-1a5463c98d94" (UID: "30ae46dd-e9ca-4cd8-b78c-1a5463c98d94"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:55.735807 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.735771    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30ae46dd-e9ca-4cd8-b78c-1a5463c98d94-istio-envoy" pod "30ae46dd-e9ca-4cd8-b78c-1a5463c98d94" (UID: "30ae46dd-e9ca-4cd8-b78c-1a5463c98d94"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:55.752047 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.751227    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30ae46dd-e9ca-4cd8-b78c-1a5463c98d94-kube-api-access-vwwq2" pod "30ae46dd-e9ca-4cd8-b78c-1a5463c98d94" (UID: "30ae46dd-e9ca-4cd8-b78c-1a5463c98d94"). InnerVolumeSpecName "kube-api-access-vwwq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:02:55.752047 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.751386    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/30ae46dd-e9ca-4cd8-b78c-1a5463c98d94-istio-podinfo" pod "30ae46dd-e9ca-4cd8-b78c-1a5463c98d94" (UID: "30ae46dd-e9ca-4cd8-b78c-1a5463c98d94"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:02:55.753938 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.753869    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30ae46dd-e9ca-4cd8-b78c-1a5463c98d94-istio-token" pod "30ae46dd-e9ca-4cd8-b78c-1a5463c98d94" (UID: "30ae46dd-e9ca-4cd8-b78c-1a5463c98d94"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:02:55.829269 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.829122    4921 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30ae46dd-e9ca-4cd8-b78c-1a5463c98d94-workload-socket\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:55.829269 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.829154    4921 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30ae46dd-e9ca-4cd8-b78c-1a5463c98d94-istio-envoy\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:55.829269 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.829164    4921 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30ae46dd-e9ca-4cd8-b78c-1a5463c98d94-istio-token\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:55.829269 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.829172    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwwq2\" (UniqueName: \"kubernetes.io/projected/30ae46dd-e9ca-4cd8-b78c-1a5463c98d94-kube-api-access-vwwq2\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:55.829269 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.829182    4921 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30ae46dd-e9ca-4cd8-b78c-1a5463c98d94-istio-data\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:55.829269 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.829192    4921 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30ae46dd-e9ca-4cd8-b78c-1a5463c98d94-workload-certs\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:55.829269 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.829205    4921 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30ae46dd-e9ca-4cd8-b78c-1a5463c98d94-istiod-ca-cert\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:55.829269 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.829224    4921 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30ae46dd-e9ca-4cd8-b78c-1a5463c98d94-credential-socket\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:55.829269 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.829233    4921 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30ae46dd-e9ca-4cd8-b78c-1a5463c98d94-istio-podinfo\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:55.865260 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.865233    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m"
Nov 03 20:02:55.930059 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.930025    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee6baa84-41ee-4bee-98c9-a9d0f078d4b9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee6baa84-41ee-4bee-98c9-a9d0f078d4b9-workload-socket\") pod \"ee6baa84-41ee-4bee-98c9-a9d0f078d4b9\" (UID: \"ee6baa84-41ee-4bee-98c9-a9d0f078d4b9\") "
Nov 03 20:02:55.930319 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.930115    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee6baa84-41ee-4bee-98c9-a9d0f078d4b9-istio-token\" (UniqueName: \"kubernetes.io/projected/ee6baa84-41ee-4bee-98c9-a9d0f078d4b9-istio-token\") pod \"ee6baa84-41ee-4bee-98c9-a9d0f078d4b9\" (UID: \"ee6baa84-41ee-4bee-98c9-a9d0f078d4b9\") "
Nov 03 20:02:55.930319 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.930178    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee6baa84-41ee-4bee-98c9-a9d0f078d4b9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee6baa84-41ee-4bee-98c9-a9d0f078d4b9-istio-data\") pod \"ee6baa84-41ee-4bee-98c9-a9d0f078d4b9\" (UID: \"ee6baa84-41ee-4bee-98c9-a9d0f078d4b9\") "
Nov 03 20:02:55.930319 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.930270    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee6baa84-41ee-4bee-98c9-a9d0f078d4b9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee6baa84-41ee-4bee-98c9-a9d0f078d4b9-istiod-ca-cert\") pod \"ee6baa84-41ee-4bee-98c9-a9d0f078d4b9\" (UID: \"ee6baa84-41ee-4bee-98c9-a9d0f078d4b9\") "
Nov 03 20:02:55.930489 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.930342    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee6baa84-41ee-4bee-98c9-a9d0f078d4b9-workload-socket" pod "ee6baa84-41ee-4bee-98c9-a9d0f078d4b9" (UID: "ee6baa84-41ee-4bee-98c9-a9d0f078d4b9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:55.930489 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.930353    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ee6baa84-41ee-4bee-98c9-a9d0f078d4b9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee6baa84-41ee-4bee-98c9-a9d0f078d4b9-istio-podinfo\") pod \"ee6baa84-41ee-4bee-98c9-a9d0f078d4b9\" (UID: \"ee6baa84-41ee-4bee-98c9-a9d0f078d4b9\") "
Nov 03 20:02:55.930596 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.930485    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee6baa84-41ee-4bee-98c9-a9d0f078d4b9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee6baa84-41ee-4bee-98c9-a9d0f078d4b9-credential-socket\") pod \"ee6baa84-41ee-4bee-98c9-a9d0f078d4b9\" (UID: \"ee6baa84-41ee-4bee-98c9-a9d0f078d4b9\") "
Nov 03 20:02:55.930596 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.930531    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee6baa84-41ee-4bee-98c9-a9d0f078d4b9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee6baa84-41ee-4bee-98c9-a9d0f078d4b9-istio-envoy\") pod \"ee6baa84-41ee-4bee-98c9-a9d0f078d4b9\" (UID: \"ee6baa84-41ee-4bee-98c9-a9d0f078d4b9\") "
Nov 03 20:02:55.930727 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.930600    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee6baa84-41ee-4bee-98c9-a9d0f078d4b9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee6baa84-41ee-4bee-98c9-a9d0f078d4b9-workload-certs\") pod \"ee6baa84-41ee-4bee-98c9-a9d0f078d4b9\" (UID: \"ee6baa84-41ee-4bee-98c9-a9d0f078d4b9\") "
Nov 03 20:02:55.930786 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.930721    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee6baa84-41ee-4bee-98c9-a9d0f078d4b9-kube-api-access-h55np\" (UniqueName: \"kubernetes.io/projected/ee6baa84-41ee-4bee-98c9-a9d0f078d4b9-kube-api-access-h55np\") pod \"ee6baa84-41ee-4bee-98c9-a9d0f078d4b9\" (UID: \"ee6baa84-41ee-4bee-98c9-a9d0f078d4b9\") "
Nov 03 20:02:55.931784 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.931045    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee6baa84-41ee-4bee-98c9-a9d0f078d4b9-istio-data" pod "ee6baa84-41ee-4bee-98c9-a9d0f078d4b9" (UID: "ee6baa84-41ee-4bee-98c9-a9d0f078d4b9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:55.931784 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.931458    4921 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee6baa84-41ee-4bee-98c9-a9d0f078d4b9-workload-socket\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:55.931784 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.931480    4921 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee6baa84-41ee-4bee-98c9-a9d0f078d4b9-istio-data\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:55.932858 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.932767    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee6baa84-41ee-4bee-98c9-a9d0f078d4b9-workload-certs" pod "ee6baa84-41ee-4bee-98c9-a9d0f078d4b9" (UID: "ee6baa84-41ee-4bee-98c9-a9d0f078d4b9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:55.932858 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.932833    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee6baa84-41ee-4bee-98c9-a9d0f078d4b9-credential-socket" pod "ee6baa84-41ee-4bee-98c9-a9d0f078d4b9" (UID: "ee6baa84-41ee-4bee-98c9-a9d0f078d4b9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:55.933153 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.933115    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee6baa84-41ee-4bee-98c9-a9d0f078d4b9-istio-podinfo" pod "ee6baa84-41ee-4bee-98c9-a9d0f078d4b9" (UID: "ee6baa84-41ee-4bee-98c9-a9d0f078d4b9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:02:55.933626 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.933605    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee6baa84-41ee-4bee-98c9-a9d0f078d4b9-istiod-ca-cert" pod "ee6baa84-41ee-4bee-98c9-a9d0f078d4b9" (UID: "ee6baa84-41ee-4bee-98c9-a9d0f078d4b9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:02:55.933783 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.933723    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee6baa84-41ee-4bee-98c9-a9d0f078d4b9-kube-api-access-h55np" pod "ee6baa84-41ee-4bee-98c9-a9d0f078d4b9" (UID: "ee6baa84-41ee-4bee-98c9-a9d0f078d4b9"). InnerVolumeSpecName "kube-api-access-h55np". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:02:55.938282 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.938253    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee6baa84-41ee-4bee-98c9-a9d0f078d4b9-istio-token" pod "ee6baa84-41ee-4bee-98c9-a9d0f078d4b9" (UID: "ee6baa84-41ee-4bee-98c9-a9d0f078d4b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:02:55.939827 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:55.939799    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee6baa84-41ee-4bee-98c9-a9d0f078d4b9-istio-envoy" pod "ee6baa84-41ee-4bee-98c9-a9d0f078d4b9" (UID: "ee6baa84-41ee-4bee-98c9-a9d0f078d4b9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:56.033326 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:56.033230    4921 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee6baa84-41ee-4bee-98c9-a9d0f078d4b9-istio-podinfo\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:56.033326 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:56.033267    4921 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee6baa84-41ee-4bee-98c9-a9d0f078d4b9-credential-socket\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:56.033326 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:56.033281    4921 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee6baa84-41ee-4bee-98c9-a9d0f078d4b9-istio-envoy\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:56.033326 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:56.033292    4921 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee6baa84-41ee-4bee-98c9-a9d0f078d4b9-workload-certs\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:56.033326 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:56.033305    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h55np\" (UniqueName: \"kubernetes.io/projected/ee6baa84-41ee-4bee-98c9-a9d0f078d4b9-kube-api-access-h55np\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:56.033326 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:56.033317    4921 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee6baa84-41ee-4bee-98c9-a9d0f078d4b9-istio-token\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:56.033326 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:56.033329    4921 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee6baa84-41ee-4bee-98c9-a9d0f078d4b9-istiod-ca-cert\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:56.463893 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:56.463846    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj"
Nov 03 20:02:56.464052 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:56.463844    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj" event={"ID":"30ae46dd-e9ca-4cd8-b78c-1a5463c98d94","Type":"ContainerDied","Data":"262eb77e6558b08d9b4f554e1aba96e1b3b65a9d9f2f627fad81e11c7d55f468"}
Nov 03 20:02:56.464052 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:56.464035    4921 scope.go:122] "RemoveContainer" containerID="7260a03f31c099a7e261da6eb837d29cb48f6c8aafe972f88f5557465d2ca813"
Nov 03 20:02:56.466064 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:56.466042    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m"
Nov 03 20:02:56.466615 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:56.466580    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m" event={"ID":"ee6baa84-41ee-4bee-98c9-a9d0f078d4b9","Type":"ContainerDied","Data":"d653f6d08b96bd9d6c233b0e1fcd2b0988059908c72214d37a3567e0122c780e"}
Nov 03 20:02:56.484342 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:56.484318    4921 scope.go:122] "RemoveContainer" containerID="a8f40df2f93cafd7ce8790e309e58a23dad94a4e563c39af9ee3fe65cc9dc7c3"
Nov 03 20:02:56.496397 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:56.496362    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj"]
Nov 03 20:02:56.500440 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:56.500420    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-l59gj"]
Nov 03 20:02:56.507003 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:56.506984    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m"]
Nov 03 20:02:56.512165 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:56.512145    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-qz45m"]
Nov 03 20:02:56.920391 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:56.920353    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30ae46dd-e9ca-4cd8-b78c-1a5463c98d94" path="/var/lib/kubelet/pods/30ae46dd-e9ca-4cd8-b78c-1a5463c98d94/volumes"
Nov 03 20:02:56.922136 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:56.921172    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49c66530-de54-4ec9-83cb-61cafce44e7b" path="/var/lib/kubelet/pods/49c66530-de54-4ec9-83cb-61cafce44e7b/volumes"
Nov 03 20:02:56.922136 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:56.921960    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52048b65-d34b-412d-a473-36c9c72ed8e9" path="/var/lib/kubelet/pods/52048b65-d34b-412d-a473-36c9c72ed8e9/volumes"
Nov 03 20:02:56.923095 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:56.923071    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e96267a-d738-4325-90e7-d1c9ce87eeb5" path="/var/lib/kubelet/pods/7e96267a-d738-4325-90e7-d1c9ce87eeb5/volumes"
Nov 03 20:02:56.923610 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:56.923591    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97dc926b-00fc-4a42-9283-938cac033482" path="/var/lib/kubelet/pods/97dc926b-00fc-4a42-9283-938cac033482/volumes"
Nov 03 20:02:56.924201 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:56.924180    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee6baa84-41ee-4bee-98c9-a9d0f078d4b9" path="/var/lib/kubelet/pods/ee6baa84-41ee-4bee-98c9-a9d0f078d4b9/volumes"
Nov 03 20:02:57.152743 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:57.152710    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb"]
Nov 03 20:02:57.152927 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:57.152906    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb" podUID="8423f401-70db-4f4a-9bf5-0597acb06be6" containerName="istio-proxy" containerID="cri-o://01974174fc1b417c37ea4625b302d901dba257d096d3d9455d313e3bff633938" gracePeriod=30
Nov 03 20:02:57.904814 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:57.900047    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dce558-77fccb9bff-qgtj8"]
Nov 03 20:02:57.904814 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:57.901681    4921 cpu_manager.go:414] "containerMap: removing container" podUID="0ed39371-700a-43b3-a66b-a500d13cfd77" containerName="payload-processing"
Nov 03 20:02:57.904814 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:57.901704    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ed39371-700a-43b3-a66b-a500d13cfd77" containerName="payload-processing"
Nov 03 20:02:57.904814 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:57.901732    4921 cpu_manager.go:414] "containerMap: removing container" podUID="7e96267a-d738-4325-90e7-d1c9ce87eeb5" containerName="istio-proxy"
Nov 03 20:02:57.904814 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:57.901740    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e96267a-d738-4325-90e7-d1c9ce87eeb5" containerName="istio-proxy"
Nov 03 20:02:57.904814 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:57.901759    4921 cpu_manager.go:414] "containerMap: removing container" podUID="6dec4941-e24a-4b8b-852c-a9d0128c533c" containerName="istio-proxy"
Nov 03 20:02:57.904814 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:57.901767    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dec4941-e24a-4b8b-852c-a9d0128c533c" containerName="istio-proxy"
Nov 03 20:02:57.904814 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:57.901781    4921 cpu_manager.go:414] "containerMap: removing container" podUID="52048b65-d34b-412d-a473-36c9c72ed8e9" containerName="istio-proxy"
Nov 03 20:02:57.904814 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:57.901788    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52048b65-d34b-412d-a473-36c9c72ed8e9" containerName="istio-proxy"
Nov 03 20:02:57.904814 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:57.901819    4921 cpu_manager.go:414] "containerMap: removing container" podUID="2fd7fd41-37ac-4e6c-8042-2c7b4a1f0674" containerName="payload-processing"
Nov 03 20:02:57.904814 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:57.901826    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fd7fd41-37ac-4e6c-8042-2c7b4a1f0674" containerName="payload-processing"
Nov 03 20:02:57.904814 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:57.901837    4921 cpu_manager.go:414] "containerMap: removing container" podUID="cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd" containerName="istio-proxy"
Nov 03 20:02:57.904814 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:57.901844    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd" containerName="istio-proxy"
Nov 03 20:02:57.904814 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:57.901855    4921 cpu_manager.go:414] "containerMap: removing container" podUID="30ae46dd-e9ca-4cd8-b78c-1a5463c98d94" containerName="istio-proxy"
Nov 03 20:02:57.904814 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:57.901862    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30ae46dd-e9ca-4cd8-b78c-1a5463c98d94" containerName="istio-proxy"
Nov 03 20:02:57.904814 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:57.901885    4921 cpu_manager.go:414] "containerMap: removing container" podUID="9be42b54-1aa8-4751-9671-b83f78cf6d3a" containerName="main"
Nov 03 20:02:57.904814 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:57.901892    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9be42b54-1aa8-4751-9671-b83f78cf6d3a" containerName="main"
Nov 03 20:02:57.904814 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:57.901908    4921 cpu_manager.go:414] "containerMap: removing container" podUID="06971289-0da8-4921-9df7-140ff477056c" containerName="payload-pre-processing"
Nov 03 20:02:57.904814 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:57.901915    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06971289-0da8-4921-9df7-140ff477056c" containerName="payload-pre-processing"
Nov 03 20:02:57.904814 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:57.901926    4921 cpu_manager.go:414] "containerMap: removing container" podUID="6f79e6cd-82c7-4e8b-b40b-67f5c8948173" containerName="maas-api"
Nov 03 20:02:57.904814 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:57.901933    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f79e6cd-82c7-4e8b-b40b-67f5c8948173" containerName="maas-api"
Nov 03 20:02:57.904814 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:57.901951    4921 cpu_manager.go:414] "containerMap: removing container" podUID="97dc926b-00fc-4a42-9283-938cac033482" containerName="istio-proxy"
Nov 03 20:02:57.904814 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:57.901959    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97dc926b-00fc-4a42-9283-938cac033482" containerName="istio-proxy"
Nov 03 20:02:57.904814 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:57.901977    4921 cpu_manager.go:414] "containerMap: removing container" podUID="b642f55a-3767-464c-b9ff-280627b3e277" containerName="payload-pre-processing"
Nov 03 20:02:57.904814 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:57.901984    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b642f55a-3767-464c-b9ff-280627b3e277" containerName="payload-pre-processing"
Nov 03 20:02:57.904814 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:57.901997    4921 cpu_manager.go:414] "containerMap: removing container" podUID="e6a71aa3-33b3-42bd-b270-64ce3260cc2a" containerName="main"
Nov 03 20:02:57.904814 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:57.902004    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6a71aa3-33b3-42bd-b270-64ce3260cc2a" containerName="main"
Nov 03 20:02:57.904814 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:57.902017    4921 cpu_manager.go:414] "containerMap: removing container" podUID="49c66530-de54-4ec9-83cb-61cafce44e7b" containerName="istio-proxy"
Nov 03 20:02:57.904814 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:57.902023    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49c66530-de54-4ec9-83cb-61cafce44e7b" containerName="istio-proxy"
Nov 03 20:02:57.904814 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:57.902036    4921 cpu_manager.go:414] "containerMap: removing container" podUID="ee6baa84-41ee-4bee-98c9-a9d0f078d4b9" containerName="istio-proxy"
Nov 03 20:02:57.904814 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:57.902042    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee6baa84-41ee-4bee-98c9-a9d0f078d4b9" containerName="istio-proxy"
Nov 03 20:02:57.904814 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:57.902056    4921 cpu_manager.go:414] "containerMap: removing container" podUID="ca28dda5-c003-4070-b331-43c5a51d9f87" containerName="istio-proxy"
Nov 03 20:02:57.904814 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:57.902063    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca28dda5-c003-4070-b331-43c5a51d9f87" containerName="istio-proxy"
Nov 03 20:02:57.904814 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:57.902255    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="9be42b54-1aa8-4751-9671-b83f78cf6d3a" containerName="main"
Nov 03 20:02:57.904814 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:57.902269    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="06971289-0da8-4921-9df7-140ff477056c" containerName="payload-pre-processing"
Nov 03 20:02:57.904814 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:57.902281    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f79e6cd-82c7-4e8b-b40b-67f5c8948173" containerName="maas-api"
Nov 03 20:02:57.904814 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:57.902754    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="97dc926b-00fc-4a42-9283-938cac033482" containerName="istio-proxy"
Nov 03 20:02:57.904814 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:57.902777    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="b642f55a-3767-464c-b9ff-280627b3e277" containerName="payload-pre-processing"
Nov 03 20:02:57.904814 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:57.902791    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6a71aa3-33b3-42bd-b270-64ce3260cc2a" containerName="main"
Nov 03 20:02:57.904814 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:57.902802    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="49c66530-de54-4ec9-83cb-61cafce44e7b" containerName="istio-proxy"
Nov 03 20:02:57.904814 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:57.902813    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee6baa84-41ee-4bee-98c9-a9d0f078d4b9" containerName="istio-proxy"
Nov 03 20:02:57.904814 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:57.902825    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca28dda5-c003-4070-b331-43c5a51d9f87" containerName="istio-proxy"
Nov 03 20:02:57.904814 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:57.902836    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ed39371-700a-43b3-a66b-a500d13cfd77" containerName="payload-processing"
Nov 03 20:02:57.904814 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:57.902847    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e96267a-d738-4325-90e7-d1c9ce87eeb5" containerName="istio-proxy"
Nov 03 20:02:57.904814 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:57.902857    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dec4941-e24a-4b8b-852c-a9d0128c533c" containerName="istio-proxy"
Nov 03 20:02:57.904814 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:57.902866    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="52048b65-d34b-412d-a473-36c9c72ed8e9" containerName="istio-proxy"
Nov 03 20:02:57.904814 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:57.902880    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fd7fd41-37ac-4e6c-8042-2c7b4a1f0674" containerName="payload-processing"
Nov 03 20:02:57.904814 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:57.902891    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="cce688e1-b7b2-49b6-bb88-21bbc4d1c9cd" containerName="istio-proxy"
Nov 03 20:02:57.904814 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:57.902901    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="30ae46dd-e9ca-4cd8-b78c-1a5463c98d94" containerName="istio-proxy"
Nov 03 20:02:57.912150 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:57.912127    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dce558-77fccb9bff-qgtj8"
Nov 03 20:02:57.913898 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:57.913868    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dce558-77fccb9bff-qgtj8"]
Nov 03 20:02:57.917721 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:57.917650    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-dce558\""
Nov 03 20:02:57.921166 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:57.921144    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-dce558-dockercfg-bhx6c\""
Nov 03 20:02:57.949921 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:57.949858    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-dce558-566ddc874c-ctw65"]
Nov 03 20:02:57.964007 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:57.963971    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-dce558-566ddc874c-ctw65"
Nov 03 20:02:57.986924 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:57.986893    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-dce558-566ddc874c-ctw65"]
Nov 03 20:02:57.989390 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:57.989357    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c554c44-d20a-4efe-9faf-5c30ec3a3f5e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-dce558-77fccb9bff-qgtj8\" (UID: \"6c554c44-d20a-4efe-9faf-5c30ec3a3f5e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dce558-77fccb9bff-qgtj8"
Nov 03 20:02:57.989885 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:57.989859    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wz4t5\" (UniqueName: \"kubernetes.io/projected/6c554c44-d20a-4efe-9faf-5c30ec3a3f5e-kube-api-access-wz4t5\") pod \"payload-pre-processing-e2e-shared-b-w0-dce558-77fccb9bff-qgtj8\" (UID: \"6c554c44-d20a-4efe-9faf-5c30ec3a3f5e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dce558-77fccb9bff-qgtj8"
Nov 03 20:02:58.098693 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:58.097596    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr2w7\" (UniqueName: \"kubernetes.io/projected/8d974919-e441-47b7-9044-9bb57982bdc1-kube-api-access-rr2w7\") pod \"payload-processing-e2e-shared-b-w0-dce558-566ddc874c-ctw65\" (UID: \"8d974919-e441-47b7-9044-9bb57982bdc1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-dce558-566ddc874c-ctw65"
Nov 03 20:02:58.098693 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:58.097802    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wz4t5\" (UniqueName: \"kubernetes.io/projected/6c554c44-d20a-4efe-9faf-5c30ec3a3f5e-kube-api-access-wz4t5\") pod \"payload-pre-processing-e2e-shared-b-w0-dce558-77fccb9bff-qgtj8\" (UID: \"6c554c44-d20a-4efe-9faf-5c30ec3a3f5e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dce558-77fccb9bff-qgtj8"
Nov 03 20:02:58.098693 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:58.098411    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d974919-e441-47b7-9044-9bb57982bdc1-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-dce558-566ddc874c-ctw65\" (UID: \"8d974919-e441-47b7-9044-9bb57982bdc1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-dce558-566ddc874c-ctw65"
Nov 03 20:02:58.098693 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:58.098581    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c554c44-d20a-4efe-9faf-5c30ec3a3f5e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-dce558-77fccb9bff-qgtj8\" (UID: \"6c554c44-d20a-4efe-9faf-5c30ec3a3f5e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dce558-77fccb9bff-qgtj8"
Nov 03 20:02:58.100413 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:58.100386    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c554c44-d20a-4efe-9faf-5c30ec3a3f5e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-dce558-77fccb9bff-qgtj8\" (UID: \"6c554c44-d20a-4efe-9faf-5c30ec3a3f5e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dce558-77fccb9bff-qgtj8"
Nov 03 20:02:58.133296 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:58.133247    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wz4t5\" (UniqueName: \"kubernetes.io/projected/6c554c44-d20a-4efe-9faf-5c30ec3a3f5e-kube-api-access-wz4t5\") pod \"payload-pre-processing-e2e-shared-b-w0-dce558-77fccb9bff-qgtj8\" (UID: \"6c554c44-d20a-4efe-9faf-5c30ec3a3f5e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dce558-77fccb9bff-qgtj8"
Nov 03 20:02:58.202742 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:58.202165    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d974919-e441-47b7-9044-9bb57982bdc1-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-dce558-566ddc874c-ctw65\" (UID: \"8d974919-e441-47b7-9044-9bb57982bdc1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-dce558-566ddc874c-ctw65"
Nov 03 20:02:58.202742 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:58.202429    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr2w7\" (UniqueName: \"kubernetes.io/projected/8d974919-e441-47b7-9044-9bb57982bdc1-kube-api-access-rr2w7\") pod \"payload-processing-e2e-shared-b-w0-dce558-566ddc874c-ctw65\" (UID: \"8d974919-e441-47b7-9044-9bb57982bdc1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-dce558-566ddc874c-ctw65"
Nov 03 20:02:58.207715 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:58.205656    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d974919-e441-47b7-9044-9bb57982bdc1-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-dce558-566ddc874c-ctw65\" (UID: \"8d974919-e441-47b7-9044-9bb57982bdc1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-dce558-566ddc874c-ctw65"
Nov 03 20:02:58.241692 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:58.240339    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dce558-77fccb9bff-qgtj8"
Nov 03 20:02:58.269431 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:58.263365    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr2w7\" (UniqueName: \"kubernetes.io/projected/8d974919-e441-47b7-9044-9bb57982bdc1-kube-api-access-rr2w7\") pod \"payload-processing-e2e-shared-b-w0-dce558-566ddc874c-ctw65\" (UID: \"8d974919-e441-47b7-9044-9bb57982bdc1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-dce558-566ddc874c-ctw65"
Nov 03 20:02:58.426266 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:58.426214    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-dce558-566ddc874c-ctw65"
Nov 03 20:02:58.804698 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:58.803940    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dce558-77fccb9bff-qgtj8"]
Nov 03 20:02:59.007222 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:59.007189    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-dce558-566ddc874c-ctw65"]
Nov 03 20:02:59.008147 ip-10-0-134-222 kubenswrapper[4921]: W1103 20:02:59.008112    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d974919_e441_47b7_9044_9bb57982bdc1.slice/crio-80f5dc380d2fa29c7166db1a47139fe512fe93779e898695ec53d36ff5a3b413 WatchSource:0}: Error finding container 80f5dc380d2fa29c7166db1a47139fe512fe93779e898695ec53d36ff5a3b413: Status 404 returned error can't find the container with id 80f5dc380d2fa29c7166db1a47139fe512fe93779e898695ec53d36ff5a3b413
Nov 03 20:02:59.702783 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:59.702730    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-dce558-566ddc874c-ctw65" event={"ID":"8d974919-e441-47b7-9044-9bb57982bdc1","Type":"ContainerStarted","Data":"5ce027893323fce2895f873269ccdb3dfd0e039f5e1fd9827bd6f3e13ba85436"}
Nov 03 20:02:59.702969 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:59.702815    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-dce558-566ddc874c-ctw65" event={"ID":"8d974919-e441-47b7-9044-9bb57982bdc1","Type":"ContainerStarted","Data":"80f5dc380d2fa29c7166db1a47139fe512fe93779e898695ec53d36ff5a3b413"}
Nov 03 20:02:59.702969 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:59.702860    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-dce558-566ddc874c-ctw65"
Nov 03 20:02:59.705067 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:59.705028    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dce558-77fccb9bff-qgtj8" event={"ID":"6c554c44-d20a-4efe-9faf-5c30ec3a3f5e","Type":"ContainerStarted","Data":"8eec240a6663240347160fa2c893eb54a925408b75bee523c5313a91310a1a37"}
Nov 03 20:02:59.705226 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:59.705093    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dce558-77fccb9bff-qgtj8" event={"ID":"6c554c44-d20a-4efe-9faf-5c30ec3a3f5e","Type":"ContainerStarted","Data":"d8d13f45ce6cafa28158ba3203a5ace7d0b789ad37bb85056aa3517c58132e29"}
Nov 03 20:02:59.705226 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:59.705124    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dce558-77fccb9bff-qgtj8"
Nov 03 20:02:59.718356 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:59.718292    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-dce558-566ddc874c-ctw65" podStartSLOduration=2.718273146 podStartE2EDuration="2.718273146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:02:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:02:59.716485107 +0000 UTC m=+1892.960450585" watchObservedRunningTime="2026-11-03 20:02:59.718273146 +0000 UTC m=+1892.962238612"
Nov 03 20:02:59.732118 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:02:59.732067    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dce558-77fccb9bff-qgtj8" podStartSLOduration=2.732052633 podStartE2EDuration="2.732052633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:02:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:02:59.729962098 +0000 UTC m=+1892.973927564" watchObservedRunningTime="2026-11-03 20:02:59.732052633 +0000 UTC m=+1892.976018097"
Nov 03 20:03:02.475328 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:02.475301    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb"
Nov 03 20:03:02.561941 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:02.561899    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8423f401-70db-4f4a-9bf5-0597acb06be6-istio-token\" (UniqueName: \"kubernetes.io/projected/8423f401-70db-4f4a-9bf5-0597acb06be6-istio-token\") pod \"8423f401-70db-4f4a-9bf5-0597acb06be6\" (UID: \"8423f401-70db-4f4a-9bf5-0597acb06be6\") "
Nov 03 20:03:02.562122 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:02.562101    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8423f401-70db-4f4a-9bf5-0597acb06be6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8423f401-70db-4f4a-9bf5-0597acb06be6-istio-data\") pod \"8423f401-70db-4f4a-9bf5-0597acb06be6\" (UID: \"8423f401-70db-4f4a-9bf5-0597acb06be6\") "
Nov 03 20:03:02.562195 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:02.562154    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8423f401-70db-4f4a-9bf5-0597acb06be6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8423f401-70db-4f4a-9bf5-0597acb06be6-workload-certs\") pod \"8423f401-70db-4f4a-9bf5-0597acb06be6\" (UID: \"8423f401-70db-4f4a-9bf5-0597acb06be6\") "
Nov 03 20:03:02.562195 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:02.562183    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8423f401-70db-4f4a-9bf5-0597acb06be6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8423f401-70db-4f4a-9bf5-0597acb06be6-istio-envoy\") pod \"8423f401-70db-4f4a-9bf5-0597acb06be6\" (UID: \"8423f401-70db-4f4a-9bf5-0597acb06be6\") "
Nov 03 20:03:02.562314 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:02.562232    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8423f401-70db-4f4a-9bf5-0597acb06be6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8423f401-70db-4f4a-9bf5-0597acb06be6-istiod-ca-cert\") pod \"8423f401-70db-4f4a-9bf5-0597acb06be6\" (UID: \"8423f401-70db-4f4a-9bf5-0597acb06be6\") "
Nov 03 20:03:02.562314 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:02.562269    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8423f401-70db-4f4a-9bf5-0597acb06be6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8423f401-70db-4f4a-9bf5-0597acb06be6-credential-socket\") pod \"8423f401-70db-4f4a-9bf5-0597acb06be6\" (UID: \"8423f401-70db-4f4a-9bf5-0597acb06be6\") "
Nov 03 20:03:02.562452 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:02.562424    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8423f401-70db-4f4a-9bf5-0597acb06be6-kube-api-access-hfhjj\" (UniqueName: \"kubernetes.io/projected/8423f401-70db-4f4a-9bf5-0597acb06be6-kube-api-access-hfhjj\") pod \"8423f401-70db-4f4a-9bf5-0597acb06be6\" (UID: \"8423f401-70db-4f4a-9bf5-0597acb06be6\") "
Nov 03 20:03:02.562513 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:02.562464    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8423f401-70db-4f4a-9bf5-0597acb06be6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8423f401-70db-4f4a-9bf5-0597acb06be6-istio-podinfo\") pod \"8423f401-70db-4f4a-9bf5-0597acb06be6\" (UID: \"8423f401-70db-4f4a-9bf5-0597acb06be6\") "
Nov 03 20:03:02.562591 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:02.562526    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8423f401-70db-4f4a-9bf5-0597acb06be6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8423f401-70db-4f4a-9bf5-0597acb06be6-workload-socket\") pod \"8423f401-70db-4f4a-9bf5-0597acb06be6\" (UID: \"8423f401-70db-4f4a-9bf5-0597acb06be6\") "
Nov 03 20:03:02.562680 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:02.562639    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8423f401-70db-4f4a-9bf5-0597acb06be6-credential-socket" pod "8423f401-70db-4f4a-9bf5-0597acb06be6" (UID: "8423f401-70db-4f4a-9bf5-0597acb06be6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:03:02.562736 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:02.562696    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8423f401-70db-4f4a-9bf5-0597acb06be6-istiod-ca-cert" pod "8423f401-70db-4f4a-9bf5-0597acb06be6" (UID: "8423f401-70db-4f4a-9bf5-0597acb06be6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:03:02.562808 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:02.562774    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8423f401-70db-4f4a-9bf5-0597acb06be6-istio-data" pod "8423f401-70db-4f4a-9bf5-0597acb06be6" (UID: "8423f401-70db-4f4a-9bf5-0597acb06be6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:03:02.563065 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:02.563029    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8423f401-70db-4f4a-9bf5-0597acb06be6-workload-socket" pod "8423f401-70db-4f4a-9bf5-0597acb06be6" (UID: "8423f401-70db-4f4a-9bf5-0597acb06be6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:03:02.563423 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:02.563398    4921 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8423f401-70db-4f4a-9bf5-0597acb06be6-istio-data\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:02.563536 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:02.563430    4921 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8423f401-70db-4f4a-9bf5-0597acb06be6-istiod-ca-cert\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:02.563536 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:02.563447    4921 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8423f401-70db-4f4a-9bf5-0597acb06be6-credential-socket\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:02.563536 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:02.563462    4921 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8423f401-70db-4f4a-9bf5-0597acb06be6-workload-socket\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:02.565969 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:02.563885    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8423f401-70db-4f4a-9bf5-0597acb06be6-workload-certs" pod "8423f401-70db-4f4a-9bf5-0597acb06be6" (UID: "8423f401-70db-4f4a-9bf5-0597acb06be6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:03:02.570355 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:02.570304    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8423f401-70db-4f4a-9bf5-0597acb06be6-istio-envoy" pod "8423f401-70db-4f4a-9bf5-0597acb06be6" (UID: "8423f401-70db-4f4a-9bf5-0597acb06be6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:03:02.570553 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:02.570357    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8423f401-70db-4f4a-9bf5-0597acb06be6-istio-token" pod "8423f401-70db-4f4a-9bf5-0597acb06be6" (UID: "8423f401-70db-4f4a-9bf5-0597acb06be6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:03:02.571081 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:02.571043    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8423f401-70db-4f4a-9bf5-0597acb06be6-istio-podinfo" pod "8423f401-70db-4f4a-9bf5-0597acb06be6" (UID: "8423f401-70db-4f4a-9bf5-0597acb06be6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:03:02.574313 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:02.574238    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8423f401-70db-4f4a-9bf5-0597acb06be6-kube-api-access-hfhjj" pod "8423f401-70db-4f4a-9bf5-0597acb06be6" (UID: "8423f401-70db-4f4a-9bf5-0597acb06be6"). InnerVolumeSpecName "kube-api-access-hfhjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:03:02.664995 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:02.664955    4921 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8423f401-70db-4f4a-9bf5-0597acb06be6-workload-certs\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:02.665256 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:02.665234    4921 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8423f401-70db-4f4a-9bf5-0597acb06be6-istio-envoy\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:02.665476 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:02.665436    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfhjj\" (UniqueName: \"kubernetes.io/projected/8423f401-70db-4f4a-9bf5-0597acb06be6-kube-api-access-hfhjj\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:02.665629 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:02.665613    4921 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8423f401-70db-4f4a-9bf5-0597acb06be6-istio-podinfo\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:02.666005 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:02.665977    4921 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8423f401-70db-4f4a-9bf5-0597acb06be6-istio-token\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:02.744532 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:02.744494    4921 generic.go:356] "Generic (PLEG): container finished" podID="8423f401-70db-4f4a-9bf5-0597acb06be6" containerID="01974174fc1b417c37ea4625b302d901dba257d096d3d9455d313e3bff633938" exitCode=0
Nov 03 20:03:02.744752 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:02.744549    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb"
Nov 03 20:03:02.744752 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:02.744575    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb" event={"ID":"8423f401-70db-4f4a-9bf5-0597acb06be6","Type":"ContainerDied","Data":"01974174fc1b417c37ea4625b302d901dba257d096d3d9455d313e3bff633938"}
Nov 03 20:03:02.744752 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:02.744632    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb" event={"ID":"8423f401-70db-4f4a-9bf5-0597acb06be6","Type":"ContainerDied","Data":"604d7ee62ad8c7e009724ec948831a9434eb1e273d1fdb5c367da56bc717ba51"}
Nov 03 20:03:02.744752 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:02.744660    4921 scope.go:122] "RemoveContainer" containerID="01974174fc1b417c37ea4625b302d901dba257d096d3d9455d313e3bff633938"
Nov 03 20:03:02.758170 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:02.758150    4921 scope.go:122] "RemoveContainer" containerID="01974174fc1b417c37ea4625b302d901dba257d096d3d9455d313e3bff633938"
Nov 03 20:03:02.758438 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:03:02.758422    4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01974174fc1b417c37ea4625b302d901dba257d096d3d9455d313e3bff633938\": container with ID starting with 01974174fc1b417c37ea4625b302d901dba257d096d3d9455d313e3bff633938 not found: ID does not exist" containerID="01974174fc1b417c37ea4625b302d901dba257d096d3d9455d313e3bff633938"
Nov 03 20:03:02.758497 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:02.758448    4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01974174fc1b417c37ea4625b302d901dba257d096d3d9455d313e3bff633938"} err="failed to get container status \"01974174fc1b417c37ea4625b302d901dba257d096d3d9455d313e3bff633938\": rpc error: code = NotFound desc = could not find container \"01974174fc1b417c37ea4625b302d901dba257d096d3d9455d313e3bff633938\": container with ID starting with 01974174fc1b417c37ea4625b302d901dba257d096d3d9455d313e3bff633938 not found: ID does not exist"
Nov 03 20:03:02.770304 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:02.770265    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb"]
Nov 03 20:03:02.774381 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:02.774349    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-a99fad-openshift-default-8475f7788f-lzlqb"]
Nov 03 20:03:02.890864 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:02.890822    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8423f401-70db-4f4a-9bf5-0597acb06be6" path="/var/lib/kubelet/pods/8423f401-70db-4f4a-9bf5-0597acb06be6/volumes"
Nov 03 20:03:02.924167 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:03:02.924132    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 20:03:03.934709 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:03.934659    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-801162-74c864f4c-kwqd4"]
Nov 03 20:03:03.935409 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:03.935393    4921 cpu_manager.go:414] "containerMap: removing container" podUID="8423f401-70db-4f4a-9bf5-0597acb06be6" containerName="istio-proxy"
Nov 03 20:03:03.935409 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:03.935409    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8423f401-70db-4f4a-9bf5-0597acb06be6" containerName="istio-proxy"
Nov 03 20:03:03.935551 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:03.935542    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="8423f401-70db-4f4a-9bf5-0597acb06be6" containerName="istio-proxy"
Nov 03 20:03:03.941164 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:03.941141    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-801162-74c864f4c-kwqd4"
Nov 03 20:03:03.947581 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:03.947547    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w5-801162\""
Nov 03 20:03:03.947753 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:03.947716    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w5-801162-dockercfg-jmzvg\""
Nov 03 20:03:03.948529 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:03.948498    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-801162-74c864f4c-kwqd4"]
Nov 03 20:03:03.981416 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:03.981378    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-801162-74759cf659-bdg8m"]
Nov 03 20:03:03.987586 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:03.987558    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-801162-74759cf659-bdg8m"
Nov 03 20:03:03.988821 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:03.988789    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86ww2\" (UniqueName: \"kubernetes.io/projected/cbdab358-70c7-4c98-a2c4-4dc4a924e8d9-kube-api-access-86ww2\") pod \"payload-pre-processing-e2e-shared-a-w5-801162-74c864f4c-kwqd4\" (UID: \"cbdab358-70c7-4c98-a2c4-4dc4a924e8d9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-801162-74c864f4c-kwqd4"
Nov 03 20:03:03.989009 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:03.988901    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbdab358-70c7-4c98-a2c4-4dc4a924e8d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-801162-74c864f4c-kwqd4\" (UID: \"cbdab358-70c7-4c98-a2c4-4dc4a924e8d9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-801162-74c864f4c-kwqd4"
Nov 03 20:03:04.024374 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:04.024312    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-801162-74759cf659-bdg8m"]
Nov 03 20:03:04.111709 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:04.109226    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvsxx\" (UniqueName: \"kubernetes.io/projected/0df8b636-6a8f-4a9d-ab89-2dabedc2bc21-kube-api-access-tvsxx\") pod \"payload-processing-e2e-shared-a-w5-801162-74759cf659-bdg8m\" (UID: \"0df8b636-6a8f-4a9d-ab89-2dabedc2bc21\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-801162-74759cf659-bdg8m"
Nov 03 20:03:04.111709 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:04.109354    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86ww2\" (UniqueName: \"kubernetes.io/projected/cbdab358-70c7-4c98-a2c4-4dc4a924e8d9-kube-api-access-86ww2\") pod \"payload-pre-processing-e2e-shared-a-w5-801162-74c864f4c-kwqd4\" (UID: \"cbdab358-70c7-4c98-a2c4-4dc4a924e8d9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-801162-74c864f4c-kwqd4"
Nov 03 20:03:04.111709 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:04.109432    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbdab358-70c7-4c98-a2c4-4dc4a924e8d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-801162-74c864f4c-kwqd4\" (UID: \"cbdab358-70c7-4c98-a2c4-4dc4a924e8d9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-801162-74c864f4c-kwqd4"
Nov 03 20:03:04.111709 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:04.109472    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0df8b636-6a8f-4a9d-ab89-2dabedc2bc21-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-801162-74759cf659-bdg8m\" (UID: \"0df8b636-6a8f-4a9d-ab89-2dabedc2bc21\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-801162-74759cf659-bdg8m"
Nov 03 20:03:04.112979 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:04.112951    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbdab358-70c7-4c98-a2c4-4dc4a924e8d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-801162-74c864f4c-kwqd4\" (UID: \"cbdab358-70c7-4c98-a2c4-4dc4a924e8d9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-801162-74c864f4c-kwqd4"
Nov 03 20:03:04.139885 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:04.139852    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86ww2\" (UniqueName: \"kubernetes.io/projected/cbdab358-70c7-4c98-a2c4-4dc4a924e8d9-kube-api-access-86ww2\") pod \"payload-pre-processing-e2e-shared-a-w5-801162-74c864f4c-kwqd4\" (UID: \"cbdab358-70c7-4c98-a2c4-4dc4a924e8d9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-801162-74c864f4c-kwqd4"
Nov 03 20:03:04.222745 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:04.220076    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0df8b636-6a8f-4a9d-ab89-2dabedc2bc21-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-801162-74759cf659-bdg8m\" (UID: \"0df8b636-6a8f-4a9d-ab89-2dabedc2bc21\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-801162-74759cf659-bdg8m"
Nov 03 20:03:04.222745 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:04.220228    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvsxx\" (UniqueName: \"kubernetes.io/projected/0df8b636-6a8f-4a9d-ab89-2dabedc2bc21-kube-api-access-tvsxx\") pod \"payload-processing-e2e-shared-a-w5-801162-74759cf659-bdg8m\" (UID: \"0df8b636-6a8f-4a9d-ab89-2dabedc2bc21\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-801162-74759cf659-bdg8m"
Nov 03 20:03:04.266654 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:04.266621    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvsxx\" (UniqueName: \"kubernetes.io/projected/0df8b636-6a8f-4a9d-ab89-2dabedc2bc21-kube-api-access-tvsxx\") pod \"payload-processing-e2e-shared-a-w5-801162-74759cf659-bdg8m\" (UID: \"0df8b636-6a8f-4a9d-ab89-2dabedc2bc21\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-801162-74759cf659-bdg8m"
Nov 03 20:03:04.270937 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:04.270904    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-801162-74c864f4c-kwqd4"
Nov 03 20:03:04.271527 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:04.271499    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0df8b636-6a8f-4a9d-ab89-2dabedc2bc21-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-801162-74759cf659-bdg8m\" (UID: \"0df8b636-6a8f-4a9d-ab89-2dabedc2bc21\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-801162-74759cf659-bdg8m"
Nov 03 20:03:04.352585 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:04.352555    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-801162-74759cf659-bdg8m"
Nov 03 20:03:04.643872 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:04.643834    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-225668-65675d6655-qdr5l" podUID="2105ca77-5b9c-45b2-9256-782b998a0041" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:03:04.650721 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:04.650682    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-225668-6d7866c8dc-29bbm" podUID="e2c9f88f-de8e-406e-ad7d-2038e0faba39" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:03:04.666288 ip-10-0-134-222 kubenswrapper[4921]: W1103 20:03:04.666249    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbdab358_70c7_4c98_a2c4_4dc4a924e8d9.slice/crio-5691f25b141e55994421ade9c42e2c8ed473dc77a13d335caf6b575588605632 WatchSource:0}: Error finding container 5691f25b141e55994421ade9c42e2c8ed473dc77a13d335caf6b575588605632: Status 404 returned error can't find the container with id 5691f25b141e55994421ade9c42e2c8ed473dc77a13d335caf6b575588605632
Nov 03 20:03:04.666440 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:04.666394    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-801162-74c864f4c-kwqd4"]
Nov 03 20:03:04.683407 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:04.683369    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-801162-74759cf659-bdg8m"]
Nov 03 20:03:04.693094 ip-10-0-134-222 kubenswrapper[4921]: W1103 20:03:04.693049    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0df8b636_6a8f_4a9d_ab89_2dabedc2bc21.slice/crio-eda212e404d652116073cf1a1f0b72a1d707e24b5b2ef4df5db5e220cefa2895 WatchSource:0}: Error finding container eda212e404d652116073cf1a1f0b72a1d707e24b5b2ef4df5db5e220cefa2895: Status 404 returned error can't find the container with id eda212e404d652116073cf1a1f0b72a1d707e24b5b2ef4df5db5e220cefa2895
Nov 03 20:03:04.766945 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:04.766914    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-801162-74c864f4c-kwqd4" event={"ID":"cbdab358-70c7-4c98-a2c4-4dc4a924e8d9","Type":"ContainerStarted","Data":"5691f25b141e55994421ade9c42e2c8ed473dc77a13d335caf6b575588605632"}
Nov 03 20:03:04.769167 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:04.769047    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-801162-74759cf659-bdg8m" event={"ID":"0df8b636-6a8f-4a9d-ab89-2dabedc2bc21","Type":"ContainerStarted","Data":"eda212e404d652116073cf1a1f0b72a1d707e24b5b2ef4df5db5e220cefa2895"}
Nov 03 20:03:05.801871 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:05.801831    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-801162-74759cf659-bdg8m" event={"ID":"0df8b636-6a8f-4a9d-ab89-2dabedc2bc21","Type":"ContainerStarted","Data":"df4113ff6e49262a34805e323c91ceb7c186c5b1c36dccae631fd70e206eb4c0"}
Nov 03 20:03:05.802637 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:05.802616    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-801162-74759cf659-bdg8m"
Nov 03 20:03:05.806162 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:05.806129    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-801162-74c864f4c-kwqd4" event={"ID":"cbdab358-70c7-4c98-a2c4-4dc4a924e8d9","Type":"ContainerStarted","Data":"10feec450e6673f6a0b644e19d8bdc6ea0a6994747257faeba9230a38a598ba3"}
Nov 03 20:03:05.806311 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:05.806234    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-801162-74c864f4c-kwqd4"
Nov 03 20:03:05.828693 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:05.826848    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-801162-74759cf659-bdg8m" podStartSLOduration=2.8268304840000003 podStartE2EDuration="2.826830484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:05.817532502 +0000 UTC m=+1899.061497968" watchObservedRunningTime="2026-11-03 20:03:05.826830484 +0000 UTC m=+1899.070795950"
Nov 03 20:03:05.840521 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:05.840461    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-801162-74c864f4c-kwqd4" podStartSLOduration=2.840441553 podStartE2EDuration="2.840441553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:05.835935293 +0000 UTC m=+1899.079900758" watchObservedRunningTime="2026-11-03 20:03:05.840441553 +0000 UTC m=+1899.084407020"
Nov 03 20:03:07.834680 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:07.834629    4921 generic.go:356] "Generic (PLEG): container finished" podID="2105ca77-5b9c-45b2-9256-782b998a0041" containerID="925ce0d76832c7f770773101fd88e6ee410b0ce690e8577656f6fb1702ed5cb3" exitCode=0
Nov 03 20:03:07.835266 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:07.834757    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-225668-65675d6655-qdr5l" event={"ID":"2105ca77-5b9c-45b2-9256-782b998a0041","Type":"ContainerDied","Data":"925ce0d76832c7f770773101fd88e6ee410b0ce690e8577656f6fb1702ed5cb3"}
Nov 03 20:03:07.837847 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:07.837800    4921 generic.go:356] "Generic (PLEG): container finished" podID="e2c9f88f-de8e-406e-ad7d-2038e0faba39" containerID="3b0c6a0e30cb1fcb599d3a582876cbaaa9dcd837ed8e58ed55f26a4e6d29ec73" exitCode=0
Nov 03 20:03:07.838021 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:07.837966    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-225668-6d7866c8dc-29bbm" event={"ID":"e2c9f88f-de8e-406e-ad7d-2038e0faba39","Type":"ContainerDied","Data":"3b0c6a0e30cb1fcb599d3a582876cbaaa9dcd837ed8e58ed55f26a4e6d29ec73"}
Nov 03 20:03:07.996947 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:07.996912    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-225668-6d7866c8dc-29bbm"
Nov 03 20:03:08.004372 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:08.004341    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-225668-65675d6655-qdr5l"
Nov 03 20:03:08.126439 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:08.126405    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2c9f88f-de8e-406e-ad7d-2038e0faba39-kube-api-access-phljb\" (UniqueName: \"kubernetes.io/projected/e2c9f88f-de8e-406e-ad7d-2038e0faba39-kube-api-access-phljb\") pod \"e2c9f88f-de8e-406e-ad7d-2038e0faba39\" (UID: \"e2c9f88f-de8e-406e-ad7d-2038e0faba39\") "
Nov 03 20:03:08.126610 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:08.126496    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2105ca77-5b9c-45b2-9256-782b998a0041-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2105ca77-5b9c-45b2-9256-782b998a0041-plugins-config-volume\") pod \"2105ca77-5b9c-45b2-9256-782b998a0041\" (UID: \"2105ca77-5b9c-45b2-9256-782b998a0041\") "
Nov 03 20:03:08.126610 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:08.126593    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2c9f88f-de8e-406e-ad7d-2038e0faba39-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2c9f88f-de8e-406e-ad7d-2038e0faba39-plugins-config-volume\") pod \"e2c9f88f-de8e-406e-ad7d-2038e0faba39\" (UID: \"e2c9f88f-de8e-406e-ad7d-2038e0faba39\") "
Nov 03 20:03:08.126787 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:08.126722    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2105ca77-5b9c-45b2-9256-782b998a0041-kube-api-access-d5lkl\" (UniqueName: \"kubernetes.io/projected/2105ca77-5b9c-45b2-9256-782b998a0041-kube-api-access-d5lkl\") pod \"2105ca77-5b9c-45b2-9256-782b998a0041\" (UID: \"2105ca77-5b9c-45b2-9256-782b998a0041\") "
Nov 03 20:03:08.127005 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:08.126977    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2105ca77-5b9c-45b2-9256-782b998a0041-plugins-config-volume" pod "2105ca77-5b9c-45b2-9256-782b998a0041" (UID: "2105ca77-5b9c-45b2-9256-782b998a0041"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:03:08.127078 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:08.126995    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2c9f88f-de8e-406e-ad7d-2038e0faba39-plugins-config-volume" pod "e2c9f88f-de8e-406e-ad7d-2038e0faba39" (UID: "e2c9f88f-de8e-406e-ad7d-2038e0faba39"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:03:08.127374 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:08.127354    4921 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2c9f88f-de8e-406e-ad7d-2038e0faba39-plugins-config-volume\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:08.127494 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:08.127375    4921 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2105ca77-5b9c-45b2-9256-782b998a0041-plugins-config-volume\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:08.129116 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:08.129088    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2c9f88f-de8e-406e-ad7d-2038e0faba39-kube-api-access-phljb" pod "e2c9f88f-de8e-406e-ad7d-2038e0faba39" (UID: "e2c9f88f-de8e-406e-ad7d-2038e0faba39"). InnerVolumeSpecName "kube-api-access-phljb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:03:08.129195 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:08.129129    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2105ca77-5b9c-45b2-9256-782b998a0041-kube-api-access-d5lkl" pod "2105ca77-5b9c-45b2-9256-782b998a0041" (UID: "2105ca77-5b9c-45b2-9256-782b998a0041"). InnerVolumeSpecName "kube-api-access-d5lkl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:03:08.228306 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:08.228268    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phljb\" (UniqueName: \"kubernetes.io/projected/e2c9f88f-de8e-406e-ad7d-2038e0faba39-kube-api-access-phljb\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:08.228306 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:08.228304    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5lkl\" (UniqueName: \"kubernetes.io/projected/2105ca77-5b9c-45b2-9256-782b998a0041-kube-api-access-d5lkl\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:08.847635 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:08.847605    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-225668-65675d6655-qdr5l"
Nov 03 20:03:08.847635 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:08.847613    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-225668-65675d6655-qdr5l" event={"ID":"2105ca77-5b9c-45b2-9256-782b998a0041","Type":"ContainerDied","Data":"35e9259dd57d1adefb45a4b4f5fb61eab195ae0d50f7e316ed23a9eccfc63eea"}
Nov 03 20:03:08.848259 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:08.847741    4921 scope.go:122] "RemoveContainer" containerID="925ce0d76832c7f770773101fd88e6ee410b0ce690e8577656f6fb1702ed5cb3"
Nov 03 20:03:08.849300 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:08.849275    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-225668-6d7866c8dc-29bbm"
Nov 03 20:03:08.849300 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:08.849299    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-225668-6d7866c8dc-29bbm" event={"ID":"e2c9f88f-de8e-406e-ad7d-2038e0faba39","Type":"ContainerDied","Data":"03a9883bb8a17a31e79fa45397a76987fa72bd89caad6c083a44fbcfca515e4e"}
Nov 03 20:03:08.865364 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:08.865344    4921 scope.go:122] "RemoveContainer" containerID="3b0c6a0e30cb1fcb599d3a582876cbaaa9dcd837ed8e58ed55f26a4e6d29ec73"
Nov 03 20:03:08.874845 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:08.874819    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-225668-65675d6655-qdr5l"]
Nov 03 20:03:08.879731 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:08.879708    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-225668-65675d6655-qdr5l"]
Nov 03 20:03:08.893117 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:08.893092    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2105ca77-5b9c-45b2-9256-782b998a0041" path="/var/lib/kubelet/pods/2105ca77-5b9c-45b2-9256-782b998a0041/volumes"
Nov 03 20:03:08.893535 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:08.893514    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-225668-6d7866c8dc-29bbm"]
Nov 03 20:03:08.894538 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:08.894513    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-225668-6d7866c8dc-29bbm"]
Nov 03 20:03:10.720728 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:10.718923    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-dce558-566ddc874c-ctw65"
Nov 03 20:03:10.729511 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:10.727485    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dce558-77fccb9bff-qgtj8"
Nov 03 20:03:10.889436 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:10.889406    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2c9f88f-de8e-406e-ad7d-2038e0faba39" path="/var/lib/kubelet/pods/e2c9f88f-de8e-406e-ad7d-2038e0faba39/volumes"
Nov 03 20:03:10.985616 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:10.985581    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr"]
Nov 03 20:03:10.986093 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:10.986046    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr" podUID="c05f083e-3052-44f9-ac8f-746ea890191b" containerName="istio-proxy" containerID="cri-o://4c1bcfe0668dc1be5482f113148b76ff1946205adf4e00c1b2427a7eb896fa57" gracePeriod=30
Nov 03 20:03:11.551571 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:11.551529    4921 patch_prober.go:29] interesting pod/e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:03:11.551806 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:11.551590    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr" podUID="c05f083e-3052-44f9-ac8f-746ea890191b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:03:16.028419 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:16.028390    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dce558-7bcc44d6c-s92vv"]
Nov 03 20:03:16.029592 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:16.029567    4921 cpu_manager.go:414] "containerMap: removing container" podUID="e2c9f88f-de8e-406e-ad7d-2038e0faba39" containerName="payload-pre-processing"
Nov 03 20:03:16.029712 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:16.029594    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2c9f88f-de8e-406e-ad7d-2038e0faba39" containerName="payload-pre-processing"
Nov 03 20:03:16.029712 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:16.029638    4921 cpu_manager.go:414] "containerMap: removing container" podUID="2105ca77-5b9c-45b2-9256-782b998a0041" containerName="payload-processing"
Nov 03 20:03:16.029712 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:16.029647    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2105ca77-5b9c-45b2-9256-782b998a0041" containerName="payload-processing"
Nov 03 20:03:16.030976 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:16.029858    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2c9f88f-de8e-406e-ad7d-2038e0faba39" containerName="payload-pre-processing"
Nov 03 20:03:16.030976 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:16.029922    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="2105ca77-5b9c-45b2-9256-782b998a0041" containerName="payload-processing"
Nov 03 20:03:16.034784 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:16.034756    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dce558-7bcc44d6c-s92vv"
Nov 03 20:03:16.038338 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:16.038308    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-dce558-serving-cert\""
Nov 03 20:03:16.039990 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:16.039961    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dce558-7bcc44d6c-s92vv"]
Nov 03 20:03:16.159717 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:16.156880    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cb2ad5f-35b4-4c92-9499-d7ac8152a72c-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-dce558-7bcc44d6c-s92vv\" (UID: \"6cb2ad5f-35b4-4c92-9499-d7ac8152a72c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dce558-7bcc44d6c-s92vv"
Nov 03 20:03:16.159717 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:16.157063    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cb2ad5f-35b4-4c92-9499-d7ac8152a72c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-dce558-7bcc44d6c-s92vv\" (UID: \"6cb2ad5f-35b4-4c92-9499-d7ac8152a72c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dce558-7bcc44d6c-s92vv"
Nov 03 20:03:16.159717 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:16.157179    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcgvh\" (UniqueName: \"kubernetes.io/projected/6cb2ad5f-35b4-4c92-9499-d7ac8152a72c-kube-api-access-mcgvh\") pod \"maas-api-e2e-shared-b-w0-dce558-7bcc44d6c-s92vv\" (UID: \"6cb2ad5f-35b4-4c92-9499-d7ac8152a72c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dce558-7bcc44d6c-s92vv"
Nov 03 20:03:16.159717 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:16.157306    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cb2ad5f-35b4-4c92-9499-d7ac8152a72c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-dce558-7bcc44d6c-s92vv\" (UID: \"6cb2ad5f-35b4-4c92-9499-d7ac8152a72c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dce558-7bcc44d6c-s92vv"
Nov 03 20:03:16.259822 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:16.259746    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cb2ad5f-35b4-4c92-9499-d7ac8152a72c-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-dce558-7bcc44d6c-s92vv\" (UID: \"6cb2ad5f-35b4-4c92-9499-d7ac8152a72c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dce558-7bcc44d6c-s92vv"
Nov 03 20:03:16.260683 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:16.260605    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cb2ad5f-35b4-4c92-9499-d7ac8152a72c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-dce558-7bcc44d6c-s92vv\" (UID: \"6cb2ad5f-35b4-4c92-9499-d7ac8152a72c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dce558-7bcc44d6c-s92vv"
Nov 03 20:03:16.261344 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:16.261320    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcgvh\" (UniqueName: \"kubernetes.io/projected/6cb2ad5f-35b4-4c92-9499-d7ac8152a72c-kube-api-access-mcgvh\") pod \"maas-api-e2e-shared-b-w0-dce558-7bcc44d6c-s92vv\" (UID: \"6cb2ad5f-35b4-4c92-9499-d7ac8152a72c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dce558-7bcc44d6c-s92vv"
Nov 03 20:03:16.261590 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:16.261571    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cb2ad5f-35b4-4c92-9499-d7ac8152a72c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-dce558-7bcc44d6c-s92vv\" (UID: \"6cb2ad5f-35b4-4c92-9499-d7ac8152a72c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dce558-7bcc44d6c-s92vv"
Nov 03 20:03:16.262589 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:16.262568    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cb2ad5f-35b4-4c92-9499-d7ac8152a72c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-dce558-7bcc44d6c-s92vv\" (UID: \"6cb2ad5f-35b4-4c92-9499-d7ac8152a72c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dce558-7bcc44d6c-s92vv"
Nov 03 20:03:16.277799 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:16.277403    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cb2ad5f-35b4-4c92-9499-d7ac8152a72c-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-dce558-7bcc44d6c-s92vv\" (UID: \"6cb2ad5f-35b4-4c92-9499-d7ac8152a72c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dce558-7bcc44d6c-s92vv"
Nov 03 20:03:16.279840 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:16.279468    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cb2ad5f-35b4-4c92-9499-d7ac8152a72c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-dce558-7bcc44d6c-s92vv\" (UID: \"6cb2ad5f-35b4-4c92-9499-d7ac8152a72c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dce558-7bcc44d6c-s92vv"
Nov 03 20:03:16.282961 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:16.282935    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcgvh\" (UniqueName: \"kubernetes.io/projected/6cb2ad5f-35b4-4c92-9499-d7ac8152a72c-kube-api-access-mcgvh\") pod \"maas-api-e2e-shared-b-w0-dce558-7bcc44d6c-s92vv\" (UID: \"6cb2ad5f-35b4-4c92-9499-d7ac8152a72c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dce558-7bcc44d6c-s92vv"
Nov 03 20:03:16.314603 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:16.314577    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr"
Nov 03 20:03:16.393516 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:16.393483    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dce558-7bcc44d6c-s92vv"
Nov 03 20:03:16.464970 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:16.464935    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c05f083e-3052-44f9-ac8f-746ea890191b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c05f083e-3052-44f9-ac8f-746ea890191b-credential-socket\") pod \"c05f083e-3052-44f9-ac8f-746ea890191b\" (UID: \"c05f083e-3052-44f9-ac8f-746ea890191b\") "
Nov 03 20:03:16.465142 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:16.465008    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c05f083e-3052-44f9-ac8f-746ea890191b-kube-api-access-lsqqd\" (UniqueName: \"kubernetes.io/projected/c05f083e-3052-44f9-ac8f-746ea890191b-kube-api-access-lsqqd\") pod \"c05f083e-3052-44f9-ac8f-746ea890191b\" (UID: \"c05f083e-3052-44f9-ac8f-746ea890191b\") "
Nov 03 20:03:16.465142 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:16.465056    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c05f083e-3052-44f9-ac8f-746ea890191b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c05f083e-3052-44f9-ac8f-746ea890191b-istio-envoy\") pod \"c05f083e-3052-44f9-ac8f-746ea890191b\" (UID: \"c05f083e-3052-44f9-ac8f-746ea890191b\") "
Nov 03 20:03:16.465142 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:16.465119    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c05f083e-3052-44f9-ac8f-746ea890191b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c05f083e-3052-44f9-ac8f-746ea890191b-workload-socket\") pod \"c05f083e-3052-44f9-ac8f-746ea890191b\" (UID: \"c05f083e-3052-44f9-ac8f-746ea890191b\") "
Nov 03 20:03:16.465267 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:16.465202    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c05f083e-3052-44f9-ac8f-746ea890191b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c05f083e-3052-44f9-ac8f-746ea890191b-workload-certs\") pod \"c05f083e-3052-44f9-ac8f-746ea890191b\" (UID: \"c05f083e-3052-44f9-ac8f-746ea890191b\") "
Nov 03 20:03:16.465310 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:16.465271    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c05f083e-3052-44f9-ac8f-746ea890191b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c05f083e-3052-44f9-ac8f-746ea890191b-istiod-ca-cert\") pod \"c05f083e-3052-44f9-ac8f-746ea890191b\" (UID: \"c05f083e-3052-44f9-ac8f-746ea890191b\") "
Nov 03 20:03:16.465355 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:16.465338    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c05f083e-3052-44f9-ac8f-746ea890191b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c05f083e-3052-44f9-ac8f-746ea890191b-istio-data\") pod \"c05f083e-3052-44f9-ac8f-746ea890191b\" (UID: \"c05f083e-3052-44f9-ac8f-746ea890191b\") "
Nov 03 20:03:16.465554 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:16.465535    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c05f083e-3052-44f9-ac8f-746ea890191b-istio-token\" (UniqueName: \"kubernetes.io/projected/c05f083e-3052-44f9-ac8f-746ea890191b-istio-token\") pod \"c05f083e-3052-44f9-ac8f-746ea890191b\" (UID: \"c05f083e-3052-44f9-ac8f-746ea890191b\") "
Nov 03 20:03:16.465648 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:16.465600    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c05f083e-3052-44f9-ac8f-746ea890191b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c05f083e-3052-44f9-ac8f-746ea890191b-istio-podinfo\") pod \"c05f083e-3052-44f9-ac8f-746ea890191b\" (UID: \"c05f083e-3052-44f9-ac8f-746ea890191b\") "
Nov 03 20:03:16.466033 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:16.466006    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c05f083e-3052-44f9-ac8f-746ea890191b-credential-socket" pod "c05f083e-3052-44f9-ac8f-746ea890191b" (UID: "c05f083e-3052-44f9-ac8f-746ea890191b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:03:16.466266 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:16.466248    4921 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c05f083e-3052-44f9-ac8f-746ea890191b-credential-socket\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:16.466635 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:16.466614    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c05f083e-3052-44f9-ac8f-746ea890191b-istiod-ca-cert" pod "c05f083e-3052-44f9-ac8f-746ea890191b" (UID: "c05f083e-3052-44f9-ac8f-746ea890191b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:03:16.466961 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:16.466799    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c05f083e-3052-44f9-ac8f-746ea890191b-workload-socket" pod "c05f083e-3052-44f9-ac8f-746ea890191b" (UID: "c05f083e-3052-44f9-ac8f-746ea890191b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:03:16.466961 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:16.466859    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c05f083e-3052-44f9-ac8f-746ea890191b-workload-certs" pod "c05f083e-3052-44f9-ac8f-746ea890191b" (UID: "c05f083e-3052-44f9-ac8f-746ea890191b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:03:16.466961 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:16.466927    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c05f083e-3052-44f9-ac8f-746ea890191b-istio-data" pod "c05f083e-3052-44f9-ac8f-746ea890191b" (UID: "c05f083e-3052-44f9-ac8f-746ea890191b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:03:16.478539 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:16.478491    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c05f083e-3052-44f9-ac8f-746ea890191b-kube-api-access-lsqqd" pod "c05f083e-3052-44f9-ac8f-746ea890191b" (UID: "c05f083e-3052-44f9-ac8f-746ea890191b"). InnerVolumeSpecName "kube-api-access-lsqqd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:03:16.485069 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:16.485043    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c05f083e-3052-44f9-ac8f-746ea890191b-istio-envoy" pod "c05f083e-3052-44f9-ac8f-746ea890191b" (UID: "c05f083e-3052-44f9-ac8f-746ea890191b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:03:16.486263 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:16.486238    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c05f083e-3052-44f9-ac8f-746ea890191b-istio-token" pod "c05f083e-3052-44f9-ac8f-746ea890191b" (UID: "c05f083e-3052-44f9-ac8f-746ea890191b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:03:16.487855 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:16.487832    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c05f083e-3052-44f9-ac8f-746ea890191b-istio-podinfo" pod "c05f083e-3052-44f9-ac8f-746ea890191b" (UID: "c05f083e-3052-44f9-ac8f-746ea890191b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:03:16.572023 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:16.571935    4921 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c05f083e-3052-44f9-ac8f-746ea890191b-workload-certs\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:16.572023 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:16.571975    4921 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c05f083e-3052-44f9-ac8f-746ea890191b-istiod-ca-cert\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:16.572023 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:16.571990    4921 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c05f083e-3052-44f9-ac8f-746ea890191b-istio-data\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:16.572023 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:16.572006    4921 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c05f083e-3052-44f9-ac8f-746ea890191b-istio-token\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:16.572023 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:16.572019    4921 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c05f083e-3052-44f9-ac8f-746ea890191b-istio-podinfo\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:16.573950 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:16.572034    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsqqd\" (UniqueName: \"kubernetes.io/projected/c05f083e-3052-44f9-ac8f-746ea890191b-kube-api-access-lsqqd\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:16.573950 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:16.572049    4921 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c05f083e-3052-44f9-ac8f-746ea890191b-istio-envoy\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:16.573950 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:16.572064    4921 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c05f083e-3052-44f9-ac8f-746ea890191b-workload-socket\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:16.805933 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:16.805864    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dce558-7bcc44d6c-s92vv"]
Nov 03 20:03:16.806461 ip-10-0-134-222 kubenswrapper[4921]: W1103 20:03:16.806420    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cb2ad5f_35b4_4c92_9499_d7ac8152a72c.slice/crio-7ccfffc1db40d26dca2d10ce6dda5ec22d1849744484ae1122cdd427285fa124 WatchSource:0}: Error finding container 7ccfffc1db40d26dca2d10ce6dda5ec22d1849744484ae1122cdd427285fa124: Status 404 returned error can't find the container with id 7ccfffc1db40d26dca2d10ce6dda5ec22d1849744484ae1122cdd427285fa124
Nov 03 20:03:16.807717 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:16.807623    4921 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 20:03:16.818126 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:16.818103    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-801162-74c864f4c-kwqd4"
Nov 03 20:03:16.818237 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:16.818183    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-801162-74759cf659-bdg8m"
Nov 03 20:03:16.933076 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:16.933045    4921 generic.go:356] "Generic (PLEG): container finished" podID="c05f083e-3052-44f9-ac8f-746ea890191b" containerID="4c1bcfe0668dc1be5482f113148b76ff1946205adf4e00c1b2427a7eb896fa57" exitCode=0
Nov 03 20:03:16.933247 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:16.933109    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr"
Nov 03 20:03:16.933247 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:16.933187    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr" event={"ID":"c05f083e-3052-44f9-ac8f-746ea890191b","Type":"ContainerDied","Data":"4c1bcfe0668dc1be5482f113148b76ff1946205adf4e00c1b2427a7eb896fa57"}
Nov 03 20:03:16.933247 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:16.933239    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr" event={"ID":"c05f083e-3052-44f9-ac8f-746ea890191b","Type":"ContainerDied","Data":"eab8e0760f1f6435aabe5434457ea4132af5fe2a9007d7705dc0c86304276ea1"}
Nov 03 20:03:16.933435 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:16.933267    4921 scope.go:122] "RemoveContainer" containerID="4c1bcfe0668dc1be5482f113148b76ff1946205adf4e00c1b2427a7eb896fa57"
Nov 03 20:03:16.935158 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:16.935131    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dce558-7bcc44d6c-s92vv" event={"ID":"6cb2ad5f-35b4-4c92-9499-d7ac8152a72c","Type":"ContainerStarted","Data":"7ccfffc1db40d26dca2d10ce6dda5ec22d1849744484ae1122cdd427285fa124"}
Nov 03 20:03:16.948440 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:16.948419    4921 scope.go:122] "RemoveContainer" containerID="4c1bcfe0668dc1be5482f113148b76ff1946205adf4e00c1b2427a7eb896fa57"
Nov 03 20:03:16.948763 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:03:16.948742    4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c1bcfe0668dc1be5482f113148b76ff1946205adf4e00c1b2427a7eb896fa57\": container with ID starting with 4c1bcfe0668dc1be5482f113148b76ff1946205adf4e00c1b2427a7eb896fa57 not found: ID does not exist" containerID="4c1bcfe0668dc1be5482f113148b76ff1946205adf4e00c1b2427a7eb896fa57"
Nov 03 20:03:16.948871 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:16.948768    4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c1bcfe0668dc1be5482f113148b76ff1946205adf4e00c1b2427a7eb896fa57"} err="failed to get container status \"4c1bcfe0668dc1be5482f113148b76ff1946205adf4e00c1b2427a7eb896fa57\": rpc error: code = NotFound desc = could not find container \"4c1bcfe0668dc1be5482f113148b76ff1946205adf4e00c1b2427a7eb896fa57\": container with ID starting with 4c1bcfe0668dc1be5482f113148b76ff1946205adf4e00c1b2427a7eb896fa57 not found: ID does not exist"
Nov 03 20:03:16.950501 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:16.950479    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr"]
Nov 03 20:03:16.955910 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:16.955889    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-102379-openshift-default-69dd54f7df-8bmvr"]
Nov 03 20:03:17.945495 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:17.945460    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dce558-7bcc44d6c-s92vv" event={"ID":"6cb2ad5f-35b4-4c92-9499-d7ac8152a72c","Type":"ContainerStarted","Data":"c794af3dc9751f62d719af8054db882f33d445e48fc32054cd4f774e40b29d2f"}
Nov 03 20:03:17.945935 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:17.945587    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dce558-7bcc44d6c-s92vv"
Nov 03 20:03:17.964220 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:17.964163    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dce558-7bcc44d6c-s92vv" podStartSLOduration=1.510625806 podStartE2EDuration="1.964147876s" totalImagesPullingTime="453.52207ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:17.959337552 +0000 UTC m=+1911.203303019" watchObservedRunningTime="2026-11-03 20:03:17.964147876 +0000 UTC m=+1911.208113341"
Nov 03 20:03:18.741099 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:18.741065    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-79c47f-5dfb89f897-kwzwd"]
Nov 03 20:03:18.741298 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:18.741274    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-79c47f-5dfb89f897-kwzwd" podUID="c38ce101-aa00-4700-8a03-854d3e56ee34" containerName="maas-api" containerID="cri-o://b7d633906df2c647305971a12c48549330b9649c5f4b5013cf4bb0e505a41ccb" gracePeriod=30
Nov 03 20:03:18.895599 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:18.895555    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c05f083e-3052-44f9-ac8f-746ea890191b" path="/var/lib/kubelet/pods/c05f083e-3052-44f9-ac8f-746ea890191b/volumes"
Nov 03 20:03:18.900903 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:18.900867    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-79c47f-5dfb89f897-kwzwd" podUID="c38ce101-aa00-4700-8a03-854d3e56ee34" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.152:8443/health\": dial tcp 10.132.0.152:8443: connect: connection refused"
Nov 03 20:03:18.971147 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:18.971110    4921 generic.go:356] "Generic (PLEG): container finished" podID="c38ce101-aa00-4700-8a03-854d3e56ee34" containerID="b7d633906df2c647305971a12c48549330b9649c5f4b5013cf4bb0e505a41ccb" exitCode=0
Nov 03 20:03:18.987335 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:18.972239    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-79c47f-5dfb89f897-kwzwd" event={"ID":"c38ce101-aa00-4700-8a03-854d3e56ee34","Type":"ContainerDied","Data":"b7d633906df2c647305971a12c48549330b9649c5f4b5013cf4bb0e505a41ccb"}
Nov 03 20:03:19.314239 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:19.314207    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-79c47f-5dfb89f897-kwzwd"
Nov 03 20:03:19.391045 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:19.391011    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c38ce101-aa00-4700-8a03-854d3e56ee34-kube-api-access-q7wnp\" (UniqueName: \"kubernetes.io/projected/c38ce101-aa00-4700-8a03-854d3e56ee34-kube-api-access-q7wnp\") pod \"c38ce101-aa00-4700-8a03-854d3e56ee34\" (UID: \"c38ce101-aa00-4700-8a03-854d3e56ee34\") "
Nov 03 20:03:19.391266 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:19.391243    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c38ce101-aa00-4700-8a03-854d3e56ee34-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c38ce101-aa00-4700-8a03-854d3e56ee34-metrics-certs\") pod \"c38ce101-aa00-4700-8a03-854d3e56ee34\" (UID: \"c38ce101-aa00-4700-8a03-854d3e56ee34\") "
Nov 03 20:03:19.391359 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:19.391292    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c38ce101-aa00-4700-8a03-854d3e56ee34-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c38ce101-aa00-4700-8a03-854d3e56ee34-maas-api-tls\") pod \"c38ce101-aa00-4700-8a03-854d3e56ee34\" (UID: \"c38ce101-aa00-4700-8a03-854d3e56ee34\") "
Nov 03 20:03:19.391418 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:19.391386    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c38ce101-aa00-4700-8a03-854d3e56ee34-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c38ce101-aa00-4700-8a03-854d3e56ee34-openshift-service-ca\") pod \"c38ce101-aa00-4700-8a03-854d3e56ee34\" (UID: \"c38ce101-aa00-4700-8a03-854d3e56ee34\") "
Nov 03 20:03:19.392236 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:19.392200    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c38ce101-aa00-4700-8a03-854d3e56ee34-openshift-service-ca" pod "c38ce101-aa00-4700-8a03-854d3e56ee34" (UID: "c38ce101-aa00-4700-8a03-854d3e56ee34"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:03:19.394008 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:19.393973    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c38ce101-aa00-4700-8a03-854d3e56ee34-metrics-certs" pod "c38ce101-aa00-4700-8a03-854d3e56ee34" (UID: "c38ce101-aa00-4700-8a03-854d3e56ee34"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:03:19.394460 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:19.394438    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c38ce101-aa00-4700-8a03-854d3e56ee34-maas-api-tls" pod "c38ce101-aa00-4700-8a03-854d3e56ee34" (UID: "c38ce101-aa00-4700-8a03-854d3e56ee34"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:03:19.399315 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:19.399285    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c38ce101-aa00-4700-8a03-854d3e56ee34-kube-api-access-q7wnp" pod "c38ce101-aa00-4700-8a03-854d3e56ee34" (UID: "c38ce101-aa00-4700-8a03-854d3e56ee34"). InnerVolumeSpecName "kube-api-access-q7wnp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:03:19.492878 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:19.492839    4921 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c38ce101-aa00-4700-8a03-854d3e56ee34-metrics-certs\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:19.492878 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:19.492871    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c38ce101-aa00-4700-8a03-854d3e56ee34-maas-api-tls\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:19.492878 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:19.492882    4921 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c38ce101-aa00-4700-8a03-854d3e56ee34-openshift-service-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:19.493100 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:19.492894    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7wnp\" (UniqueName: \"kubernetes.io/projected/c38ce101-aa00-4700-8a03-854d3e56ee34-kube-api-access-q7wnp\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:19.980574 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:19.980545    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-79c47f-5dfb89f897-kwzwd"
Nov 03 20:03:19.981023 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:19.980588    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-79c47f-5dfb89f897-kwzwd" event={"ID":"c38ce101-aa00-4700-8a03-854d3e56ee34","Type":"ContainerDied","Data":"7b3336bd9477acfb2e71e6a87f002d7fec35b17148e8ff34ef936c5ee48271ee"}
Nov 03 20:03:19.981023 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:19.980645    4921 scope.go:122] "RemoveContainer" containerID="b7d633906df2c647305971a12c48549330b9649c5f4b5013cf4bb0e505a41ccb"
Nov 03 20:03:20.004075 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:20.004045    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-79c47f-5dfb89f897-kwzwd"]
Nov 03 20:03:20.008527 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:20.008501    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-79c47f-5dfb89f897-kwzwd"]
Nov 03 20:03:20.890321 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:20.890288    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c38ce101-aa00-4700-8a03-854d3e56ee34" path="/var/lib/kubelet/pods/c38ce101-aa00-4700-8a03-854d3e56ee34/volumes"
Nov 03 20:03:21.949053 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:21.949014    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-801162-695dd8cc6c-hl47x"]
Nov 03 20:03:21.950133 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:21.950106    4921 cpu_manager.go:414] "containerMap: removing container" podUID="c05f083e-3052-44f9-ac8f-746ea890191b" containerName="istio-proxy"
Nov 03 20:03:21.950133 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:21.950130    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c05f083e-3052-44f9-ac8f-746ea890191b" containerName="istio-proxy"
Nov 03 20:03:21.950309 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:21.950202    4921 cpu_manager.go:414] "containerMap: removing container" podUID="c38ce101-aa00-4700-8a03-854d3e56ee34" containerName="maas-api"
Nov 03 20:03:21.950309 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:21.950210    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c38ce101-aa00-4700-8a03-854d3e56ee34" containerName="maas-api"
Nov 03 20:03:21.950374 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:21.950363    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="c05f083e-3052-44f9-ac8f-746ea890191b" containerName="istio-proxy"
Nov 03 20:03:21.950404 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:21.950383    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="c38ce101-aa00-4700-8a03-854d3e56ee34" containerName="maas-api"
Nov 03 20:03:21.956167 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:21.956144    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-801162-695dd8cc6c-hl47x"
Nov 03 20:03:21.960341 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:21.960313    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w5-801162-serving-cert\""
Nov 03 20:03:21.968329 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:21.968301    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-801162-695dd8cc6c-hl47x"]
Nov 03 20:03:22.051005 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:22.050974    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28288186-9960-4f98-8506-95c3dbf53cfd-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-801162-695dd8cc6c-hl47x\" (UID: \"28288186-9960-4f98-8506-95c3dbf53cfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-801162-695dd8cc6c-hl47x"
Nov 03 20:03:22.051210 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:22.051018    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28288186-9960-4f98-8506-95c3dbf53cfd-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-801162-695dd8cc6c-hl47x\" (UID: \"28288186-9960-4f98-8506-95c3dbf53cfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-801162-695dd8cc6c-hl47x"
Nov 03 20:03:22.051210 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:22.051182    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28288186-9960-4f98-8506-95c3dbf53cfd-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-801162-695dd8cc6c-hl47x\" (UID: \"28288186-9960-4f98-8506-95c3dbf53cfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-801162-695dd8cc6c-hl47x"
Nov 03 20:03:22.051360 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:22.051340    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q54g\" (UniqueName: \"kubernetes.io/projected/28288186-9960-4f98-8506-95c3dbf53cfd-kube-api-access-8q54g\") pod \"maas-api-e2e-shared-a-w5-801162-695dd8cc6c-hl47x\" (UID: \"28288186-9960-4f98-8506-95c3dbf53cfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-801162-695dd8cc6c-hl47x"
Nov 03 20:03:22.153987 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:22.153943    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28288186-9960-4f98-8506-95c3dbf53cfd-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-801162-695dd8cc6c-hl47x\" (UID: \"28288186-9960-4f98-8506-95c3dbf53cfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-801162-695dd8cc6c-hl47x"
Nov 03 20:03:22.154992 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:22.154962    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28288186-9960-4f98-8506-95c3dbf53cfd-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-801162-695dd8cc6c-hl47x\" (UID: \"28288186-9960-4f98-8506-95c3dbf53cfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-801162-695dd8cc6c-hl47x"
Nov 03 20:03:22.156819 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:22.156794    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28288186-9960-4f98-8506-95c3dbf53cfd-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-801162-695dd8cc6c-hl47x\" (UID: \"28288186-9960-4f98-8506-95c3dbf53cfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-801162-695dd8cc6c-hl47x"
Nov 03 20:03:22.157376 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:22.157195    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28288186-9960-4f98-8506-95c3dbf53cfd-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-801162-695dd8cc6c-hl47x\" (UID: \"28288186-9960-4f98-8506-95c3dbf53cfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-801162-695dd8cc6c-hl47x"
Nov 03 20:03:22.157551 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:22.157526    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q54g\" (UniqueName: \"kubernetes.io/projected/28288186-9960-4f98-8506-95c3dbf53cfd-kube-api-access-8q54g\") pod \"maas-api-e2e-shared-a-w5-801162-695dd8cc6c-hl47x\" (UID: \"28288186-9960-4f98-8506-95c3dbf53cfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-801162-695dd8cc6c-hl47x"
Nov 03 20:03:22.161714 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:22.161293    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28288186-9960-4f98-8506-95c3dbf53cfd-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-801162-695dd8cc6c-hl47x\" (UID: \"28288186-9960-4f98-8506-95c3dbf53cfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-801162-695dd8cc6c-hl47x"
Nov 03 20:03:22.168722 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:22.162532    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28288186-9960-4f98-8506-95c3dbf53cfd-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-801162-695dd8cc6c-hl47x\" (UID: \"28288186-9960-4f98-8506-95c3dbf53cfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-801162-695dd8cc6c-hl47x"
Nov 03 20:03:22.175096 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:22.174780    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q54g\" (UniqueName: \"kubernetes.io/projected/28288186-9960-4f98-8506-95c3dbf53cfd-kube-api-access-8q54g\") pod \"maas-api-e2e-shared-a-w5-801162-695dd8cc6c-hl47x\" (UID: \"28288186-9960-4f98-8506-95c3dbf53cfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-801162-695dd8cc6c-hl47x"
Nov 03 20:03:22.281059 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:22.281022    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-801162-695dd8cc6c-hl47x"
Nov 03 20:03:22.567791 ip-10-0-134-222 kubenswrapper[4921]: W1103 20:03:22.567731    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28288186_9960_4f98_8506_95c3dbf53cfd.slice/crio-31265a8bdb5825f26711f89f15772e3dd891e58e69881352e261370712aa9cde WatchSource:0}: Error finding container 31265a8bdb5825f26711f89f15772e3dd891e58e69881352e261370712aa9cde: Status 404 returned error can't find the container with id 31265a8bdb5825f26711f89f15772e3dd891e58e69881352e261370712aa9cde
Nov 03 20:03:22.568533 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:22.568444    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-801162-695dd8cc6c-hl47x"]
Nov 03 20:03:23.008062 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:23.007951    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-801162-695dd8cc6c-hl47x" event={"ID":"28288186-9960-4f98-8506-95c3dbf53cfd","Type":"ContainerStarted","Data":"31265a8bdb5825f26711f89f15772e3dd891e58e69881352e261370712aa9cde"}
Nov 03 20:03:23.978217 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:23.978188    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dce558-7bcc44d6c-s92vv"
Nov 03 20:03:24.019591 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:24.019531    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-801162-695dd8cc6c-hl47x" event={"ID":"28288186-9960-4f98-8506-95c3dbf53cfd","Type":"ContainerStarted","Data":"40d43bcefc6d10b4f36bb9063d6a6b70877fca16125196b3add451cd63bc5248"}
Nov 03 20:03:24.020385 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:24.020353    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-801162-695dd8cc6c-hl47x"
Nov 03 20:03:24.044600 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:24.044533    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-801162-695dd8cc6c-hl47x" podStartSLOduration=2.606037656 podStartE2EDuration="3.044398219s" totalImagesPullingTime="438.360563ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:24.038243341 +0000 UTC m=+1917.282208806" watchObservedRunningTime="2026-11-03 20:03:24.044398219 +0000 UTC m=+1917.288363685"
Nov 03 20:03:30.035919 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:30.035888    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-801162-695dd8cc6c-hl47x"
Nov 03 20:03:31.177969 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:31.177936    4921 scope.go:122] "RemoveContainer" containerID="76fb700ed57858ed7215c3174791b2cfba78745c09e53fedd4ea62b8e92c7075"
Nov 03 20:03:32.932324 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:03:32.932274    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 20:03:36.223791 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:36.223707    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-11c1a8/test-model-11c1a8-kserve-6c47c9d749-pvxrt"]
Nov 03 20:03:49.048869 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:49.048800    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-f58171/test-model-f58171-kserve-5c596fcddc-vgqtt"]
Nov 03 20:03:50.150850 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:50.150814    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-b30676-5b9668d5b5-xgdqw"]
Nov 03 20:03:50.155357 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:50.155337    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-b30676-5b9668d5b5-xgdqw"
Nov 03 20:03:50.166213 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:50.166165    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-b30676-5b9668d5b5-xgdqw"]
Nov 03 20:03:50.168955 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:50.168927    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w5-b30676\""
Nov 03 20:03:50.169085 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:50.168935    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w5-b30676-dockercfg-qwn75\""
Nov 03 20:03:50.171011 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:50.170988    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-b30676-7bd74677b7-vw8sk"]
Nov 03 20:03:50.203026 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:50.184379    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-b30676-7bd74677b7-vw8sk"
Nov 03 20:03:50.222818 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:50.208616    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c51313a1-0e14-4824-89a8-c715b2f7ba9b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-b30676-5b9668d5b5-xgdqw\" (UID: \"c51313a1-0e14-4824-89a8-c715b2f7ba9b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-b30676-5b9668d5b5-xgdqw"
Nov 03 20:03:50.222818 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:50.208759    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf4gm\" (UniqueName: \"kubernetes.io/projected/b677e48e-572f-4c27-86de-dd30566c8acb-kube-api-access-kf4gm\") pod \"payload-pre-processing-e2e-shared-b-w5-b30676-7bd74677b7-vw8sk\" (UID: \"b677e48e-572f-4c27-86de-dd30566c8acb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-b30676-7bd74677b7-vw8sk"
Nov 03 20:03:50.222818 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:50.208805    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffcjs\" (UniqueName: \"kubernetes.io/projected/c51313a1-0e14-4824-89a8-c715b2f7ba9b-kube-api-access-ffcjs\") pod \"payload-processing-e2e-shared-b-w5-b30676-5b9668d5b5-xgdqw\" (UID: \"c51313a1-0e14-4824-89a8-c715b2f7ba9b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-b30676-5b9668d5b5-xgdqw"
Nov 03 20:03:50.222818 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:50.208868    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b677e48e-572f-4c27-86de-dd30566c8acb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-b30676-7bd74677b7-vw8sk\" (UID: \"b677e48e-572f-4c27-86de-dd30566c8acb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-b30676-7bd74677b7-vw8sk"
Nov 03 20:03:50.230580 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:50.230304    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-b30676-7bd74677b7-vw8sk"]
Nov 03 20:03:50.391697 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:50.385151    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c51313a1-0e14-4824-89a8-c715b2f7ba9b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-b30676-5b9668d5b5-xgdqw\" (UID: \"c51313a1-0e14-4824-89a8-c715b2f7ba9b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-b30676-5b9668d5b5-xgdqw"
Nov 03 20:03:50.391697 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:50.385398    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf4gm\" (UniqueName: \"kubernetes.io/projected/b677e48e-572f-4c27-86de-dd30566c8acb-kube-api-access-kf4gm\") pod \"payload-pre-processing-e2e-shared-b-w5-b30676-7bd74677b7-vw8sk\" (UID: \"b677e48e-572f-4c27-86de-dd30566c8acb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-b30676-7bd74677b7-vw8sk"
Nov 03 20:03:50.391697 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:50.385453    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffcjs\" (UniqueName: \"kubernetes.io/projected/c51313a1-0e14-4824-89a8-c715b2f7ba9b-kube-api-access-ffcjs\") pod \"payload-processing-e2e-shared-b-w5-b30676-5b9668d5b5-xgdqw\" (UID: \"c51313a1-0e14-4824-89a8-c715b2f7ba9b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-b30676-5b9668d5b5-xgdqw"
Nov 03 20:03:50.391697 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:50.385536    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b677e48e-572f-4c27-86de-dd30566c8acb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-b30676-7bd74677b7-vw8sk\" (UID: \"b677e48e-572f-4c27-86de-dd30566c8acb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-b30676-7bd74677b7-vw8sk"
Nov 03 20:03:50.414774 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:50.408205    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c51313a1-0e14-4824-89a8-c715b2f7ba9b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-b30676-5b9668d5b5-xgdqw\" (UID: \"c51313a1-0e14-4824-89a8-c715b2f7ba9b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-b30676-5b9668d5b5-xgdqw"
Nov 03 20:03:50.445960 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:50.429626    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf4gm\" (UniqueName: \"kubernetes.io/projected/b677e48e-572f-4c27-86de-dd30566c8acb-kube-api-access-kf4gm\") pod \"payload-pre-processing-e2e-shared-b-w5-b30676-7bd74677b7-vw8sk\" (UID: \"b677e48e-572f-4c27-86de-dd30566c8acb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-b30676-7bd74677b7-vw8sk"
Nov 03 20:03:50.445960 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:50.438721    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffcjs\" (UniqueName: \"kubernetes.io/projected/c51313a1-0e14-4824-89a8-c715b2f7ba9b-kube-api-access-ffcjs\") pod \"payload-processing-e2e-shared-b-w5-b30676-5b9668d5b5-xgdqw\" (UID: \"c51313a1-0e14-4824-89a8-c715b2f7ba9b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-b30676-5b9668d5b5-xgdqw"
Nov 03 20:03:50.445960 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:50.438788    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b677e48e-572f-4c27-86de-dd30566c8acb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-b30676-7bd74677b7-vw8sk\" (UID: \"b677e48e-572f-4c27-86de-dd30566c8acb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-b30676-7bd74677b7-vw8sk"
Nov 03 20:03:50.473066 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:50.473020    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-b30676-5b9668d5b5-xgdqw"
Nov 03 20:03:50.539975 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:50.539939    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-b30676-7bd74677b7-vw8sk"
Nov 03 20:03:51.000626 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:51.000595    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-b30676-5b9668d5b5-xgdqw"]
Nov 03 20:03:51.004857 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:51.004827    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-b30676-7bd74677b7-vw8sk"]
Nov 03 20:03:51.005835 ip-10-0-134-222 kubenswrapper[4921]: W1103 20:03:51.005807    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb677e48e_572f_4c27_86de_dd30566c8acb.slice/crio-f1b641fabaeee87c050ecfc05cc3b473ab087a419e5773b9e77ff781a0b2c397 WatchSource:0}: Error finding container f1b641fabaeee87c050ecfc05cc3b473ab087a419e5773b9e77ff781a0b2c397: Status 404 returned error can't find the container with id f1b641fabaeee87c050ecfc05cc3b473ab087a419e5773b9e77ff781a0b2c397
Nov 03 20:03:51.248541 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:51.248492    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-11c1a8/test-model-11c1a8-kserve-6c47c9d749-pvxrt" podUID="8f1dd983-8a22-4bfb-8d24-37b1c67c8070" containerName="main" containerID="cri-o://93b985490eaec58f8e32e127ffe404fdd832a68bd3cbdd4d9b2a8b5fc891f713" gracePeriod=45
Nov 03 20:03:51.463686 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:51.460962    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-b30676-7bd74677b7-vw8sk" event={"ID":"b677e48e-572f-4c27-86de-dd30566c8acb","Type":"ContainerStarted","Data":"d38dadd6324eece9309158ff441fd5324ae5df23a8d80cb2eaf17b3fcbc6c7f1"}
Nov 03 20:03:51.463686 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:51.461033    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-b30676-7bd74677b7-vw8sk" event={"ID":"b677e48e-572f-4c27-86de-dd30566c8acb","Type":"ContainerStarted","Data":"f1b641fabaeee87c050ecfc05cc3b473ab087a419e5773b9e77ff781a0b2c397"}
Nov 03 20:03:51.463686 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:51.462200    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-b30676-7bd74677b7-vw8sk"
Nov 03 20:03:51.465919 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:51.465423    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-b30676-5b9668d5b5-xgdqw" event={"ID":"c51313a1-0e14-4824-89a8-c715b2f7ba9b","Type":"ContainerStarted","Data":"cf2e4aa5ef68419a1ca9e8299bb1f69a3644e57a5f54c019bba6bfd38f171b8e"}
Nov 03 20:03:51.465919 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:51.465477    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-b30676-5b9668d5b5-xgdqw" event={"ID":"c51313a1-0e14-4824-89a8-c715b2f7ba9b","Type":"ContainerStarted","Data":"a10dae68691aa4f5b4d23323bf97bba56956da646ff9a2bd348a7ecb1c66c7fd"}
Nov 03 20:03:51.466602 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:51.466530    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-b30676-5b9668d5b5-xgdqw"
Nov 03 20:03:51.468820 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:51.468798    4921 generic.go:356] "Generic (PLEG): container finished" podID="8f1dd983-8a22-4bfb-8d24-37b1c67c8070" containerID="93b985490eaec58f8e32e127ffe404fdd832a68bd3cbdd4d9b2a8b5fc891f713" exitCode=0
Nov 03 20:03:51.469068 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:51.469048    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-11c1a8/test-model-11c1a8-kserve-6c47c9d749-pvxrt" event={"ID":"8f1dd983-8a22-4bfb-8d24-37b1c67c8070","Type":"ContainerDied","Data":"93b985490eaec58f8e32e127ffe404fdd832a68bd3cbdd4d9b2a8b5fc891f713"}
Nov 03 20:03:51.491096 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:51.491038    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-b30676-7bd74677b7-vw8sk" podStartSLOduration=1.4910197539999999 podStartE2EDuration="1.491019754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:51.47890662 +0000 UTC m=+1944.722872085" watchObservedRunningTime="2026-11-03 20:03:51.491019754 +0000 UTC m=+1944.734985238"
Nov 03 20:03:51.501775 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:51.499639    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-b30676-5b9668d5b5-xgdqw" podStartSLOduration=1.499620525 podStartE2EDuration="1.499620525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:51.498082247 +0000 UTC m=+1944.742047712" watchObservedRunningTime="2026-11-03 20:03:51.499620525 +0000 UTC m=+1944.743585994"
Nov 03 20:03:51.559224 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:51.559193    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-11c1a8/test-model-11c1a8-kserve-6c47c9d749-pvxrt"
Nov 03 20:03:51.686982 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:51.686293    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f1dd983-8a22-4bfb-8d24-37b1c67c8070-kube-api-access-7lfpb\" (UniqueName: \"kubernetes.io/projected/8f1dd983-8a22-4bfb-8d24-37b1c67c8070-kube-api-access-7lfpb\") pod \"8f1dd983-8a22-4bfb-8d24-37b1c67c8070\" (UID: \"8f1dd983-8a22-4bfb-8d24-37b1c67c8070\") "
Nov 03 20:03:51.686982 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:51.686451    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f1dd983-8a22-4bfb-8d24-37b1c67c8070-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f1dd983-8a22-4bfb-8d24-37b1c67c8070-tmp-dir\") pod \"8f1dd983-8a22-4bfb-8d24-37b1c67c8070\" (UID: \"8f1dd983-8a22-4bfb-8d24-37b1c67c8070\") "
Nov 03 20:03:51.686982 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:51.686487    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f1dd983-8a22-4bfb-8d24-37b1c67c8070-tls-certs\" (UniqueName: \"kubernetes.io/secret/8f1dd983-8a22-4bfb-8d24-37b1c67c8070-tls-certs\") pod \"8f1dd983-8a22-4bfb-8d24-37b1c67c8070\" (UID: \"8f1dd983-8a22-4bfb-8d24-37b1c67c8070\") "
Nov 03 20:03:51.687424 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:51.687378    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f1dd983-8a22-4bfb-8d24-37b1c67c8070-tmp-dir" pod "8f1dd983-8a22-4bfb-8d24-37b1c67c8070" (UID: "8f1dd983-8a22-4bfb-8d24-37b1c67c8070"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:03:51.688084 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:51.686721    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f1dd983-8a22-4bfb-8d24-37b1c67c8070-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f1dd983-8a22-4bfb-8d24-37b1c67c8070-model-cache\") pod \"8f1dd983-8a22-4bfb-8d24-37b1c67c8070\" (UID: \"8f1dd983-8a22-4bfb-8d24-37b1c67c8070\") "
Nov 03 20:03:51.688084 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:51.687713    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f1dd983-8a22-4bfb-8d24-37b1c67c8070-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f1dd983-8a22-4bfb-8d24-37b1c67c8070-dshm\") pod \"8f1dd983-8a22-4bfb-8d24-37b1c67c8070\" (UID: \"8f1dd983-8a22-4bfb-8d24-37b1c67c8070\") "
Nov 03 20:03:51.688084 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:51.687774    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f1dd983-8a22-4bfb-8d24-37b1c67c8070-home\" (UniqueName: \"kubernetes.io/empty-dir/8f1dd983-8a22-4bfb-8d24-37b1c67c8070-home\") pod \"8f1dd983-8a22-4bfb-8d24-37b1c67c8070\" (UID: \"8f1dd983-8a22-4bfb-8d24-37b1c67c8070\") "
Nov 03 20:03:51.688388 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:51.688335    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f1dd983-8a22-4bfb-8d24-37b1c67c8070-home" pod "8f1dd983-8a22-4bfb-8d24-37b1c67c8070" (UID: "8f1dd983-8a22-4bfb-8d24-37b1c67c8070"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:03:51.689023 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:51.688973    4921 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f1dd983-8a22-4bfb-8d24-37b1c67c8070-home\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:51.689023 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:51.689002    4921 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f1dd983-8a22-4bfb-8d24-37b1c67c8070-tmp-dir\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:51.691942 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:51.691916    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f1dd983-8a22-4bfb-8d24-37b1c67c8070-dshm" pod "8f1dd983-8a22-4bfb-8d24-37b1c67c8070" (UID: "8f1dd983-8a22-4bfb-8d24-37b1c67c8070"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:03:51.692057 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:51.692026    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f1dd983-8a22-4bfb-8d24-37b1c67c8070-tls-certs" pod "8f1dd983-8a22-4bfb-8d24-37b1c67c8070" (UID: "8f1dd983-8a22-4bfb-8d24-37b1c67c8070"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:03:51.692776 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:51.692748    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f1dd983-8a22-4bfb-8d24-37b1c67c8070-model-cache" pod "8f1dd983-8a22-4bfb-8d24-37b1c67c8070" (UID: "8f1dd983-8a22-4bfb-8d24-37b1c67c8070"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:03:51.693464 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:51.693434    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f1dd983-8a22-4bfb-8d24-37b1c67c8070-kube-api-access-7lfpb" pod "8f1dd983-8a22-4bfb-8d24-37b1c67c8070" (UID: "8f1dd983-8a22-4bfb-8d24-37b1c67c8070"). InnerVolumeSpecName "kube-api-access-7lfpb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:03:51.790778 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:51.790693    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lfpb\" (UniqueName: \"kubernetes.io/projected/8f1dd983-8a22-4bfb-8d24-37b1c67c8070-kube-api-access-7lfpb\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:51.790778 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:51.790724    4921 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f1dd983-8a22-4bfb-8d24-37b1c67c8070-tls-certs\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:51.790778 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:51.790735    4921 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f1dd983-8a22-4bfb-8d24-37b1c67c8070-model-cache\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:51.790778 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:51.790745    4921 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f1dd983-8a22-4bfb-8d24-37b1c67c8070-dshm\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:52.498807 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:52.491451    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-11c1a8/test-model-11c1a8-kserve-6c47c9d749-pvxrt" event={"ID":"8f1dd983-8a22-4bfb-8d24-37b1c67c8070","Type":"ContainerDied","Data":"00994291297dcef58d86c0d7521ea5685cc5d4ee8775d9ca8ff2f709ae07475b"}
Nov 03 20:03:52.498807 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:52.491530    4921 scope.go:122] "RemoveContainer" containerID="93b985490eaec58f8e32e127ffe404fdd832a68bd3cbdd4d9b2a8b5fc891f713"
Nov 03 20:03:52.498807 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:52.491819    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-11c1a8/test-model-11c1a8-kserve-6c47c9d749-pvxrt"
Nov 03 20:03:52.526956 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:52.526879    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-11c1a8/test-model-11c1a8-kserve-6c47c9d749-pvxrt"]
Nov 03 20:03:52.531419 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:52.531393    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-11c1a8/test-model-11c1a8-kserve-6c47c9d749-pvxrt"]
Nov 03 20:03:52.897400 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:52.897308    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f1dd983-8a22-4bfb-8d24-37b1c67c8070" path="/var/lib/kubelet/pods/8f1dd983-8a22-4bfb-8d24-37b1c67c8070/volumes"
Nov 03 20:03:59.523795 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:03:59.523762    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-11c1a8-598fbd7d99-fzmxn"]
Nov 03 20:04:02.924391 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:04:02.924346    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 20:04:03.505943 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:03.505915    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-b30676-7bd74677b7-vw8sk"
Nov 03 20:04:03.506125 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:03.506013    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-b30676-5b9668d5b5-xgdqw"
Nov 03 20:04:04.084484 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:04.084443    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-f58171/test-model-f58171-kserve-5c596fcddc-vgqtt" podUID="e839d396-3136-41e8-b900-1194d9067c88" containerName="main" containerID="cri-o://626bf069fede1710a5d55ceaf3ef690c8fcf23559dbbc99bdebf661bc85dea19" gracePeriod=45
Nov 03 20:04:04.384696 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:04.380516    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f58171/test-model-f58171-kserve-5c596fcddc-vgqtt"
Nov 03 20:04:04.513589 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:04.513553    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e839d396-3136-41e8-b900-1194d9067c88-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e839d396-3136-41e8-b900-1194d9067c88-tmp-dir\") pod \"e839d396-3136-41e8-b900-1194d9067c88\" (UID: \"e839d396-3136-41e8-b900-1194d9067c88\") "
Nov 03 20:04:04.513794 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:04.513661    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e839d396-3136-41e8-b900-1194d9067c88-kube-api-access-lct7x\" (UniqueName: \"kubernetes.io/projected/e839d396-3136-41e8-b900-1194d9067c88-kube-api-access-lct7x\") pod \"e839d396-3136-41e8-b900-1194d9067c88\" (UID: \"e839d396-3136-41e8-b900-1194d9067c88\") "
Nov 03 20:04:04.513794 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:04.513752    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e839d396-3136-41e8-b900-1194d9067c88-tls-certs\" (UniqueName: \"kubernetes.io/secret/e839d396-3136-41e8-b900-1194d9067c88-tls-certs\") pod \"e839d396-3136-41e8-b900-1194d9067c88\" (UID: \"e839d396-3136-41e8-b900-1194d9067c88\") "
Nov 03 20:04:04.513898 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:04.513866    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e839d396-3136-41e8-b900-1194d9067c88-tmp-dir" pod "e839d396-3136-41e8-b900-1194d9067c88" (UID: "e839d396-3136-41e8-b900-1194d9067c88"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:04:04.513945 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:04.513928    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e839d396-3136-41e8-b900-1194d9067c88-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e839d396-3136-41e8-b900-1194d9067c88-model-cache\") pod \"e839d396-3136-41e8-b900-1194d9067c88\" (UID: \"e839d396-3136-41e8-b900-1194d9067c88\") "
Nov 03 20:04:04.513985 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:04.513972    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e839d396-3136-41e8-b900-1194d9067c88-home\" (UniqueName: \"kubernetes.io/empty-dir/e839d396-3136-41e8-b900-1194d9067c88-home\") pod \"e839d396-3136-41e8-b900-1194d9067c88\" (UID: \"e839d396-3136-41e8-b900-1194d9067c88\") "
Nov 03 20:04:04.514033 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:04.514011    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e839d396-3136-41e8-b900-1194d9067c88-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e839d396-3136-41e8-b900-1194d9067c88-dshm\") pod \"e839d396-3136-41e8-b900-1194d9067c88\" (UID: \"e839d396-3136-41e8-b900-1194d9067c88\") "
Nov 03 20:04:04.514241 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:04.514181    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e839d396-3136-41e8-b900-1194d9067c88-model-cache" pod "e839d396-3136-41e8-b900-1194d9067c88" (UID: "e839d396-3136-41e8-b900-1194d9067c88"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:04:04.514241 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:04.514219    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e839d396-3136-41e8-b900-1194d9067c88-home" pod "e839d396-3136-41e8-b900-1194d9067c88" (UID: "e839d396-3136-41e8-b900-1194d9067c88"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:04:04.514720 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:04.514659    4921 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e839d396-3136-41e8-b900-1194d9067c88-model-cache\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:04.514807 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:04.514728    4921 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e839d396-3136-41e8-b900-1194d9067c88-home\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:04.514807 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:04.514746    4921 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e839d396-3136-41e8-b900-1194d9067c88-tmp-dir\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:04.516207 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:04.516182    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e839d396-3136-41e8-b900-1194d9067c88-dshm" pod "e839d396-3136-41e8-b900-1194d9067c88" (UID: "e839d396-3136-41e8-b900-1194d9067c88"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:04:04.516383 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:04.516363    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e839d396-3136-41e8-b900-1194d9067c88-tls-certs" pod "e839d396-3136-41e8-b900-1194d9067c88" (UID: "e839d396-3136-41e8-b900-1194d9067c88"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:04:04.516383 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:04.516374    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e839d396-3136-41e8-b900-1194d9067c88-kube-api-access-lct7x" pod "e839d396-3136-41e8-b900-1194d9067c88" (UID: "e839d396-3136-41e8-b900-1194d9067c88"). InnerVolumeSpecName "kube-api-access-lct7x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:04:04.524423 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:04.524395    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-11c1a8-598fbd7d99-fzmxn" podUID="c76485a0-b175-405c-8b6e-ff94c6b9ae1f" containerName="payload-processing" containerID="cri-o://6eb4de70602cb3c2e9acfddb47ea90fa306e2aa5113764982b51720f862f44f7" gracePeriod=25
Nov 03 20:04:04.608608 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:04.608527    4921 generic.go:356] "Generic (PLEG): container finished" podID="e839d396-3136-41e8-b900-1194d9067c88" containerID="626bf069fede1710a5d55ceaf3ef690c8fcf23559dbbc99bdebf661bc85dea19" exitCode=0
Nov 03 20:04:04.608608 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:04.608576    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f58171/test-model-f58171-kserve-5c596fcddc-vgqtt" event={"ID":"e839d396-3136-41e8-b900-1194d9067c88","Type":"ContainerDied","Data":"626bf069fede1710a5d55ceaf3ef690c8fcf23559dbbc99bdebf661bc85dea19"}
Nov 03 20:04:04.608608 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:04.608585    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f58171/test-model-f58171-kserve-5c596fcddc-vgqtt"
Nov 03 20:04:04.608854 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:04.608607    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f58171/test-model-f58171-kserve-5c596fcddc-vgqtt" event={"ID":"e839d396-3136-41e8-b900-1194d9067c88","Type":"ContainerDied","Data":"09dafb51f19ed55324915456a0e0499bdb2d4b78db8aa7e186009ad798711d61"}
Nov 03 20:04:04.608854 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:04.608627    4921 scope.go:122] "RemoveContainer" containerID="626bf069fede1710a5d55ceaf3ef690c8fcf23559dbbc99bdebf661bc85dea19"
Nov 03 20:04:04.615844 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:04.615812    4921 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e839d396-3136-41e8-b900-1194d9067c88-tls-certs\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:04.615995 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:04.615847    4921 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e839d396-3136-41e8-b900-1194d9067c88-dshm\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:04.615995 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:04.615864    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lct7x\" (UniqueName: \"kubernetes.io/projected/e839d396-3136-41e8-b900-1194d9067c88-kube-api-access-lct7x\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:04.631217 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:04.630509    4921 scope.go:122] "RemoveContainer" containerID="626bf069fede1710a5d55ceaf3ef690c8fcf23559dbbc99bdebf661bc85dea19"
Nov 03 20:04:04.631217 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:04:04.631029    4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"626bf069fede1710a5d55ceaf3ef690c8fcf23559dbbc99bdebf661bc85dea19\": container with ID starting with 626bf069fede1710a5d55ceaf3ef690c8fcf23559dbbc99bdebf661bc85dea19 not found: ID does not exist" containerID="626bf069fede1710a5d55ceaf3ef690c8fcf23559dbbc99bdebf661bc85dea19"
Nov 03 20:04:04.631217 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:04.631068    4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"626bf069fede1710a5d55ceaf3ef690c8fcf23559dbbc99bdebf661bc85dea19"} err="failed to get container status \"626bf069fede1710a5d55ceaf3ef690c8fcf23559dbbc99bdebf661bc85dea19\": rpc error: code = NotFound desc = could not find container \"626bf069fede1710a5d55ceaf3ef690c8fcf23559dbbc99bdebf661bc85dea19\": container with ID starting with 626bf069fede1710a5d55ceaf3ef690c8fcf23559dbbc99bdebf661bc85dea19 not found: ID does not exist"
Nov 03 20:04:04.640585 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:04.640534    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-f58171/test-model-f58171-kserve-5c596fcddc-vgqtt"]
Nov 03 20:04:04.644375 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:04.644345    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-f58171/test-model-f58171-kserve-5c596fcddc-vgqtt"]
Nov 03 20:04:04.893126 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:04.893043    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e839d396-3136-41e8-b900-1194d9067c88" path="/var/lib/kubelet/pods/e839d396-3136-41e8-b900-1194d9067c88/volumes"
Nov 03 20:04:05.038936 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:05.038896    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766"]
Nov 03 20:04:05.039487 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:05.039160    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766" podUID="7ad21605-a58c-494c-89e0-6aca910b76b1" containerName="istio-proxy" containerID="cri-o://2ce740bd67ada8c9acf3c3b091c62171649d15c1f554157cdc8dd661c65a3926" gracePeriod=30
Nov 03 20:04:05.326214 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:05.326165    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-f58171/test-model-f58171-kserve-5c596fcddc-vgqtt" podUID="e839d396-3136-41e8-b900-1194d9067c88" containerName="main" probeResult="failure" output="Get \"https://10.132.0.157:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:04:10.367899 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:10.367872    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766"
Nov 03 20:04:10.525881 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:10.525849    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ad21605-a58c-494c-89e0-6aca910b76b1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ad21605-a58c-494c-89e0-6aca910b76b1-workload-socket\") pod \"7ad21605-a58c-494c-89e0-6aca910b76b1\" (UID: \"7ad21605-a58c-494c-89e0-6aca910b76b1\") "
Nov 03 20:04:10.526052 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:10.525904    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ad21605-a58c-494c-89e0-6aca910b76b1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ad21605-a58c-494c-89e0-6aca910b76b1-istio-envoy\") pod \"7ad21605-a58c-494c-89e0-6aca910b76b1\" (UID: \"7ad21605-a58c-494c-89e0-6aca910b76b1\") "
Nov 03 20:04:10.526052 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:10.525927    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7ad21605-a58c-494c-89e0-6aca910b76b1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ad21605-a58c-494c-89e0-6aca910b76b1-istio-podinfo\") pod \"7ad21605-a58c-494c-89e0-6aca910b76b1\" (UID: \"7ad21605-a58c-494c-89e0-6aca910b76b1\") "
Nov 03 20:04:10.526052 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:10.525958    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ad21605-a58c-494c-89e0-6aca910b76b1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ad21605-a58c-494c-89e0-6aca910b76b1-istio-data\") pod \"7ad21605-a58c-494c-89e0-6aca910b76b1\" (UID: \"7ad21605-a58c-494c-89e0-6aca910b76b1\") "
Nov 03 20:04:10.526052 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:10.526007    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ad21605-a58c-494c-89e0-6aca910b76b1-istio-token\" (UniqueName: \"kubernetes.io/projected/7ad21605-a58c-494c-89e0-6aca910b76b1-istio-token\") pod \"7ad21605-a58c-494c-89e0-6aca910b76b1\" (UID: \"7ad21605-a58c-494c-89e0-6aca910b76b1\") "
Nov 03 20:04:10.526052 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:10.526041    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ad21605-a58c-494c-89e0-6aca910b76b1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ad21605-a58c-494c-89e0-6aca910b76b1-credential-socket\") pod \"7ad21605-a58c-494c-89e0-6aca910b76b1\" (UID: \"7ad21605-a58c-494c-89e0-6aca910b76b1\") "
Nov 03 20:04:10.526361 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:10.526123    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ad21605-a58c-494c-89e0-6aca910b76b1-workload-socket" pod "7ad21605-a58c-494c-89e0-6aca910b76b1" (UID: "7ad21605-a58c-494c-89e0-6aca910b76b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:04:10.526361 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:10.526185    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ad21605-a58c-494c-89e0-6aca910b76b1-kube-api-access-fdkz7\" (UniqueName: \"kubernetes.io/projected/7ad21605-a58c-494c-89e0-6aca910b76b1-kube-api-access-fdkz7\") pod \"7ad21605-a58c-494c-89e0-6aca910b76b1\" (UID: \"7ad21605-a58c-494c-89e0-6aca910b76b1\") "
Nov 03 20:04:10.526361 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:10.526225    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ad21605-a58c-494c-89e0-6aca910b76b1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ad21605-a58c-494c-89e0-6aca910b76b1-istiod-ca-cert\") pod \"7ad21605-a58c-494c-89e0-6aca910b76b1\" (UID: \"7ad21605-a58c-494c-89e0-6aca910b76b1\") "
Nov 03 20:04:10.526361 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:10.526257    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ad21605-a58c-494c-89e0-6aca910b76b1-istio-data" pod "7ad21605-a58c-494c-89e0-6aca910b76b1" (UID: "7ad21605-a58c-494c-89e0-6aca910b76b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:04:10.526361 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:10.526269    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ad21605-a58c-494c-89e0-6aca910b76b1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ad21605-a58c-494c-89e0-6aca910b76b1-workload-certs\") pod \"7ad21605-a58c-494c-89e0-6aca910b76b1\" (UID: \"7ad21605-a58c-494c-89e0-6aca910b76b1\") "
Nov 03 20:04:10.526642 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:10.526394    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ad21605-a58c-494c-89e0-6aca910b76b1-credential-socket" pod "7ad21605-a58c-494c-89e0-6aca910b76b1" (UID: "7ad21605-a58c-494c-89e0-6aca910b76b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:04:10.526751 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:10.526693    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ad21605-a58c-494c-89e0-6aca910b76b1-workload-certs" pod "7ad21605-a58c-494c-89e0-6aca910b76b1" (UID: "7ad21605-a58c-494c-89e0-6aca910b76b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:04:10.526751 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:10.526609    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ad21605-a58c-494c-89e0-6aca910b76b1-istiod-ca-cert" pod "7ad21605-a58c-494c-89e0-6aca910b76b1" (UID: "7ad21605-a58c-494c-89e0-6aca910b76b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:04:10.526979 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:10.526953    4921 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ad21605-a58c-494c-89e0-6aca910b76b1-istiod-ca-cert\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:10.526979 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:10.526979    4921 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ad21605-a58c-494c-89e0-6aca910b76b1-workload-certs\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:10.527167 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:10.526990    4921 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ad21605-a58c-494c-89e0-6aca910b76b1-workload-socket\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:10.527167 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:10.526999    4921 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ad21605-a58c-494c-89e0-6aca910b76b1-istio-data\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:10.527167 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:10.527007    4921 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ad21605-a58c-494c-89e0-6aca910b76b1-credential-socket\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:10.528418 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:10.528393    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ad21605-a58c-494c-89e0-6aca910b76b1-istio-token" pod "7ad21605-a58c-494c-89e0-6aca910b76b1" (UID: "7ad21605-a58c-494c-89e0-6aca910b76b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:04:10.528519 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:10.528497    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ad21605-a58c-494c-89e0-6aca910b76b1-kube-api-access-fdkz7" pod "7ad21605-a58c-494c-89e0-6aca910b76b1" (UID: "7ad21605-a58c-494c-89e0-6aca910b76b1"). InnerVolumeSpecName "kube-api-access-fdkz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:04:10.528697 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:10.528652    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ad21605-a58c-494c-89e0-6aca910b76b1-istio-envoy" pod "7ad21605-a58c-494c-89e0-6aca910b76b1" (UID: "7ad21605-a58c-494c-89e0-6aca910b76b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:04:10.528798 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:10.528774    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ad21605-a58c-494c-89e0-6aca910b76b1-istio-podinfo" pod "7ad21605-a58c-494c-89e0-6aca910b76b1" (UID: "7ad21605-a58c-494c-89e0-6aca910b76b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:04:10.628273 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:10.628191    4921 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ad21605-a58c-494c-89e0-6aca910b76b1-istio-token\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:10.628273 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:10.628223    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdkz7\" (UniqueName: \"kubernetes.io/projected/7ad21605-a58c-494c-89e0-6aca910b76b1-kube-api-access-fdkz7\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:10.628273 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:10.628233    4921 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ad21605-a58c-494c-89e0-6aca910b76b1-istio-envoy\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:10.628273 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:10.628241    4921 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ad21605-a58c-494c-89e0-6aca910b76b1-istio-podinfo\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:10.702062 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:10.702027    4921 generic.go:356] "Generic (PLEG): container finished" podID="7ad21605-a58c-494c-89e0-6aca910b76b1" containerID="2ce740bd67ada8c9acf3c3b091c62171649d15c1f554157cdc8dd661c65a3926" exitCode=0
Nov 03 20:04:10.702263 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:10.702088    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766"
Nov 03 20:04:10.702263 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:10.702111    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766" event={"ID":"7ad21605-a58c-494c-89e0-6aca910b76b1","Type":"ContainerDied","Data":"2ce740bd67ada8c9acf3c3b091c62171649d15c1f554157cdc8dd661c65a3926"}
Nov 03 20:04:10.702263 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:10.702168    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766" event={"ID":"7ad21605-a58c-494c-89e0-6aca910b76b1","Type":"ContainerDied","Data":"d912cf58fb150f6c9c67b773adf8e8d4f97477658a1f9e45a6a1fae87cd8cd55"}
Nov 03 20:04:10.702263 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:10.702190    4921 scope.go:122] "RemoveContainer" containerID="2ce740bd67ada8c9acf3c3b091c62171649d15c1f554157cdc8dd661c65a3926"
Nov 03 20:04:10.715260 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:10.715241    4921 scope.go:122] "RemoveContainer" containerID="2ce740bd67ada8c9acf3c3b091c62171649d15c1f554157cdc8dd661c65a3926"
Nov 03 20:04:10.715544 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:04:10.715526    4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ce740bd67ada8c9acf3c3b091c62171649d15c1f554157cdc8dd661c65a3926\": container with ID starting with 2ce740bd67ada8c9acf3c3b091c62171649d15c1f554157cdc8dd661c65a3926 not found: ID does not exist" containerID="2ce740bd67ada8c9acf3c3b091c62171649d15c1f554157cdc8dd661c65a3926"
Nov 03 20:04:10.715598 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:10.715553    4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ce740bd67ada8c9acf3c3b091c62171649d15c1f554157cdc8dd661c65a3926"} err="failed to get container status \"2ce740bd67ada8c9acf3c3b091c62171649d15c1f554157cdc8dd661c65a3926\": rpc error: code = NotFound desc = could not find container \"2ce740bd67ada8c9acf3c3b091c62171649d15c1f554157cdc8dd661c65a3926\": container with ID starting with 2ce740bd67ada8c9acf3c3b091c62171649d15c1f554157cdc8dd661c65a3926 not found: ID does not exist"
Nov 03 20:04:10.725824 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:10.725800    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766"]
Nov 03 20:04:10.729981 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:10.729960    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-5j766"]
Nov 03 20:04:10.890545 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:10.890463    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ad21605-a58c-494c-89e0-6aca910b76b1" path="/var/lib/kubelet/pods/7ad21605-a58c-494c-89e0-6aca910b76b1/volumes"
Nov 03 20:04:10.917752 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:10.917719    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv"]
Nov 03 20:04:10.918504 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:10.918473    4921 cpu_manager.go:414] "containerMap: removing container" podUID="8f1dd983-8a22-4bfb-8d24-37b1c67c8070" containerName="main"
Nov 03 20:04:10.918504 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:10.918499    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f1dd983-8a22-4bfb-8d24-37b1c67c8070" containerName="main"
Nov 03 20:04:10.918724 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:10.918539    4921 cpu_manager.go:414] "containerMap: removing container" podUID="e839d396-3136-41e8-b900-1194d9067c88" containerName="main"
Nov 03 20:04:10.918724 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:10.918547    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e839d396-3136-41e8-b900-1194d9067c88" containerName="main"
Nov 03 20:04:10.918724 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:10.918576    4921 cpu_manager.go:414] "containerMap: removing container" podUID="7ad21605-a58c-494c-89e0-6aca910b76b1" containerName="istio-proxy"
Nov 03 20:04:10.918724 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:10.918585    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ad21605-a58c-494c-89e0-6aca910b76b1" containerName="istio-proxy"
Nov 03 20:04:10.918856 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:10.918783    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f1dd983-8a22-4bfb-8d24-37b1c67c8070" containerName="main"
Nov 03 20:04:10.918856 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:10.918795    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="e839d396-3136-41e8-b900-1194d9067c88" containerName="main"
Nov 03 20:04:10.918856 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:10.918805    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ad21605-a58c-494c-89e0-6aca910b76b1" containerName="istio-proxy"
Nov 03 20:04:10.924358 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:10.924338    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv"
Nov 03 20:04:10.932851 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:10.932826    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6"]
Nov 03 20:04:10.942177 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:10.942131    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6"
Nov 03 20:04:10.942319 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:10.942211    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv"]
Nov 03 20:04:10.968848 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:10.968804    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6"]
Nov 03 20:04:11.034198 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:11.034165    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb55b062-fcb5-4fac-8089-09ad9549b7f9-istio-envoy\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv\" (UID: \"bb55b062-fcb5-4fac-8089-09ad9549b7f9\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv"
Nov 03 20:04:11.034367 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:11.034257    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb55b062-fcb5-4fac-8089-09ad9549b7f9-istiod-ca-cert\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv\" (UID: \"bb55b062-fcb5-4fac-8089-09ad9549b7f9\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv"
Nov 03 20:04:11.034518 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:11.034499    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb55b062-fcb5-4fac-8089-09ad9549b7f9-istio-token\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv\" (UID: \"bb55b062-fcb5-4fac-8089-09ad9549b7f9\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv"
Nov 03 20:04:11.034651 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:11.034636    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb55b062-fcb5-4fac-8089-09ad9549b7f9-istio-podinfo\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv\" (UID: \"bb55b062-fcb5-4fac-8089-09ad9549b7f9\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv"
Nov 03 20:04:11.034708 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:11.034662    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb55b062-fcb5-4fac-8089-09ad9549b7f9-workload-socket\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv\" (UID: \"bb55b062-fcb5-4fac-8089-09ad9549b7f9\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv"
Nov 03 20:04:11.034767 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:11.034746    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzcb4\" (UniqueName: \"kubernetes.io/projected/bb55b062-fcb5-4fac-8089-09ad9549b7f9-kube-api-access-mzcb4\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv\" (UID: \"bb55b062-fcb5-4fac-8089-09ad9549b7f9\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv"
Nov 03 20:04:11.034834 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:11.034819    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb55b062-fcb5-4fac-8089-09ad9549b7f9-credential-socket\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv\" (UID: \"bb55b062-fcb5-4fac-8089-09ad9549b7f9\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv"
Nov 03 20:04:11.034876 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:11.034849    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb55b062-fcb5-4fac-8089-09ad9549b7f9-workload-certs\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv\" (UID: \"bb55b062-fcb5-4fac-8089-09ad9549b7f9\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv"
Nov 03 20:04:11.034907 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:11.034888    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb55b062-fcb5-4fac-8089-09ad9549b7f9-istio-data\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv\" (UID: \"bb55b062-fcb5-4fac-8089-09ad9549b7f9\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv"
Nov 03 20:04:11.135957 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:11.135917    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb55b062-fcb5-4fac-8089-09ad9549b7f9-istiod-ca-cert\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv\" (UID: \"bb55b062-fcb5-4fac-8089-09ad9549b7f9\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv"
Nov 03 20:04:11.135957 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:11.135961    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40248459-fffd-4588-94fd-5569ee6a9e8e-workload-socket\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6\" (UID: \"40248459-fffd-4588-94fd-5569ee6a9e8e\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6"
Nov 03 20:04:11.136164 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:11.136078    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40248459-fffd-4588-94fd-5569ee6a9e8e-workload-certs\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6\" (UID: \"40248459-fffd-4588-94fd-5569ee6a9e8e\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6"
Nov 03 20:04:11.136164 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:11.136132    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40248459-fffd-4588-94fd-5569ee6a9e8e-istio-token\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6\" (UID: \"40248459-fffd-4588-94fd-5569ee6a9e8e\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6"
Nov 03 20:04:11.136236 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:11.136205    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb55b062-fcb5-4fac-8089-09ad9549b7f9-istio-token\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv\" (UID: \"bb55b062-fcb5-4fac-8089-09ad9549b7f9\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv"
Nov 03 20:04:11.136236 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:11.136230    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngnbt\" (UniqueName: \"kubernetes.io/projected/40248459-fffd-4588-94fd-5569ee6a9e8e-kube-api-access-ngnbt\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6\" (UID: \"40248459-fffd-4588-94fd-5569ee6a9e8e\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6"
Nov 03 20:04:11.136309 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:11.136269    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40248459-fffd-4588-94fd-5569ee6a9e8e-credential-socket\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6\" (UID: \"40248459-fffd-4588-94fd-5569ee6a9e8e\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6"
Nov 03 20:04:11.136347 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:11.136323    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb55b062-fcb5-4fac-8089-09ad9549b7f9-istio-podinfo\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv\" (UID: \"bb55b062-fcb5-4fac-8089-09ad9549b7f9\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv"
Nov 03 20:04:11.136379 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:11.136345    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb55b062-fcb5-4fac-8089-09ad9549b7f9-workload-socket\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv\" (UID: \"bb55b062-fcb5-4fac-8089-09ad9549b7f9\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv"
Nov 03 20:04:11.136413 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:11.136392    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzcb4\" (UniqueName: \"kubernetes.io/projected/bb55b062-fcb5-4fac-8089-09ad9549b7f9-kube-api-access-mzcb4\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv\" (UID: \"bb55b062-fcb5-4fac-8089-09ad9549b7f9\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv"
Nov 03 20:04:11.136463 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:11.136447    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40248459-fffd-4588-94fd-5569ee6a9e8e-istio-data\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6\" (UID: \"40248459-fffd-4588-94fd-5569ee6a9e8e\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6"
Nov 03 20:04:11.136618 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:11.136596    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb55b062-fcb5-4fac-8089-09ad9549b7f9-credential-socket\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv\" (UID: \"bb55b062-fcb5-4fac-8089-09ad9549b7f9\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv"
Nov 03 20:04:11.136732 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:11.136645    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb55b062-fcb5-4fac-8089-09ad9549b7f9-workload-certs\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv\" (UID: \"bb55b062-fcb5-4fac-8089-09ad9549b7f9\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv"
Nov 03 20:04:11.136732 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:11.136705    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb55b062-fcb5-4fac-8089-09ad9549b7f9-istiod-ca-cert\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv\" (UID: \"bb55b062-fcb5-4fac-8089-09ad9549b7f9\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv"
Nov 03 20:04:11.136732 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:11.136720    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb55b062-fcb5-4fac-8089-09ad9549b7f9-istio-data\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv\" (UID: \"bb55b062-fcb5-4fac-8089-09ad9549b7f9\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv"
Nov 03 20:04:11.136923 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:11.136757    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40248459-fffd-4588-94fd-5569ee6a9e8e-istio-envoy\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6\" (UID: \"40248459-fffd-4588-94fd-5569ee6a9e8e\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6"
Nov 03 20:04:11.136923 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:11.136845    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb55b062-fcb5-4fac-8089-09ad9549b7f9-workload-socket\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv\" (UID: \"bb55b062-fcb5-4fac-8089-09ad9549b7f9\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv"
Nov 03 20:04:11.136923 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:11.136880    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40248459-fffd-4588-94fd-5569ee6a9e8e-istio-podinfo\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6\" (UID: \"40248459-fffd-4588-94fd-5569ee6a9e8e\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6"
Nov 03 20:04:11.137091 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:11.136942    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40248459-fffd-4588-94fd-5569ee6a9e8e-istiod-ca-cert\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6\" (UID: \"40248459-fffd-4588-94fd-5569ee6a9e8e\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6"
Nov 03 20:04:11.137091 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:11.137004    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb55b062-fcb5-4fac-8089-09ad9549b7f9-istio-envoy\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv\" (UID: \"bb55b062-fcb5-4fac-8089-09ad9549b7f9\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv"
Nov 03 20:04:11.137091 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:11.137025    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb55b062-fcb5-4fac-8089-09ad9549b7f9-credential-socket\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv\" (UID: \"bb55b062-fcb5-4fac-8089-09ad9549b7f9\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv"
Nov 03 20:04:11.137091 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:11.137042    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb55b062-fcb5-4fac-8089-09ad9549b7f9-workload-certs\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv\" (UID: \"bb55b062-fcb5-4fac-8089-09ad9549b7f9\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv"
Nov 03 20:04:11.137241 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:11.137154    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb55b062-fcb5-4fac-8089-09ad9549b7f9-istio-data\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv\" (UID: \"bb55b062-fcb5-4fac-8089-09ad9549b7f9\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv"
Nov 03 20:04:11.138950 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:11.138929    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb55b062-fcb5-4fac-8089-09ad9549b7f9-istio-podinfo\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv\" (UID: \"bb55b062-fcb5-4fac-8089-09ad9549b7f9\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv"
Nov 03 20:04:11.139073 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:11.139056    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb55b062-fcb5-4fac-8089-09ad9549b7f9-istio-envoy\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv\" (UID: \"bb55b062-fcb5-4fac-8089-09ad9549b7f9\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv"
Nov 03 20:04:11.146044 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:11.145991    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb55b062-fcb5-4fac-8089-09ad9549b7f9-istio-token\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv\" (UID: \"bb55b062-fcb5-4fac-8089-09ad9549b7f9\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv"
Nov 03 20:04:11.146639 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:11.146622    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzcb4\" (UniqueName: \"kubernetes.io/projected/bb55b062-fcb5-4fac-8089-09ad9549b7f9-kube-api-access-mzcb4\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv\" (UID: \"bb55b062-fcb5-4fac-8089-09ad9549b7f9\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv"
Nov 03 20:04:11.238897 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:11.238862    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40248459-fffd-4588-94fd-5569ee6a9e8e-istio-data\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6\" (UID: \"40248459-fffd-4588-94fd-5569ee6a9e8e\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6"
Nov 03 20:04:11.239044 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:11.238921    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40248459-fffd-4588-94fd-5569ee6a9e8e-istio-envoy\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6\" (UID: \"40248459-fffd-4588-94fd-5569ee6a9e8e\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6"
Nov 03 20:04:11.239044 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:11.239004    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40248459-fffd-4588-94fd-5569ee6a9e8e-istio-podinfo\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6\" (UID: \"40248459-fffd-4588-94fd-5569ee6a9e8e\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6"
Nov 03 20:04:11.239044 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:11.239027    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40248459-fffd-4588-94fd-5569ee6a9e8e-istiod-ca-cert\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6\" (UID: \"40248459-fffd-4588-94fd-5569ee6a9e8e\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6"
Nov 03 20:04:11.239199 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:11.239116    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40248459-fffd-4588-94fd-5569ee6a9e8e-workload-socket\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6\" (UID: \"40248459-fffd-4588-94fd-5569ee6a9e8e\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6"
Nov 03 20:04:11.239282 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:11.239253    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40248459-fffd-4588-94fd-5569ee6a9e8e-workload-certs\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6\" (UID: \"40248459-fffd-4588-94fd-5569ee6a9e8e\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6"
Nov 03 20:04:11.239339 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:11.239329    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40248459-fffd-4588-94fd-5569ee6a9e8e-istio-token\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6\" (UID: \"40248459-fffd-4588-94fd-5569ee6a9e8e\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6"
Nov 03 20:04:11.239405 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:11.239384    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngnbt\" (UniqueName: \"kubernetes.io/projected/40248459-fffd-4588-94fd-5569ee6a9e8e-kube-api-access-ngnbt\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6\" (UID: \"40248459-fffd-4588-94fd-5569ee6a9e8e\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6"
Nov 03 20:04:11.239468 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:11.239418    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40248459-fffd-4588-94fd-5569ee6a9e8e-istio-data\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6\" (UID: \"40248459-fffd-4588-94fd-5569ee6a9e8e\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6"
Nov 03 20:04:11.239468 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:11.239438    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40248459-fffd-4588-94fd-5569ee6a9e8e-credential-socket\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6\" (UID: \"40248459-fffd-4588-94fd-5569ee6a9e8e\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6"
Nov 03 20:04:11.239575 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:11.239466    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40248459-fffd-4588-94fd-5569ee6a9e8e-workload-socket\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6\" (UID: \"40248459-fffd-4588-94fd-5569ee6a9e8e\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6"
Nov 03 20:04:11.239635 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:11.239602    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40248459-fffd-4588-94fd-5569ee6a9e8e-workload-certs\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6\" (UID: \"40248459-fffd-4588-94fd-5569ee6a9e8e\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6"
Nov 03 20:04:11.239915 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:11.239896    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40248459-fffd-4588-94fd-5569ee6a9e8e-credential-socket\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6\" (UID: \"40248459-fffd-4588-94fd-5569ee6a9e8e\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6"
Nov 03 20:04:11.239915 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:11.239903    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40248459-fffd-4588-94fd-5569ee6a9e8e-istiod-ca-cert\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6\" (UID: \"40248459-fffd-4588-94fd-5569ee6a9e8e\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6"
Nov 03 20:04:11.241821 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:11.241800    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40248459-fffd-4588-94fd-5569ee6a9e8e-istio-envoy\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6\" (UID: \"40248459-fffd-4588-94fd-5569ee6a9e8e\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6"
Nov 03 20:04:11.241975 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:11.241958    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40248459-fffd-4588-94fd-5569ee6a9e8e-istio-podinfo\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6\" (UID: \"40248459-fffd-4588-94fd-5569ee6a9e8e\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6"
Nov 03 20:04:11.245962 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:11.245942    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv"
Nov 03 20:04:11.246747 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:11.246727    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40248459-fffd-4588-94fd-5569ee6a9e8e-istio-token\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6\" (UID: \"40248459-fffd-4588-94fd-5569ee6a9e8e\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6"
Nov 03 20:04:11.246907 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:11.246887    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngnbt\" (UniqueName: \"kubernetes.io/projected/40248459-fffd-4588-94fd-5569ee6a9e8e-kube-api-access-ngnbt\") pod \"e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6\" (UID: \"40248459-fffd-4588-94fd-5569ee6a9e8e\") " pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6"
Nov 03 20:04:11.267573 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:11.267544    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6"
Nov 03 20:04:11.535827 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:11.535796    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv"]
Nov 03 20:04:11.536329 ip-10-0-134-222 kubenswrapper[4921]: W1103 20:04:11.536302    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb55b062_fcb5_4fac_8089_09ad9549b7f9.slice/crio-4e940f73c1cda0c4ca2b4055bc9e3ef6dc80ac7248aea13c6f67c6a99eda8bee WatchSource:0}: Error finding container 4e940f73c1cda0c4ca2b4055bc9e3ef6dc80ac7248aea13c6f67c6a99eda8bee: Status 404 returned error can't find the container with id 4e940f73c1cda0c4ca2b4055bc9e3ef6dc80ac7248aea13c6f67c6a99eda8bee
Nov 03 20:04:11.539246 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:11.539201    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:04:11.539303 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:11.539282    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:04:11.539334 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:11.539312    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:04:11.580387 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:11.580357    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6"]
Nov 03 20:04:11.581517 ip-10-0-134-222 kubenswrapper[4921]: W1103 20:04:11.581428    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40248459_fffd_4588_94fd_5569ee6a9e8e.slice/crio-7f04b538793d12db4b2d782ed7fea4e7563f3e5935ba2c81ddace89c5d76691f WatchSource:0}: Error finding container 7f04b538793d12db4b2d782ed7fea4e7563f3e5935ba2c81ddace89c5d76691f: Status 404 returned error can't find the container with id 7f04b538793d12db4b2d782ed7fea4e7563f3e5935ba2c81ddace89c5d76691f
Nov 03 20:04:11.586450 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:11.585414    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:04:11.586450 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:11.585510    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:04:11.586450 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:11.585552    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:04:11.719645 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:11.719606    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6" event={"ID":"40248459-fffd-4588-94fd-5569ee6a9e8e","Type":"ContainerStarted","Data":"007b593652233cb3dd86e355118fe7d9cdba4012b9d854780a3a58eb9eb00189"}
Nov 03 20:04:11.719937 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:11.719711    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6" event={"ID":"40248459-fffd-4588-94fd-5569ee6a9e8e","Type":"ContainerStarted","Data":"7f04b538793d12db4b2d782ed7fea4e7563f3e5935ba2c81ddace89c5d76691f"}
Nov 03 20:04:11.726970 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:11.726936    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv" event={"ID":"bb55b062-fcb5-4fac-8089-09ad9549b7f9","Type":"ContainerStarted","Data":"88d2068aa9e5168eea1004b0eb4db7cad409df8b83e63b185dfce1783f35c34c"}
Nov 03 20:04:11.727131 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:11.726991    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv" event={"ID":"bb55b062-fcb5-4fac-8089-09ad9549b7f9","Type":"ContainerStarted","Data":"4e940f73c1cda0c4ca2b4055bc9e3ef6dc80ac7248aea13c6f67c6a99eda8bee"}
Nov 03 20:04:11.746162 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:11.746105    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6" podStartSLOduration=1.746086417 podStartE2EDuration="1.746086417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:11.738518553 +0000 UTC m=+1964.982484021" watchObservedRunningTime="2026-11-03 20:04:11.746086417 +0000 UTC m=+1964.990051886"
Nov 03 20:04:11.763689 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:11.760855    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv" podStartSLOduration=1.760808988 podStartE2EDuration="1.760808988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:11.759145575 +0000 UTC m=+1965.003111038" watchObservedRunningTime="2026-11-03 20:04:11.760808988 +0000 UTC m=+1965.004774451"
Nov 03 20:04:12.246141 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:12.246107    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv"
Nov 03 20:04:12.248259 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:12.248219    4921 patch_prober.go:29] interesting pod/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Nov 03 20:04:12.248378 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:12.248278    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv" podUID="bb55b062-fcb5-4fac-8089-09ad9549b7f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Nov 03 20:04:12.268071 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:12.268022    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6"
Nov 03 20:04:12.269981 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:12.269953    4921 patch_prober.go:29] interesting pod/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Nov 03 20:04:12.270162 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:12.270007    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6" podUID="40248459-fffd-4588-94fd-5569ee6a9e8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Nov 03 20:04:13.246505 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:13.246472    4921 patch_prober.go:29] interesting pod/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Nov 03 20:04:13.246942 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:13.246525    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv" podUID="bb55b062-fcb5-4fac-8089-09ad9549b7f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Nov 03 20:04:13.272690 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:13.272132    4921 patch_prober.go:29] interesting pod/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Nov 03 20:04:13.272690 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:13.272207    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6" podUID="40248459-fffd-4588-94fd-5569ee6a9e8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Nov 03 20:04:13.482822 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:13.482774    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-11c1a8-598fbd7d99-fzmxn" podUID="c76485a0-b175-405c-8b6e-ff94c6b9ae1f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:04:13.626820 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:13.626734    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-b30676-57f49c45c6-pkbfh"]
Nov 03 20:04:13.631903 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:13.631876    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-b30676-57f49c45c6-pkbfh"]
Nov 03 20:04:13.632043 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:13.631964    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-b30676-57f49c45c6-pkbfh"
Nov 03 20:04:13.635038 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:13.635009    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w5-b30676-serving-cert\""
Nov 03 20:04:13.840130 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:13.835158    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/817f547d-db14-409a-a162-9dc0a49be5af-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-b30676-57f49c45c6-pkbfh\" (UID: \"817f547d-db14-409a-a162-9dc0a49be5af\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-b30676-57f49c45c6-pkbfh"
Nov 03 20:04:13.840130 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:13.835330    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/817f547d-db14-409a-a162-9dc0a49be5af-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-b30676-57f49c45c6-pkbfh\" (UID: \"817f547d-db14-409a-a162-9dc0a49be5af\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-b30676-57f49c45c6-pkbfh"
Nov 03 20:04:13.840130 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:13.835367    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/817f547d-db14-409a-a162-9dc0a49be5af-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-b30676-57f49c45c6-pkbfh\" (UID: \"817f547d-db14-409a-a162-9dc0a49be5af\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-b30676-57f49c45c6-pkbfh"
Nov 03 20:04:13.840130 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:13.835492    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg547\" (UniqueName: \"kubernetes.io/projected/817f547d-db14-409a-a162-9dc0a49be5af-kube-api-access-fg547\") pod \"maas-api-e2e-shared-b-w5-b30676-57f49c45c6-pkbfh\" (UID: \"817f547d-db14-409a-a162-9dc0a49be5af\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-b30676-57f49c45c6-pkbfh"
Nov 03 20:04:13.936828 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:13.936739    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/817f547d-db14-409a-a162-9dc0a49be5af-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-b30676-57f49c45c6-pkbfh\" (UID: \"817f547d-db14-409a-a162-9dc0a49be5af\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-b30676-57f49c45c6-pkbfh"
Nov 03 20:04:13.937018 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:13.936855    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/817f547d-db14-409a-a162-9dc0a49be5af-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-b30676-57f49c45c6-pkbfh\" (UID: \"817f547d-db14-409a-a162-9dc0a49be5af\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-b30676-57f49c45c6-pkbfh"
Nov 03 20:04:13.937018 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:13.936884    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/817f547d-db14-409a-a162-9dc0a49be5af-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-b30676-57f49c45c6-pkbfh\" (UID: \"817f547d-db14-409a-a162-9dc0a49be5af\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-b30676-57f49c45c6-pkbfh"
Nov 03 20:04:13.937018 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:13.936974    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg547\" (UniqueName: \"kubernetes.io/projected/817f547d-db14-409a-a162-9dc0a49be5af-kube-api-access-fg547\") pod \"maas-api-e2e-shared-b-w5-b30676-57f49c45c6-pkbfh\" (UID: \"817f547d-db14-409a-a162-9dc0a49be5af\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-b30676-57f49c45c6-pkbfh"
Nov 03 20:04:13.937217 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:04:13.937137    4921 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-b30676-serving-cert: secret "maas-api-e2e-shared-b-w5-b30676-serving-cert" not found
Nov 03 20:04:13.937274 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:04:13.937220    4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/817f547d-db14-409a-a162-9dc0a49be5af-maas-api-tls podName:817f547d-db14-409a-a162-9dc0a49be5af nodeName:}" failed. No retries permitted until 2026-11-03 20:04:14.437184631 +0000 UTC m=+1967.681150090 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/817f547d-db14-409a-a162-9dc0a49be5af-maas-api-tls") pod "maas-api-e2e-shared-b-w5-b30676-57f49c45c6-pkbfh" (UID: "817f547d-db14-409a-a162-9dc0a49be5af") : secret "maas-api-e2e-shared-b-w5-b30676-serving-cert" not found
Nov 03 20:04:13.937722 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:13.937700    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/817f547d-db14-409a-a162-9dc0a49be5af-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-b30676-57f49c45c6-pkbfh\" (UID: \"817f547d-db14-409a-a162-9dc0a49be5af\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-b30676-57f49c45c6-pkbfh"
Nov 03 20:04:13.939772 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:13.939751    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/817f547d-db14-409a-a162-9dc0a49be5af-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-b30676-57f49c45c6-pkbfh\" (UID: \"817f547d-db14-409a-a162-9dc0a49be5af\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-b30676-57f49c45c6-pkbfh"
Nov 03 20:04:13.944810 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:13.944792    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg547\" (UniqueName: \"kubernetes.io/projected/817f547d-db14-409a-a162-9dc0a49be5af-kube-api-access-fg547\") pod \"maas-api-e2e-shared-b-w5-b30676-57f49c45c6-pkbfh\" (UID: \"817f547d-db14-409a-a162-9dc0a49be5af\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-b30676-57f49c45c6-pkbfh"
Nov 03 20:04:14.247014 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:14.246980    4921 patch_prober.go:29] interesting pod/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Nov 03 20:04:14.247426 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:14.247029    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv" podUID="bb55b062-fcb5-4fac-8089-09ad9549b7f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Nov 03 20:04:14.272690 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:14.272099    4921 patch_prober.go:29] interesting pod/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Nov 03 20:04:14.272690 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:14.272161    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6" podUID="40248459-fffd-4588-94fd-5569ee6a9e8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Nov 03 20:04:14.488576 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:14.487892    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/817f547d-db14-409a-a162-9dc0a49be5af-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-b30676-57f49c45c6-pkbfh\" (UID: \"817f547d-db14-409a-a162-9dc0a49be5af\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-b30676-57f49c45c6-pkbfh"
Nov 03 20:04:14.542231 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:14.542119    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/817f547d-db14-409a-a162-9dc0a49be5af-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-b30676-57f49c45c6-pkbfh\" (UID: \"817f547d-db14-409a-a162-9dc0a49be5af\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-b30676-57f49c45c6-pkbfh"
Nov 03 20:04:14.594691 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:14.592045    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-b30676-57f49c45c6-pkbfh"
Nov 03 20:04:15.309046 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:15.251788    4921 patch_prober.go:29] interesting pod/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Nov 03 20:04:15.309046 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:15.251851    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv" podUID="bb55b062-fcb5-4fac-8089-09ad9549b7f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Nov 03 20:04:15.311163 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:15.311131    4921 patch_prober.go:29] interesting pod/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Nov 03 20:04:15.311480 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:15.311408    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6" podUID="40248459-fffd-4588-94fd-5569ee6a9e8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Nov 03 20:04:15.641763 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:15.639720    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-b30676-57f49c45c6-pkbfh"]
Nov 03 20:04:16.125397 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:16.125363    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-b30676-57f49c45c6-pkbfh" event={"ID":"817f547d-db14-409a-a162-9dc0a49be5af","Type":"ContainerStarted","Data":"21c480f12462f48b1d568a2202082e4dad1c289d2b3b835e6bac74027b345cbc"}
Nov 03 20:04:16.246329 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:16.246299    4921 patch_prober.go:29] interesting pod/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Nov 03 20:04:16.246423 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:16.246354    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv" podUID="bb55b062-fcb5-4fac-8089-09ad9549b7f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Nov 03 20:04:16.288704 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:16.268019    4921 patch_prober.go:29] interesting pod/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Nov 03 20:04:16.288704 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:16.268074    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6" podUID="40248459-fffd-4588-94fd-5569ee6a9e8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Nov 03 20:04:17.134691 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:17.134504    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-b30676-57f49c45c6-pkbfh" event={"ID":"817f547d-db14-409a-a162-9dc0a49be5af","Type":"ContainerStarted","Data":"30915a3149e65d675c0b262bd72bb982608e93eddd6a0e6a4c590e8fd2822c26"}
Nov 03 20:04:17.134691 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:17.134597    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-b30676-57f49c45c6-pkbfh"
Nov 03 20:04:17.185694 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:17.184818    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-b30676-57f49c45c6-pkbfh" podStartSLOduration=3.673257973 podStartE2EDuration="4.184798994s" totalImagesPullingTime="511.541021ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:17.168007588 +0000 UTC m=+1970.411973056" watchObservedRunningTime="2026-11-03 20:04:17.184798994 +0000 UTC m=+1970.428764460"
Nov 03 20:04:17.246505 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:17.246357    4921 patch_prober.go:29] interesting pod/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Nov 03 20:04:17.246505 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:17.246425    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv" podUID="bb55b062-fcb5-4fac-8089-09ad9549b7f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Nov 03 20:04:17.279691 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:17.274758    4921 patch_prober.go:29] interesting pod/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Nov 03 20:04:17.279691 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:17.274815    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6" podUID="40248459-fffd-4588-94fd-5569ee6a9e8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Nov 03 20:04:18.247224 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:18.247192    4921 patch_prober.go:29] interesting pod/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Nov 03 20:04:18.247612 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:18.247241    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv" podUID="bb55b062-fcb5-4fac-8089-09ad9549b7f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Nov 03 20:04:18.295335 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:18.295295    4921 patch_prober.go:29] interesting pod/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Nov 03 20:04:18.295629 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:18.295596    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6" podUID="40248459-fffd-4588-94fd-5569ee6a9e8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Nov 03 20:04:19.251389 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:19.251356    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv"
Nov 03 20:04:19.251908 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:19.251886    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv"
Nov 03 20:04:19.252500 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:19.252481    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv"
Nov 03 20:04:19.273896 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:19.273871    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6"
Nov 03 20:04:19.274259 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:19.274239    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6"
Nov 03 20:04:19.274955 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:19.274938    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6"
Nov 03 20:04:22.327237 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:22.327192    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-11c1a8-6c664c46cf-9pcm6"]
Nov 03 20:04:22.329204 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:22.329169    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11c1a8-6c664c46cf-9pcm6" podUID="63c60ac9-edc9-41ff-88d8-80197876106c" containerName="maas-api" containerID="cri-o://1149d38940fc9b963f7de4fe41a69155868d54ab0534244da4c2c67bb9b96c68" gracePeriod=30
Nov 03 20:04:23.034611 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:23.034268    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11c1a8-6c664c46cf-9pcm6"
Nov 03 20:04:23.087234 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:23.087200    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63c60ac9-edc9-41ff-88d8-80197876106c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63c60ac9-edc9-41ff-88d8-80197876106c-openshift-service-ca\") pod \"63c60ac9-edc9-41ff-88d8-80197876106c\" (UID: \"63c60ac9-edc9-41ff-88d8-80197876106c\") "
Nov 03 20:04:23.087437 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:23.087266    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63c60ac9-edc9-41ff-88d8-80197876106c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63c60ac9-edc9-41ff-88d8-80197876106c-maas-api-tls\") pod \"63c60ac9-edc9-41ff-88d8-80197876106c\" (UID: \"63c60ac9-edc9-41ff-88d8-80197876106c\") "
Nov 03 20:04:23.087437 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:23.087303    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63c60ac9-edc9-41ff-88d8-80197876106c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/63c60ac9-edc9-41ff-88d8-80197876106c-metrics-certs\") pod \"63c60ac9-edc9-41ff-88d8-80197876106c\" (UID: \"63c60ac9-edc9-41ff-88d8-80197876106c\") "
Nov 03 20:04:23.087437 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:23.087327    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63c60ac9-edc9-41ff-88d8-80197876106c-kube-api-access-kgdzs\" (UniqueName: \"kubernetes.io/projected/63c60ac9-edc9-41ff-88d8-80197876106c-kube-api-access-kgdzs\") pod \"63c60ac9-edc9-41ff-88d8-80197876106c\" (UID: \"63c60ac9-edc9-41ff-88d8-80197876106c\") "
Nov 03 20:04:23.095973 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:23.090349    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63c60ac9-edc9-41ff-88d8-80197876106c-openshift-service-ca" pod "63c60ac9-edc9-41ff-88d8-80197876106c" (UID: "63c60ac9-edc9-41ff-88d8-80197876106c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:04:23.095973 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:23.090562    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63c60ac9-edc9-41ff-88d8-80197876106c-kube-api-access-kgdzs" pod "63c60ac9-edc9-41ff-88d8-80197876106c" (UID: "63c60ac9-edc9-41ff-88d8-80197876106c"). InnerVolumeSpecName "kube-api-access-kgdzs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:04:23.095973 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:23.092066    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63c60ac9-edc9-41ff-88d8-80197876106c-maas-api-tls" pod "63c60ac9-edc9-41ff-88d8-80197876106c" (UID: "63c60ac9-edc9-41ff-88d8-80197876106c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:04:23.095973 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:23.093001    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63c60ac9-edc9-41ff-88d8-80197876106c-metrics-certs" pod "63c60ac9-edc9-41ff-88d8-80197876106c" (UID: "63c60ac9-edc9-41ff-88d8-80197876106c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:04:23.171078 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:23.167859    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-b30676-57f49c45c6-pkbfh"
Nov 03 20:04:23.206190 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:23.206112    4921 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63c60ac9-edc9-41ff-88d8-80197876106c-openshift-service-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:23.218677 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:23.218637    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63c60ac9-edc9-41ff-88d8-80197876106c-maas-api-tls\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:23.218889 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:23.218872    4921 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63c60ac9-edc9-41ff-88d8-80197876106c-metrics-certs\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:23.218984 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:23.218972    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgdzs\" (UniqueName: \"kubernetes.io/projected/63c60ac9-edc9-41ff-88d8-80197876106c-kube-api-access-kgdzs\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:23.229344 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:23.229310    4921 generic.go:356] "Generic (PLEG): container finished" podID="63c60ac9-edc9-41ff-88d8-80197876106c" containerID="1149d38940fc9b963f7de4fe41a69155868d54ab0534244da4c2c67bb9b96c68" exitCode=0
Nov 03 20:04:23.229571 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:23.229547    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11c1a8-6c664c46cf-9pcm6" event={"ID":"63c60ac9-edc9-41ff-88d8-80197876106c","Type":"ContainerDied","Data":"1149d38940fc9b963f7de4fe41a69155868d54ab0534244da4c2c67bb9b96c68"}
Nov 03 20:04:23.229797 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:23.229775    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11c1a8-6c664c46cf-9pcm6" event={"ID":"63c60ac9-edc9-41ff-88d8-80197876106c","Type":"ContainerDied","Data":"c96d5778d5cfb82d9915b6bcb5fe9a206a3fc9e295db3b9b4747f81478607f13"}
Nov 03 20:04:23.229953 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:23.229938    4921 scope.go:122] "RemoveContainer" containerID="1149d38940fc9b963f7de4fe41a69155868d54ab0534244da4c2c67bb9b96c68"
Nov 03 20:04:23.230231 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:23.230212    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11c1a8-6c664c46cf-9pcm6"
Nov 03 20:04:23.331775 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:23.327576    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-11c1a8-6c664c46cf-9pcm6"]
Nov 03 20:04:23.331775 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:23.327632    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-11c1a8-6c664c46cf-9pcm6"]
Nov 03 20:04:23.343750 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:23.343441    4921 scope.go:122] "RemoveContainer" containerID="1149d38940fc9b963f7de4fe41a69155868d54ab0534244da4c2c67bb9b96c68"
Nov 03 20:04:23.343943 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:04:23.343839    4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1149d38940fc9b963f7de4fe41a69155868d54ab0534244da4c2c67bb9b96c68\": container with ID starting with 1149d38940fc9b963f7de4fe41a69155868d54ab0534244da4c2c67bb9b96c68 not found: ID does not exist" containerID="1149d38940fc9b963f7de4fe41a69155868d54ab0534244da4c2c67bb9b96c68"
Nov 03 20:04:23.343943 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:23.343873    4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1149d38940fc9b963f7de4fe41a69155868d54ab0534244da4c2c67bb9b96c68"} err="failed to get container status \"1149d38940fc9b963f7de4fe41a69155868d54ab0534244da4c2c67bb9b96c68\": rpc error: code = NotFound desc = could not find container \"1149d38940fc9b963f7de4fe41a69155868d54ab0534244da4c2c67bb9b96c68\": container with ID starting with 1149d38940fc9b963f7de4fe41a69155868d54ab0534244da4c2c67bb9b96c68 not found: ID does not exist"
Nov 03 20:04:23.460916 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:23.460836    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-11c1a8-598fbd7d99-fzmxn" podUID="c76485a0-b175-405c-8b6e-ff94c6b9ae1f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:04:24.877468 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:24.877430    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-11c1a8-598fbd7d99-fzmxn"
Nov 03 20:04:24.890869 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:24.890839    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63c60ac9-edc9-41ff-88d8-80197876106c" path="/var/lib/kubelet/pods/63c60ac9-edc9-41ff-88d8-80197876106c/volumes"
Nov 03 20:04:24.963946 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:24.963906    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c76485a0-b175-405c-8b6e-ff94c6b9ae1f-kube-api-access-sbwtd\" (UniqueName: \"kubernetes.io/projected/c76485a0-b175-405c-8b6e-ff94c6b9ae1f-kube-api-access-sbwtd\") pod \"c76485a0-b175-405c-8b6e-ff94c6b9ae1f\" (UID: \"c76485a0-b175-405c-8b6e-ff94c6b9ae1f\") "
Nov 03 20:04:24.964105 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:24.964006    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c76485a0-b175-405c-8b6e-ff94c6b9ae1f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c76485a0-b175-405c-8b6e-ff94c6b9ae1f-plugins-config-volume\") pod \"c76485a0-b175-405c-8b6e-ff94c6b9ae1f\" (UID: \"c76485a0-b175-405c-8b6e-ff94c6b9ae1f\") "
Nov 03 20:04:24.964405 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:24.964381    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c76485a0-b175-405c-8b6e-ff94c6b9ae1f-plugins-config-volume" pod "c76485a0-b175-405c-8b6e-ff94c6b9ae1f" (UID: "c76485a0-b175-405c-8b6e-ff94c6b9ae1f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:04:24.964780 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:24.964755    4921 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c76485a0-b175-405c-8b6e-ff94c6b9ae1f-plugins-config-volume\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:24.966340 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:24.966318    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c76485a0-b175-405c-8b6e-ff94c6b9ae1f-kube-api-access-sbwtd" pod "c76485a0-b175-405c-8b6e-ff94c6b9ae1f" (UID: "c76485a0-b175-405c-8b6e-ff94c6b9ae1f"). InnerVolumeSpecName "kube-api-access-sbwtd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:04:25.066706 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:25.066590    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbwtd\" (UniqueName: \"kubernetes.io/projected/c76485a0-b175-405c-8b6e-ff94c6b9ae1f-kube-api-access-sbwtd\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:25.251444 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:25.251412    4921 generic.go:356] "Generic (PLEG): container finished" podID="c76485a0-b175-405c-8b6e-ff94c6b9ae1f" containerID="6eb4de70602cb3c2e9acfddb47ea90fa306e2aa5113764982b51720f862f44f7" exitCode=0
Nov 03 20:04:25.251628 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:25.251453    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-11c1a8-598fbd7d99-fzmxn" event={"ID":"c76485a0-b175-405c-8b6e-ff94c6b9ae1f","Type":"ContainerDied","Data":"6eb4de70602cb3c2e9acfddb47ea90fa306e2aa5113764982b51720f862f44f7"}
Nov 03 20:04:25.251628 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:25.251458    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-11c1a8-598fbd7d99-fzmxn"
Nov 03 20:04:25.251628 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:25.251493    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-11c1a8-598fbd7d99-fzmxn" event={"ID":"c76485a0-b175-405c-8b6e-ff94c6b9ae1f","Type":"ContainerDied","Data":"520849455f16408a9934b9e72b908262bc1579b34ecaa5eff2051e069897270b"}
Nov 03 20:04:25.251628 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:25.251514    4921 scope.go:122] "RemoveContainer" containerID="6eb4de70602cb3c2e9acfddb47ea90fa306e2aa5113764982b51720f862f44f7"
Nov 03 20:04:25.263885 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:25.263859    4921 scope.go:122] "RemoveContainer" containerID="6eb4de70602cb3c2e9acfddb47ea90fa306e2aa5113764982b51720f862f44f7"
Nov 03 20:04:25.264188 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:04:25.264166    4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6eb4de70602cb3c2e9acfddb47ea90fa306e2aa5113764982b51720f862f44f7\": container with ID starting with 6eb4de70602cb3c2e9acfddb47ea90fa306e2aa5113764982b51720f862f44f7 not found: ID does not exist" containerID="6eb4de70602cb3c2e9acfddb47ea90fa306e2aa5113764982b51720f862f44f7"
Nov 03 20:04:25.264241 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:25.264224    4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6eb4de70602cb3c2e9acfddb47ea90fa306e2aa5113764982b51720f862f44f7"} err="failed to get container status \"6eb4de70602cb3c2e9acfddb47ea90fa306e2aa5113764982b51720f862f44f7\": rpc error: code = NotFound desc = could not find container \"6eb4de70602cb3c2e9acfddb47ea90fa306e2aa5113764982b51720f862f44f7\": container with ID starting with 6eb4de70602cb3c2e9acfddb47ea90fa306e2aa5113764982b51720f862f44f7 not found: ID does not exist"
Nov 03 20:04:25.277307 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:25.277277    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-11c1a8-598fbd7d99-fzmxn"]
Nov 03 20:04:25.281390 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:25.281363    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-11c1a8-598fbd7d99-fzmxn"]
Nov 03 20:04:26.891852 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:26.891819    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c76485a0-b175-405c-8b6e-ff94c6b9ae1f" path="/var/lib/kubelet/pods/c76485a0-b175-405c-8b6e-ff94c6b9ae1f/volumes"
Nov 03 20:04:29.552774 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:29.552739    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6"]
Nov 03 20:04:29.553692 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:29.553646    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6" podUID="40248459-fffd-4588-94fd-5569ee6a9e8e" containerName="istio-proxy" containerID="cri-o://007b593652233cb3dd86e355118fe7d9cdba4012b9d854780a3a58eb9eb00189" gracePeriod=30
Nov 03 20:04:29.569874 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:29.569846    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx"]
Nov 03 20:04:29.570322 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:29.570271    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx" podUID="204760c8-b3c3-4b1d-8b67-5bb6cf918407" containerName="istio-proxy" containerID="cri-o://d8db868e7d3754b44916f08023a06e1f3503ee59fc6156bd976cfc873dfa26ee" gracePeriod=30
Nov 03 20:04:29.573585 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:29.573564    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv"]
Nov 03 20:04:29.573866 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:29.573845    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv" podUID="bb55b062-fcb5-4fac-8089-09ad9549b7f9" containerName="istio-proxy" containerID="cri-o://88d2068aa9e5168eea1004b0eb4db7cad409df8b83e63b185dfce1783f35c34c" gracePeriod=30
Nov 03 20:04:32.923435 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:04:32.923391    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 20:04:34.254325 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:34.254277    4921 patch_prober.go:29] interesting pod/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Nov 03 20:04:34.254825 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:34.254350    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv" podUID="bb55b062-fcb5-4fac-8089-09ad9549b7f9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:04:34.276802 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:34.276759    4921 patch_prober.go:29] interesting pod/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:04:34.276994 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:34.276827    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6" podUID="40248459-fffd-4588-94fd-5569ee6a9e8e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:04:34.371511 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:34.371473    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt"]
Nov 03 20:04:34.372863 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:34.372836    4921 cpu_manager.go:414] "containerMap: removing container" podUID="c76485a0-b175-405c-8b6e-ff94c6b9ae1f" containerName="payload-processing"
Nov 03 20:04:34.373154 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:34.373116    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c76485a0-b175-405c-8b6e-ff94c6b9ae1f" containerName="payload-processing"
Nov 03 20:04:34.373716 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:34.373192    4921 cpu_manager.go:414] "containerMap: removing container" podUID="63c60ac9-edc9-41ff-88d8-80197876106c" containerName="maas-api"
Nov 03 20:04:34.373716 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:34.373203    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63c60ac9-edc9-41ff-88d8-80197876106c" containerName="maas-api"
Nov 03 20:04:34.373716 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:34.373376    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="c76485a0-b175-405c-8b6e-ff94c6b9ae1f" containerName="payload-processing"
Nov 03 20:04:34.373716 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:34.373400    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="63c60ac9-edc9-41ff-88d8-80197876106c" containerName="maas-api"
Nov 03 20:04:34.382017 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:34.380077    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt"
Nov 03 20:04:34.393473 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:34.393439    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt"]
Nov 03 20:04:34.476780 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:34.476748    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-9f3188-openshift-default-dockercfg-xp46f\""
Nov 03 20:04:34.477740 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:34.477710    4921 patch_prober.go:29] interesting pod/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:04:34.477862 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:34.477765    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx" podUID="204760c8-b3c3-4b1d-8b67-5bb6cf918407" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:04:34.517937 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:34.517848    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09a4d5b2-79a2-4149-99a9-5447b1999110-istio-data\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt\" (UID: \"09a4d5b2-79a2-4149-99a9-5447b1999110\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt"
Nov 03 20:04:34.517937 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:34.517895    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09a4d5b2-79a2-4149-99a9-5447b1999110-credential-socket\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt\" (UID: \"09a4d5b2-79a2-4149-99a9-5447b1999110\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt"
Nov 03 20:04:34.518155 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:34.517994    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09a4d5b2-79a2-4149-99a9-5447b1999110-istio-podinfo\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt\" (UID: \"09a4d5b2-79a2-4149-99a9-5447b1999110\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt"
Nov 03 20:04:34.518155 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:34.518057    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7wkq\" (UniqueName: \"kubernetes.io/projected/09a4d5b2-79a2-4149-99a9-5447b1999110-kube-api-access-t7wkq\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt\" (UID: \"09a4d5b2-79a2-4149-99a9-5447b1999110\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt"
Nov 03 20:04:34.518225 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:34.518198    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09a4d5b2-79a2-4149-99a9-5447b1999110-workload-certs\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt\" (UID: \"09a4d5b2-79a2-4149-99a9-5447b1999110\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt"
Nov 03 20:04:34.518260 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:34.518231    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09a4d5b2-79a2-4149-99a9-5447b1999110-istiod-ca-cert\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt\" (UID: \"09a4d5b2-79a2-4149-99a9-5447b1999110\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt"
Nov 03 20:04:34.518333 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:34.518317    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09a4d5b2-79a2-4149-99a9-5447b1999110-istio-token\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt\" (UID: \"09a4d5b2-79a2-4149-99a9-5447b1999110\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt"
Nov 03 20:04:34.518377 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:34.518359    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09a4d5b2-79a2-4149-99a9-5447b1999110-istio-envoy\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt\" (UID: \"09a4d5b2-79a2-4149-99a9-5447b1999110\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt"
Nov 03 20:04:34.518495 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:34.518483    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09a4d5b2-79a2-4149-99a9-5447b1999110-workload-socket\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt\" (UID: \"09a4d5b2-79a2-4149-99a9-5447b1999110\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt"
Nov 03 20:04:34.619738 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:34.619701    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09a4d5b2-79a2-4149-99a9-5447b1999110-workload-certs\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt\" (UID: \"09a4d5b2-79a2-4149-99a9-5447b1999110\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt"
Nov 03 20:04:34.619918 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:34.619749    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09a4d5b2-79a2-4149-99a9-5447b1999110-istiod-ca-cert\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt\" (UID: \"09a4d5b2-79a2-4149-99a9-5447b1999110\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt"
Nov 03 20:04:34.619918 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:34.619790    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09a4d5b2-79a2-4149-99a9-5447b1999110-istio-token\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt\" (UID: \"09a4d5b2-79a2-4149-99a9-5447b1999110\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt"
Nov 03 20:04:34.619918 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:34.619823    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09a4d5b2-79a2-4149-99a9-5447b1999110-istio-envoy\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt\" (UID: \"09a4d5b2-79a2-4149-99a9-5447b1999110\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt"
Nov 03 20:04:34.619918 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:34.619907    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09a4d5b2-79a2-4149-99a9-5447b1999110-workload-socket\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt\" (UID: \"09a4d5b2-79a2-4149-99a9-5447b1999110\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt"
Nov 03 20:04:34.620120 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:34.619949    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09a4d5b2-79a2-4149-99a9-5447b1999110-istio-data\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt\" (UID: \"09a4d5b2-79a2-4149-99a9-5447b1999110\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt"
Nov 03 20:04:34.620120 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:34.619972    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09a4d5b2-79a2-4149-99a9-5447b1999110-credential-socket\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt\" (UID: \"09a4d5b2-79a2-4149-99a9-5447b1999110\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt"
Nov 03 20:04:34.620120 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:34.619993    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09a4d5b2-79a2-4149-99a9-5447b1999110-istio-podinfo\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt\" (UID: \"09a4d5b2-79a2-4149-99a9-5447b1999110\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt"
Nov 03 20:04:34.620120 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:34.620036    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7wkq\" (UniqueName: \"kubernetes.io/projected/09a4d5b2-79a2-4149-99a9-5447b1999110-kube-api-access-t7wkq\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt\" (UID: \"09a4d5b2-79a2-4149-99a9-5447b1999110\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt"
Nov 03 20:04:34.620351 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:34.620141    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09a4d5b2-79a2-4149-99a9-5447b1999110-workload-certs\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt\" (UID: \"09a4d5b2-79a2-4149-99a9-5447b1999110\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt"
Nov 03 20:04:34.620412 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:34.620347    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09a4d5b2-79a2-4149-99a9-5447b1999110-istio-data\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt\" (UID: \"09a4d5b2-79a2-4149-99a9-5447b1999110\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt"
Nov 03 20:04:34.620506 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:34.620475    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09a4d5b2-79a2-4149-99a9-5447b1999110-workload-socket\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt\" (UID: \"09a4d5b2-79a2-4149-99a9-5447b1999110\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt"
Nov 03 20:04:34.620561 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:34.620542    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09a4d5b2-79a2-4149-99a9-5447b1999110-credential-socket\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt\" (UID: \"09a4d5b2-79a2-4149-99a9-5447b1999110\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt"
Nov 03 20:04:34.620600 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:34.620581    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09a4d5b2-79a2-4149-99a9-5447b1999110-istiod-ca-cert\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt\" (UID: \"09a4d5b2-79a2-4149-99a9-5447b1999110\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt"
Nov 03 20:04:34.622904 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:34.622878    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09a4d5b2-79a2-4149-99a9-5447b1999110-istio-envoy\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt\" (UID: \"09a4d5b2-79a2-4149-99a9-5447b1999110\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt"
Nov 03 20:04:34.623159 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:34.623137    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09a4d5b2-79a2-4149-99a9-5447b1999110-istio-podinfo\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt\" (UID: \"09a4d5b2-79a2-4149-99a9-5447b1999110\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt"
Nov 03 20:04:34.630143 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:34.630120    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09a4d5b2-79a2-4149-99a9-5447b1999110-istio-token\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt\" (UID: \"09a4d5b2-79a2-4149-99a9-5447b1999110\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt"
Nov 03 20:04:34.630723 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:34.630537    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7wkq\" (UniqueName: \"kubernetes.io/projected/09a4d5b2-79a2-4149-99a9-5447b1999110-kube-api-access-t7wkq\") pod \"e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt\" (UID: \"09a4d5b2-79a2-4149-99a9-5447b1999110\") " pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt"
Nov 03 20:04:34.714316 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:34.714060    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt"
Nov 03 20:04:34.946147 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:34.946116    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6"
Nov 03 20:04:34.996779 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:34.996748    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx"
Nov 03 20:04:35.027143 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.027054    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40248459-fffd-4588-94fd-5569ee6a9e8e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40248459-fffd-4588-94fd-5569ee6a9e8e-istio-envoy\") pod \"40248459-fffd-4588-94fd-5569ee6a9e8e\" (UID: \"40248459-fffd-4588-94fd-5569ee6a9e8e\") "
Nov 03 20:04:35.027299 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.027284    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40248459-fffd-4588-94fd-5569ee6a9e8e-istio-token\" (UniqueName: \"kubernetes.io/projected/40248459-fffd-4588-94fd-5569ee6a9e8e-istio-token\") pod \"40248459-fffd-4588-94fd-5569ee6a9e8e\" (UID: \"40248459-fffd-4588-94fd-5569ee6a9e8e\") "
Nov 03 20:04:35.027471 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.027452    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40248459-fffd-4588-94fd-5569ee6a9e8e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40248459-fffd-4588-94fd-5569ee6a9e8e-istiod-ca-cert\") pod \"40248459-fffd-4588-94fd-5569ee6a9e8e\" (UID: \"40248459-fffd-4588-94fd-5569ee6a9e8e\") "
Nov 03 20:04:35.027564 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.027537    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/40248459-fffd-4588-94fd-5569ee6a9e8e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40248459-fffd-4588-94fd-5569ee6a9e8e-istio-podinfo\") pod \"40248459-fffd-4588-94fd-5569ee6a9e8e\" (UID: \"40248459-fffd-4588-94fd-5569ee6a9e8e\") "
Nov 03 20:04:35.027629 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.027612    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40248459-fffd-4588-94fd-5569ee6a9e8e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40248459-fffd-4588-94fd-5569ee6a9e8e-credential-socket\") pod \"40248459-fffd-4588-94fd-5569ee6a9e8e\" (UID: \"40248459-fffd-4588-94fd-5569ee6a9e8e\") "
Nov 03 20:04:35.027719 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.027698    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40248459-fffd-4588-94fd-5569ee6a9e8e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40248459-fffd-4588-94fd-5569ee6a9e8e-istio-data\") pod \"40248459-fffd-4588-94fd-5569ee6a9e8e\" (UID: \"40248459-fffd-4588-94fd-5569ee6a9e8e\") "
Nov 03 20:04:35.027773 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.027744    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40248459-fffd-4588-94fd-5569ee6a9e8e-kube-api-access-ngnbt\" (UniqueName: \"kubernetes.io/projected/40248459-fffd-4588-94fd-5569ee6a9e8e-kube-api-access-ngnbt\") pod \"40248459-fffd-4588-94fd-5569ee6a9e8e\" (UID: \"40248459-fffd-4588-94fd-5569ee6a9e8e\") "
Nov 03 20:04:35.027810 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.027795    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40248459-fffd-4588-94fd-5569ee6a9e8e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40248459-fffd-4588-94fd-5569ee6a9e8e-workload-socket\") pod \"40248459-fffd-4588-94fd-5569ee6a9e8e\" (UID: \"40248459-fffd-4588-94fd-5569ee6a9e8e\") "
Nov 03 20:04:35.027862 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.027842    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40248459-fffd-4588-94fd-5569ee6a9e8e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40248459-fffd-4588-94fd-5569ee6a9e8e-workload-certs\") pod \"40248459-fffd-4588-94fd-5569ee6a9e8e\" (UID: \"40248459-fffd-4588-94fd-5569ee6a9e8e\") "
Nov 03 20:04:35.033010 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.030252    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40248459-fffd-4588-94fd-5569ee6a9e8e-credential-socket" pod "40248459-fffd-4588-94fd-5569ee6a9e8e" (UID: "40248459-fffd-4588-94fd-5569ee6a9e8e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:04:35.033010 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.030653    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40248459-fffd-4588-94fd-5569ee6a9e8e-istiod-ca-cert" pod "40248459-fffd-4588-94fd-5569ee6a9e8e" (UID: "40248459-fffd-4588-94fd-5569ee6a9e8e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:04:35.033010 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.031600    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40248459-fffd-4588-94fd-5569ee6a9e8e-istio-token" pod "40248459-fffd-4588-94fd-5569ee6a9e8e" (UID: "40248459-fffd-4588-94fd-5569ee6a9e8e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:04:35.033010 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.031826    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40248459-fffd-4588-94fd-5569ee6a9e8e-workload-socket" pod "40248459-fffd-4588-94fd-5569ee6a9e8e" (UID: "40248459-fffd-4588-94fd-5569ee6a9e8e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:04:35.033010 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.031976    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40248459-fffd-4588-94fd-5569ee6a9e8e-istio-data" pod "40248459-fffd-4588-94fd-5569ee6a9e8e" (UID: "40248459-fffd-4588-94fd-5569ee6a9e8e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:04:35.033010 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.032209    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40248459-fffd-4588-94fd-5569ee6a9e8e-workload-certs" pod "40248459-fffd-4588-94fd-5569ee6a9e8e" (UID: "40248459-fffd-4588-94fd-5569ee6a9e8e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:04:35.033461 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.033321    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40248459-fffd-4588-94fd-5569ee6a9e8e-istio-envoy" pod "40248459-fffd-4588-94fd-5569ee6a9e8e" (UID: "40248459-fffd-4588-94fd-5569ee6a9e8e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:04:35.033617 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.033596    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/40248459-fffd-4588-94fd-5569ee6a9e8e-istio-podinfo" pod "40248459-fffd-4588-94fd-5569ee6a9e8e" (UID: "40248459-fffd-4588-94fd-5569ee6a9e8e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:04:35.037410 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.037382    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40248459-fffd-4588-94fd-5569ee6a9e8e-kube-api-access-ngnbt" pod "40248459-fffd-4588-94fd-5569ee6a9e8e" (UID: "40248459-fffd-4588-94fd-5569ee6a9e8e"). InnerVolumeSpecName "kube-api-access-ngnbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:04:35.068711 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.068641    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv"
Nov 03 20:04:35.103112 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.103081    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt"]
Nov 03 20:04:35.103543 ip-10-0-134-222 kubenswrapper[4921]: W1103 20:04:35.103514    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09a4d5b2_79a2_4149_99a9_5447b1999110.slice/crio-faa774d13fada30dc8ff7a729a9b4623198b709481fe39df44ccbf29476d54e3 WatchSource:0}: Error finding container faa774d13fada30dc8ff7a729a9b4623198b709481fe39df44ccbf29476d54e3: Status 404 returned error can't find the container with id faa774d13fada30dc8ff7a729a9b4623198b709481fe39df44ccbf29476d54e3
Nov 03 20:04:35.106335 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.106274    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:04:35.106445 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.106373    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:04:35.106445 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.106426    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:04:35.129271 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.129241    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/204760c8-b3c3-4b1d-8b67-5bb6cf918407-kube-api-access-hdv2r\" (UniqueName: \"kubernetes.io/projected/204760c8-b3c3-4b1d-8b67-5bb6cf918407-kube-api-access-hdv2r\") pod \"204760c8-b3c3-4b1d-8b67-5bb6cf918407\" (UID: \"204760c8-b3c3-4b1d-8b67-5bb6cf918407\") "
Nov 03 20:04:35.129462 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.129302    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/204760c8-b3c3-4b1d-8b67-5bb6cf918407-istio-token\" (UniqueName: \"kubernetes.io/projected/204760c8-b3c3-4b1d-8b67-5bb6cf918407-istio-token\") pod \"204760c8-b3c3-4b1d-8b67-5bb6cf918407\" (UID: \"204760c8-b3c3-4b1d-8b67-5bb6cf918407\") "
Nov 03 20:04:35.129462 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.129372    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/204760c8-b3c3-4b1d-8b67-5bb6cf918407-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/204760c8-b3c3-4b1d-8b67-5bb6cf918407-istio-podinfo\") pod \"204760c8-b3c3-4b1d-8b67-5bb6cf918407\" (UID: \"204760c8-b3c3-4b1d-8b67-5bb6cf918407\") "
Nov 03 20:04:35.129462 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.129423    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/204760c8-b3c3-4b1d-8b67-5bb6cf918407-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/204760c8-b3c3-4b1d-8b67-5bb6cf918407-workload-socket\") pod \"204760c8-b3c3-4b1d-8b67-5bb6cf918407\" (UID: \"204760c8-b3c3-4b1d-8b67-5bb6cf918407\") "
Nov 03 20:04:35.129637 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.129607    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/204760c8-b3c3-4b1d-8b67-5bb6cf918407-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/204760c8-b3c3-4b1d-8b67-5bb6cf918407-credential-socket\") pod \"204760c8-b3c3-4b1d-8b67-5bb6cf918407\" (UID: \"204760c8-b3c3-4b1d-8b67-5bb6cf918407\") "
Nov 03 20:04:35.129769 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.129734    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/204760c8-b3c3-4b1d-8b67-5bb6cf918407-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/204760c8-b3c3-4b1d-8b67-5bb6cf918407-workload-certs\") pod \"204760c8-b3c3-4b1d-8b67-5bb6cf918407\" (UID: \"204760c8-b3c3-4b1d-8b67-5bb6cf918407\") "
Nov 03 20:04:35.129850 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.129783    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/204760c8-b3c3-4b1d-8b67-5bb6cf918407-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/204760c8-b3c3-4b1d-8b67-5bb6cf918407-istiod-ca-cert\") pod \"204760c8-b3c3-4b1d-8b67-5bb6cf918407\" (UID: \"204760c8-b3c3-4b1d-8b67-5bb6cf918407\") "
Nov 03 20:04:35.129912 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.129868    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/204760c8-b3c3-4b1d-8b67-5bb6cf918407-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/204760c8-b3c3-4b1d-8b67-5bb6cf918407-istio-data\") pod \"204760c8-b3c3-4b1d-8b67-5bb6cf918407\" (UID: \"204760c8-b3c3-4b1d-8b67-5bb6cf918407\") "
Nov 03 20:04:35.129985 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.129971    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/204760c8-b3c3-4b1d-8b67-5bb6cf918407-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/204760c8-b3c3-4b1d-8b67-5bb6cf918407-istio-envoy\") pod \"204760c8-b3c3-4b1d-8b67-5bb6cf918407\" (UID: \"204760c8-b3c3-4b1d-8b67-5bb6cf918407\") "
Nov 03 20:04:35.130571 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.130551    4921 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40248459-fffd-4588-94fd-5569ee6a9e8e-istio-envoy\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:35.130718 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.130575    4921 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40248459-fffd-4588-94fd-5569ee6a9e8e-istio-token\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:35.130718 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.130588    4921 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40248459-fffd-4588-94fd-5569ee6a9e8e-istiod-ca-cert\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:35.130718 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.130601    4921 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40248459-fffd-4588-94fd-5569ee6a9e8e-istio-podinfo\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:35.130718 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.130614    4921 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40248459-fffd-4588-94fd-5569ee6a9e8e-credential-socket\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:35.130718 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.130627    4921 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40248459-fffd-4588-94fd-5569ee6a9e8e-istio-data\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:35.130718 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.130640    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngnbt\" (UniqueName: \"kubernetes.io/projected/40248459-fffd-4588-94fd-5569ee6a9e8e-kube-api-access-ngnbt\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:35.130718 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.130655    4921 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40248459-fffd-4588-94fd-5569ee6a9e8e-workload-socket\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:35.130718 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.130696    4921 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40248459-fffd-4588-94fd-5569ee6a9e8e-workload-certs\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:35.131281 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.131257    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/204760c8-b3c3-4b1d-8b67-5bb6cf918407-workload-socket" pod "204760c8-b3c3-4b1d-8b67-5bb6cf918407" (UID: "204760c8-b3c3-4b1d-8b67-5bb6cf918407"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:04:35.132293 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.131988    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/204760c8-b3c3-4b1d-8b67-5bb6cf918407-istio-data" pod "204760c8-b3c3-4b1d-8b67-5bb6cf918407" (UID: "204760c8-b3c3-4b1d-8b67-5bb6cf918407"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:04:35.132401 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.132313    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/204760c8-b3c3-4b1d-8b67-5bb6cf918407-credential-socket" pod "204760c8-b3c3-4b1d-8b67-5bb6cf918407" (UID: "204760c8-b3c3-4b1d-8b67-5bb6cf918407"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:04:35.132757 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.132736    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/204760c8-b3c3-4b1d-8b67-5bb6cf918407-workload-certs" pod "204760c8-b3c3-4b1d-8b67-5bb6cf918407" (UID: "204760c8-b3c3-4b1d-8b67-5bb6cf918407"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:04:35.133028 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.132986    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/204760c8-b3c3-4b1d-8b67-5bb6cf918407-istiod-ca-cert" pod "204760c8-b3c3-4b1d-8b67-5bb6cf918407" (UID: "204760c8-b3c3-4b1d-8b67-5bb6cf918407"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:04:35.135132 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.134967    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/204760c8-b3c3-4b1d-8b67-5bb6cf918407-istio-envoy" pod "204760c8-b3c3-4b1d-8b67-5bb6cf918407" (UID: "204760c8-b3c3-4b1d-8b67-5bb6cf918407"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:04:35.135132 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.135093    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/204760c8-b3c3-4b1d-8b67-5bb6cf918407-kube-api-access-hdv2r" pod "204760c8-b3c3-4b1d-8b67-5bb6cf918407" (UID: "204760c8-b3c3-4b1d-8b67-5bb6cf918407"). InnerVolumeSpecName "kube-api-access-hdv2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:04:35.135418 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.135392    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/204760c8-b3c3-4b1d-8b67-5bb6cf918407-istio-podinfo" pod "204760c8-b3c3-4b1d-8b67-5bb6cf918407" (UID: "204760c8-b3c3-4b1d-8b67-5bb6cf918407"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:04:35.139466 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.139435    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/204760c8-b3c3-4b1d-8b67-5bb6cf918407-istio-token" pod "204760c8-b3c3-4b1d-8b67-5bb6cf918407" (UID: "204760c8-b3c3-4b1d-8b67-5bb6cf918407"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:04:35.231649 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.231611    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb55b062-fcb5-4fac-8089-09ad9549b7f9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb55b062-fcb5-4fac-8089-09ad9549b7f9-credential-socket\") pod \"bb55b062-fcb5-4fac-8089-09ad9549b7f9\" (UID: \"bb55b062-fcb5-4fac-8089-09ad9549b7f9\") "
Nov 03 20:04:35.231857 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.231723    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb55b062-fcb5-4fac-8089-09ad9549b7f9-kube-api-access-mzcb4\" (UniqueName: \"kubernetes.io/projected/bb55b062-fcb5-4fac-8089-09ad9549b7f9-kube-api-access-mzcb4\") pod \"bb55b062-fcb5-4fac-8089-09ad9549b7f9\" (UID: \"bb55b062-fcb5-4fac-8089-09ad9549b7f9\") "
Nov 03 20:04:35.233181 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.233151    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb55b062-fcb5-4fac-8089-09ad9549b7f9-credential-socket" pod "bb55b062-fcb5-4fac-8089-09ad9549b7f9" (UID: "bb55b062-fcb5-4fac-8089-09ad9549b7f9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:04:35.233327 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.233215    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb55b062-fcb5-4fac-8089-09ad9549b7f9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb55b062-fcb5-4fac-8089-09ad9549b7f9-istio-data\") pod \"bb55b062-fcb5-4fac-8089-09ad9549b7f9\" (UID: \"bb55b062-fcb5-4fac-8089-09ad9549b7f9\") "
Nov 03 20:04:35.233429 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.233403    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb55b062-fcb5-4fac-8089-09ad9549b7f9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb55b062-fcb5-4fac-8089-09ad9549b7f9-workload-certs\") pod \"bb55b062-fcb5-4fac-8089-09ad9549b7f9\" (UID: \"bb55b062-fcb5-4fac-8089-09ad9549b7f9\") "
Nov 03 20:04:35.233509 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.233463    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb55b062-fcb5-4fac-8089-09ad9549b7f9-istio-token\" (UniqueName: \"kubernetes.io/projected/bb55b062-fcb5-4fac-8089-09ad9549b7f9-istio-token\") pod \"bb55b062-fcb5-4fac-8089-09ad9549b7f9\" (UID: \"bb55b062-fcb5-4fac-8089-09ad9549b7f9\") "
Nov 03 20:04:35.233509 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.233494    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb55b062-fcb5-4fac-8089-09ad9549b7f9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb55b062-fcb5-4fac-8089-09ad9549b7f9-workload-socket\") pod \"bb55b062-fcb5-4fac-8089-09ad9549b7f9\" (UID: \"bb55b062-fcb5-4fac-8089-09ad9549b7f9\") "
Nov 03 20:04:35.233627 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.233525    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb55b062-fcb5-4fac-8089-09ad9549b7f9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb55b062-fcb5-4fac-8089-09ad9549b7f9-istiod-ca-cert\") pod \"bb55b062-fcb5-4fac-8089-09ad9549b7f9\" (UID: \"bb55b062-fcb5-4fac-8089-09ad9549b7f9\") "
Nov 03 20:04:35.233627 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.233575    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb55b062-fcb5-4fac-8089-09ad9549b7f9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb55b062-fcb5-4fac-8089-09ad9549b7f9-istio-envoy\") pod \"bb55b062-fcb5-4fac-8089-09ad9549b7f9\" (UID: \"bb55b062-fcb5-4fac-8089-09ad9549b7f9\") "
Nov 03 20:04:35.233921 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.233626    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bb55b062-fcb5-4fac-8089-09ad9549b7f9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb55b062-fcb5-4fac-8089-09ad9549b7f9-istio-podinfo\") pod \"bb55b062-fcb5-4fac-8089-09ad9549b7f9\" (UID: \"bb55b062-fcb5-4fac-8089-09ad9549b7f9\") "
Nov 03 20:04:35.234449 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.234424    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdv2r\" (UniqueName: \"kubernetes.io/projected/204760c8-b3c3-4b1d-8b67-5bb6cf918407-kube-api-access-hdv2r\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:35.234558 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.234456    4921 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/204760c8-b3c3-4b1d-8b67-5bb6cf918407-istio-token\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:35.234558 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.234472    4921 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/204760c8-b3c3-4b1d-8b67-5bb6cf918407-istio-podinfo\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:35.234558 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.234488    4921 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/204760c8-b3c3-4b1d-8b67-5bb6cf918407-workload-socket\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:35.234558 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.234501    4921 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/204760c8-b3c3-4b1d-8b67-5bb6cf918407-credential-socket\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:35.234558 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.234514    4921 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/204760c8-b3c3-4b1d-8b67-5bb6cf918407-workload-certs\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:35.234558 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.234529    4921 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/204760c8-b3c3-4b1d-8b67-5bb6cf918407-istiod-ca-cert\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:35.234558 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.234541    4921 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/204760c8-b3c3-4b1d-8b67-5bb6cf918407-istio-data\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:35.234558 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.234556    4921 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb55b062-fcb5-4fac-8089-09ad9549b7f9-credential-socket\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:35.235129 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.234570    4921 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/204760c8-b3c3-4b1d-8b67-5bb6cf918407-istio-envoy\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:35.235129 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.234971    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb55b062-fcb5-4fac-8089-09ad9549b7f9-workload-socket" pod "bb55b062-fcb5-4fac-8089-09ad9549b7f9" (UID: "bb55b062-fcb5-4fac-8089-09ad9549b7f9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:04:35.235790 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.235252    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb55b062-fcb5-4fac-8089-09ad9549b7f9-istio-data" pod "bb55b062-fcb5-4fac-8089-09ad9549b7f9" (UID: "bb55b062-fcb5-4fac-8089-09ad9549b7f9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:04:35.235790 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.235595    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb55b062-fcb5-4fac-8089-09ad9549b7f9-workload-certs" pod "bb55b062-fcb5-4fac-8089-09ad9549b7f9" (UID: "bb55b062-fcb5-4fac-8089-09ad9549b7f9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:04:35.235790 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.235692    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb55b062-fcb5-4fac-8089-09ad9549b7f9-istiod-ca-cert" pod "bb55b062-fcb5-4fac-8089-09ad9549b7f9" (UID: "bb55b062-fcb5-4fac-8089-09ad9549b7f9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:04:35.238304 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.238275    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb55b062-fcb5-4fac-8089-09ad9549b7f9-kube-api-access-mzcb4" pod "bb55b062-fcb5-4fac-8089-09ad9549b7f9" (UID: "bb55b062-fcb5-4fac-8089-09ad9549b7f9"). InnerVolumeSpecName "kube-api-access-mzcb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:04:35.239772 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.239746    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb55b062-fcb5-4fac-8089-09ad9549b7f9-istio-token" pod "bb55b062-fcb5-4fac-8089-09ad9549b7f9" (UID: "bb55b062-fcb5-4fac-8089-09ad9549b7f9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:04:35.243139 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.243100    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bb55b062-fcb5-4fac-8089-09ad9549b7f9-istio-podinfo" pod "bb55b062-fcb5-4fac-8089-09ad9549b7f9" (UID: "bb55b062-fcb5-4fac-8089-09ad9549b7f9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:04:35.244993 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.244958    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb55b062-fcb5-4fac-8089-09ad9549b7f9-istio-envoy" pod "bb55b062-fcb5-4fac-8089-09ad9549b7f9" (UID: "bb55b062-fcb5-4fac-8089-09ad9549b7f9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:04:35.336089 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.335930    4921 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb55b062-fcb5-4fac-8089-09ad9549b7f9-workload-certs\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:35.336089 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.335963    4921 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb55b062-fcb5-4fac-8089-09ad9549b7f9-istio-token\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:35.336089 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.335973    4921 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb55b062-fcb5-4fac-8089-09ad9549b7f9-workload-socket\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:35.336089 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.335982    4921 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb55b062-fcb5-4fac-8089-09ad9549b7f9-istiod-ca-cert\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:35.336089 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.335989    4921 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb55b062-fcb5-4fac-8089-09ad9549b7f9-istio-envoy\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:35.336089 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.335998    4921 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb55b062-fcb5-4fac-8089-09ad9549b7f9-istio-podinfo\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:35.336089 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.336007    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzcb4\" (UniqueName: \"kubernetes.io/projected/bb55b062-fcb5-4fac-8089-09ad9549b7f9-kube-api-access-mzcb4\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:35.336089 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.336015    4921 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb55b062-fcb5-4fac-8089-09ad9549b7f9-istio-data\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:35.354581 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.354381    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt" event={"ID":"09a4d5b2-79a2-4149-99a9-5447b1999110","Type":"ContainerStarted","Data":"fa5a75a060b6286fe77ca5cca1590a265d0048d2ef08b0ec84337b3de7c908e2"}
Nov 03 20:04:35.355021 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.354955    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt" event={"ID":"09a4d5b2-79a2-4149-99a9-5447b1999110","Type":"ContainerStarted","Data":"faa774d13fada30dc8ff7a729a9b4623198b709481fe39df44ccbf29476d54e3"}
Nov 03 20:04:35.358632 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.357074    4921 generic.go:356] "Generic (PLEG): container finished" podID="204760c8-b3c3-4b1d-8b67-5bb6cf918407" containerID="d8db868e7d3754b44916f08023a06e1f3503ee59fc6156bd976cfc873dfa26ee" exitCode=0
Nov 03 20:04:35.358632 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.357230    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx"
Nov 03 20:04:35.358632 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.357405    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx" event={"ID":"204760c8-b3c3-4b1d-8b67-5bb6cf918407","Type":"ContainerDied","Data":"d8db868e7d3754b44916f08023a06e1f3503ee59fc6156bd976cfc873dfa26ee"}
Nov 03 20:04:35.358632 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.357462    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx" event={"ID":"204760c8-b3c3-4b1d-8b67-5bb6cf918407","Type":"ContainerDied","Data":"10537d31d73f4d93ec48cb12a59a4c7be02b35c86ec19c236a28eebece834464"}
Nov 03 20:04:35.358632 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.357486    4921 scope.go:122] "RemoveContainer" containerID="d8db868e7d3754b44916f08023a06e1f3503ee59fc6156bd976cfc873dfa26ee"
Nov 03 20:04:35.365701 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.362221    4921 generic.go:356] "Generic (PLEG): container finished" podID="40248459-fffd-4588-94fd-5569ee6a9e8e" containerID="007b593652233cb3dd86e355118fe7d9cdba4012b9d854780a3a58eb9eb00189" exitCode=0
Nov 03 20:04:35.365701 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.362362    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6" event={"ID":"40248459-fffd-4588-94fd-5569ee6a9e8e","Type":"ContainerDied","Data":"007b593652233cb3dd86e355118fe7d9cdba4012b9d854780a3a58eb9eb00189"}
Nov 03 20:04:35.365701 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.362410    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6" event={"ID":"40248459-fffd-4588-94fd-5569ee6a9e8e","Type":"ContainerDied","Data":"7f04b538793d12db4b2d782ed7fea4e7563f3e5935ba2c81ddace89c5d76691f"}
Nov 03 20:04:35.365701 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.362491    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6"
Nov 03 20:04:35.368879 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.368842    4921 generic.go:356] "Generic (PLEG): container finished" podID="bb55b062-fcb5-4fac-8089-09ad9549b7f9" containerID="88d2068aa9e5168eea1004b0eb4db7cad409df8b83e63b185dfce1783f35c34c" exitCode=0
Nov 03 20:04:35.369024 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.368975    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv" event={"ID":"bb55b062-fcb5-4fac-8089-09ad9549b7f9","Type":"ContainerDied","Data":"88d2068aa9e5168eea1004b0eb4db7cad409df8b83e63b185dfce1783f35c34c"}
Nov 03 20:04:35.369096 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.369025    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv" event={"ID":"bb55b062-fcb5-4fac-8089-09ad9549b7f9","Type":"ContainerDied","Data":"4e940f73c1cda0c4ca2b4055bc9e3ef6dc80ac7248aea13c6f67c6a99eda8bee"}
Nov 03 20:04:35.369156 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.369123    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv"
Nov 03 20:04:35.384770 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.381172    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt" podStartSLOduration=1.381153824 podStartE2EDuration="1.381153824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:35.380418414 +0000 UTC m=+1988.624383880" watchObservedRunningTime="2026-11-03 20:04:35.381153824 +0000 UTC m=+1988.625119290"
Nov 03 20:04:35.396308 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.396283    4921 scope.go:122] "RemoveContainer" containerID="d8db868e7d3754b44916f08023a06e1f3503ee59fc6156bd976cfc873dfa26ee"
Nov 03 20:04:35.396781 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:04:35.396752    4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8db868e7d3754b44916f08023a06e1f3503ee59fc6156bd976cfc873dfa26ee\": container with ID starting with d8db868e7d3754b44916f08023a06e1f3503ee59fc6156bd976cfc873dfa26ee not found: ID does not exist" containerID="d8db868e7d3754b44916f08023a06e1f3503ee59fc6156bd976cfc873dfa26ee"
Nov 03 20:04:35.396880 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.396791    4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8db868e7d3754b44916f08023a06e1f3503ee59fc6156bd976cfc873dfa26ee"} err="failed to get container status \"d8db868e7d3754b44916f08023a06e1f3503ee59fc6156bd976cfc873dfa26ee\": rpc error: code = NotFound desc = could not find container \"d8db868e7d3754b44916f08023a06e1f3503ee59fc6156bd976cfc873dfa26ee\": container with ID starting with d8db868e7d3754b44916f08023a06e1f3503ee59fc6156bd976cfc873dfa26ee not found: ID does not exist"
Nov 03 20:04:35.396880 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.396818    4921 scope.go:122] "RemoveContainer" containerID="007b593652233cb3dd86e355118fe7d9cdba4012b9d854780a3a58eb9eb00189"
Nov 03 20:04:35.423056 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.423034    4921 scope.go:122] "RemoveContainer" containerID="007b593652233cb3dd86e355118fe7d9cdba4012b9d854780a3a58eb9eb00189"
Nov 03 20:04:35.423440 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:04:35.423411    4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"007b593652233cb3dd86e355118fe7d9cdba4012b9d854780a3a58eb9eb00189\": container with ID starting with 007b593652233cb3dd86e355118fe7d9cdba4012b9d854780a3a58eb9eb00189 not found: ID does not exist" containerID="007b593652233cb3dd86e355118fe7d9cdba4012b9d854780a3a58eb9eb00189"
Nov 03 20:04:35.423568 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.423445    4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"007b593652233cb3dd86e355118fe7d9cdba4012b9d854780a3a58eb9eb00189"} err="failed to get container status \"007b593652233cb3dd86e355118fe7d9cdba4012b9d854780a3a58eb9eb00189\": rpc error: code = NotFound desc = could not find container \"007b593652233cb3dd86e355118fe7d9cdba4012b9d854780a3a58eb9eb00189\": container with ID starting with 007b593652233cb3dd86e355118fe7d9cdba4012b9d854780a3a58eb9eb00189 not found: ID does not exist"
Nov 03 20:04:35.423568 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.423462    4921 scope.go:122] "RemoveContainer" containerID="88d2068aa9e5168eea1004b0eb4db7cad409df8b83e63b185dfce1783f35c34c"
Nov 03 20:04:35.435588 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.435564    4921 scope.go:122] "RemoveContainer" containerID="88d2068aa9e5168eea1004b0eb4db7cad409df8b83e63b185dfce1783f35c34c"
Nov 03 20:04:35.436002 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:04:35.435980    4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88d2068aa9e5168eea1004b0eb4db7cad409df8b83e63b185dfce1783f35c34c\": container with ID starting with 88d2068aa9e5168eea1004b0eb4db7cad409df8b83e63b185dfce1783f35c34c not found: ID does not exist" containerID="88d2068aa9e5168eea1004b0eb4db7cad409df8b83e63b185dfce1783f35c34c"
Nov 03 20:04:35.436107 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.436009    4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88d2068aa9e5168eea1004b0eb4db7cad409df8b83e63b185dfce1783f35c34c"} err="failed to get container status \"88d2068aa9e5168eea1004b0eb4db7cad409df8b83e63b185dfce1783f35c34c\": rpc error: code = NotFound desc = could not find container \"88d2068aa9e5168eea1004b0eb4db7cad409df8b83e63b185dfce1783f35c34c\": container with ID starting with 88d2068aa9e5168eea1004b0eb4db7cad409df8b83e63b185dfce1783f35c34c not found: ID does not exist"
Nov 03 20:04:35.452649 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.452610    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx"]
Nov 03 20:04:35.465107 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.465075    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-jzmtx"]
Nov 03 20:04:35.479017 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.478984    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6"]
Nov 03 20:04:35.495094 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.495057    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-kt8f6"]
Nov 03 20:04:35.519376 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.519334    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv"]
Nov 03 20:04:35.714163 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.714128    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt"
Nov 03 20:04:35.740895 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:35.739020    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-11c1a8-openshift-default-6f96865fb5-74snv"]
Nov 03 20:04:36.716237 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:36.716201    4921 patch_prober.go:29] interesting pod/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body=
Nov 03 20:04:36.716694 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:36.716295    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt" podUID="09a4d5b2-79a2-4149-99a9-5447b1999110" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused"
Nov 03 20:04:36.893098 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:36.893063    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="204760c8-b3c3-4b1d-8b67-5bb6cf918407" path="/var/lib/kubelet/pods/204760c8-b3c3-4b1d-8b67-5bb6cf918407/volumes"
Nov 03 20:04:36.893602 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:36.893587    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40248459-fffd-4588-94fd-5569ee6a9e8e" path="/var/lib/kubelet/pods/40248459-fffd-4588-94fd-5569ee6a9e8e/volumes"
Nov 03 20:04:36.894065 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:36.894052    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb55b062-fcb5-4fac-8089-09ad9549b7f9" path="/var/lib/kubelet/pods/bb55b062-fcb5-4fac-8089-09ad9549b7f9/volumes"
Nov 03 20:04:37.715188 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:37.715151    4921 patch_prober.go:29] interesting pod/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body=
Nov 03 20:04:37.715344 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:37.715225    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt" podUID="09a4d5b2-79a2-4149-99a9-5447b1999110" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused"
Nov 03 20:04:38.715281 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:38.715243    4921 patch_prober.go:29] interesting pod/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body=
Nov 03 20:04:38.715736 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:38.715298    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt" podUID="09a4d5b2-79a2-4149-99a9-5447b1999110" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused"
Nov 03 20:04:39.723692 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:39.723016    4921 patch_prober.go:29] interesting pod/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body=
Nov 03 20:04:39.723692 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:39.723075    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt" podUID="09a4d5b2-79a2-4149-99a9-5447b1999110" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused"
Nov 03 20:04:40.180226 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:40.180182    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f58171-57c45cc64c-2xsx5"]
Nov 03 20:04:40.719745 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:40.716943    4921 patch_prober.go:29] interesting pod/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body=
Nov 03 20:04:40.719745 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:40.717017    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt" podUID="09a4d5b2-79a2-4149-99a9-5447b1999110" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused"
Nov 03 20:04:41.715068 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:41.715032    4921 patch_prober.go:29] interesting pod/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body=
Nov 03 20:04:41.715658 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:41.715628    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt" podUID="09a4d5b2-79a2-4149-99a9-5447b1999110" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused"
Nov 03 20:04:42.777877 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:42.777842    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt"
Nov 03 20:04:42.778443 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:42.778407    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt"
Nov 03 20:04:42.780097 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:42.780073    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt"
Nov 03 20:04:45.182270 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:45.182084    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f58171-57c45cc64c-2xsx5" podUID="5ba7b96b-f2c0-4ddf-8117-c578d6ba1d69" containerName="payload-pre-processing" containerID="cri-o://a00db52098aab2cdf265bdb589432764d81f2d05a3f2cd437ec73b9964f3cf6d" gracePeriod=25
Nov 03 20:04:49.239935 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:49.239902    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt"]
Nov 03 20:04:49.240740 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:49.240705    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt" podUID="09a4d5b2-79a2-4149-99a9-5447b1999110" containerName="istio-proxy" containerID="cri-o://fa5a75a060b6286fe77ca5cca1590a265d0048d2ef08b0ec84337b3de7c908e2" gracePeriod=30
Nov 03 20:04:52.468487 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:52.468449    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f58171-57c45cc64c-2xsx5" podUID="5ba7b96b-f2c0-4ddf-8117-c578d6ba1d69" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:04:54.562616 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:54.562588    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt"
Nov 03 20:04:54.593492 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:54.593457    4921 generic.go:356] "Generic (PLEG): container finished" podID="09a4d5b2-79a2-4149-99a9-5447b1999110" containerID="fa5a75a060b6286fe77ca5cca1590a265d0048d2ef08b0ec84337b3de7c908e2" exitCode=0
Nov 03 20:04:54.593764 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:54.593518    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt"
Nov 03 20:04:54.594138 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:54.593534    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt" event={"ID":"09a4d5b2-79a2-4149-99a9-5447b1999110","Type":"ContainerDied","Data":"fa5a75a060b6286fe77ca5cca1590a265d0048d2ef08b0ec84337b3de7c908e2"}
Nov 03 20:04:54.594244 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:54.594199    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt" event={"ID":"09a4d5b2-79a2-4149-99a9-5447b1999110","Type":"ContainerDied","Data":"faa774d13fada30dc8ff7a729a9b4623198b709481fe39df44ccbf29476d54e3"}
Nov 03 20:04:54.594244 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:54.594234    4921 scope.go:122] "RemoveContainer" containerID="fa5a75a060b6286fe77ca5cca1590a265d0048d2ef08b0ec84337b3de7c908e2"
Nov 03 20:04:54.619614 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:54.619595    4921 scope.go:122] "RemoveContainer" containerID="fa5a75a060b6286fe77ca5cca1590a265d0048d2ef08b0ec84337b3de7c908e2"
Nov 03 20:04:54.619985 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:04:54.619964    4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa5a75a060b6286fe77ca5cca1590a265d0048d2ef08b0ec84337b3de7c908e2\": container with ID starting with fa5a75a060b6286fe77ca5cca1590a265d0048d2ef08b0ec84337b3de7c908e2 not found: ID does not exist" containerID="fa5a75a060b6286fe77ca5cca1590a265d0048d2ef08b0ec84337b3de7c908e2"
Nov 03 20:04:54.620078 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:54.619994    4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa5a75a060b6286fe77ca5cca1590a265d0048d2ef08b0ec84337b3de7c908e2"} err="failed to get container status \"fa5a75a060b6286fe77ca5cca1590a265d0048d2ef08b0ec84337b3de7c908e2\": rpc error: code = NotFound desc = could not find container \"fa5a75a060b6286fe77ca5cca1590a265d0048d2ef08b0ec84337b3de7c908e2\": container with ID starting with fa5a75a060b6286fe77ca5cca1590a265d0048d2ef08b0ec84337b3de7c908e2 not found: ID does not exist"
Nov 03 20:04:54.657724 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:54.657689    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09a4d5b2-79a2-4149-99a9-5447b1999110-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09a4d5b2-79a2-4149-99a9-5447b1999110-credential-socket\") pod \"09a4d5b2-79a2-4149-99a9-5447b1999110\" (UID: \"09a4d5b2-79a2-4149-99a9-5447b1999110\") "
Nov 03 20:04:54.657951 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:54.657931    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09a4d5b2-79a2-4149-99a9-5447b1999110-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09a4d5b2-79a2-4149-99a9-5447b1999110-istio-data\") pod \"09a4d5b2-79a2-4149-99a9-5447b1999110\" (UID: \"09a4d5b2-79a2-4149-99a9-5447b1999110\") "
Nov 03 20:04:54.658163 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:54.658144    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09a4d5b2-79a2-4149-99a9-5447b1999110-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09a4d5b2-79a2-4149-99a9-5447b1999110-istiod-ca-cert\") pod \"09a4d5b2-79a2-4149-99a9-5447b1999110\" (UID: \"09a4d5b2-79a2-4149-99a9-5447b1999110\") "
Nov 03 20:04:54.658349 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:54.658332    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09a4d5b2-79a2-4149-99a9-5447b1999110-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09a4d5b2-79a2-4149-99a9-5447b1999110-workload-socket\") pod \"09a4d5b2-79a2-4149-99a9-5447b1999110\" (UID: \"09a4d5b2-79a2-4149-99a9-5447b1999110\") "
Nov 03 20:04:54.658476 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:54.658460    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09a4d5b2-79a2-4149-99a9-5447b1999110-istio-token\" (UniqueName: \"kubernetes.io/projected/09a4d5b2-79a2-4149-99a9-5447b1999110-istio-token\") pod \"09a4d5b2-79a2-4149-99a9-5447b1999110\" (UID: \"09a4d5b2-79a2-4149-99a9-5447b1999110\") "
Nov 03 20:04:54.658623 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:54.658607    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/09a4d5b2-79a2-4149-99a9-5447b1999110-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09a4d5b2-79a2-4149-99a9-5447b1999110-istio-podinfo\") pod \"09a4d5b2-79a2-4149-99a9-5447b1999110\" (UID: \"09a4d5b2-79a2-4149-99a9-5447b1999110\") "
Nov 03 20:04:54.658793 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:54.658775    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09a4d5b2-79a2-4149-99a9-5447b1999110-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09a4d5b2-79a2-4149-99a9-5447b1999110-workload-certs\") pod \"09a4d5b2-79a2-4149-99a9-5447b1999110\" (UID: \"09a4d5b2-79a2-4149-99a9-5447b1999110\") "
Nov 03 20:04:54.658940 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:54.658924    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09a4d5b2-79a2-4149-99a9-5447b1999110-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09a4d5b2-79a2-4149-99a9-5447b1999110-istio-envoy\") pod \"09a4d5b2-79a2-4149-99a9-5447b1999110\" (UID: \"09a4d5b2-79a2-4149-99a9-5447b1999110\") "
Nov 03 20:04:54.659073 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:54.659056    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09a4d5b2-79a2-4149-99a9-5447b1999110-kube-api-access-t7wkq\" (UniqueName: \"kubernetes.io/projected/09a4d5b2-79a2-4149-99a9-5447b1999110-kube-api-access-t7wkq\") pod \"09a4d5b2-79a2-4149-99a9-5447b1999110\" (UID: \"09a4d5b2-79a2-4149-99a9-5447b1999110\") "
Nov 03 20:04:54.659285 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:54.658009    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09a4d5b2-79a2-4149-99a9-5447b1999110-credential-socket" pod "09a4d5b2-79a2-4149-99a9-5447b1999110" (UID: "09a4d5b2-79a2-4149-99a9-5447b1999110"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:04:54.659896 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:54.659870    4921 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09a4d5b2-79a2-4149-99a9-5447b1999110-credential-socket\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:54.660025 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:54.658215    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09a4d5b2-79a2-4149-99a9-5447b1999110-istio-data" pod "09a4d5b2-79a2-4149-99a9-5447b1999110" (UID: "09a4d5b2-79a2-4149-99a9-5447b1999110"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:04:54.660120 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:54.658533    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09a4d5b2-79a2-4149-99a9-5447b1999110-istiod-ca-cert" pod "09a4d5b2-79a2-4149-99a9-5447b1999110" (UID: "09a4d5b2-79a2-4149-99a9-5447b1999110"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:04:54.660202 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:54.658745    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09a4d5b2-79a2-4149-99a9-5447b1999110-workload-socket" pod "09a4d5b2-79a2-4149-99a9-5447b1999110" (UID: "09a4d5b2-79a2-4149-99a9-5447b1999110"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:04:54.660280 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:54.659252    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09a4d5b2-79a2-4149-99a9-5447b1999110-workload-certs" pod "09a4d5b2-79a2-4149-99a9-5447b1999110" (UID: "09a4d5b2-79a2-4149-99a9-5447b1999110"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:04:54.662160 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:54.662131    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09a4d5b2-79a2-4149-99a9-5447b1999110-kube-api-access-t7wkq" pod "09a4d5b2-79a2-4149-99a9-5447b1999110" (UID: "09a4d5b2-79a2-4149-99a9-5447b1999110"). InnerVolumeSpecName "kube-api-access-t7wkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:04:54.662719 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:54.662661    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09a4d5b2-79a2-4149-99a9-5447b1999110-istio-envoy" pod "09a4d5b2-79a2-4149-99a9-5447b1999110" (UID: "09a4d5b2-79a2-4149-99a9-5447b1999110"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:04:54.682273 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:54.682235    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09a4d5b2-79a2-4149-99a9-5447b1999110-istio-token" pod "09a4d5b2-79a2-4149-99a9-5447b1999110" (UID: "09a4d5b2-79a2-4149-99a9-5447b1999110"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:04:54.692973 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:54.692928    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/09a4d5b2-79a2-4149-99a9-5447b1999110-istio-podinfo" pod "09a4d5b2-79a2-4149-99a9-5447b1999110" (UID: "09a4d5b2-79a2-4149-99a9-5447b1999110"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:04:54.761149 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:54.761112    4921 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09a4d5b2-79a2-4149-99a9-5447b1999110-istio-data\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:54.761149 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:54.761152    4921 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09a4d5b2-79a2-4149-99a9-5447b1999110-istiod-ca-cert\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:54.761355 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:54.761170    4921 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09a4d5b2-79a2-4149-99a9-5447b1999110-workload-socket\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:54.761355 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:54.761185    4921 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09a4d5b2-79a2-4149-99a9-5447b1999110-istio-token\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:54.761355 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:54.761199    4921 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09a4d5b2-79a2-4149-99a9-5447b1999110-istio-podinfo\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:54.761355 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:54.761221    4921 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09a4d5b2-79a2-4149-99a9-5447b1999110-workload-certs\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:54.761355 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:54.761236    4921 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09a4d5b2-79a2-4149-99a9-5447b1999110-istio-envoy\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:54.761355 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:54.761249    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7wkq\" (UniqueName: \"kubernetes.io/projected/09a4d5b2-79a2-4149-99a9-5447b1999110-kube-api-access-t7wkq\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:04:54.940991 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:54.940947    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt"]
Nov 03 20:04:54.945805 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:54.945773    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-9f3188-openshift-default-679fbdfc87-zcpqt"]
Nov 03 20:04:55.432102 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:55.432065    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d"]
Nov 03 20:04:55.432364 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:55.432337    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d" podUID="33d97e3e-9c11-4ca2-bddd-b2c057ab41e9" containerName="istio-proxy" containerID="cri-o://30b912486eeaf1f1f3fb846ff6d8148564788180791de064cb5a9585d7217c28" gracePeriod=30
Nov 03 20:04:56.889523 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:04:56.889489    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09a4d5b2-79a2-4149-99a9-5447b1999110" path="/var/lib/kubelet/pods/09a4d5b2-79a2-4149-99a9-5447b1999110/volumes"
Nov 03 20:05:00.683539 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:00.683089    4921 generic.go:356] "Generic (PLEG): container finished" podID="33d97e3e-9c11-4ca2-bddd-b2c057ab41e9" containerID="30b912486eeaf1f1f3fb846ff6d8148564788180791de064cb5a9585d7217c28" exitCode=0
Nov 03 20:05:00.683539 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:00.683373    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d" event={"ID":"33d97e3e-9c11-4ca2-bddd-b2c057ab41e9","Type":"ContainerDied","Data":"30b912486eeaf1f1f3fb846ff6d8148564788180791de064cb5a9585d7217c28"}
Nov 03 20:05:00.803839 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:00.803806    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d"
Nov 03 20:05:00.985376 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:00.983445    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33d97e3e-9c11-4ca2-bddd-b2c057ab41e9-istio-token\" (UniqueName: \"kubernetes.io/projected/33d97e3e-9c11-4ca2-bddd-b2c057ab41e9-istio-token\") pod \"33d97e3e-9c11-4ca2-bddd-b2c057ab41e9\" (UID: \"33d97e3e-9c11-4ca2-bddd-b2c057ab41e9\") "
Nov 03 20:05:00.985376 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:00.983545    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33d97e3e-9c11-4ca2-bddd-b2c057ab41e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33d97e3e-9c11-4ca2-bddd-b2c057ab41e9-workload-certs\") pod \"33d97e3e-9c11-4ca2-bddd-b2c057ab41e9\" (UID: \"33d97e3e-9c11-4ca2-bddd-b2c057ab41e9\") "
Nov 03 20:05:00.985376 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:00.983586    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33d97e3e-9c11-4ca2-bddd-b2c057ab41e9-kube-api-access-tf6jt\" (UniqueName: \"kubernetes.io/projected/33d97e3e-9c11-4ca2-bddd-b2c057ab41e9-kube-api-access-tf6jt\") pod \"33d97e3e-9c11-4ca2-bddd-b2c057ab41e9\" (UID: \"33d97e3e-9c11-4ca2-bddd-b2c057ab41e9\") "
Nov 03 20:05:00.985376 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:00.983727    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33d97e3e-9c11-4ca2-bddd-b2c057ab41e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33d97e3e-9c11-4ca2-bddd-b2c057ab41e9-istio-envoy\") pod \"33d97e3e-9c11-4ca2-bddd-b2c057ab41e9\" (UID: \"33d97e3e-9c11-4ca2-bddd-b2c057ab41e9\") "
Nov 03 20:05:00.985376 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:00.983763    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/33d97e3e-9c11-4ca2-bddd-b2c057ab41e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33d97e3e-9c11-4ca2-bddd-b2c057ab41e9-istio-podinfo\") pod \"33d97e3e-9c11-4ca2-bddd-b2c057ab41e9\" (UID: \"33d97e3e-9c11-4ca2-bddd-b2c057ab41e9\") "
Nov 03 20:05:00.985376 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:00.983967    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33d97e3e-9c11-4ca2-bddd-b2c057ab41e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33d97e3e-9c11-4ca2-bddd-b2c057ab41e9-credential-socket\") pod \"33d97e3e-9c11-4ca2-bddd-b2c057ab41e9\" (UID: \"33d97e3e-9c11-4ca2-bddd-b2c057ab41e9\") "
Nov 03 20:05:00.985376 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:00.984003    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33d97e3e-9c11-4ca2-bddd-b2c057ab41e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33d97e3e-9c11-4ca2-bddd-b2c057ab41e9-istiod-ca-cert\") pod \"33d97e3e-9c11-4ca2-bddd-b2c057ab41e9\" (UID: \"33d97e3e-9c11-4ca2-bddd-b2c057ab41e9\") "
Nov 03 20:05:00.985376 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:00.984039    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33d97e3e-9c11-4ca2-bddd-b2c057ab41e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33d97e3e-9c11-4ca2-bddd-b2c057ab41e9-istio-data\") pod \"33d97e3e-9c11-4ca2-bddd-b2c057ab41e9\" (UID: \"33d97e3e-9c11-4ca2-bddd-b2c057ab41e9\") "
Nov 03 20:05:00.985376 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:00.984091    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33d97e3e-9c11-4ca2-bddd-b2c057ab41e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33d97e3e-9c11-4ca2-bddd-b2c057ab41e9-workload-socket\") pod \"33d97e3e-9c11-4ca2-bddd-b2c057ab41e9\" (UID: \"33d97e3e-9c11-4ca2-bddd-b2c057ab41e9\") "
Nov 03 20:05:00.987970 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:00.987315    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33d97e3e-9c11-4ca2-bddd-b2c057ab41e9-istio-data" pod "33d97e3e-9c11-4ca2-bddd-b2c057ab41e9" (UID: "33d97e3e-9c11-4ca2-bddd-b2c057ab41e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:05:00.987970 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:00.987504    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33d97e3e-9c11-4ca2-bddd-b2c057ab41e9-workload-certs" pod "33d97e3e-9c11-4ca2-bddd-b2c057ab41e9" (UID: "33d97e3e-9c11-4ca2-bddd-b2c057ab41e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:05:00.987970 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:00.987628    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33d97e3e-9c11-4ca2-bddd-b2c057ab41e9-workload-socket" pod "33d97e3e-9c11-4ca2-bddd-b2c057ab41e9" (UID: "33d97e3e-9c11-4ca2-bddd-b2c057ab41e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:05:00.987970 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:00.987941    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33d97e3e-9c11-4ca2-bddd-b2c057ab41e9-credential-socket" pod "33d97e3e-9c11-4ca2-bddd-b2c057ab41e9" (UID: "33d97e3e-9c11-4ca2-bddd-b2c057ab41e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:05:00.988761 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:00.988737    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33d97e3e-9c11-4ca2-bddd-b2c057ab41e9-istiod-ca-cert" pod "33d97e3e-9c11-4ca2-bddd-b2c057ab41e9" (UID: "33d97e3e-9c11-4ca2-bddd-b2c057ab41e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:05:01.016412 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:01.016348    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33d97e3e-9c11-4ca2-bddd-b2c057ab41e9-istio-envoy" pod "33d97e3e-9c11-4ca2-bddd-b2c057ab41e9" (UID: "33d97e3e-9c11-4ca2-bddd-b2c057ab41e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:05:01.017107 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:01.017061    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33d97e3e-9c11-4ca2-bddd-b2c057ab41e9-kube-api-access-tf6jt" pod "33d97e3e-9c11-4ca2-bddd-b2c057ab41e9" (UID: "33d97e3e-9c11-4ca2-bddd-b2c057ab41e9"). InnerVolumeSpecName "kube-api-access-tf6jt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:05:01.017373 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:01.017343    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33d97e3e-9c11-4ca2-bddd-b2c057ab41e9-istio-token" pod "33d97e3e-9c11-4ca2-bddd-b2c057ab41e9" (UID: "33d97e3e-9c11-4ca2-bddd-b2c057ab41e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:05:01.047502 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:01.047460    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/33d97e3e-9c11-4ca2-bddd-b2c057ab41e9-istio-podinfo" pod "33d97e3e-9c11-4ca2-bddd-b2c057ab41e9" (UID: "33d97e3e-9c11-4ca2-bddd-b2c057ab41e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:05:01.086857 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:01.086489    4921 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33d97e3e-9c11-4ca2-bddd-b2c057ab41e9-credential-socket\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:01.086857 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:01.086534    4921 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33d97e3e-9c11-4ca2-bddd-b2c057ab41e9-istiod-ca-cert\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:01.086857 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:01.086560    4921 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33d97e3e-9c11-4ca2-bddd-b2c057ab41e9-istio-data\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:01.086857 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:01.086575    4921 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33d97e3e-9c11-4ca2-bddd-b2c057ab41e9-workload-socket\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:01.086857 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:01.086589    4921 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33d97e3e-9c11-4ca2-bddd-b2c057ab41e9-istio-token\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:01.086857 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:01.086605    4921 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33d97e3e-9c11-4ca2-bddd-b2c057ab41e9-workload-certs\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:01.086857 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:01.086619    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tf6jt\" (UniqueName: \"kubernetes.io/projected/33d97e3e-9c11-4ca2-bddd-b2c057ab41e9-kube-api-access-tf6jt\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:01.086857 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:01.086632    4921 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33d97e3e-9c11-4ca2-bddd-b2c057ab41e9-istio-envoy\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:01.086857 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:01.086646    4921 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33d97e3e-9c11-4ca2-bddd-b2c057ab41e9-istio-podinfo\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:01.693594 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:01.693565    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d"
Nov 03 20:05:01.694071 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:01.693599    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d" event={"ID":"33d97e3e-9c11-4ca2-bddd-b2c057ab41e9","Type":"ContainerDied","Data":"ea16b2d9ea06d43399d0411fd14dcd3c345e154357fbb5838e4db96c688b982c"}
Nov 03 20:05:01.694071 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:01.693689    4921 scope.go:122] "RemoveContainer" containerID="30b912486eeaf1f1f3fb846ff6d8148564788180791de064cb5a9585d7217c28"
Nov 03 20:05:01.717576 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:01.717533    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d"]
Nov 03 20:05:01.721563 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:01.721534    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-f58171-openshift-default-77bc76bfb5-jvl9d"]
Nov 03 20:05:02.468520 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:02.468486    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f58171-57c45cc64c-2xsx5" podUID="5ba7b96b-f2c0-4ddf-8117-c578d6ba1d69" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:05:02.892039 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:02.891998    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33d97e3e-9c11-4ca2-bddd-b2c057ab41e9" path="/var/lib/kubelet/pods/33d97e3e-9c11-4ca2-bddd-b2c057ab41e9/volumes"
Nov 03 20:05:02.946081 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:05:02.946043    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 20:05:03.864332 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:03.864297    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3cee6a-799d7c674c-fj9vk"]
Nov 03 20:05:03.876214 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:03.876178    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3cee6a-544fd5876c-56hwf"]
Nov 03 20:05:05.466021 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:05.465996    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f58171-57c45cc64c-2xsx5"
Nov 03 20:05:05.554899 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:05.554870    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ba7b96b-f2c0-4ddf-8117-c578d6ba1d69-kube-api-access-hmbgp\" (UniqueName: \"kubernetes.io/projected/5ba7b96b-f2c0-4ddf-8117-c578d6ba1d69-kube-api-access-hmbgp\") pod \"5ba7b96b-f2c0-4ddf-8117-c578d6ba1d69\" (UID: \"5ba7b96b-f2c0-4ddf-8117-c578d6ba1d69\") "
Nov 03 20:05:05.555073 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:05.555044    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ba7b96b-f2c0-4ddf-8117-c578d6ba1d69-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ba7b96b-f2c0-4ddf-8117-c578d6ba1d69-plugins-config-volume\") pod \"5ba7b96b-f2c0-4ddf-8117-c578d6ba1d69\" (UID: \"5ba7b96b-f2c0-4ddf-8117-c578d6ba1d69\") "
Nov 03 20:05:05.555454 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:05.555429    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ba7b96b-f2c0-4ddf-8117-c578d6ba1d69-plugins-config-volume" pod "5ba7b96b-f2c0-4ddf-8117-c578d6ba1d69" (UID: "5ba7b96b-f2c0-4ddf-8117-c578d6ba1d69"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:05:05.555593 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:05.555579    4921 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ba7b96b-f2c0-4ddf-8117-c578d6ba1d69-plugins-config-volume\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:05.557082 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:05.557056    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ba7b96b-f2c0-4ddf-8117-c578d6ba1d69-kube-api-access-hmbgp" pod "5ba7b96b-f2c0-4ddf-8117-c578d6ba1d69" (UID: "5ba7b96b-f2c0-4ddf-8117-c578d6ba1d69"). InnerVolumeSpecName "kube-api-access-hmbgp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:05:05.656576 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:05.656545    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmbgp\" (UniqueName: \"kubernetes.io/projected/5ba7b96b-f2c0-4ddf-8117-c578d6ba1d69-kube-api-access-hmbgp\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:05.754945 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:05.754907    4921 generic.go:356] "Generic (PLEG): container finished" podID="5ba7b96b-f2c0-4ddf-8117-c578d6ba1d69" containerID="a00db52098aab2cdf265bdb589432764d81f2d05a3f2cd437ec73b9964f3cf6d" exitCode=0
Nov 03 20:05:05.755138 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:05.755009    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f58171-57c45cc64c-2xsx5"
Nov 03 20:05:05.755138 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:05.755026    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f58171-57c45cc64c-2xsx5" event={"ID":"5ba7b96b-f2c0-4ddf-8117-c578d6ba1d69","Type":"ContainerDied","Data":"a00db52098aab2cdf265bdb589432764d81f2d05a3f2cd437ec73b9964f3cf6d"}
Nov 03 20:05:05.755138 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:05.755079    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f58171-57c45cc64c-2xsx5" event={"ID":"5ba7b96b-f2c0-4ddf-8117-c578d6ba1d69","Type":"ContainerDied","Data":"9b51fdd8c07e519d6918069a9c7468e527d4867b6b9bb58107f6637eb0a2c7c2"}
Nov 03 20:05:05.755138 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:05.755107    4921 scope.go:122] "RemoveContainer" containerID="a00db52098aab2cdf265bdb589432764d81f2d05a3f2cd437ec73b9964f3cf6d"
Nov 03 20:05:05.768226 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:05.768203    4921 scope.go:122] "RemoveContainer" containerID="a00db52098aab2cdf265bdb589432764d81f2d05a3f2cd437ec73b9964f3cf6d"
Nov 03 20:05:05.768557 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:05:05.768536    4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a00db52098aab2cdf265bdb589432764d81f2d05a3f2cd437ec73b9964f3cf6d\": container with ID starting with a00db52098aab2cdf265bdb589432764d81f2d05a3f2cd437ec73b9964f3cf6d not found: ID does not exist" containerID="a00db52098aab2cdf265bdb589432764d81f2d05a3f2cd437ec73b9964f3cf6d"
Nov 03 20:05:05.768623 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:05.768567    4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a00db52098aab2cdf265bdb589432764d81f2d05a3f2cd437ec73b9964f3cf6d"} err="failed to get container status \"a00db52098aab2cdf265bdb589432764d81f2d05a3f2cd437ec73b9964f3cf6d\": rpc error: code = NotFound desc = could not find container \"a00db52098aab2cdf265bdb589432764d81f2d05a3f2cd437ec73b9964f3cf6d\": container with ID starting with a00db52098aab2cdf265bdb589432764d81f2d05a3f2cd437ec73b9964f3cf6d not found: ID does not exist"
Nov 03 20:05:05.779515 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:05.779486    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f58171-57c45cc64c-2xsx5"]
Nov 03 20:05:05.784021 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:05.783997    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f58171-57c45cc64c-2xsx5"]
Nov 03 20:05:06.050643 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:06.050529    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-fa9526-55fcd67459-dv9g7"]
Nov 03 20:05:06.060726 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:06.060691    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fa9526-7b98b75779-6gbh9"]
Nov 03 20:05:06.889875 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:06.889843    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ba7b96b-f2c0-4ddf-8117-c578d6ba1d69" path="/var/lib/kubelet/pods/5ba7b96b-f2c0-4ddf-8117-c578d6ba1d69/volumes"
Nov 03 20:05:07.851718 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:07.841482    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd"]
Nov 03 20:05:07.851718 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:07.841752    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd" podUID="4736bbc2-086a-4d21-bafa-3abd36c25fa6" containerName="istio-proxy" containerID="cri-o://1d7ffece2c1e3a1e71c9c10f7d5445f3702abd027a7601aefc4b0c0483872c2a" gracePeriod=30
Nov 03 20:05:08.019637 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:07.998978    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-32a584-5bc48c4649-xdrrx"]
Nov 03 20:05:08.020438 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:08.020396    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-32a584-5bc48c4649-xdrrx" podUID="1068a9b9-6312-4246-b139-679922ea7a5a" containerName="maas-api" containerID="cri-o://2d7518a66331807813225c4e6e9cee8b4cef2220a7c2cfbcf41dbafee7dafa7c" gracePeriod=30
Nov 03 20:05:08.539996 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:08.539964    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-32a584-5bc48c4649-xdrrx"
Nov 03 20:05:08.664810 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:08.664770    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1068a9b9-6312-4246-b139-679922ea7a5a-kube-api-access-b9n5w\" (UniqueName: \"kubernetes.io/projected/1068a9b9-6312-4246-b139-679922ea7a5a-kube-api-access-b9n5w\") pod \"1068a9b9-6312-4246-b139-679922ea7a5a\" (UID: \"1068a9b9-6312-4246-b139-679922ea7a5a\") "
Nov 03 20:05:08.665003 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:08.664908    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1068a9b9-6312-4246-b139-679922ea7a5a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1068a9b9-6312-4246-b139-679922ea7a5a-metrics-certs\") pod \"1068a9b9-6312-4246-b139-679922ea7a5a\" (UID: \"1068a9b9-6312-4246-b139-679922ea7a5a\") "
Nov 03 20:05:08.665003 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:08.664947    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1068a9b9-6312-4246-b139-679922ea7a5a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1068a9b9-6312-4246-b139-679922ea7a5a-openshift-service-ca\") pod \"1068a9b9-6312-4246-b139-679922ea7a5a\" (UID: \"1068a9b9-6312-4246-b139-679922ea7a5a\") "
Nov 03 20:05:08.665113 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:08.665029    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1068a9b9-6312-4246-b139-679922ea7a5a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1068a9b9-6312-4246-b139-679922ea7a5a-maas-api-tls\") pod \"1068a9b9-6312-4246-b139-679922ea7a5a\" (UID: \"1068a9b9-6312-4246-b139-679922ea7a5a\") "
Nov 03 20:05:08.665394 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:08.665364    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1068a9b9-6312-4246-b139-679922ea7a5a-openshift-service-ca" pod "1068a9b9-6312-4246-b139-679922ea7a5a" (UID: "1068a9b9-6312-4246-b139-679922ea7a5a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:05:08.665730 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:08.665706    4921 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1068a9b9-6312-4246-b139-679922ea7a5a-openshift-service-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:08.667347 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:08.667321    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1068a9b9-6312-4246-b139-679922ea7a5a-maas-api-tls" pod "1068a9b9-6312-4246-b139-679922ea7a5a" (UID: "1068a9b9-6312-4246-b139-679922ea7a5a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:05:08.667469 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:08.667358    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1068a9b9-6312-4246-b139-679922ea7a5a-kube-api-access-b9n5w" pod "1068a9b9-6312-4246-b139-679922ea7a5a" (UID: "1068a9b9-6312-4246-b139-679922ea7a5a"). InnerVolumeSpecName "kube-api-access-b9n5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:05:08.667721 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:08.667695    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1068a9b9-6312-4246-b139-679922ea7a5a-metrics-certs" pod "1068a9b9-6312-4246-b139-679922ea7a5a" (UID: "1068a9b9-6312-4246-b139-679922ea7a5a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:05:08.766989 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:08.766899    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1068a9b9-6312-4246-b139-679922ea7a5a-maas-api-tls\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:08.766989 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:08.766934    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9n5w\" (UniqueName: \"kubernetes.io/projected/1068a9b9-6312-4246-b139-679922ea7a5a-kube-api-access-b9n5w\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:08.766989 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:08.766950    4921 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1068a9b9-6312-4246-b139-679922ea7a5a-metrics-certs\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:08.781875 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:08.781841    4921 generic.go:356] "Generic (PLEG): container finished" podID="1068a9b9-6312-4246-b139-679922ea7a5a" containerID="2d7518a66331807813225c4e6e9cee8b4cef2220a7c2cfbcf41dbafee7dafa7c" exitCode=0
Nov 03 20:05:08.782059 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:08.781895    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-32a584-5bc48c4649-xdrrx"
Nov 03 20:05:08.782059 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:08.781929    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-32a584-5bc48c4649-xdrrx" event={"ID":"1068a9b9-6312-4246-b139-679922ea7a5a","Type":"ContainerDied","Data":"2d7518a66331807813225c4e6e9cee8b4cef2220a7c2cfbcf41dbafee7dafa7c"}
Nov 03 20:05:08.782059 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:08.781995    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-32a584-5bc48c4649-xdrrx" event={"ID":"1068a9b9-6312-4246-b139-679922ea7a5a","Type":"ContainerDied","Data":"9faaf6861c6f1411f1ed70b245b4dc295778a045eb720a030b8ea7ef48528556"}
Nov 03 20:05:08.782059 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:08.782025    4921 scope.go:122] "RemoveContainer" containerID="2d7518a66331807813225c4e6e9cee8b4cef2220a7c2cfbcf41dbafee7dafa7c"
Nov 03 20:05:08.795445 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:08.795422    4921 scope.go:122] "RemoveContainer" containerID="2d7518a66331807813225c4e6e9cee8b4cef2220a7c2cfbcf41dbafee7dafa7c"
Nov 03 20:05:08.795732 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:05:08.795710    4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d7518a66331807813225c4e6e9cee8b4cef2220a7c2cfbcf41dbafee7dafa7c\": container with ID starting with 2d7518a66331807813225c4e6e9cee8b4cef2220a7c2cfbcf41dbafee7dafa7c not found: ID does not exist" containerID="2d7518a66331807813225c4e6e9cee8b4cef2220a7c2cfbcf41dbafee7dafa7c"
Nov 03 20:05:08.795802 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:08.795740    4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d7518a66331807813225c4e6e9cee8b4cef2220a7c2cfbcf41dbafee7dafa7c"} err="failed to get container status \"2d7518a66331807813225c4e6e9cee8b4cef2220a7c2cfbcf41dbafee7dafa7c\": rpc error: code = NotFound desc = could not find container \"2d7518a66331807813225c4e6e9cee8b4cef2220a7c2cfbcf41dbafee7dafa7c\": container with ID starting with 2d7518a66331807813225c4e6e9cee8b4cef2220a7c2cfbcf41dbafee7dafa7c not found: ID does not exist"
Nov 03 20:05:08.805984 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:08.805936    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-32a584-5bc48c4649-xdrrx"]
Nov 03 20:05:08.809709 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:08.809652    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-32a584-5bc48c4649-xdrrx"]
Nov 03 20:05:08.865779 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:08.865729    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-3cee6a-799d7c674c-fj9vk" podUID="4aa9c292-f06c-45d9-9a58-8aeae3b7b6ab" containerName="payload-processing" containerID="cri-o://c22dd48c7c79a0e292ef4e7ee62a718aa5c150899e124577e4bea7f0abf2ea3c" gracePeriod=25
Nov 03 20:05:08.884696 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:08.877779    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3cee6a-544fd5876c-56hwf" podUID="26379d10-3da7-4549-8b8c-41730a41792b" containerName="payload-pre-processing" containerID="cri-o://03ff696cb9d480ac235973dcde8e5a58b5d22ac3a8144b098460f74873cc28a4" gracePeriod=25
Nov 03 20:05:08.989653 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:08.989144    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1068a9b9-6312-4246-b139-679922ea7a5a" path="/var/lib/kubelet/pods/1068a9b9-6312-4246-b139-679922ea7a5a/volumes"
Nov 03 20:05:11.051223 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:11.051176    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fa9526-55fcd67459-dv9g7" podUID="65b8374e-d983-45f0-8b27-8de1000e1f69" containerName="payload-processing" containerID="cri-o://a86dd6bd6ca4e217007ca74ab46dfa2f9622db6fec2fe417c84f13a0376e47c3" gracePeriod=25
Nov 03 20:05:11.061890 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:11.061853    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fa9526-7b98b75779-6gbh9" podUID="aeb289bd-62bd-45a1-9168-6e0e023e504f" containerName="payload-pre-processing" containerID="cri-o://035b56bfd3b74216aef011d5f8b4f934d6a4af74989501b17a10d359d7c9b9d5" gracePeriod=25
Nov 03 20:05:11.346609 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:11.346579    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fa9526-55fcd67459-dv9g7"
Nov 03 20:05:11.405238 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:11.405189    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65b8374e-d983-45f0-8b27-8de1000e1f69-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65b8374e-d983-45f0-8b27-8de1000e1f69-plugins-config-volume\") pod \"65b8374e-d983-45f0-8b27-8de1000e1f69\" (UID: \"65b8374e-d983-45f0-8b27-8de1000e1f69\") "
Nov 03 20:05:11.405392 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:11.405329    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65b8374e-d983-45f0-8b27-8de1000e1f69-kube-api-access-kc8gv\" (UniqueName: \"kubernetes.io/projected/65b8374e-d983-45f0-8b27-8de1000e1f69-kube-api-access-kc8gv\") pod \"65b8374e-d983-45f0-8b27-8de1000e1f69\" (UID: \"65b8374e-d983-45f0-8b27-8de1000e1f69\") "
Nov 03 20:05:11.405682 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:11.405640    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65b8374e-d983-45f0-8b27-8de1000e1f69-plugins-config-volume" pod "65b8374e-d983-45f0-8b27-8de1000e1f69" (UID: "65b8374e-d983-45f0-8b27-8de1000e1f69"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:05:11.405829 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:11.405812    4921 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65b8374e-d983-45f0-8b27-8de1000e1f69-plugins-config-volume\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:11.407959 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:11.407935    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65b8374e-d983-45f0-8b27-8de1000e1f69-kube-api-access-kc8gv" pod "65b8374e-d983-45f0-8b27-8de1000e1f69" (UID: "65b8374e-d983-45f0-8b27-8de1000e1f69"). InnerVolumeSpecName "kube-api-access-kc8gv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:05:11.420249 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:11.420228    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fa9526-7b98b75779-6gbh9"
Nov 03 20:05:11.506866 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:11.506835    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aeb289bd-62bd-45a1-9168-6e0e023e504f-kube-api-access-lzgnp\" (UniqueName: \"kubernetes.io/projected/aeb289bd-62bd-45a1-9168-6e0e023e504f-kube-api-access-lzgnp\") pod \"aeb289bd-62bd-45a1-9168-6e0e023e504f\" (UID: \"aeb289bd-62bd-45a1-9168-6e0e023e504f\") "
Nov 03 20:05:11.507055 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:11.506935    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aeb289bd-62bd-45a1-9168-6e0e023e504f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aeb289bd-62bd-45a1-9168-6e0e023e504f-plugins-config-volume\") pod \"aeb289bd-62bd-45a1-9168-6e0e023e504f\" (UID: \"aeb289bd-62bd-45a1-9168-6e0e023e504f\") "
Nov 03 20:05:11.507292 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:11.507277    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kc8gv\" (UniqueName: \"kubernetes.io/projected/65b8374e-d983-45f0-8b27-8de1000e1f69-kube-api-access-kc8gv\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:11.507355 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:11.507296    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aeb289bd-62bd-45a1-9168-6e0e023e504f-plugins-config-volume" pod "aeb289bd-62bd-45a1-9168-6e0e023e504f" (UID: "aeb289bd-62bd-45a1-9168-6e0e023e504f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:05:11.509090 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:11.509065    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aeb289bd-62bd-45a1-9168-6e0e023e504f-kube-api-access-lzgnp" pod "aeb289bd-62bd-45a1-9168-6e0e023e504f" (UID: "aeb289bd-62bd-45a1-9168-6e0e023e504f"). InnerVolumeSpecName "kube-api-access-lzgnp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:05:11.608913 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:11.608826    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzgnp\" (UniqueName: \"kubernetes.io/projected/aeb289bd-62bd-45a1-9168-6e0e023e504f-kube-api-access-lzgnp\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:11.608913 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:11.608857    4921 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aeb289bd-62bd-45a1-9168-6e0e023e504f-plugins-config-volume\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:11.811274 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:11.811233    4921 generic.go:356] "Generic (PLEG): container finished" podID="65b8374e-d983-45f0-8b27-8de1000e1f69" containerID="a86dd6bd6ca4e217007ca74ab46dfa2f9622db6fec2fe417c84f13a0376e47c3" exitCode=0
Nov 03 20:05:11.811480 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:11.811337    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fa9526-55fcd67459-dv9g7"
Nov 03 20:05:11.811480 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:11.811342    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fa9526-55fcd67459-dv9g7" event={"ID":"65b8374e-d983-45f0-8b27-8de1000e1f69","Type":"ContainerDied","Data":"a86dd6bd6ca4e217007ca74ab46dfa2f9622db6fec2fe417c84f13a0376e47c3"}
Nov 03 20:05:11.811480 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:11.811457    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fa9526-55fcd67459-dv9g7" event={"ID":"65b8374e-d983-45f0-8b27-8de1000e1f69","Type":"ContainerDied","Data":"48b0423a5eda1c3ce25ce6b3548273ba145f3a601c3e089ea99c25033bd8444b"}
Nov 03 20:05:11.811663 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:11.811524    4921 scope.go:122] "RemoveContainer" containerID="a86dd6bd6ca4e217007ca74ab46dfa2f9622db6fec2fe417c84f13a0376e47c3"
Nov 03 20:05:11.814776 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:11.813553    4921 generic.go:356] "Generic (PLEG): container finished" podID="aeb289bd-62bd-45a1-9168-6e0e023e504f" containerID="035b56bfd3b74216aef011d5f8b4f934d6a4af74989501b17a10d359d7c9b9d5" exitCode=0
Nov 03 20:05:11.814776 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:11.813733    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fa9526-7b98b75779-6gbh9" event={"ID":"aeb289bd-62bd-45a1-9168-6e0e023e504f","Type":"ContainerDied","Data":"035b56bfd3b74216aef011d5f8b4f934d6a4af74989501b17a10d359d7c9b9d5"}
Nov 03 20:05:11.814776 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:11.813780    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fa9526-7b98b75779-6gbh9" event={"ID":"aeb289bd-62bd-45a1-9168-6e0e023e504f","Type":"ContainerDied","Data":"50f2ff14adaf796c6e189d98446d6e5f4b5f17329f6fbc775f563fd4d45bc7eb"}
Nov 03 20:05:11.814776 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:11.813849    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fa9526-7b98b75779-6gbh9"
Nov 03 20:05:11.827944 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:11.827926    4921 scope.go:122] "RemoveContainer" containerID="a86dd6bd6ca4e217007ca74ab46dfa2f9622db6fec2fe417c84f13a0376e47c3"
Nov 03 20:05:11.828263 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:05:11.828235    4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a86dd6bd6ca4e217007ca74ab46dfa2f9622db6fec2fe417c84f13a0376e47c3\": container with ID starting with a86dd6bd6ca4e217007ca74ab46dfa2f9622db6fec2fe417c84f13a0376e47c3 not found: ID does not exist" containerID="a86dd6bd6ca4e217007ca74ab46dfa2f9622db6fec2fe417c84f13a0376e47c3"
Nov 03 20:05:11.828349 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:11.828273    4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a86dd6bd6ca4e217007ca74ab46dfa2f9622db6fec2fe417c84f13a0376e47c3"} err="failed to get container status \"a86dd6bd6ca4e217007ca74ab46dfa2f9622db6fec2fe417c84f13a0376e47c3\": rpc error: code = NotFound desc = could not find container \"a86dd6bd6ca4e217007ca74ab46dfa2f9622db6fec2fe417c84f13a0376e47c3\": container with ID starting with a86dd6bd6ca4e217007ca74ab46dfa2f9622db6fec2fe417c84f13a0376e47c3 not found: ID does not exist"
Nov 03 20:05:11.828349 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:11.828293    4921 scope.go:122] "RemoveContainer" containerID="035b56bfd3b74216aef011d5f8b4f934d6a4af74989501b17a10d359d7c9b9d5"
Nov 03 20:05:11.839651 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:11.839630    4921 scope.go:122] "RemoveContainer" containerID="035b56bfd3b74216aef011d5f8b4f934d6a4af74989501b17a10d359d7c9b9d5"
Nov 03 20:05:11.839951 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:05:11.839929    4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"035b56bfd3b74216aef011d5f8b4f934d6a4af74989501b17a10d359d7c9b9d5\": container with ID starting with 035b56bfd3b74216aef011d5f8b4f934d6a4af74989501b17a10d359d7c9b9d5 not found: ID does not exist" containerID="035b56bfd3b74216aef011d5f8b4f934d6a4af74989501b17a10d359d7c9b9d5"
Nov 03 20:05:11.840018 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:11.839960    4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"035b56bfd3b74216aef011d5f8b4f934d6a4af74989501b17a10d359d7c9b9d5"} err="failed to get container status \"035b56bfd3b74216aef011d5f8b4f934d6a4af74989501b17a10d359d7c9b9d5\": rpc error: code = NotFound desc = could not find container \"035b56bfd3b74216aef011d5f8b4f934d6a4af74989501b17a10d359d7c9b9d5\": container with ID starting with 035b56bfd3b74216aef011d5f8b4f934d6a4af74989501b17a10d359d7c9b9d5 not found: ID does not exist"
Nov 03 20:05:11.841029 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:11.841008    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-fa9526-55fcd67459-dv9g7"]
Nov 03 20:05:11.846798 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:11.846777    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-fa9526-55fcd67459-dv9g7"]
Nov 03 20:05:11.855260 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:11.855238    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fa9526-7b98b75779-6gbh9"]
Nov 03 20:05:11.859827 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:11.859769    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fa9526-7b98b75779-6gbh9"]
Nov 03 20:05:12.919754 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:12.919720    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65b8374e-d983-45f0-8b27-8de1000e1f69" path="/var/lib/kubelet/pods/65b8374e-d983-45f0-8b27-8de1000e1f69/volumes"
Nov 03 20:05:12.920929 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:12.920906    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aeb289bd-62bd-45a1-9168-6e0e023e504f" path="/var/lib/kubelet/pods/aeb289bd-62bd-45a1-9168-6e0e023e504f/volumes"
Nov 03 20:05:13.291992 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:13.291951    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd"
Nov 03 20:05:13.366273 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:13.366037    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4736bbc2-086a-4d21-bafa-3abd36c25fa6-kube-api-access-w4vdp\" (UniqueName: \"kubernetes.io/projected/4736bbc2-086a-4d21-bafa-3abd36c25fa6-kube-api-access-w4vdp\") pod \"4736bbc2-086a-4d21-bafa-3abd36c25fa6\" (UID: \"4736bbc2-086a-4d21-bafa-3abd36c25fa6\") "
Nov 03 20:05:13.366273 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:13.366138    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4736bbc2-086a-4d21-bafa-3abd36c25fa6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4736bbc2-086a-4d21-bafa-3abd36c25fa6-workload-socket\") pod \"4736bbc2-086a-4d21-bafa-3abd36c25fa6\" (UID: \"4736bbc2-086a-4d21-bafa-3abd36c25fa6\") "
Nov 03 20:05:13.366273 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:13.366208    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4736bbc2-086a-4d21-bafa-3abd36c25fa6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4736bbc2-086a-4d21-bafa-3abd36c25fa6-credential-socket\") pod \"4736bbc2-086a-4d21-bafa-3abd36c25fa6\" (UID: \"4736bbc2-086a-4d21-bafa-3abd36c25fa6\") "
Nov 03 20:05:13.366273 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:13.366260    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4736bbc2-086a-4d21-bafa-3abd36c25fa6-istio-token\" (UniqueName: \"kubernetes.io/projected/4736bbc2-086a-4d21-bafa-3abd36c25fa6-istio-token\") pod \"4736bbc2-086a-4d21-bafa-3abd36c25fa6\" (UID: \"4736bbc2-086a-4d21-bafa-3abd36c25fa6\") "
Nov 03 20:05:13.366620 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:13.366297    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4736bbc2-086a-4d21-bafa-3abd36c25fa6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4736bbc2-086a-4d21-bafa-3abd36c25fa6-workload-certs\") pod \"4736bbc2-086a-4d21-bafa-3abd36c25fa6\" (UID: \"4736bbc2-086a-4d21-bafa-3abd36c25fa6\") "
Nov 03 20:05:13.366620 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:13.366349    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4736bbc2-086a-4d21-bafa-3abd36c25fa6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4736bbc2-086a-4d21-bafa-3abd36c25fa6-istiod-ca-cert\") pod \"4736bbc2-086a-4d21-bafa-3abd36c25fa6\" (UID: \"4736bbc2-086a-4d21-bafa-3abd36c25fa6\") "
Nov 03 20:05:13.366620 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:13.366468    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4736bbc2-086a-4d21-bafa-3abd36c25fa6-workload-socket" pod "4736bbc2-086a-4d21-bafa-3abd36c25fa6" (UID: "4736bbc2-086a-4d21-bafa-3abd36c25fa6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:05:13.366620 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:13.366504    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4736bbc2-086a-4d21-bafa-3abd36c25fa6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4736bbc2-086a-4d21-bafa-3abd36c25fa6-istio-envoy\") pod \"4736bbc2-086a-4d21-bafa-3abd36c25fa6\" (UID: \"4736bbc2-086a-4d21-bafa-3abd36c25fa6\") "
Nov 03 20:05:13.366620 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:13.366593    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4736bbc2-086a-4d21-bafa-3abd36c25fa6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4736bbc2-086a-4d21-bafa-3abd36c25fa6-istio-data\") pod \"4736bbc2-086a-4d21-bafa-3abd36c25fa6\" (UID: \"4736bbc2-086a-4d21-bafa-3abd36c25fa6\") "
Nov 03 20:05:13.366892 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:13.366645    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4736bbc2-086a-4d21-bafa-3abd36c25fa6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4736bbc2-086a-4d21-bafa-3abd36c25fa6-istio-podinfo\") pod \"4736bbc2-086a-4d21-bafa-3abd36c25fa6\" (UID: \"4736bbc2-086a-4d21-bafa-3abd36c25fa6\") "
Nov 03 20:05:13.370450 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:13.367328    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4736bbc2-086a-4d21-bafa-3abd36c25fa6-istiod-ca-cert" pod "4736bbc2-086a-4d21-bafa-3abd36c25fa6" (UID: "4736bbc2-086a-4d21-bafa-3abd36c25fa6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:05:13.370450 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:13.367797    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4736bbc2-086a-4d21-bafa-3abd36c25fa6-workload-certs" pod "4736bbc2-086a-4d21-bafa-3abd36c25fa6" (UID: "4736bbc2-086a-4d21-bafa-3abd36c25fa6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:05:13.370450 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:13.368349    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4736bbc2-086a-4d21-bafa-3abd36c25fa6-istio-data" pod "4736bbc2-086a-4d21-bafa-3abd36c25fa6" (UID: "4736bbc2-086a-4d21-bafa-3abd36c25fa6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:05:13.370450 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:13.368378    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4736bbc2-086a-4d21-bafa-3abd36c25fa6-credential-socket" pod "4736bbc2-086a-4d21-bafa-3abd36c25fa6" (UID: "4736bbc2-086a-4d21-bafa-3abd36c25fa6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:05:13.370450 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:13.368843    4921 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4736bbc2-086a-4d21-bafa-3abd36c25fa6-istio-data\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:13.370450 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:13.368869    4921 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4736bbc2-086a-4d21-bafa-3abd36c25fa6-workload-socket\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:13.370450 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:13.368884    4921 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4736bbc2-086a-4d21-bafa-3abd36c25fa6-credential-socket\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:13.370450 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:13.368899    4921 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4736bbc2-086a-4d21-bafa-3abd36c25fa6-workload-certs\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:13.370450 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:13.368929    4921 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4736bbc2-086a-4d21-bafa-3abd36c25fa6-istiod-ca-cert\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:13.372461 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:13.372431    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4736bbc2-086a-4d21-bafa-3abd36c25fa6-istio-envoy" pod "4736bbc2-086a-4d21-bafa-3abd36c25fa6" (UID: "4736bbc2-086a-4d21-bafa-3abd36c25fa6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:05:13.372899 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:13.372876    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4736bbc2-086a-4d21-bafa-3abd36c25fa6-istio-podinfo" pod "4736bbc2-086a-4d21-bafa-3abd36c25fa6" (UID: "4736bbc2-086a-4d21-bafa-3abd36c25fa6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:05:13.375870 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:13.375843    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4736bbc2-086a-4d21-bafa-3abd36c25fa6-kube-api-access-w4vdp" pod "4736bbc2-086a-4d21-bafa-3abd36c25fa6" (UID: "4736bbc2-086a-4d21-bafa-3abd36c25fa6"). InnerVolumeSpecName "kube-api-access-w4vdp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:05:13.376291 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:13.376272    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4736bbc2-086a-4d21-bafa-3abd36c25fa6-istio-token" pod "4736bbc2-086a-4d21-bafa-3abd36c25fa6" (UID: "4736bbc2-086a-4d21-bafa-3abd36c25fa6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:05:13.471041 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:13.470919    4921 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4736bbc2-086a-4d21-bafa-3abd36c25fa6-istio-envoy\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:13.471041 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:13.470960    4921 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4736bbc2-086a-4d21-bafa-3abd36c25fa6-istio-podinfo\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:13.471041 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:13.470978    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4vdp\" (UniqueName: \"kubernetes.io/projected/4736bbc2-086a-4d21-bafa-3abd36c25fa6-kube-api-access-w4vdp\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:13.471041 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:13.470993    4921 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4736bbc2-086a-4d21-bafa-3abd36c25fa6-istio-token\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:13.822484 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:13.822447    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3cee6a-544fd5876c-56hwf" podUID="26379d10-3da7-4549-8b8c-41730a41792b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:05:13.823369 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:13.823339    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-3cee6a-799d7c674c-fj9vk" podUID="4aa9c292-f06c-45d9-9a58-8aeae3b7b6ab" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:05:13.838234 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:13.838209    4921 generic.go:356] "Generic (PLEG): container finished" podID="4736bbc2-086a-4d21-bafa-3abd36c25fa6" containerID="1d7ffece2c1e3a1e71c9c10f7d5445f3702abd027a7601aefc4b0c0483872c2a" exitCode=0
Nov 03 20:05:13.838360 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:13.838256    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd"
Nov 03 20:05:13.838360 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:13.838295    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd" event={"ID":"4736bbc2-086a-4d21-bafa-3abd36c25fa6","Type":"ContainerDied","Data":"1d7ffece2c1e3a1e71c9c10f7d5445f3702abd027a7601aefc4b0c0483872c2a"}
Nov 03 20:05:13.838360 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:13.838349    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd" event={"ID":"4736bbc2-086a-4d21-bafa-3abd36c25fa6","Type":"ContainerDied","Data":"f18e447ad99cf87d0e8eda8c656f269a66145188b266a0658f85cf9aadca90a2"}
Nov 03 20:05:13.838466 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:13.838378    4921 scope.go:122] "RemoveContainer" containerID="1d7ffece2c1e3a1e71c9c10f7d5445f3702abd027a7601aefc4b0c0483872c2a"
Nov 03 20:05:13.850422 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:13.850404    4921 scope.go:122] "RemoveContainer" containerID="1d7ffece2c1e3a1e71c9c10f7d5445f3702abd027a7601aefc4b0c0483872c2a"
Nov 03 20:05:13.850745 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:05:13.850713    4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d7ffece2c1e3a1e71c9c10f7d5445f3702abd027a7601aefc4b0c0483872c2a\": container with ID starting with 1d7ffece2c1e3a1e71c9c10f7d5445f3702abd027a7601aefc4b0c0483872c2a not found: ID does not exist" containerID="1d7ffece2c1e3a1e71c9c10f7d5445f3702abd027a7601aefc4b0c0483872c2a"
Nov 03 20:05:13.850834 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:13.850754    4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d7ffece2c1e3a1e71c9c10f7d5445f3702abd027a7601aefc4b0c0483872c2a"} err="failed to get container status \"1d7ffece2c1e3a1e71c9c10f7d5445f3702abd027a7601aefc4b0c0483872c2a\": rpc error: code = NotFound desc = could not find container \"1d7ffece2c1e3a1e71c9c10f7d5445f3702abd027a7601aefc4b0c0483872c2a\": container with ID starting with 1d7ffece2c1e3a1e71c9c10f7d5445f3702abd027a7601aefc4b0c0483872c2a not found: ID does not exist"
Nov 03 20:05:13.865236 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:13.865200    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd"]
Nov 03 20:05:13.868752 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:13.868733    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-fcddeb-openshift-default-55c6ddb6f-p26jd"]
Nov 03 20:05:14.889202 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:14.889171    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4736bbc2-086a-4d21-bafa-3abd36c25fa6" path="/var/lib/kubelet/pods/4736bbc2-086a-4d21-bafa-3abd36c25fa6/volumes"
Nov 03 20:05:14.915290 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:14.915259    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3cee6a-5f99576765-9s4g2"]
Nov 03 20:05:14.915508 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:14.915482    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3cee6a-5f99576765-9s4g2" podUID="9a2ce54f-1cb3-47b0-aa6e-e8b699ea8241" containerName="maas-api" containerID="cri-o://b10ad26b42709406642991c89e224e16e03f2e8e7be48ee478ca64bd25ed2d21" gracePeriod=30
Nov 03 20:05:15.677711 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:15.677684    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3cee6a-5f99576765-9s4g2"
Nov 03 20:05:15.817282 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:15.817243    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a2ce54f-1cb3-47b0-aa6e-e8b699ea8241-kube-api-access-bx72n\" (UniqueName: \"kubernetes.io/projected/9a2ce54f-1cb3-47b0-aa6e-e8b699ea8241-kube-api-access-bx72n\") pod \"9a2ce54f-1cb3-47b0-aa6e-e8b699ea8241\" (UID: \"9a2ce54f-1cb3-47b0-aa6e-e8b699ea8241\") "
Nov 03 20:05:15.817474 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:15.817326    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a2ce54f-1cb3-47b0-aa6e-e8b699ea8241-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a2ce54f-1cb3-47b0-aa6e-e8b699ea8241-metrics-certs\") pod \"9a2ce54f-1cb3-47b0-aa6e-e8b699ea8241\" (UID: \"9a2ce54f-1cb3-47b0-aa6e-e8b699ea8241\") "
Nov 03 20:05:15.817474 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:15.817440    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a2ce54f-1cb3-47b0-aa6e-e8b699ea8241-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a2ce54f-1cb3-47b0-aa6e-e8b699ea8241-maas-api-tls\") pod \"9a2ce54f-1cb3-47b0-aa6e-e8b699ea8241\" (UID: \"9a2ce54f-1cb3-47b0-aa6e-e8b699ea8241\") "
Nov 03 20:05:15.817611 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:15.817525    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a2ce54f-1cb3-47b0-aa6e-e8b699ea8241-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a2ce54f-1cb3-47b0-aa6e-e8b699ea8241-openshift-service-ca\") pod \"9a2ce54f-1cb3-47b0-aa6e-e8b699ea8241\" (UID: \"9a2ce54f-1cb3-47b0-aa6e-e8b699ea8241\") "
Nov 03 20:05:15.817960 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:15.817925    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a2ce54f-1cb3-47b0-aa6e-e8b699ea8241-openshift-service-ca" pod "9a2ce54f-1cb3-47b0-aa6e-e8b699ea8241" (UID: "9a2ce54f-1cb3-47b0-aa6e-e8b699ea8241"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:05:15.818257 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:15.818230    4921 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a2ce54f-1cb3-47b0-aa6e-e8b699ea8241-openshift-service-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:15.819826 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:15.819793    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a2ce54f-1cb3-47b0-aa6e-e8b699ea8241-metrics-certs" pod "9a2ce54f-1cb3-47b0-aa6e-e8b699ea8241" (UID: "9a2ce54f-1cb3-47b0-aa6e-e8b699ea8241"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:05:15.820024 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:15.820001    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a2ce54f-1cb3-47b0-aa6e-e8b699ea8241-maas-api-tls" pod "9a2ce54f-1cb3-47b0-aa6e-e8b699ea8241" (UID: "9a2ce54f-1cb3-47b0-aa6e-e8b699ea8241"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:05:15.820587 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:15.820562    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a2ce54f-1cb3-47b0-aa6e-e8b699ea8241-kube-api-access-bx72n" pod "9a2ce54f-1cb3-47b0-aa6e-e8b699ea8241" (UID: "9a2ce54f-1cb3-47b0-aa6e-e8b699ea8241"). InnerVolumeSpecName "kube-api-access-bx72n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:05:15.857898 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:15.857866    4921 generic.go:356] "Generic (PLEG): container finished" podID="9a2ce54f-1cb3-47b0-aa6e-e8b699ea8241" containerID="b10ad26b42709406642991c89e224e16e03f2e8e7be48ee478ca64bd25ed2d21" exitCode=0
Nov 03 20:05:15.858041 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:15.857941    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3cee6a-5f99576765-9s4g2"
Nov 03 20:05:15.858041 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:15.857950    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3cee6a-5f99576765-9s4g2" event={"ID":"9a2ce54f-1cb3-47b0-aa6e-e8b699ea8241","Type":"ContainerDied","Data":"b10ad26b42709406642991c89e224e16e03f2e8e7be48ee478ca64bd25ed2d21"}
Nov 03 20:05:15.858153 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:15.858042    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3cee6a-5f99576765-9s4g2" event={"ID":"9a2ce54f-1cb3-47b0-aa6e-e8b699ea8241","Type":"ContainerDied","Data":"dbb222211d8b38cd14a1fff35b9a25e31d49507e5915ee0e9d4cec99dab3f062"}
Nov 03 20:05:15.858153 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:15.858074    4921 scope.go:122] "RemoveContainer" containerID="b10ad26b42709406642991c89e224e16e03f2e8e7be48ee478ca64bd25ed2d21"
Nov 03 20:05:15.871653 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:15.871632    4921 scope.go:122] "RemoveContainer" containerID="b10ad26b42709406642991c89e224e16e03f2e8e7be48ee478ca64bd25ed2d21"
Nov 03 20:05:15.871988 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:05:15.871964    4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b10ad26b42709406642991c89e224e16e03f2e8e7be48ee478ca64bd25ed2d21\": container with ID starting with b10ad26b42709406642991c89e224e16e03f2e8e7be48ee478ca64bd25ed2d21 not found: ID does not exist" containerID="b10ad26b42709406642991c89e224e16e03f2e8e7be48ee478ca64bd25ed2d21"
Nov 03 20:05:15.872057 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:15.872003    4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b10ad26b42709406642991c89e224e16e03f2e8e7be48ee478ca64bd25ed2d21"} err="failed to get container status \"b10ad26b42709406642991c89e224e16e03f2e8e7be48ee478ca64bd25ed2d21\": rpc error: code = NotFound desc = could not find container \"b10ad26b42709406642991c89e224e16e03f2e8e7be48ee478ca64bd25ed2d21\": container with ID starting with b10ad26b42709406642991c89e224e16e03f2e8e7be48ee478ca64bd25ed2d21 not found: ID does not exist"
Nov 03 20:05:15.886132 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:15.886069    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3cee6a-5f99576765-9s4g2"]
Nov 03 20:05:15.890175 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:15.890154    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3cee6a-5f99576765-9s4g2"]
Nov 03 20:05:15.919416 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:15.919382    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bx72n\" (UniqueName: \"kubernetes.io/projected/9a2ce54f-1cb3-47b0-aa6e-e8b699ea8241-kube-api-access-bx72n\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:15.919416 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:15.919414    4921 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a2ce54f-1cb3-47b0-aa6e-e8b699ea8241-metrics-certs\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:15.919416 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:15.919424    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a2ce54f-1cb3-47b0-aa6e-e8b699ea8241-maas-api-tls\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:16.890137 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:16.890107    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a2ce54f-1cb3-47b0-aa6e-e8b699ea8241" path="/var/lib/kubelet/pods/9a2ce54f-1cb3-47b0-aa6e-e8b699ea8241/volumes"
Nov 03 20:05:22.986636 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:22.986601    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c"]
Nov 03 20:05:22.987220 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:22.987207    4921 cpu_manager.go:414] "containerMap: removing container" podUID="5ba7b96b-f2c0-4ddf-8117-c578d6ba1d69" containerName="payload-pre-processing"
Nov 03 20:05:22.987267 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:22.987222    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ba7b96b-f2c0-4ddf-8117-c578d6ba1d69" containerName="payload-pre-processing"
Nov 03 20:05:22.987267 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:22.987237    4921 cpu_manager.go:414] "containerMap: removing container" podUID="09a4d5b2-79a2-4149-99a9-5447b1999110" containerName="istio-proxy"
Nov 03 20:05:22.987267 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:22.987243    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09a4d5b2-79a2-4149-99a9-5447b1999110" containerName="istio-proxy"
Nov 03 20:05:22.987267 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:22.987256    4921 cpu_manager.go:414] "containerMap: removing container" podUID="204760c8-b3c3-4b1d-8b67-5bb6cf918407" containerName="istio-proxy"
Nov 03 20:05:22.987267 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:22.987261    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="204760c8-b3c3-4b1d-8b67-5bb6cf918407" containerName="istio-proxy"
Nov 03 20:05:22.987450 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:22.987269    4921 cpu_manager.go:414] "containerMap: removing container" podUID="40248459-fffd-4588-94fd-5569ee6a9e8e" containerName="istio-proxy"
Nov 03 20:05:22.987450 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:22.987277    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40248459-fffd-4588-94fd-5569ee6a9e8e" containerName="istio-proxy"
Nov 03 20:05:22.987450 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:22.987286    4921 cpu_manager.go:414] "containerMap: removing container" podUID="1068a9b9-6312-4246-b139-679922ea7a5a" containerName="maas-api"
Nov 03 20:05:22.987450 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:22.987291    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1068a9b9-6312-4246-b139-679922ea7a5a" containerName="maas-api"
Nov 03 20:05:22.987450 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:22.987298    4921 cpu_manager.go:414] "containerMap: removing container" podUID="aeb289bd-62bd-45a1-9168-6e0e023e504f" containerName="payload-pre-processing"
Nov 03 20:05:22.987450 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:22.987303    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aeb289bd-62bd-45a1-9168-6e0e023e504f" containerName="payload-pre-processing"
Nov 03 20:05:22.987450 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:22.987312    4921 cpu_manager.go:414] "containerMap: removing container" podUID="9a2ce54f-1cb3-47b0-aa6e-e8b699ea8241" containerName="maas-api"
Nov 03 20:05:22.987450 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:22.987317    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a2ce54f-1cb3-47b0-aa6e-e8b699ea8241" containerName="maas-api"
Nov 03 20:05:22.987450 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:22.987333    4921 cpu_manager.go:414] "containerMap: removing container" podUID="4736bbc2-086a-4d21-bafa-3abd36c25fa6" containerName="istio-proxy"
Nov 03 20:05:22.987450 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:22.987338    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4736bbc2-086a-4d21-bafa-3abd36c25fa6" containerName="istio-proxy"
Nov 03 20:05:22.987450 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:22.987348    4921 cpu_manager.go:414] "containerMap: removing container" podUID="33d97e3e-9c11-4ca2-bddd-b2c057ab41e9" containerName="istio-proxy"
Nov 03 20:05:22.987450 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:22.987352    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33d97e3e-9c11-4ca2-bddd-b2c057ab41e9" containerName="istio-proxy"
Nov 03 20:05:22.987450 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:22.987361    4921 cpu_manager.go:414] "containerMap: removing container" podUID="65b8374e-d983-45f0-8b27-8de1000e1f69" containerName="payload-processing"
Nov 03 20:05:22.987450 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:22.987365    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65b8374e-d983-45f0-8b27-8de1000e1f69" containerName="payload-processing"
Nov 03 20:05:22.987450 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:22.987373    4921 cpu_manager.go:414] "containerMap: removing container" podUID="bb55b062-fcb5-4fac-8089-09ad9549b7f9" containerName="istio-proxy"
Nov 03 20:05:22.987450 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:22.987377    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb55b062-fcb5-4fac-8089-09ad9549b7f9" containerName="istio-proxy"
Nov 03 20:05:22.987948 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:22.987473    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="4736bbc2-086a-4d21-bafa-3abd36c25fa6" containerName="istio-proxy"
Nov 03 20:05:22.987948 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:22.987483    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="33d97e3e-9c11-4ca2-bddd-b2c057ab41e9" containerName="istio-proxy"
Nov 03 20:05:22.987948 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:22.987489    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="65b8374e-d983-45f0-8b27-8de1000e1f69" containerName="payload-processing"
Nov 03 20:05:22.987948 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:22.987496    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb55b062-fcb5-4fac-8089-09ad9549b7f9" containerName="istio-proxy"
Nov 03 20:05:22.987948 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:22.987506    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ba7b96b-f2c0-4ddf-8117-c578d6ba1d69" containerName="payload-pre-processing"
Nov 03 20:05:22.987948 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:22.987512    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="09a4d5b2-79a2-4149-99a9-5447b1999110" containerName="istio-proxy"
Nov 03 20:05:22.987948 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:22.987518    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="204760c8-b3c3-4b1d-8b67-5bb6cf918407" containerName="istio-proxy"
Nov 03 20:05:22.987948 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:22.987524    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="40248459-fffd-4588-94fd-5569ee6a9e8e" containerName="istio-proxy"
Nov 03 20:05:22.987948 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:22.987530    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="1068a9b9-6312-4246-b139-679922ea7a5a" containerName="maas-api"
Nov 03 20:05:22.987948 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:22.987537    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="aeb289bd-62bd-45a1-9168-6e0e023e504f" containerName="payload-pre-processing"
Nov 03 20:05:22.987948 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:22.987542    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a2ce54f-1cb3-47b0-aa6e-e8b699ea8241" containerName="maas-api"
Nov 03 20:05:22.991166 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:22.991146    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c"
Nov 03 20:05:22.996998 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:22.996971    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-dce558-openshift-default-dockercfg-rshjr\""
Nov 03 20:05:23.002274 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:23.002248    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c"]
Nov 03 20:05:23.132211 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:23.132173    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b9000a2-e11e-4afb-9ea7-4e4f1446d918-credential-socket\") pod \"e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c\" (UID: \"1b9000a2-e11e-4afb-9ea7-4e4f1446d918\") " pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c"
Nov 03 20:05:23.132364 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:23.132232    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b9000a2-e11e-4afb-9ea7-4e4f1446d918-istio-token\") pod \"e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c\" (UID: \"1b9000a2-e11e-4afb-9ea7-4e4f1446d918\") " pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c"
Nov 03 20:05:23.132364 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:23.132341    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b9000a2-e11e-4afb-9ea7-4e4f1446d918-istio-data\") pod \"e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c\" (UID: \"1b9000a2-e11e-4afb-9ea7-4e4f1446d918\") " pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c"
Nov 03 20:05:23.132458 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:23.132387    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b9000a2-e11e-4afb-9ea7-4e4f1446d918-istiod-ca-cert\") pod \"e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c\" (UID: \"1b9000a2-e11e-4afb-9ea7-4e4f1446d918\") " pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c"
Nov 03 20:05:23.132458 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:23.132415    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57pth\" (UniqueName: \"kubernetes.io/projected/1b9000a2-e11e-4afb-9ea7-4e4f1446d918-kube-api-access-57pth\") pod \"e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c\" (UID: \"1b9000a2-e11e-4afb-9ea7-4e4f1446d918\") " pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c"
Nov 03 20:05:23.132520 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:23.132489    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b9000a2-e11e-4afb-9ea7-4e4f1446d918-istio-envoy\") pod \"e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c\" (UID: \"1b9000a2-e11e-4afb-9ea7-4e4f1446d918\") " pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c"
Nov 03 20:05:23.132520 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:23.132517    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b9000a2-e11e-4afb-9ea7-4e4f1446d918-workload-certs\") pod \"e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c\" (UID: \"1b9000a2-e11e-4afb-9ea7-4e4f1446d918\") " pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c"
Nov 03 20:05:23.132586 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:23.132541    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b9000a2-e11e-4afb-9ea7-4e4f1446d918-workload-socket\") pod \"e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c\" (UID: \"1b9000a2-e11e-4afb-9ea7-4e4f1446d918\") " pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c"
Nov 03 20:05:23.132713 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:23.132699    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b9000a2-e11e-4afb-9ea7-4e4f1446d918-istio-podinfo\") pod \"e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c\" (UID: \"1b9000a2-e11e-4afb-9ea7-4e4f1446d918\") " pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c"
Nov 03 20:05:23.234342 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:23.234305    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b9000a2-e11e-4afb-9ea7-4e4f1446d918-istio-data\") pod \"e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c\" (UID: \"1b9000a2-e11e-4afb-9ea7-4e4f1446d918\") " pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c"
Nov 03 20:05:23.234540 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:23.234354    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b9000a2-e11e-4afb-9ea7-4e4f1446d918-istiod-ca-cert\") pod \"e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c\" (UID: \"1b9000a2-e11e-4afb-9ea7-4e4f1446d918\") " pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c"
Nov 03 20:05:23.234540 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:23.234467    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57pth\" (UniqueName: \"kubernetes.io/projected/1b9000a2-e11e-4afb-9ea7-4e4f1446d918-kube-api-access-57pth\") pod \"e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c\" (UID: \"1b9000a2-e11e-4afb-9ea7-4e4f1446d918\") " pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c"
Nov 03 20:05:23.234789 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:23.234753    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b9000a2-e11e-4afb-9ea7-4e4f1446d918-istio-envoy\") pod \"e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c\" (UID: \"1b9000a2-e11e-4afb-9ea7-4e4f1446d918\") " pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c"
Nov 03 20:05:23.234939 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:23.234797    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b9000a2-e11e-4afb-9ea7-4e4f1446d918-workload-certs\") pod \"e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c\" (UID: \"1b9000a2-e11e-4afb-9ea7-4e4f1446d918\") " pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c"
Nov 03 20:05:23.234939 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:23.234840    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b9000a2-e11e-4afb-9ea7-4e4f1446d918-workload-socket\") pod \"e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c\" (UID: \"1b9000a2-e11e-4afb-9ea7-4e4f1446d918\") " pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c"
Nov 03 20:05:23.234939 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:23.234885    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b9000a2-e11e-4afb-9ea7-4e4f1446d918-istio-data\") pod \"e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c\" (UID: \"1b9000a2-e11e-4afb-9ea7-4e4f1446d918\") " pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c"
Nov 03 20:05:23.235237 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:23.235007    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b9000a2-e11e-4afb-9ea7-4e4f1446d918-istio-podinfo\") pod \"e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c\" (UID: \"1b9000a2-e11e-4afb-9ea7-4e4f1446d918\") " pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c"
Nov 03 20:05:23.235237 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:23.235093    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b9000a2-e11e-4afb-9ea7-4e4f1446d918-credential-socket\") pod \"e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c\" (UID: \"1b9000a2-e11e-4afb-9ea7-4e4f1446d918\") " pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c"
Nov 03 20:05:23.235237 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:23.235188    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b9000a2-e11e-4afb-9ea7-4e4f1446d918-istio-token\") pod \"e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c\" (UID: \"1b9000a2-e11e-4afb-9ea7-4e4f1446d918\") " pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c"
Nov 03 20:05:23.235402 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:23.235286    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b9000a2-e11e-4afb-9ea7-4e4f1446d918-istiod-ca-cert\") pod \"e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c\" (UID: \"1b9000a2-e11e-4afb-9ea7-4e4f1446d918\") " pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c"
Nov 03 20:05:23.235402 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:23.235360    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b9000a2-e11e-4afb-9ea7-4e4f1446d918-credential-socket\") pod \"e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c\" (UID: \"1b9000a2-e11e-4afb-9ea7-4e4f1446d918\") " pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c"
Nov 03 20:05:23.235521 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:23.235481    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b9000a2-e11e-4afb-9ea7-4e4f1446d918-workload-socket\") pod \"e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c\" (UID: \"1b9000a2-e11e-4afb-9ea7-4e4f1446d918\") " pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c"
Nov 03 20:05:23.235777 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:23.235733    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b9000a2-e11e-4afb-9ea7-4e4f1446d918-workload-certs\") pod \"e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c\" (UID: \"1b9000a2-e11e-4afb-9ea7-4e4f1446d918\") " pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c"
Nov 03 20:05:23.237414 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:23.237348    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b9000a2-e11e-4afb-9ea7-4e4f1446d918-istio-envoy\") pod \"e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c\" (UID: \"1b9000a2-e11e-4afb-9ea7-4e4f1446d918\") " pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c"
Nov 03 20:05:23.238140 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:23.238089    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b9000a2-e11e-4afb-9ea7-4e4f1446d918-istio-podinfo\") pod \"e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c\" (UID: \"1b9000a2-e11e-4afb-9ea7-4e4f1446d918\") " pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c"
Nov 03 20:05:23.242180 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:23.242157    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b9000a2-e11e-4afb-9ea7-4e4f1446d918-istio-token\") pod \"e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c\" (UID: \"1b9000a2-e11e-4afb-9ea7-4e4f1446d918\") " pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c"
Nov 03 20:05:23.242568 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:23.242550    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57pth\" (UniqueName: \"kubernetes.io/projected/1b9000a2-e11e-4afb-9ea7-4e4f1446d918-kube-api-access-57pth\") pod \"e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c\" (UID: \"1b9000a2-e11e-4afb-9ea7-4e4f1446d918\") " pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c"
Nov 03 20:05:23.308685 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:23.308640    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c"
Nov 03 20:05:23.594498 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:23.594474    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c"]
Nov 03 20:05:23.595757 ip-10-0-134-222 kubenswrapper[4921]: W1103 20:05:23.595729    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b9000a2_e11e_4afb_9ea7_4e4f1446d918.slice/crio-74656a60a921f88ff54820ddced059200af6a8ff5568952ecbac63e0f4ddc638 WatchSource:0}: Error finding container 74656a60a921f88ff54820ddced059200af6a8ff5568952ecbac63e0f4ddc638: Status 404 returned error can't find the container with id 74656a60a921f88ff54820ddced059200af6a8ff5568952ecbac63e0f4ddc638
Nov 03 20:05:23.598248 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:23.598215    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:05:23.598346 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:23.598279    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:05:23.598346 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:23.598311    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:05:23.822458 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:23.822419    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-3cee6a-799d7c674c-fj9vk" podUID="4aa9c292-f06c-45d9-9a58-8aeae3b7b6ab" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:05:23.822662 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:23.822533    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3cee6a-544fd5876c-56hwf" podUID="26379d10-3da7-4549-8b8c-41730a41792b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:05:23.948745 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:23.948645    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c" event={"ID":"1b9000a2-e11e-4afb-9ea7-4e4f1446d918","Type":"ContainerStarted","Data":"7ccd826447701663e2f015beb307be7f4751b6292b68aba7f43b41b959fcbe92"}
Nov 03 20:05:23.948893 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:23.948742    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c" event={"ID":"1b9000a2-e11e-4afb-9ea7-4e4f1446d918","Type":"ContainerStarted","Data":"74656a60a921f88ff54820ddced059200af6a8ff5568952ecbac63e0f4ddc638"}
Nov 03 20:05:23.973939 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:23.973884    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c" podStartSLOduration=1.973869658 podStartE2EDuration="1.973869658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:05:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:05:23.969663799 +0000 UTC m=+2037.213629263" watchObservedRunningTime="2026-11-03 20:05:23.973869658 +0000 UTC m=+2037.217835122"
Nov 03 20:05:24.309763 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:24.309720    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c"
Nov 03 20:05:24.311135 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:24.311115    4921 patch_prober.go:29] interesting pod/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Nov 03 20:05:24.311204 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:24.311167    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c" podUID="1b9000a2-e11e-4afb-9ea7-4e4f1446d918" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Nov 03 20:05:25.309746 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:25.309639    4921 patch_prober.go:29] interesting pod/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Nov 03 20:05:25.310304 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:25.310264    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c" podUID="1b9000a2-e11e-4afb-9ea7-4e4f1446d918" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Nov 03 20:05:25.722746 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:25.722631    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2d78ea-6496b8c67-4wtxz"]
Nov 03 20:05:25.723197 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:25.723161    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2d78ea-6496b8c67-4wtxz" podUID="d1fd32d8-1484-4622-86a3-24b16903084b" containerName="maas-api" containerID="cri-o://50f3cc2ac0d7bfafdb4640589b9735df46d142135165830a93db9e8624a81f93" gracePeriod=30
Nov 03 20:05:26.002002 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:26.001963    4921 generic.go:356] "Generic (PLEG): container finished" podID="d1fd32d8-1484-4622-86a3-24b16903084b" containerID="50f3cc2ac0d7bfafdb4640589b9735df46d142135165830a93db9e8624a81f93" exitCode=0
Nov 03 20:05:26.003140 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:26.003108    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2d78ea-6496b8c67-4wtxz" event={"ID":"d1fd32d8-1484-4622-86a3-24b16903084b","Type":"ContainerDied","Data":"50f3cc2ac0d7bfafdb4640589b9735df46d142135165830a93db9e8624a81f93"}
Nov 03 20:05:26.312269 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:26.311008    4921 patch_prober.go:29] interesting pod/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Nov 03 20:05:26.312269 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:26.311067    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c" podUID="1b9000a2-e11e-4afb-9ea7-4e4f1446d918" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Nov 03 20:05:26.346261 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:26.346225    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2d78ea-6496b8c67-4wtxz"
Nov 03 20:05:26.513316 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:26.513279    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1fd32d8-1484-4622-86a3-24b16903084b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1fd32d8-1484-4622-86a3-24b16903084b-metrics-certs\") pod \"d1fd32d8-1484-4622-86a3-24b16903084b\" (UID: \"d1fd32d8-1484-4622-86a3-24b16903084b\") "
Nov 03 20:05:26.513493 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:26.513389    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1fd32d8-1484-4622-86a3-24b16903084b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1fd32d8-1484-4622-86a3-24b16903084b-maas-api-tls\") pod \"d1fd32d8-1484-4622-86a3-24b16903084b\" (UID: \"d1fd32d8-1484-4622-86a3-24b16903084b\") "
Nov 03 20:05:26.513493 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:26.513477    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1fd32d8-1484-4622-86a3-24b16903084b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1fd32d8-1484-4622-86a3-24b16903084b-openshift-service-ca\") pod \"d1fd32d8-1484-4622-86a3-24b16903084b\" (UID: \"d1fd32d8-1484-4622-86a3-24b16903084b\") "
Nov 03 20:05:26.513624 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:26.513544    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1fd32d8-1484-4622-86a3-24b16903084b-kube-api-access-4zvhh\" (UniqueName: \"kubernetes.io/projected/d1fd32d8-1484-4622-86a3-24b16903084b-kube-api-access-4zvhh\") pod \"d1fd32d8-1484-4622-86a3-24b16903084b\" (UID: \"d1fd32d8-1484-4622-86a3-24b16903084b\") "
Nov 03 20:05:26.513906 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:26.513870    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1fd32d8-1484-4622-86a3-24b16903084b-openshift-service-ca" pod "d1fd32d8-1484-4622-86a3-24b16903084b" (UID: "d1fd32d8-1484-4622-86a3-24b16903084b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:05:26.514244 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:26.514218    4921 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1fd32d8-1484-4622-86a3-24b16903084b-openshift-service-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:26.515595 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:26.515566    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1fd32d8-1484-4622-86a3-24b16903084b-metrics-certs" pod "d1fd32d8-1484-4622-86a3-24b16903084b" (UID: "d1fd32d8-1484-4622-86a3-24b16903084b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:05:26.515702 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:26.515594    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1fd32d8-1484-4622-86a3-24b16903084b-maas-api-tls" pod "d1fd32d8-1484-4622-86a3-24b16903084b" (UID: "d1fd32d8-1484-4622-86a3-24b16903084b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:05:26.515762 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:26.515727    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1fd32d8-1484-4622-86a3-24b16903084b-kube-api-access-4zvhh" pod "d1fd32d8-1484-4622-86a3-24b16903084b" (UID: "d1fd32d8-1484-4622-86a3-24b16903084b"). InnerVolumeSpecName "kube-api-access-4zvhh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:05:26.621051 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:26.620947    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1fd32d8-1484-4622-86a3-24b16903084b-maas-api-tls\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:26.621321 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:26.621302    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zvhh\" (UniqueName: \"kubernetes.io/projected/d1fd32d8-1484-4622-86a3-24b16903084b-kube-api-access-4zvhh\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:26.621424 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:26.621412    4921 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1fd32d8-1484-4622-86a3-24b16903084b-metrics-certs\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:27.011517 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:27.011487    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2d78ea-6496b8c67-4wtxz" event={"ID":"d1fd32d8-1484-4622-86a3-24b16903084b","Type":"ContainerDied","Data":"45bad26b45693de9d9cf3c88a4f991a75c4ffa7147a762dc8d20f4d5cfaed37a"}
Nov 03 20:05:27.011701 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:27.011527    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2d78ea-6496b8c67-4wtxz"
Nov 03 20:05:27.011701 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:27.011546    4921 scope.go:122] "RemoveContainer" containerID="50f3cc2ac0d7bfafdb4640589b9735df46d142135165830a93db9e8624a81f93"
Nov 03 20:05:27.029464 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:27.029439    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2d78ea-6496b8c67-4wtxz"]
Nov 03 20:05:27.033160 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:27.033131    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-2d78ea-6496b8c67-4wtxz"]
Nov 03 20:05:27.309489 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:27.309453    4921 patch_prober.go:29] interesting pod/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Nov 03 20:05:27.309684 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:27.309519    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c" podUID="1b9000a2-e11e-4afb-9ea7-4e4f1446d918" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Nov 03 20:05:28.309278 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:28.309244    4921 patch_prober.go:29] interesting pod/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Nov 03 20:05:28.309715 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:28.309310    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c" podUID="1b9000a2-e11e-4afb-9ea7-4e4f1446d918" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Nov 03 20:05:28.889596 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:28.889519    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1fd32d8-1484-4622-86a3-24b16903084b" path="/var/lib/kubelet/pods/d1fd32d8-1484-4622-86a3-24b16903084b/volumes"
Nov 03 20:05:29.039554 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:29.039328    4921 generic.go:356] "Generic (PLEG): container finished" podID="26379d10-3da7-4549-8b8c-41730a41792b" containerID="03ff696cb9d480ac235973dcde8e5a58b5d22ac3a8144b098460f74873cc28a4" exitCode=0
Nov 03 20:05:29.039554 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:29.039465    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3cee6a-544fd5876c-56hwf" event={"ID":"26379d10-3da7-4549-8b8c-41730a41792b","Type":"ContainerDied","Data":"03ff696cb9d480ac235973dcde8e5a58b5d22ac3a8144b098460f74873cc28a4"}
Nov 03 20:05:29.041705 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:29.041660    4921 generic.go:356] "Generic (PLEG): container finished" podID="4aa9c292-f06c-45d9-9a58-8aeae3b7b6ab" containerID="c22dd48c7c79a0e292ef4e7ee62a718aa5c150899e124577e4bea7f0abf2ea3c" exitCode=0
Nov 03 20:05:29.041814 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:29.041714    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3cee6a-799d7c674c-fj9vk" event={"ID":"4aa9c292-f06c-45d9-9a58-8aeae3b7b6ab","Type":"ContainerDied","Data":"c22dd48c7c79a0e292ef4e7ee62a718aa5c150899e124577e4bea7f0abf2ea3c"}
Nov 03 20:05:29.127382 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:29.127350    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3cee6a-799d7c674c-fj9vk"
Nov 03 20:05:29.205412 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:29.205387    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3cee6a-544fd5876c-56hwf"
Nov 03 20:05:29.290319 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:29.290277    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4aa9c292-f06c-45d9-9a58-8aeae3b7b6ab-kube-api-access-zngl8\" (UniqueName: \"kubernetes.io/projected/4aa9c292-f06c-45d9-9a58-8aeae3b7b6ab-kube-api-access-zngl8\") pod \"4aa9c292-f06c-45d9-9a58-8aeae3b7b6ab\" (UID: \"4aa9c292-f06c-45d9-9a58-8aeae3b7b6ab\") "
Nov 03 20:05:29.290524 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:29.290440    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4aa9c292-f06c-45d9-9a58-8aeae3b7b6ab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4aa9c292-f06c-45d9-9a58-8aeae3b7b6ab-plugins-config-volume\") pod \"4aa9c292-f06c-45d9-9a58-8aeae3b7b6ab\" (UID: \"4aa9c292-f06c-45d9-9a58-8aeae3b7b6ab\") "
Nov 03 20:05:29.290524 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:29.290509    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26379d10-3da7-4549-8b8c-41730a41792b-kube-api-access-tr6rd\" (UniqueName: \"kubernetes.io/projected/26379d10-3da7-4549-8b8c-41730a41792b-kube-api-access-tr6rd\") pod \"26379d10-3da7-4549-8b8c-41730a41792b\" (UID: \"26379d10-3da7-4549-8b8c-41730a41792b\") "
Nov 03 20:05:29.290658 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:29.290613    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26379d10-3da7-4549-8b8c-41730a41792b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26379d10-3da7-4549-8b8c-41730a41792b-plugins-config-volume\") pod \"26379d10-3da7-4549-8b8c-41730a41792b\" (UID: \"26379d10-3da7-4549-8b8c-41730a41792b\") "
Nov 03 20:05:29.291635 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:29.291604    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26379d10-3da7-4549-8b8c-41730a41792b-plugins-config-volume" pod "26379d10-3da7-4549-8b8c-41730a41792b" (UID: "26379d10-3da7-4549-8b8c-41730a41792b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:05:29.292102 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:29.292079    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4aa9c292-f06c-45d9-9a58-8aeae3b7b6ab-plugins-config-volume" pod "4aa9c292-f06c-45d9-9a58-8aeae3b7b6ab" (UID: "4aa9c292-f06c-45d9-9a58-8aeae3b7b6ab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:05:29.293041 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:29.293018    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4aa9c292-f06c-45d9-9a58-8aeae3b7b6ab-kube-api-access-zngl8" pod "4aa9c292-f06c-45d9-9a58-8aeae3b7b6ab" (UID: "4aa9c292-f06c-45d9-9a58-8aeae3b7b6ab"). InnerVolumeSpecName "kube-api-access-zngl8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:05:29.293425 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:29.293403    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26379d10-3da7-4549-8b8c-41730a41792b-kube-api-access-tr6rd" pod "26379d10-3da7-4549-8b8c-41730a41792b" (UID: "26379d10-3da7-4549-8b8c-41730a41792b"). InnerVolumeSpecName "kube-api-access-tr6rd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:05:29.309889 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:29.309865    4921 patch_prober.go:29] interesting pod/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Nov 03 20:05:29.310197 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:29.309910    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c" podUID="1b9000a2-e11e-4afb-9ea7-4e4f1446d918" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Nov 03 20:05:29.392336 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:29.392298    4921 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4aa9c292-f06c-45d9-9a58-8aeae3b7b6ab-plugins-config-volume\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:29.392336 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:29.392330    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tr6rd\" (UniqueName: \"kubernetes.io/projected/26379d10-3da7-4549-8b8c-41730a41792b-kube-api-access-tr6rd\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:29.392502 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:29.392343    4921 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26379d10-3da7-4549-8b8c-41730a41792b-plugins-config-volume\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:29.392502 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:29.392358    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zngl8\" (UniqueName: \"kubernetes.io/projected/4aa9c292-f06c-45d9-9a58-8aeae3b7b6ab-kube-api-access-zngl8\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:30.071691 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:30.066602    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3cee6a-799d7c674c-fj9vk"
Nov 03 20:05:30.074076 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:30.073840    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3cee6a-799d7c674c-fj9vk" event={"ID":"4aa9c292-f06c-45d9-9a58-8aeae3b7b6ab","Type":"ContainerDied","Data":"c2fd821d3aec5c8c2dcf1cc062aadf8e7e79675575e5bfb1750f9178e1182eb1"}
Nov 03 20:05:30.074076 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:30.073933    4921 scope.go:122] "RemoveContainer" containerID="c22dd48c7c79a0e292ef4e7ee62a718aa5c150899e124577e4bea7f0abf2ea3c"
Nov 03 20:05:30.076308 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:30.076278    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3cee6a-544fd5876c-56hwf" event={"ID":"26379d10-3da7-4549-8b8c-41730a41792b","Type":"ContainerDied","Data":"5764911b8d3bcac9aa028b8b60f2394946bea26c43b2473d9d3d84459b269027"}
Nov 03 20:05:30.076425 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:30.076408    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3cee6a-544fd5876c-56hwf"
Nov 03 20:05:30.088489 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:30.088472    4921 scope.go:122] "RemoveContainer" containerID="03ff696cb9d480ac235973dcde8e5a58b5d22ac3a8144b098460f74873cc28a4"
Nov 03 20:05:30.094276 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:30.094253    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3cee6a-799d7c674c-fj9vk"]
Nov 03 20:05:30.098157 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:30.098135    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3cee6a-799d7c674c-fj9vk"]
Nov 03 20:05:30.105629 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:30.105597    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3cee6a-544fd5876c-56hwf"]
Nov 03 20:05:30.109067 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:30.109048    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3cee6a-544fd5876c-56hwf"]
Nov 03 20:05:30.309660 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:30.309624    4921 patch_prober.go:29] interesting pod/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Nov 03 20:05:30.309908 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:30.309712    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c" podUID="1b9000a2-e11e-4afb-9ea7-4e4f1446d918" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Nov 03 20:05:30.800608 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:30.800571    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-10572d-5bf85ff6df-7gk9c"]
Nov 03 20:05:30.812294 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:30.812254    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-10572d-d8dc7767-9lhq7"]
Nov 03 20:05:30.892246 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:30.892211    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26379d10-3da7-4549-8b8c-41730a41792b" path="/var/lib/kubelet/pods/26379d10-3da7-4549-8b8c-41730a41792b/volumes"
Nov 03 20:05:30.892639 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:30.892626    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4aa9c292-f06c-45d9-9a58-8aeae3b7b6ab" path="/var/lib/kubelet/pods/4aa9c292-f06c-45d9-9a58-8aeae3b7b6ab/volumes"
Nov 03 20:05:31.314056 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:31.314027    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c"
Nov 03 20:05:31.314536 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:31.314337    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c"
Nov 03 20:05:31.315115 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:31.315097    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c"
Nov 03 20:05:32.975737 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:05:32.969982    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 20:05:35.801950 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:35.801889    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-10572d-5bf85ff6df-7gk9c" podUID="62b4fa76-8b78-4a67-af03-b126b71aa8c6" containerName="payload-processing" containerID="cri-o://6cd08e4a269d89aea539436fa2d28446beca64a5ffdb4bcf0082a5f2fe76930a" gracePeriod=25
Nov 03 20:05:35.813596 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:35.813544    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-10572d-d8dc7767-9lhq7" podUID="48ab742a-e2bf-4751-bb31-a962f0996078" containerName="payload-pre-processing" containerID="cri-o://836e703e559eb902672de9d1b2f6659080af370b46882fd8609885f193abe10c" gracePeriod=25
Nov 03 20:05:36.086827 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:36.086801    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-10572d-5bf85ff6df-7gk9c"
Nov 03 20:05:36.117818 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:36.117793    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62b4fa76-8b78-4a67-af03-b126b71aa8c6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62b4fa76-8b78-4a67-af03-b126b71aa8c6-plugins-config-volume\") pod \"62b4fa76-8b78-4a67-af03-b126b71aa8c6\" (UID: \"62b4fa76-8b78-4a67-af03-b126b71aa8c6\") "
Nov 03 20:05:36.117954 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:36.117900    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62b4fa76-8b78-4a67-af03-b126b71aa8c6-kube-api-access-pnxnn\" (UniqueName: \"kubernetes.io/projected/62b4fa76-8b78-4a67-af03-b126b71aa8c6-kube-api-access-pnxnn\") pod \"62b4fa76-8b78-4a67-af03-b126b71aa8c6\" (UID: \"62b4fa76-8b78-4a67-af03-b126b71aa8c6\") "
Nov 03 20:05:36.118145 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:36.118123    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62b4fa76-8b78-4a67-af03-b126b71aa8c6-plugins-config-volume" pod "62b4fa76-8b78-4a67-af03-b126b71aa8c6" (UID: "62b4fa76-8b78-4a67-af03-b126b71aa8c6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:05:36.118492 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:36.118469    4921 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62b4fa76-8b78-4a67-af03-b126b71aa8c6-plugins-config-volume\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:36.120086 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:36.120064    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62b4fa76-8b78-4a67-af03-b126b71aa8c6-kube-api-access-pnxnn" pod "62b4fa76-8b78-4a67-af03-b126b71aa8c6" (UID: "62b4fa76-8b78-4a67-af03-b126b71aa8c6"). InnerVolumeSpecName "kube-api-access-pnxnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:05:36.138331 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:36.138297    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-10572d-d8dc7767-9lhq7"
Nov 03 20:05:36.173552 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:36.173519    4921 generic.go:356] "Generic (PLEG): container finished" podID="62b4fa76-8b78-4a67-af03-b126b71aa8c6" containerID="6cd08e4a269d89aea539436fa2d28446beca64a5ffdb4bcf0082a5f2fe76930a" exitCode=0
Nov 03 20:05:36.173842 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:36.173597    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-10572d-5bf85ff6df-7gk9c" event={"ID":"62b4fa76-8b78-4a67-af03-b126b71aa8c6","Type":"ContainerDied","Data":"6cd08e4a269d89aea539436fa2d28446beca64a5ffdb4bcf0082a5f2fe76930a"}
Nov 03 20:05:36.173932 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:36.173685    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-10572d-5bf85ff6df-7gk9c"
Nov 03 20:05:36.176055 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:36.176029    4921 generic.go:356] "Generic (PLEG): container finished" podID="48ab742a-e2bf-4751-bb31-a962f0996078" containerID="836e703e559eb902672de9d1b2f6659080af370b46882fd8609885f193abe10c" exitCode=0
Nov 03 20:05:36.176236 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:36.176221    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-10572d-d8dc7767-9lhq7"
Nov 03 20:05:36.176841 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:36.176746    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-10572d-5bf85ff6df-7gk9c" event={"ID":"62b4fa76-8b78-4a67-af03-b126b71aa8c6","Type":"ContainerDied","Data":"7b556b00169e06b842730c91b4aeffa03db755dc6eb10da4c63ece1dd4c2984e"}
Nov 03 20:05:36.176841 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:36.176794    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-10572d-d8dc7767-9lhq7" event={"ID":"48ab742a-e2bf-4751-bb31-a962f0996078","Type":"ContainerDied","Data":"836e703e559eb902672de9d1b2f6659080af370b46882fd8609885f193abe10c"}
Nov 03 20:05:36.176841 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:36.176813    4921 scope.go:122] "RemoveContainer" containerID="6cd08e4a269d89aea539436fa2d28446beca64a5ffdb4bcf0082a5f2fe76930a"
Nov 03 20:05:36.177046 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:36.176818    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-10572d-d8dc7767-9lhq7" event={"ID":"48ab742a-e2bf-4751-bb31-a962f0996078","Type":"ContainerDied","Data":"3c6ddf4b2009fcc2e64f80f41fca26e5eb18495120b982230a6cd69eda63b4d4"}
Nov 03 20:05:36.193332 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:36.193307    4921 scope.go:122] "RemoveContainer" containerID="6cd08e4a269d89aea539436fa2d28446beca64a5ffdb4bcf0082a5f2fe76930a"
Nov 03 20:05:36.193764 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:05:36.193741    4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cd08e4a269d89aea539436fa2d28446beca64a5ffdb4bcf0082a5f2fe76930a\": container with ID starting with 6cd08e4a269d89aea539436fa2d28446beca64a5ffdb4bcf0082a5f2fe76930a not found: ID does not exist" containerID="6cd08e4a269d89aea539436fa2d28446beca64a5ffdb4bcf0082a5f2fe76930a"
Nov 03 20:05:36.193884 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:36.193769    4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cd08e4a269d89aea539436fa2d28446beca64a5ffdb4bcf0082a5f2fe76930a"} err="failed to get container status \"6cd08e4a269d89aea539436fa2d28446beca64a5ffdb4bcf0082a5f2fe76930a\": rpc error: code = NotFound desc = could not find container \"6cd08e4a269d89aea539436fa2d28446beca64a5ffdb4bcf0082a5f2fe76930a\": container with ID starting with 6cd08e4a269d89aea539436fa2d28446beca64a5ffdb4bcf0082a5f2fe76930a not found: ID does not exist"
Nov 03 20:05:36.193884 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:36.193785    4921 scope.go:122] "RemoveContainer" containerID="836e703e559eb902672de9d1b2f6659080af370b46882fd8609885f193abe10c"
Nov 03 20:05:36.206880 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:36.206861    4921 scope.go:122] "RemoveContainer" containerID="836e703e559eb902672de9d1b2f6659080af370b46882fd8609885f193abe10c"
Nov 03 20:05:36.207282 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:05:36.207213    4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"836e703e559eb902672de9d1b2f6659080af370b46882fd8609885f193abe10c\": container with ID starting with 836e703e559eb902672de9d1b2f6659080af370b46882fd8609885f193abe10c not found: ID does not exist" containerID="836e703e559eb902672de9d1b2f6659080af370b46882fd8609885f193abe10c"
Nov 03 20:05:36.207282 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:36.207254    4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"836e703e559eb902672de9d1b2f6659080af370b46882fd8609885f193abe10c"} err="failed to get container status \"836e703e559eb902672de9d1b2f6659080af370b46882fd8609885f193abe10c\": rpc error: code = NotFound desc = could not find container \"836e703e559eb902672de9d1b2f6659080af370b46882fd8609885f193abe10c\": container with ID starting with 836e703e559eb902672de9d1b2f6659080af370b46882fd8609885f193abe10c not found: ID does not exist"
Nov 03 20:05:36.209341 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:36.209316    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-10572d-5bf85ff6df-7gk9c"]
Nov 03 20:05:36.213013 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:36.212993    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-10572d-5bf85ff6df-7gk9c"]
Nov 03 20:05:36.219375 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:36.219357    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48ab742a-e2bf-4751-bb31-a962f0996078-kube-api-access-9fg45\" (UniqueName: \"kubernetes.io/projected/48ab742a-e2bf-4751-bb31-a962f0996078-kube-api-access-9fg45\") pod \"48ab742a-e2bf-4751-bb31-a962f0996078\" (UID: \"48ab742a-e2bf-4751-bb31-a962f0996078\") "
Nov 03 20:05:36.219533 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:36.219520    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48ab742a-e2bf-4751-bb31-a962f0996078-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48ab742a-e2bf-4751-bb31-a962f0996078-plugins-config-volume\") pod \"48ab742a-e2bf-4751-bb31-a962f0996078\" (UID: \"48ab742a-e2bf-4751-bb31-a962f0996078\") "
Nov 03 20:05:36.219884 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:36.219862    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48ab742a-e2bf-4751-bb31-a962f0996078-plugins-config-volume" pod "48ab742a-e2bf-4751-bb31-a962f0996078" (UID: "48ab742a-e2bf-4751-bb31-a962f0996078"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:05:36.220066 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:36.220051    4921 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48ab742a-e2bf-4751-bb31-a962f0996078-plugins-config-volume\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:36.220128 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:36.220072    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnxnn\" (UniqueName: \"kubernetes.io/projected/62b4fa76-8b78-4a67-af03-b126b71aa8c6-kube-api-access-pnxnn\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:36.221299 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:36.221277    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48ab742a-e2bf-4751-bb31-a962f0996078-kube-api-access-9fg45" pod "48ab742a-e2bf-4751-bb31-a962f0996078" (UID: "48ab742a-e2bf-4751-bb31-a962f0996078"). InnerVolumeSpecName "kube-api-access-9fg45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:05:36.321087 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:36.321004    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fg45\" (UniqueName: \"kubernetes.io/projected/48ab742a-e2bf-4751-bb31-a962f0996078-kube-api-access-9fg45\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:36.501026 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:36.500997    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-10572d-d8dc7767-9lhq7"]
Nov 03 20:05:36.508162 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:36.508138    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-10572d-d8dc7767-9lhq7"]
Nov 03 20:05:36.889873 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:36.889838    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48ab742a-e2bf-4751-bb31-a962f0996078" path="/var/lib/kubelet/pods/48ab742a-e2bf-4751-bb31-a962f0996078/volumes"
Nov 03 20:05:36.890362 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:36.890345    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62b4fa76-8b78-4a67-af03-b126b71aa8c6" path="/var/lib/kubelet/pods/62b4fa76-8b78-4a67-af03-b126b71aa8c6/volumes"
Nov 03 20:05:39.592890 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:39.592849    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw"]
Nov 03 20:05:43.690817 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:43.690782    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c18bea-5b7994d5ff-s76l2"]
Nov 03 20:05:43.691800 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:43.691738    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c18bea-5b7994d5ff-s76l2" podUID="3f347075-1266-4c50-81ff-a8d5ab87b1b3" containerName="maas-api" containerID="cri-o://08ad2f541dbadaee372a2dd7116146b3cdbf66a389fe455ac536396c9e338c32" gracePeriod=30
Nov 03 20:05:44.097835 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:44.097809    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c18bea-5b7994d5ff-s76l2"
Nov 03 20:05:44.131520 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:44.131480    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f347075-1266-4c50-81ff-a8d5ab87b1b3-kube-api-access-jj7mt\" (UniqueName: \"kubernetes.io/projected/3f347075-1266-4c50-81ff-a8d5ab87b1b3-kube-api-access-jj7mt\") pod \"3f347075-1266-4c50-81ff-a8d5ab87b1b3\" (UID: \"3f347075-1266-4c50-81ff-a8d5ab87b1b3\") "
Nov 03 20:05:44.131715 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:44.131568    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f347075-1266-4c50-81ff-a8d5ab87b1b3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f347075-1266-4c50-81ff-a8d5ab87b1b3-openshift-service-ca\") pod \"3f347075-1266-4c50-81ff-a8d5ab87b1b3\" (UID: \"3f347075-1266-4c50-81ff-a8d5ab87b1b3\") "
Nov 03 20:05:44.131715 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:44.131603    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f347075-1266-4c50-81ff-a8d5ab87b1b3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f347075-1266-4c50-81ff-a8d5ab87b1b3-maas-api-tls\") pod \"3f347075-1266-4c50-81ff-a8d5ab87b1b3\" (UID: \"3f347075-1266-4c50-81ff-a8d5ab87b1b3\") "
Nov 03 20:05:44.131887 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:44.131761    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f347075-1266-4c50-81ff-a8d5ab87b1b3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f347075-1266-4c50-81ff-a8d5ab87b1b3-metrics-certs\") pod \"3f347075-1266-4c50-81ff-a8d5ab87b1b3\" (UID: \"3f347075-1266-4c50-81ff-a8d5ab87b1b3\") "
Nov 03 20:05:44.132290 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:44.132254    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f347075-1266-4c50-81ff-a8d5ab87b1b3-openshift-service-ca" pod "3f347075-1266-4c50-81ff-a8d5ab87b1b3" (UID: "3f347075-1266-4c50-81ff-a8d5ab87b1b3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:05:44.132413 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:44.132392    4921 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f347075-1266-4c50-81ff-a8d5ab87b1b3-openshift-service-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:44.134651 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:44.134621    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f347075-1266-4c50-81ff-a8d5ab87b1b3-kube-api-access-jj7mt" pod "3f347075-1266-4c50-81ff-a8d5ab87b1b3" (UID: "3f347075-1266-4c50-81ff-a8d5ab87b1b3"). InnerVolumeSpecName "kube-api-access-jj7mt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:05:44.136172 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:44.136143    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f347075-1266-4c50-81ff-a8d5ab87b1b3-metrics-certs" pod "3f347075-1266-4c50-81ff-a8d5ab87b1b3" (UID: "3f347075-1266-4c50-81ff-a8d5ab87b1b3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:05:44.136473 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:44.136438    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f347075-1266-4c50-81ff-a8d5ab87b1b3-maas-api-tls" pod "3f347075-1266-4c50-81ff-a8d5ab87b1b3" (UID: "3f347075-1266-4c50-81ff-a8d5ab87b1b3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:05:44.235446 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:44.235404    4921 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f347075-1266-4c50-81ff-a8d5ab87b1b3-metrics-certs\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:44.235446 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:44.235448    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jj7mt\" (UniqueName: \"kubernetes.io/projected/3f347075-1266-4c50-81ff-a8d5ab87b1b3-kube-api-access-jj7mt\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:44.235947 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:44.235466    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f347075-1266-4c50-81ff-a8d5ab87b1b3-maas-api-tls\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:44.246972 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:44.246933    4921 generic.go:356] "Generic (PLEG): container finished" podID="3f347075-1266-4c50-81ff-a8d5ab87b1b3" containerID="08ad2f541dbadaee372a2dd7116146b3cdbf66a389fe455ac536396c9e338c32" exitCode=0
Nov 03 20:05:44.247196 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:44.247042    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c18bea-5b7994d5ff-s76l2"
Nov 03 20:05:44.247287 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:44.247222    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c18bea-5b7994d5ff-s76l2" event={"ID":"3f347075-1266-4c50-81ff-a8d5ab87b1b3","Type":"ContainerDied","Data":"08ad2f541dbadaee372a2dd7116146b3cdbf66a389fe455ac536396c9e338c32"}
Nov 03 20:05:44.247287 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:44.247274    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c18bea-5b7994d5ff-s76l2" event={"ID":"3f347075-1266-4c50-81ff-a8d5ab87b1b3","Type":"ContainerDied","Data":"148fbd7921d71c9c2cffab0d74eb560eabe33a596fa06f2c56cb9a706d17e667"}
Nov 03 20:05:44.247410 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:44.247308    4921 scope.go:122] "RemoveContainer" containerID="08ad2f541dbadaee372a2dd7116146b3cdbf66a389fe455ac536396c9e338c32"
Nov 03 20:05:44.268042 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:44.268020    4921 scope.go:122] "RemoveContainer" containerID="08ad2f541dbadaee372a2dd7116146b3cdbf66a389fe455ac536396c9e338c32"
Nov 03 20:05:44.268398 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:05:44.268377    4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08ad2f541dbadaee372a2dd7116146b3cdbf66a389fe455ac536396c9e338c32\": container with ID starting with 08ad2f541dbadaee372a2dd7116146b3cdbf66a389fe455ac536396c9e338c32 not found: ID does not exist" containerID="08ad2f541dbadaee372a2dd7116146b3cdbf66a389fe455ac536396c9e338c32"
Nov 03 20:05:44.268462 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:44.268408    4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08ad2f541dbadaee372a2dd7116146b3cdbf66a389fe455ac536396c9e338c32"} err="failed to get container status \"08ad2f541dbadaee372a2dd7116146b3cdbf66a389fe455ac536396c9e338c32\": rpc error: code = NotFound desc = could not find container \"08ad2f541dbadaee372a2dd7116146b3cdbf66a389fe455ac536396c9e338c32\": container with ID starting with 08ad2f541dbadaee372a2dd7116146b3cdbf66a389fe455ac536396c9e338c32 not found: ID does not exist"
Nov 03 20:05:44.275535 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:44.275513    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c18bea-5b7994d5ff-s76l2"]
Nov 03 20:05:44.280181 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:44.280159    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c18bea-5b7994d5ff-s76l2"]
Nov 03 20:05:44.889047 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:44.889004    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f347075-1266-4c50-81ff-a8d5ab87b1b3" path="/var/lib/kubelet/pods/3f347075-1266-4c50-81ff-a8d5ab87b1b3/volumes"
Nov 03 20:05:54.614962 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:54.614911    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw" podUID="72d96abc-f3b4-4eda-b3a1-b93ba658e1d5" containerName="main" containerID="cri-o://bb1cdc51d5b9f34156315417880d99fe919d20ab677f86d0c9adfcab2f00e631" gracePeriod=45
Nov 03 20:05:54.883298 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:54.883238    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw"
Nov 03 20:05:55.006334 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:55.006293    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72d96abc-f3b4-4eda-b3a1-b93ba658e1d5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/72d96abc-f3b4-4eda-b3a1-b93ba658e1d5-dshm\") pod \"72d96abc-f3b4-4eda-b3a1-b93ba658e1d5\" (UID: \"72d96abc-f3b4-4eda-b3a1-b93ba658e1d5\") "
Nov 03 20:05:55.006532 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:55.006355    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72d96abc-f3b4-4eda-b3a1-b93ba658e1d5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72d96abc-f3b4-4eda-b3a1-b93ba658e1d5-tmp-dir\") pod \"72d96abc-f3b4-4eda-b3a1-b93ba658e1d5\" (UID: \"72d96abc-f3b4-4eda-b3a1-b93ba658e1d5\") "
Nov 03 20:05:55.006532 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:55.006430    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72d96abc-f3b4-4eda-b3a1-b93ba658e1d5-tls-certs\" (UniqueName: \"kubernetes.io/secret/72d96abc-f3b4-4eda-b3a1-b93ba658e1d5-tls-certs\") pod \"72d96abc-f3b4-4eda-b3a1-b93ba658e1d5\" (UID: \"72d96abc-f3b4-4eda-b3a1-b93ba658e1d5\") "
Nov 03 20:05:55.006532 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:55.006489    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72d96abc-f3b4-4eda-b3a1-b93ba658e1d5-home\" (UniqueName: \"kubernetes.io/empty-dir/72d96abc-f3b4-4eda-b3a1-b93ba658e1d5-home\") pod \"72d96abc-f3b4-4eda-b3a1-b93ba658e1d5\" (UID: \"72d96abc-f3b4-4eda-b3a1-b93ba658e1d5\") "
Nov 03 20:05:55.006808 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:55.006547    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72d96abc-f3b4-4eda-b3a1-b93ba658e1d5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72d96abc-f3b4-4eda-b3a1-b93ba658e1d5-model-cache\") pod \"72d96abc-f3b4-4eda-b3a1-b93ba658e1d5\" (UID: \"72d96abc-f3b4-4eda-b3a1-b93ba658e1d5\") "
Nov 03 20:05:55.006808 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:55.006717    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72d96abc-f3b4-4eda-b3a1-b93ba658e1d5-kube-api-access-jnj58\" (UniqueName: \"kubernetes.io/projected/72d96abc-f3b4-4eda-b3a1-b93ba658e1d5-kube-api-access-jnj58\") pod \"72d96abc-f3b4-4eda-b3a1-b93ba658e1d5\" (UID: \"72d96abc-f3b4-4eda-b3a1-b93ba658e1d5\") "
Nov 03 20:05:55.008370 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:55.008330    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72d96abc-f3b4-4eda-b3a1-b93ba658e1d5-tmp-dir" pod "72d96abc-f3b4-4eda-b3a1-b93ba658e1d5" (UID: "72d96abc-f3b4-4eda-b3a1-b93ba658e1d5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:05:55.009063 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:55.008536    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72d96abc-f3b4-4eda-b3a1-b93ba658e1d5-home" pod "72d96abc-f3b4-4eda-b3a1-b93ba658e1d5" (UID: "72d96abc-f3b4-4eda-b3a1-b93ba658e1d5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:05:55.009545 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:55.009420    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72d96abc-f3b4-4eda-b3a1-b93ba658e1d5-model-cache" pod "72d96abc-f3b4-4eda-b3a1-b93ba658e1d5" (UID: "72d96abc-f3b4-4eda-b3a1-b93ba658e1d5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:05:55.010895 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:55.010863    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72d96abc-f3b4-4eda-b3a1-b93ba658e1d5-dshm" pod "72d96abc-f3b4-4eda-b3a1-b93ba658e1d5" (UID: "72d96abc-f3b4-4eda-b3a1-b93ba658e1d5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:05:55.012091 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:55.012067    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72d96abc-f3b4-4eda-b3a1-b93ba658e1d5-tls-certs" pod "72d96abc-f3b4-4eda-b3a1-b93ba658e1d5" (UID: "72d96abc-f3b4-4eda-b3a1-b93ba658e1d5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:05:55.012394 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:55.012371    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72d96abc-f3b4-4eda-b3a1-b93ba658e1d5-kube-api-access-jnj58" pod "72d96abc-f3b4-4eda-b3a1-b93ba658e1d5" (UID: "72d96abc-f3b4-4eda-b3a1-b93ba658e1d5"). InnerVolumeSpecName "kube-api-access-jnj58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:05:55.108993 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:55.108953    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnj58\" (UniqueName: \"kubernetes.io/projected/72d96abc-f3b4-4eda-b3a1-b93ba658e1d5-kube-api-access-jnj58\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:55.108993 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:55.108987    4921 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72d96abc-f3b4-4eda-b3a1-b93ba658e1d5-dshm\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:55.108993 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:55.108997    4921 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72d96abc-f3b4-4eda-b3a1-b93ba658e1d5-tmp-dir\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:55.108993 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:55.109005    4921 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72d96abc-f3b4-4eda-b3a1-b93ba658e1d5-tls-certs\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:55.109285 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:55.109014    4921 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72d96abc-f3b4-4eda-b3a1-b93ba658e1d5-home\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:55.109285 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:55.109022    4921 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72d96abc-f3b4-4eda-b3a1-b93ba658e1d5-model-cache\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:05:55.348887 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:55.348819    4921 generic.go:356] "Generic (PLEG): container finished" podID="72d96abc-f3b4-4eda-b3a1-b93ba658e1d5" containerID="bb1cdc51d5b9f34156315417880d99fe919d20ab677f86d0c9adfcab2f00e631" exitCode=0
Nov 03 20:05:55.348887 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:55.348869    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw"
Nov 03 20:05:55.349185 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:55.348889    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw" event={"ID":"72d96abc-f3b4-4eda-b3a1-b93ba658e1d5","Type":"ContainerDied","Data":"bb1cdc51d5b9f34156315417880d99fe919d20ab677f86d0c9adfcab2f00e631"}
Nov 03 20:05:55.349185 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:55.348949    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw" event={"ID":"72d96abc-f3b4-4eda-b3a1-b93ba658e1d5","Type":"ContainerDied","Data":"f9107159b42d8084e76657dd9c29c6802855422da97fbf19f6f97cb214c80ba7"}
Nov 03 20:05:55.349185 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:55.348971    4921 scope.go:122] "RemoveContainer" containerID="bb1cdc51d5b9f34156315417880d99fe919d20ab677f86d0c9adfcab2f00e631"
Nov 03 20:05:55.361609 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:55.361587    4921 scope.go:122] "RemoveContainer" containerID="bb1cdc51d5b9f34156315417880d99fe919d20ab677f86d0c9adfcab2f00e631"
Nov 03 20:05:55.361988 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:05:55.361963    4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb1cdc51d5b9f34156315417880d99fe919d20ab677f86d0c9adfcab2f00e631\": container with ID starting with bb1cdc51d5b9f34156315417880d99fe919d20ab677f86d0c9adfcab2f00e631 not found: ID does not exist" containerID="bb1cdc51d5b9f34156315417880d99fe919d20ab677f86d0c9adfcab2f00e631"
Nov 03 20:05:55.362082 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:55.362001    4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb1cdc51d5b9f34156315417880d99fe919d20ab677f86d0c9adfcab2f00e631"} err="failed to get container status \"bb1cdc51d5b9f34156315417880d99fe919d20ab677f86d0c9adfcab2f00e631\": rpc error: code = NotFound desc = could not find container \"bb1cdc51d5b9f34156315417880d99fe919d20ab677f86d0c9adfcab2f00e631\": container with ID starting with bb1cdc51d5b9f34156315417880d99fe919d20ab677f86d0c9adfcab2f00e631 not found: ID does not exist"
Nov 03 20:05:55.373033 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:55.372999    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw"]
Nov 03 20:05:55.377152 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:55.377124    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-a35658/distinct-w1-a35658-kserve-54bbdfff9d-24klw"]
Nov 03 20:05:55.562109 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:55.562075    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7"]
Nov 03 20:05:55.567685 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:55.567644    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6"]
Nov 03 20:05:55.588421 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:55.585983    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt"]
Nov 03 20:05:56.889327 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:05:56.889295    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72d96abc-f3b4-4eda-b3a1-b93ba658e1d5" path="/var/lib/kubelet/pods/72d96abc-f3b4-4eda-b3a1-b93ba658e1d5/volumes"
Nov 03 20:06:00.901467 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:00.901430    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-8d4127-6fb576bf67-qr2v6"]
Nov 03 20:06:02.925446 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:06:02.925411    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 20:06:05.902197 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:05.902153    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8d4127-6fb576bf67-qr2v6" podUID="4bda8b9f-49ce-45d8-9c50-3e56a865b849" containerName="payload-pre-processing" containerID="cri-o://716c5ffad050785d674c0827a94fc0c872735446fa401b34607eadfa83ddb46c" gracePeriod=25
Nov 03 20:06:07.165834 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:07.165800    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-a35658-ff9b85698-4wqq2"]
Nov 03 20:06:07.166304 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:07.166028    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a35658-ff9b85698-4wqq2" podUID="0f48e7cf-0a2d-44ff-8949-77d277a40138" containerName="maas-api" containerID="cri-o://dd806fd9ed8b5f223b5156a6144f5d95b4d89066479f7fa2ac5b353951a94457" gracePeriod=30
Nov 03 20:06:07.530398 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:07.530197    4921 generic.go:356] "Generic (PLEG): container finished" podID="0f48e7cf-0a2d-44ff-8949-77d277a40138" containerID="dd806fd9ed8b5f223b5156a6144f5d95b4d89066479f7fa2ac5b353951a94457" exitCode=0
Nov 03 20:06:07.530398 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:07.530326    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a35658-ff9b85698-4wqq2" event={"ID":"0f48e7cf-0a2d-44ff-8949-77d277a40138","Type":"ContainerDied","Data":"dd806fd9ed8b5f223b5156a6144f5d95b4d89066479f7fa2ac5b353951a94457"}
Nov 03 20:06:07.612186 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:07.612155    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a35658-ff9b85698-4wqq2"
Nov 03 20:06:07.744610 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:07.744561    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f48e7cf-0a2d-44ff-8949-77d277a40138-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f48e7cf-0a2d-44ff-8949-77d277a40138-maas-api-tls\") pod \"0f48e7cf-0a2d-44ff-8949-77d277a40138\" (UID: \"0f48e7cf-0a2d-44ff-8949-77d277a40138\") "
Nov 03 20:06:07.745080 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:07.745051    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f48e7cf-0a2d-44ff-8949-77d277a40138-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f48e7cf-0a2d-44ff-8949-77d277a40138-metrics-certs\") pod \"0f48e7cf-0a2d-44ff-8949-77d277a40138\" (UID: \"0f48e7cf-0a2d-44ff-8949-77d277a40138\") "
Nov 03 20:06:07.745315 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:07.745280    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f48e7cf-0a2d-44ff-8949-77d277a40138-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f48e7cf-0a2d-44ff-8949-77d277a40138-openshift-service-ca\") pod \"0f48e7cf-0a2d-44ff-8949-77d277a40138\" (UID: \"0f48e7cf-0a2d-44ff-8949-77d277a40138\") "
Nov 03 20:06:07.745436 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:07.745344    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f48e7cf-0a2d-44ff-8949-77d277a40138-kube-api-access-x2pzm\" (UniqueName: \"kubernetes.io/projected/0f48e7cf-0a2d-44ff-8949-77d277a40138-kube-api-access-x2pzm\") pod \"0f48e7cf-0a2d-44ff-8949-77d277a40138\" (UID: \"0f48e7cf-0a2d-44ff-8949-77d277a40138\") "
Nov 03 20:06:07.757472 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:07.746956    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f48e7cf-0a2d-44ff-8949-77d277a40138-openshift-service-ca" pod "0f48e7cf-0a2d-44ff-8949-77d277a40138" (UID: "0f48e7cf-0a2d-44ff-8949-77d277a40138"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:06:07.757472 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:07.756614    4921 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f48e7cf-0a2d-44ff-8949-77d277a40138-openshift-service-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:07.775125 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:07.775079    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f48e7cf-0a2d-44ff-8949-77d277a40138-kube-api-access-x2pzm" pod "0f48e7cf-0a2d-44ff-8949-77d277a40138" (UID: "0f48e7cf-0a2d-44ff-8949-77d277a40138"). InnerVolumeSpecName "kube-api-access-x2pzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:06:07.777746 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:07.776926    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f48e7cf-0a2d-44ff-8949-77d277a40138-maas-api-tls" pod "0f48e7cf-0a2d-44ff-8949-77d277a40138" (UID: "0f48e7cf-0a2d-44ff-8949-77d277a40138"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:06:07.781523 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:07.781431    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f48e7cf-0a2d-44ff-8949-77d277a40138-metrics-certs" pod "0f48e7cf-0a2d-44ff-8949-77d277a40138" (UID: "0f48e7cf-0a2d-44ff-8949-77d277a40138"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:06:07.857972 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:07.857935    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2pzm\" (UniqueName: \"kubernetes.io/projected/0f48e7cf-0a2d-44ff-8949-77d277a40138-kube-api-access-x2pzm\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:07.857972 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:07.857967    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f48e7cf-0a2d-44ff-8949-77d277a40138-maas-api-tls\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:07.857972 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:07.857977    4921 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f48e7cf-0a2d-44ff-8949-77d277a40138-metrics-certs\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:08.539748 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:08.539714    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a35658-ff9b85698-4wqq2"
Nov 03 20:06:08.540238 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:08.539723    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a35658-ff9b85698-4wqq2" event={"ID":"0f48e7cf-0a2d-44ff-8949-77d277a40138","Type":"ContainerDied","Data":"f40ba4ff70c39c7c3016bbb74775e528da3688768440e79ae1d59099a831c787"}
Nov 03 20:06:08.540238 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:08.539854    4921 scope.go:122] "RemoveContainer" containerID="dd806fd9ed8b5f223b5156a6144f5d95b4d89066479f7fa2ac5b353951a94457"
Nov 03 20:06:08.569165 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:08.569134    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-a35658-ff9b85698-4wqq2"]
Nov 03 20:06:08.581909 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:08.581883    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-a35658-ff9b85698-4wqq2"]
Nov 03 20:06:08.894347 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:08.894250    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f48e7cf-0a2d-44ff-8949-77d277a40138" path="/var/lib/kubelet/pods/0f48e7cf-0a2d-44ff-8949-77d277a40138/volumes"
Nov 03 20:06:10.585332 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:10.585286    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7" podUID="d4de0090-d775-4bd7-9ee2-058b5d380638" containerName="main" containerID="cri-o://ad85a1d6af803b96c22775daff77024d9fd0d0d6badd7d4993022769516c0e08" gracePeriod=45
Nov 03 20:06:10.589340 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:10.589303    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6" podUID="1aaee61d-b036-4c2b-bee7-a692b8bed612" containerName="main" containerID="cri-o://207926fd238937e676e8b1d9164180443b545cd5d94e37d48e3ddf3f1df63e77" gracePeriod=45
Nov 03 20:06:10.608759 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:10.608697    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt" podUID="17dc0ed2-5164-449d-be12-a38bdfeb946d" containerName="main" containerID="cri-o://09809d4956de377304508ee5361a64aa2ffb2e284311ca8303ef18bb9edab5c2" gracePeriod=45
Nov 03 20:06:10.709046 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:10.709001    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6" podUID="1aaee61d-b036-4c2b-bee7-a692b8bed612" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": dial tcp 10.132.0.74:8000: connect: connection refused"
Nov 03 20:06:10.973426 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:10.970975    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6"
Nov 03 20:06:10.985946 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:10.985915    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7"
Nov 03 20:06:10.996956 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:10.996928    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt"
Nov 03 20:06:11.059207 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:11.059172    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17dc0ed2-5164-449d-be12-a38bdfeb946d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17dc0ed2-5164-449d-be12-a38bdfeb946d-model-cache\") pod \"17dc0ed2-5164-449d-be12-a38bdfeb946d\" (UID: \"17dc0ed2-5164-449d-be12-a38bdfeb946d\") "
Nov 03 20:06:11.059458 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:11.059438    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17dc0ed2-5164-449d-be12-a38bdfeb946d-tls-certs\" (UniqueName: \"kubernetes.io/secret/17dc0ed2-5164-449d-be12-a38bdfeb946d-tls-certs\") pod \"17dc0ed2-5164-449d-be12-a38bdfeb946d\" (UID: \"17dc0ed2-5164-449d-be12-a38bdfeb946d\") "
Nov 03 20:06:11.059618 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:11.059601    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4de0090-d775-4bd7-9ee2-058b5d380638-home\" (UniqueName: \"kubernetes.io/empty-dir/d4de0090-d775-4bd7-9ee2-058b5d380638-home\") pod \"d4de0090-d775-4bd7-9ee2-058b5d380638\" (UID: \"d4de0090-d775-4bd7-9ee2-058b5d380638\") "
Nov 03 20:06:11.059773 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:11.059760    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17dc0ed2-5164-449d-be12-a38bdfeb946d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17dc0ed2-5164-449d-be12-a38bdfeb946d-tmp-dir\") pod \"17dc0ed2-5164-449d-be12-a38bdfeb946d\" (UID: \"17dc0ed2-5164-449d-be12-a38bdfeb946d\") "
Nov 03 20:06:11.059877 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:11.059865    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1aaee61d-b036-4c2b-bee7-a692b8bed612-kube-api-access-hxdxg\" (UniqueName: \"kubernetes.io/projected/1aaee61d-b036-4c2b-bee7-a692b8bed612-kube-api-access-hxdxg\") pod \"1aaee61d-b036-4c2b-bee7-a692b8bed612\" (UID: \"1aaee61d-b036-4c2b-bee7-a692b8bed612\") "
Nov 03 20:06:11.059977 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:11.059967    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17dc0ed2-5164-449d-be12-a38bdfeb946d-kube-api-access-284jx\" (UniqueName: \"kubernetes.io/projected/17dc0ed2-5164-449d-be12-a38bdfeb946d-kube-api-access-284jx\") pod \"17dc0ed2-5164-449d-be12-a38bdfeb946d\" (UID: \"17dc0ed2-5164-449d-be12-a38bdfeb946d\") "
Nov 03 20:06:11.060053 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:11.060044    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4de0090-d775-4bd7-9ee2-058b5d380638-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4de0090-d775-4bd7-9ee2-058b5d380638-tmp-dir\") pod \"d4de0090-d775-4bd7-9ee2-058b5d380638\" (UID: \"d4de0090-d775-4bd7-9ee2-058b5d380638\") "
Nov 03 20:06:11.060136 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:11.060128    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17dc0ed2-5164-449d-be12-a38bdfeb946d-home\" (UniqueName: \"kubernetes.io/empty-dir/17dc0ed2-5164-449d-be12-a38bdfeb946d-home\") pod \"17dc0ed2-5164-449d-be12-a38bdfeb946d\" (UID: \"17dc0ed2-5164-449d-be12-a38bdfeb946d\") "
Nov 03 20:06:11.060208 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:11.060200    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4de0090-d775-4bd7-9ee2-058b5d380638-tls-certs\" (UniqueName: \"kubernetes.io/secret/d4de0090-d775-4bd7-9ee2-058b5d380638-tls-certs\") pod \"d4de0090-d775-4bd7-9ee2-058b5d380638\" (UID: \"d4de0090-d775-4bd7-9ee2-058b5d380638\") "
Nov 03 20:06:11.060276 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:11.060267    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aaee61d-b036-4c2b-bee7-a692b8bed612-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1aaee61d-b036-4c2b-bee7-a692b8bed612-dshm\") pod \"1aaee61d-b036-4c2b-bee7-a692b8bed612\" (UID: \"1aaee61d-b036-4c2b-bee7-a692b8bed612\") "
Nov 03 20:06:11.060362 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:11.060353    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17dc0ed2-5164-449d-be12-a38bdfeb946d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/17dc0ed2-5164-449d-be12-a38bdfeb946d-dshm\") pod \"17dc0ed2-5164-449d-be12-a38bdfeb946d\" (UID: \"17dc0ed2-5164-449d-be12-a38bdfeb946d\") "
Nov 03 20:06:11.060451 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:11.060431    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aaee61d-b036-4c2b-bee7-a692b8bed612-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1aaee61d-b036-4c2b-bee7-a692b8bed612-model-cache\") pod \"1aaee61d-b036-4c2b-bee7-a692b8bed612\" (UID: \"1aaee61d-b036-4c2b-bee7-a692b8bed612\") "
Nov 03 20:06:11.060560 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:11.060548    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aaee61d-b036-4c2b-bee7-a692b8bed612-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1aaee61d-b036-4c2b-bee7-a692b8bed612-tmp-dir\") pod \"1aaee61d-b036-4c2b-bee7-a692b8bed612\" (UID: \"1aaee61d-b036-4c2b-bee7-a692b8bed612\") "
Nov 03 20:06:11.060687 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:11.060656    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4de0090-d775-4bd7-9ee2-058b5d380638-kube-api-access-2cmbh\" (UniqueName: \"kubernetes.io/projected/d4de0090-d775-4bd7-9ee2-058b5d380638-kube-api-access-2cmbh\") pod \"d4de0090-d775-4bd7-9ee2-058b5d380638\" (UID: \"d4de0090-d775-4bd7-9ee2-058b5d380638\") "
Nov 03 20:06:11.060824 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:11.060810    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aaee61d-b036-4c2b-bee7-a692b8bed612-home\" (UniqueName: \"kubernetes.io/empty-dir/1aaee61d-b036-4c2b-bee7-a692b8bed612-home\") pod \"1aaee61d-b036-4c2b-bee7-a692b8bed612\" (UID: \"1aaee61d-b036-4c2b-bee7-a692b8bed612\") "
Nov 03 20:06:11.060976 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:11.060960    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4de0090-d775-4bd7-9ee2-058b5d380638-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4de0090-d775-4bd7-9ee2-058b5d380638-dshm\") pod \"d4de0090-d775-4bd7-9ee2-058b5d380638\" (UID: \"d4de0090-d775-4bd7-9ee2-058b5d380638\") "
Nov 03 20:06:11.061118 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:11.061102    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1aaee61d-b036-4c2b-bee7-a692b8bed612-tls-certs\" (UniqueName: \"kubernetes.io/secret/1aaee61d-b036-4c2b-bee7-a692b8bed612-tls-certs\") pod \"1aaee61d-b036-4c2b-bee7-a692b8bed612\" (UID: \"1aaee61d-b036-4c2b-bee7-a692b8bed612\") "
Nov 03 20:06:11.061280 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:11.061262    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4de0090-d775-4bd7-9ee2-058b5d380638-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4de0090-d775-4bd7-9ee2-058b5d380638-model-cache\") pod \"d4de0090-d775-4bd7-9ee2-058b5d380638\" (UID: \"d4de0090-d775-4bd7-9ee2-058b5d380638\") "
Nov 03 20:06:11.062180 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:11.059860    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4de0090-d775-4bd7-9ee2-058b5d380638-home" pod "d4de0090-d775-4bd7-9ee2-058b5d380638" (UID: "d4de0090-d775-4bd7-9ee2-058b5d380638"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:11.062306 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:11.060069    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17dc0ed2-5164-449d-be12-a38bdfeb946d-tmp-dir" pod "17dc0ed2-5164-449d-be12-a38bdfeb946d" (UID: "17dc0ed2-5164-449d-be12-a38bdfeb946d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:11.062380 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:11.060264    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4de0090-d775-4bd7-9ee2-058b5d380638-tmp-dir" pod "d4de0090-d775-4bd7-9ee2-058b5d380638" (UID: "d4de0090-d775-4bd7-9ee2-058b5d380638"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:11.062459 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:11.060442    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17dc0ed2-5164-449d-be12-a38bdfeb946d-home" pod "17dc0ed2-5164-449d-be12-a38bdfeb946d" (UID: "17dc0ed2-5164-449d-be12-a38bdfeb946d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:11.062570 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:11.060721    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17dc0ed2-5164-449d-be12-a38bdfeb946d-model-cache" pod "17dc0ed2-5164-449d-be12-a38bdfeb946d" (UID: "17dc0ed2-5164-449d-be12-a38bdfeb946d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:11.062647 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:11.061079    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aaee61d-b036-4c2b-bee7-a692b8bed612-model-cache" pod "1aaee61d-b036-4c2b-bee7-a692b8bed612" (UID: "1aaee61d-b036-4c2b-bee7-a692b8bed612"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:11.062755 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:11.062155    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4de0090-d775-4bd7-9ee2-058b5d380638-model-cache" pod "d4de0090-d775-4bd7-9ee2-058b5d380638" (UID: "d4de0090-d775-4bd7-9ee2-058b5d380638"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:11.063049 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:11.063029    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aaee61d-b036-4c2b-bee7-a692b8bed612-home" pod "1aaee61d-b036-4c2b-bee7-a692b8bed612" (UID: "1aaee61d-b036-4c2b-bee7-a692b8bed612"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:11.064132 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:11.064088    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aaee61d-b036-4c2b-bee7-a692b8bed612-tmp-dir" pod "1aaee61d-b036-4c2b-bee7-a692b8bed612" (UID: "1aaee61d-b036-4c2b-bee7-a692b8bed612"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:11.067646 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:11.067618    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1aaee61d-b036-4c2b-bee7-a692b8bed612-tls-certs" pod "1aaee61d-b036-4c2b-bee7-a692b8bed612" (UID: "1aaee61d-b036-4c2b-bee7-a692b8bed612"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:06:11.070655 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:11.070632    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4de0090-d775-4bd7-9ee2-058b5d380638-dshm" pod "d4de0090-d775-4bd7-9ee2-058b5d380638" (UID: "d4de0090-d775-4bd7-9ee2-058b5d380638"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:11.073004 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:11.072981    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17dc0ed2-5164-449d-be12-a38bdfeb946d-dshm" pod "17dc0ed2-5164-449d-be12-a38bdfeb946d" (UID: "17dc0ed2-5164-449d-be12-a38bdfeb946d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:11.073004 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:11.072994    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17dc0ed2-5164-449d-be12-a38bdfeb946d-kube-api-access-284jx" pod "17dc0ed2-5164-449d-be12-a38bdfeb946d" (UID: "17dc0ed2-5164-449d-be12-a38bdfeb946d"). InnerVolumeSpecName "kube-api-access-284jx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:06:11.073167 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:11.073145    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17dc0ed2-5164-449d-be12-a38bdfeb946d-tls-certs" pod "17dc0ed2-5164-449d-be12-a38bdfeb946d" (UID: "17dc0ed2-5164-449d-be12-a38bdfeb946d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:06:11.073245 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:11.073174    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4de0090-d775-4bd7-9ee2-058b5d380638-tls-certs" pod "d4de0090-d775-4bd7-9ee2-058b5d380638" (UID: "d4de0090-d775-4bd7-9ee2-058b5d380638"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:06:11.073245 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:11.073238    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aaee61d-b036-4c2b-bee7-a692b8bed612-dshm" pod "1aaee61d-b036-4c2b-bee7-a692b8bed612" (UID: "1aaee61d-b036-4c2b-bee7-a692b8bed612"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:11.073333 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:11.073254    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aaee61d-b036-4c2b-bee7-a692b8bed612-kube-api-access-hxdxg" pod "1aaee61d-b036-4c2b-bee7-a692b8bed612" (UID: "1aaee61d-b036-4c2b-bee7-a692b8bed612"). InnerVolumeSpecName "kube-api-access-hxdxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:06:11.075242 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:11.075215    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4de0090-d775-4bd7-9ee2-058b5d380638-kube-api-access-2cmbh" pod "d4de0090-d775-4bd7-9ee2-058b5d380638" (UID: "d4de0090-d775-4bd7-9ee2-058b5d380638"). InnerVolumeSpecName "kube-api-access-2cmbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:06:11.162759 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:11.162719    4921 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1aaee61d-b036-4c2b-bee7-a692b8bed612-home\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:11.162759 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:11.162752    4921 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4de0090-d775-4bd7-9ee2-058b5d380638-dshm\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:11.162759 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:11.162765    4921 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1aaee61d-b036-4c2b-bee7-a692b8bed612-tls-certs\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:11.163049 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:11.162777    4921 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4de0090-d775-4bd7-9ee2-058b5d380638-model-cache\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:11.163049 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:11.162791    4921 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17dc0ed2-5164-449d-be12-a38bdfeb946d-model-cache\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:11.163049 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:11.162802    4921 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17dc0ed2-5164-449d-be12-a38bdfeb946d-tls-certs\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:11.163049 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:11.162812    4921 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4de0090-d775-4bd7-9ee2-058b5d380638-home\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:11.163049 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:11.162823    4921 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17dc0ed2-5164-449d-be12-a38bdfeb946d-tmp-dir\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:11.163049 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:11.162835    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxdxg\" (UniqueName: \"kubernetes.io/projected/1aaee61d-b036-4c2b-bee7-a692b8bed612-kube-api-access-hxdxg\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:11.163049 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:11.162847    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-284jx\" (UniqueName: \"kubernetes.io/projected/17dc0ed2-5164-449d-be12-a38bdfeb946d-kube-api-access-284jx\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:11.163049 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:11.162861    4921 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4de0090-d775-4bd7-9ee2-058b5d380638-tmp-dir\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:11.163049 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:11.162872    4921 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17dc0ed2-5164-449d-be12-a38bdfeb946d-home\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:11.163049 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:11.162882    4921 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4de0090-d775-4bd7-9ee2-058b5d380638-tls-certs\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:11.163049 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:11.162893    4921 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1aaee61d-b036-4c2b-bee7-a692b8bed612-dshm\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:11.163049 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:11.162904    4921 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17dc0ed2-5164-449d-be12-a38bdfeb946d-dshm\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:11.163049 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:11.162915    4921 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1aaee61d-b036-4c2b-bee7-a692b8bed612-model-cache\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:11.163049 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:11.162926    4921 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1aaee61d-b036-4c2b-bee7-a692b8bed612-tmp-dir\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:11.163049 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:11.162938    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cmbh\" (UniqueName: \"kubernetes.io/projected/d4de0090-d775-4bd7-9ee2-058b5d380638-kube-api-access-2cmbh\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:11.582373 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:11.582332    4921 generic.go:356] "Generic (PLEG): container finished" podID="17dc0ed2-5164-449d-be12-a38bdfeb946d" containerID="09809d4956de377304508ee5361a64aa2ffb2e284311ca8303ef18bb9edab5c2" exitCode=0
Nov 03 20:06:11.582591 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:11.582372    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt" event={"ID":"17dc0ed2-5164-449d-be12-a38bdfeb946d","Type":"ContainerDied","Data":"09809d4956de377304508ee5361a64aa2ffb2e284311ca8303ef18bb9edab5c2"}
Nov 03 20:06:11.582591 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:11.582426    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt" event={"ID":"17dc0ed2-5164-449d-be12-a38bdfeb946d","Type":"ContainerDied","Data":"ff3b8a957d477abeca4d240d8c4bf29c3d1946300483167e463c35861442beb4"}
Nov 03 20:06:11.582591 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:11.582444    4921 scope.go:122] "RemoveContainer" containerID="09809d4956de377304508ee5361a64aa2ffb2e284311ca8303ef18bb9edab5c2"
Nov 03 20:06:11.582591 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:11.582386    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt"
Nov 03 20:06:11.584177 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:11.584152    4921 generic.go:356] "Generic (PLEG): container finished" podID="d4de0090-d775-4bd7-9ee2-058b5d380638" containerID="ad85a1d6af803b96c22775daff77024d9fd0d0d6badd7d4993022769516c0e08" exitCode=0
Nov 03 20:06:11.584314 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:11.584192    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7"
Nov 03 20:06:11.584314 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:11.584213    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7" event={"ID":"d4de0090-d775-4bd7-9ee2-058b5d380638","Type":"ContainerDied","Data":"ad85a1d6af803b96c22775daff77024d9fd0d0d6badd7d4993022769516c0e08"}
Nov 03 20:06:11.584314 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:11.584251    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7" event={"ID":"d4de0090-d775-4bd7-9ee2-058b5d380638","Type":"ContainerDied","Data":"5921be12b2184bd512e5ba6130609749c433dde57d133108b416ed426199968e"}
Nov 03 20:06:11.586008 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:11.585986    4921 generic.go:356] "Generic (PLEG): container finished" podID="1aaee61d-b036-4c2b-bee7-a692b8bed612" containerID="207926fd238937e676e8b1d9164180443b545cd5d94e37d48e3ddf3f1df63e77" exitCode=0
Nov 03 20:06:11.586422 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:11.586031    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6" event={"ID":"1aaee61d-b036-4c2b-bee7-a692b8bed612","Type":"ContainerDied","Data":"207926fd238937e676e8b1d9164180443b545cd5d94e37d48e3ddf3f1df63e77"}
Nov 03 20:06:11.586422 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:11.586053    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6"
Nov 03 20:06:11.586422 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:11.586060    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6" event={"ID":"1aaee61d-b036-4c2b-bee7-a692b8bed612","Type":"ContainerDied","Data":"9a8732ca19050dee4dc67889520d6a966951d78003deefeec8c979249300e560"}
Nov 03 20:06:11.600416 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:11.600191    4921 scope.go:122] "RemoveContainer" containerID="09809d4956de377304508ee5361a64aa2ffb2e284311ca8303ef18bb9edab5c2"
Nov 03 20:06:11.600534 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:06:11.600515    4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09809d4956de377304508ee5361a64aa2ffb2e284311ca8303ef18bb9edab5c2\": container with ID starting with 09809d4956de377304508ee5361a64aa2ffb2e284311ca8303ef18bb9edab5c2 not found: ID does not exist" containerID="09809d4956de377304508ee5361a64aa2ffb2e284311ca8303ef18bb9edab5c2"
Nov 03 20:06:11.600575 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:11.600543    4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09809d4956de377304508ee5361a64aa2ffb2e284311ca8303ef18bb9edab5c2"} err="failed to get container status \"09809d4956de377304508ee5361a64aa2ffb2e284311ca8303ef18bb9edab5c2\": rpc error: code = NotFound desc = could not find container \"09809d4956de377304508ee5361a64aa2ffb2e284311ca8303ef18bb9edab5c2\": container with ID starting with 09809d4956de377304508ee5361a64aa2ffb2e284311ca8303ef18bb9edab5c2 not found: ID does not exist"
Nov 03 20:06:11.600575 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:11.600560    4921 scope.go:122] "RemoveContainer" containerID="ad85a1d6af803b96c22775daff77024d9fd0d0d6badd7d4993022769516c0e08"
Nov 03 20:06:11.628953 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:11.628930    4921 scope.go:122] "RemoveContainer" containerID="ad85a1d6af803b96c22775daff77024d9fd0d0d6badd7d4993022769516c0e08"
Nov 03 20:06:11.629249 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:06:11.629225    4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad85a1d6af803b96c22775daff77024d9fd0d0d6badd7d4993022769516c0e08\": container with ID starting with ad85a1d6af803b96c22775daff77024d9fd0d0d6badd7d4993022769516c0e08 not found: ID does not exist" containerID="ad85a1d6af803b96c22775daff77024d9fd0d0d6badd7d4993022769516c0e08"
Nov 03 20:06:11.629320 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:11.629256    4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad85a1d6af803b96c22775daff77024d9fd0d0d6badd7d4993022769516c0e08"} err="failed to get container status \"ad85a1d6af803b96c22775daff77024d9fd0d0d6badd7d4993022769516c0e08\": rpc error: code = NotFound desc = could not find container \"ad85a1d6af803b96c22775daff77024d9fd0d0d6badd7d4993022769516c0e08\": container with ID starting with ad85a1d6af803b96c22775daff77024d9fd0d0d6badd7d4993022769516c0e08 not found: ID does not exist"
Nov 03 20:06:11.629320 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:11.629274    4921 scope.go:122] "RemoveContainer" containerID="207926fd238937e676e8b1d9164180443b545cd5d94e37d48e3ddf3f1df63e77"
Nov 03 20:06:11.638959 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:11.638890    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6"]
Nov 03 20:06:11.639976 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:11.639954    4921 scope.go:122] "RemoveContainer" containerID="207926fd238937e676e8b1d9164180443b545cd5d94e37d48e3ddf3f1df63e77"
Nov 03 20:06:11.640301 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:06:11.640278    4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"207926fd238937e676e8b1d9164180443b545cd5d94e37d48e3ddf3f1df63e77\": container with ID starting with 207926fd238937e676e8b1d9164180443b545cd5d94e37d48e3ddf3f1df63e77 not found: ID does not exist" containerID="207926fd238937e676e8b1d9164180443b545cd5d94e37d48e3ddf3f1df63e77"
Nov 03 20:06:11.640418 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:11.640305    4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"207926fd238937e676e8b1d9164180443b545cd5d94e37d48e3ddf3f1df63e77"} err="failed to get container status \"207926fd238937e676e8b1d9164180443b545cd5d94e37d48e3ddf3f1df63e77\": rpc error: code = NotFound desc = could not find container \"207926fd238937e676e8b1d9164180443b545cd5d94e37d48e3ddf3f1df63e77\": container with ID starting with 207926fd238937e676e8b1d9164180443b545cd5d94e37d48e3ddf3f1df63e77 not found: ID does not exist"
Nov 03 20:06:11.643025 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:11.643005    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-8d4127/distinct-w2-8d4127-kserve-c86c68c9-b5tv6"]
Nov 03 20:06:11.651256 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:11.651233    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7"]
Nov 03 20:06:11.655972 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:11.655947    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7"]
Nov 03 20:06:11.664077 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:11.664056    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt"]
Nov 03 20:06:11.669344 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:11.669317    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-8d4127/simulator-w2-8d4127-kserve-75c54bd8d5-d4tgt"]
Nov 03 20:06:11.900124 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:11.900033    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d4127/distinct-2-w2-8d4127-kserve-6bbdbfbd6d-fjcr7" podUID="d4de0090-d775-4bd7-9ee2-058b5d380638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:12.898532 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:12.898490    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17dc0ed2-5164-449d-be12-a38bdfeb946d" path="/var/lib/kubelet/pods/17dc0ed2-5164-449d-be12-a38bdfeb946d/volumes"
Nov 03 20:06:12.899147 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:12.899124    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1aaee61d-b036-4c2b-bee7-a692b8bed612" path="/var/lib/kubelet/pods/1aaee61d-b036-4c2b-bee7-a692b8bed612/volumes"
Nov 03 20:06:12.899678 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:12.899648    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4de0090-d775-4bd7-9ee2-058b5d380638" path="/var/lib/kubelet/pods/d4de0090-d775-4bd7-9ee2-058b5d380638/volumes"
Nov 03 20:06:14.635715 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:14.633527    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j"]
Nov 03 20:06:14.635715 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:14.633801    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j" podUID="e78a74a9-c32a-4047-a64f-b344f8ebaf22" containerName="istio-proxy" containerID="cri-o://1012d635dfd87be601767a1d96cda123a00bfddfd9ec6d2a97dcb84ac9f8171f" gracePeriod=30
Nov 03 20:06:14.635715 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:14.635589    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45"]
Nov 03 20:06:14.636770 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:14.635906    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45" podUID="fa3f8ec3-6ad5-4433-8dbb-71f8038dffce" containerName="istio-proxy" containerID="cri-o://f15b475c18fb67278f8a04cec8e777f2c319a4f7942bb052509323f7c2ccce1a" gracePeriod=30
Nov 03 20:06:15.532811 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:15.532771    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8d4127-6fb576bf67-qr2v6" podUID="4bda8b9f-49ce-45d8-9c50-3e56a865b849" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:06:19.942848 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:19.942818    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j"
Nov 03 20:06:19.987082 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:19.987051    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e78a74a9-c32a-4047-a64f-b344f8ebaf22-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e78a74a9-c32a-4047-a64f-b344f8ebaf22-istio-podinfo\") pod \"e78a74a9-c32a-4047-a64f-b344f8ebaf22\" (UID: \"e78a74a9-c32a-4047-a64f-b344f8ebaf22\") "
Nov 03 20:06:19.987230 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:19.987097    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e78a74a9-c32a-4047-a64f-b344f8ebaf22-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e78a74a9-c32a-4047-a64f-b344f8ebaf22-istiod-ca-cert\") pod \"e78a74a9-c32a-4047-a64f-b344f8ebaf22\" (UID: \"e78a74a9-c32a-4047-a64f-b344f8ebaf22\") "
Nov 03 20:06:19.987230 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:19.987190    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e78a74a9-c32a-4047-a64f-b344f8ebaf22-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e78a74a9-c32a-4047-a64f-b344f8ebaf22-workload-certs\") pod \"e78a74a9-c32a-4047-a64f-b344f8ebaf22\" (UID: \"e78a74a9-c32a-4047-a64f-b344f8ebaf22\") "
Nov 03 20:06:19.987367 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:19.987288    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e78a74a9-c32a-4047-a64f-b344f8ebaf22-istio-token\" (UniqueName: \"kubernetes.io/projected/e78a74a9-c32a-4047-a64f-b344f8ebaf22-istio-token\") pod \"e78a74a9-c32a-4047-a64f-b344f8ebaf22\" (UID: \"e78a74a9-c32a-4047-a64f-b344f8ebaf22\") "
Nov 03 20:06:19.987367 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:19.987333    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e78a74a9-c32a-4047-a64f-b344f8ebaf22-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e78a74a9-c32a-4047-a64f-b344f8ebaf22-workload-socket\") pod \"e78a74a9-c32a-4047-a64f-b344f8ebaf22\" (UID: \"e78a74a9-c32a-4047-a64f-b344f8ebaf22\") "
Nov 03 20:06:19.987479 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:19.987375    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e78a74a9-c32a-4047-a64f-b344f8ebaf22-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e78a74a9-c32a-4047-a64f-b344f8ebaf22-istio-data\") pod \"e78a74a9-c32a-4047-a64f-b344f8ebaf22\" (UID: \"e78a74a9-c32a-4047-a64f-b344f8ebaf22\") "
Nov 03 20:06:19.987479 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:19.987407    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e78a74a9-c32a-4047-a64f-b344f8ebaf22-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e78a74a9-c32a-4047-a64f-b344f8ebaf22-credential-socket\") pod \"e78a74a9-c32a-4047-a64f-b344f8ebaf22\" (UID: \"e78a74a9-c32a-4047-a64f-b344f8ebaf22\") "
Nov 03 20:06:19.987589 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:19.987472    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e78a74a9-c32a-4047-a64f-b344f8ebaf22-workload-certs" pod "e78a74a9-c32a-4047-a64f-b344f8ebaf22" (UID: "e78a74a9-c32a-4047-a64f-b344f8ebaf22"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:19.987589 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:19.987504    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e78a74a9-c32a-4047-a64f-b344f8ebaf22-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e78a74a9-c32a-4047-a64f-b344f8ebaf22-istio-envoy\") pod \"e78a74a9-c32a-4047-a64f-b344f8ebaf22\" (UID: \"e78a74a9-c32a-4047-a64f-b344f8ebaf22\") "
Nov 03 20:06:19.987589 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:19.987549    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e78a74a9-c32a-4047-a64f-b344f8ebaf22-kube-api-access-8fczx\" (UniqueName: \"kubernetes.io/projected/e78a74a9-c32a-4047-a64f-b344f8ebaf22-kube-api-access-8fczx\") pod \"e78a74a9-c32a-4047-a64f-b344f8ebaf22\" (UID: \"e78a74a9-c32a-4047-a64f-b344f8ebaf22\") "
Nov 03 20:06:19.987781 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:19.987585    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e78a74a9-c32a-4047-a64f-b344f8ebaf22-istiod-ca-cert" pod "e78a74a9-c32a-4047-a64f-b344f8ebaf22" (UID: "e78a74a9-c32a-4047-a64f-b344f8ebaf22"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:06:19.987781 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:19.987760    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e78a74a9-c32a-4047-a64f-b344f8ebaf22-workload-socket" pod "e78a74a9-c32a-4047-a64f-b344f8ebaf22" (UID: "e78a74a9-c32a-4047-a64f-b344f8ebaf22"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:19.988113 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:19.988086    4921 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e78a74a9-c32a-4047-a64f-b344f8ebaf22-workload-socket\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:19.988214 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:19.988121    4921 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e78a74a9-c32a-4047-a64f-b344f8ebaf22-istiod-ca-cert\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:19.988214 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:19.988139    4921 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e78a74a9-c32a-4047-a64f-b344f8ebaf22-workload-certs\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:19.988214 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:19.988155    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e78a74a9-c32a-4047-a64f-b344f8ebaf22-credential-socket" pod "e78a74a9-c32a-4047-a64f-b344f8ebaf22" (UID: "e78a74a9-c32a-4047-a64f-b344f8ebaf22"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:19.988386 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:19.988279    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e78a74a9-c32a-4047-a64f-b344f8ebaf22-istio-data" pod "e78a74a9-c32a-4047-a64f-b344f8ebaf22" (UID: "e78a74a9-c32a-4047-a64f-b344f8ebaf22"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:19.990955 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:19.990907    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e78a74a9-c32a-4047-a64f-b344f8ebaf22-istio-token" pod "e78a74a9-c32a-4047-a64f-b344f8ebaf22" (UID: "e78a74a9-c32a-4047-a64f-b344f8ebaf22"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:06:19.991151 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:19.991126    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e78a74a9-c32a-4047-a64f-b344f8ebaf22-kube-api-access-8fczx" pod "e78a74a9-c32a-4047-a64f-b344f8ebaf22" (UID: "e78a74a9-c32a-4047-a64f-b344f8ebaf22"). InnerVolumeSpecName "kube-api-access-8fczx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:06:19.991487 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:19.991466    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e78a74a9-c32a-4047-a64f-b344f8ebaf22-istio-envoy" pod "e78a74a9-c32a-4047-a64f-b344f8ebaf22" (UID: "e78a74a9-c32a-4047-a64f-b344f8ebaf22"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:19.992040 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:19.992019    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e78a74a9-c32a-4047-a64f-b344f8ebaf22-istio-podinfo" pod "e78a74a9-c32a-4047-a64f-b344f8ebaf22" (UID: "e78a74a9-c32a-4047-a64f-b344f8ebaf22"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:06:19.998075 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:19.998049    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45"
Nov 03 20:06:20.089138 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:20.089106    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa3f8ec3-6ad5-4433-8dbb-71f8038dffce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa3f8ec3-6ad5-4433-8dbb-71f8038dffce-istio-envoy\") pod \"fa3f8ec3-6ad5-4433-8dbb-71f8038dffce\" (UID: \"fa3f8ec3-6ad5-4433-8dbb-71f8038dffce\") "
Nov 03 20:06:20.089328 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:20.089148    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa3f8ec3-6ad5-4433-8dbb-71f8038dffce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa3f8ec3-6ad5-4433-8dbb-71f8038dffce-credential-socket\") pod \"fa3f8ec3-6ad5-4433-8dbb-71f8038dffce\" (UID: \"fa3f8ec3-6ad5-4433-8dbb-71f8038dffce\") "
Nov 03 20:06:20.089328 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:20.089170    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa3f8ec3-6ad5-4433-8dbb-71f8038dffce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa3f8ec3-6ad5-4433-8dbb-71f8038dffce-workload-certs\") pod \"fa3f8ec3-6ad5-4433-8dbb-71f8038dffce\" (UID: \"fa3f8ec3-6ad5-4433-8dbb-71f8038dffce\") "
Nov 03 20:06:20.089328 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:20.089224    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa3f8ec3-6ad5-4433-8dbb-71f8038dffce-kube-api-access-h7h9q\" (UniqueName: \"kubernetes.io/projected/fa3f8ec3-6ad5-4433-8dbb-71f8038dffce-kube-api-access-h7h9q\") pod \"fa3f8ec3-6ad5-4433-8dbb-71f8038dffce\" (UID: \"fa3f8ec3-6ad5-4433-8dbb-71f8038dffce\") "
Nov 03 20:06:20.089328 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:20.089253    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fa3f8ec3-6ad5-4433-8dbb-71f8038dffce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa3f8ec3-6ad5-4433-8dbb-71f8038dffce-istio-podinfo\") pod \"fa3f8ec3-6ad5-4433-8dbb-71f8038dffce\" (UID: \"fa3f8ec3-6ad5-4433-8dbb-71f8038dffce\") "
Nov 03 20:06:20.089328 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:20.089321    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa3f8ec3-6ad5-4433-8dbb-71f8038dffce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa3f8ec3-6ad5-4433-8dbb-71f8038dffce-istio-data\") pod \"fa3f8ec3-6ad5-4433-8dbb-71f8038dffce\" (UID: \"fa3f8ec3-6ad5-4433-8dbb-71f8038dffce\") "
Nov 03 20:06:20.089609 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:20.089347    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa3f8ec3-6ad5-4433-8dbb-71f8038dffce-istio-token\" (UniqueName: \"kubernetes.io/projected/fa3f8ec3-6ad5-4433-8dbb-71f8038dffce-istio-token\") pod \"fa3f8ec3-6ad5-4433-8dbb-71f8038dffce\" (UID: \"fa3f8ec3-6ad5-4433-8dbb-71f8038dffce\") "
Nov 03 20:06:20.089609 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:20.089387    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa3f8ec3-6ad5-4433-8dbb-71f8038dffce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa3f8ec3-6ad5-4433-8dbb-71f8038dffce-istiod-ca-cert\") pod \"fa3f8ec3-6ad5-4433-8dbb-71f8038dffce\" (UID: \"fa3f8ec3-6ad5-4433-8dbb-71f8038dffce\") "
Nov 03 20:06:20.089609 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:20.089407    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa3f8ec3-6ad5-4433-8dbb-71f8038dffce-credential-socket" pod "fa3f8ec3-6ad5-4433-8dbb-71f8038dffce" (UID: "fa3f8ec3-6ad5-4433-8dbb-71f8038dffce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:20.089609 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:20.089496    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa3f8ec3-6ad5-4433-8dbb-71f8038dffce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa3f8ec3-6ad5-4433-8dbb-71f8038dffce-workload-socket\") pod \"fa3f8ec3-6ad5-4433-8dbb-71f8038dffce\" (UID: \"fa3f8ec3-6ad5-4433-8dbb-71f8038dffce\") "
Nov 03 20:06:20.089609 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:20.089564    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa3f8ec3-6ad5-4433-8dbb-71f8038dffce-workload-certs" pod "fa3f8ec3-6ad5-4433-8dbb-71f8038dffce" (UID: "fa3f8ec3-6ad5-4433-8dbb-71f8038dffce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:20.090019 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:20.089883    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa3f8ec3-6ad5-4433-8dbb-71f8038dffce-workload-socket" pod "fa3f8ec3-6ad5-4433-8dbb-71f8038dffce" (UID: "fa3f8ec3-6ad5-4433-8dbb-71f8038dffce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:20.090019 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:20.089936    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa3f8ec3-6ad5-4433-8dbb-71f8038dffce-istiod-ca-cert" pod "fa3f8ec3-6ad5-4433-8dbb-71f8038dffce" (UID: "fa3f8ec3-6ad5-4433-8dbb-71f8038dffce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:06:20.090019 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:20.090010    4921 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e78a74a9-c32a-4047-a64f-b344f8ebaf22-credential-socket\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:20.090182 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:20.090028    4921 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e78a74a9-c32a-4047-a64f-b344f8ebaf22-istio-envoy\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:20.090182 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:20.090071    4921 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa3f8ec3-6ad5-4433-8dbb-71f8038dffce-workload-socket\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:20.090182 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:20.090088    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fczx\" (UniqueName: \"kubernetes.io/projected/e78a74a9-c32a-4047-a64f-b344f8ebaf22-kube-api-access-8fczx\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:20.090182 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:20.090103    4921 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e78a74a9-c32a-4047-a64f-b344f8ebaf22-istio-podinfo\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:20.090182 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:20.090115    4921 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa3f8ec3-6ad5-4433-8dbb-71f8038dffce-credential-socket\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:20.090182 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:20.090124    4921 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa3f8ec3-6ad5-4433-8dbb-71f8038dffce-workload-certs\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:20.090182 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:20.090119    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa3f8ec3-6ad5-4433-8dbb-71f8038dffce-istio-data" pod "fa3f8ec3-6ad5-4433-8dbb-71f8038dffce" (UID: "fa3f8ec3-6ad5-4433-8dbb-71f8038dffce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:20.090182 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:20.090132    4921 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e78a74a9-c32a-4047-a64f-b344f8ebaf22-istio-token\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:20.090182 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:20.090141    4921 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e78a74a9-c32a-4047-a64f-b344f8ebaf22-istio-data\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:20.091744 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:20.091718    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa3f8ec3-6ad5-4433-8dbb-71f8038dffce-kube-api-access-h7h9q" pod "fa3f8ec3-6ad5-4433-8dbb-71f8038dffce" (UID: "fa3f8ec3-6ad5-4433-8dbb-71f8038dffce"). InnerVolumeSpecName "kube-api-access-h7h9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:06:20.091856 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:20.091748    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa3f8ec3-6ad5-4433-8dbb-71f8038dffce-istio-token" pod "fa3f8ec3-6ad5-4433-8dbb-71f8038dffce" (UID: "fa3f8ec3-6ad5-4433-8dbb-71f8038dffce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:06:20.091856 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:20.091827    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa3f8ec3-6ad5-4433-8dbb-71f8038dffce-istio-envoy" pod "fa3f8ec3-6ad5-4433-8dbb-71f8038dffce" (UID: "fa3f8ec3-6ad5-4433-8dbb-71f8038dffce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:20.091856 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:20.091841    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fa3f8ec3-6ad5-4433-8dbb-71f8038dffce-istio-podinfo" pod "fa3f8ec3-6ad5-4433-8dbb-71f8038dffce" (UID: "fa3f8ec3-6ad5-4433-8dbb-71f8038dffce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:06:20.190931 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:20.190840    4921 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa3f8ec3-6ad5-4433-8dbb-71f8038dffce-istiod-ca-cert\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:20.190931 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:20.190874    4921 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa3f8ec3-6ad5-4433-8dbb-71f8038dffce-istio-envoy\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:20.190931 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:20.190884    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7h9q\" (UniqueName: \"kubernetes.io/projected/fa3f8ec3-6ad5-4433-8dbb-71f8038dffce-kube-api-access-h7h9q\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:20.190931 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:20.190894    4921 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa3f8ec3-6ad5-4433-8dbb-71f8038dffce-istio-podinfo\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:20.190931 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:20.190903    4921 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa3f8ec3-6ad5-4433-8dbb-71f8038dffce-istio-data\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:20.190931 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:20.190913    4921 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa3f8ec3-6ad5-4433-8dbb-71f8038dffce-istio-token\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:20.686860 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:20.686823    4921 generic.go:356] "Generic (PLEG): container finished" podID="e78a74a9-c32a-4047-a64f-b344f8ebaf22" containerID="1012d635dfd87be601767a1d96cda123a00bfddfd9ec6d2a97dcb84ac9f8171f" exitCode=0
Nov 03 20:06:20.687109 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:20.686867    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j"
Nov 03 20:06:20.687109 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:20.686918    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j" event={"ID":"e78a74a9-c32a-4047-a64f-b344f8ebaf22","Type":"ContainerDied","Data":"1012d635dfd87be601767a1d96cda123a00bfddfd9ec6d2a97dcb84ac9f8171f"}
Nov 03 20:06:20.687109 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:20.686972    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j" event={"ID":"e78a74a9-c32a-4047-a64f-b344f8ebaf22","Type":"ContainerDied","Data":"ee6669113bbc031960a4954c13ee2c62c30a95cbf0a00d493db490db2e9ce332"}
Nov 03 20:06:20.687109 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:20.687004    4921 scope.go:122] "RemoveContainer" containerID="1012d635dfd87be601767a1d96cda123a00bfddfd9ec6d2a97dcb84ac9f8171f"
Nov 03 20:06:20.688614 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:20.688470    4921 generic.go:356] "Generic (PLEG): container finished" podID="fa3f8ec3-6ad5-4433-8dbb-71f8038dffce" containerID="f15b475c18fb67278f8a04cec8e777f2c319a4f7942bb052509323f7c2ccce1a" exitCode=0
Nov 03 20:06:20.688614 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:20.688502    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45" event={"ID":"fa3f8ec3-6ad5-4433-8dbb-71f8038dffce","Type":"ContainerDied","Data":"f15b475c18fb67278f8a04cec8e777f2c319a4f7942bb052509323f7c2ccce1a"}
Nov 03 20:06:20.688614 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:20.688568    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45" event={"ID":"fa3f8ec3-6ad5-4433-8dbb-71f8038dffce","Type":"ContainerDied","Data":"719319ec07642acfe0818f8b9d12e80fff8101bf43843033e0bebe240ad45df2"}
Nov 03 20:06:20.688614 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:20.688518    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45"
Nov 03 20:06:20.698531 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:20.698514    4921 scope.go:122] "RemoveContainer" containerID="1012d635dfd87be601767a1d96cda123a00bfddfd9ec6d2a97dcb84ac9f8171f"
Nov 03 20:06:20.698890 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:06:20.698867    4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1012d635dfd87be601767a1d96cda123a00bfddfd9ec6d2a97dcb84ac9f8171f\": container with ID starting with 1012d635dfd87be601767a1d96cda123a00bfddfd9ec6d2a97dcb84ac9f8171f not found: ID does not exist" containerID="1012d635dfd87be601767a1d96cda123a00bfddfd9ec6d2a97dcb84ac9f8171f"
Nov 03 20:06:20.698989 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:20.698896    4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1012d635dfd87be601767a1d96cda123a00bfddfd9ec6d2a97dcb84ac9f8171f"} err="failed to get container status \"1012d635dfd87be601767a1d96cda123a00bfddfd9ec6d2a97dcb84ac9f8171f\": rpc error: code = NotFound desc = could not find container \"1012d635dfd87be601767a1d96cda123a00bfddfd9ec6d2a97dcb84ac9f8171f\": container with ID starting with 1012d635dfd87be601767a1d96cda123a00bfddfd9ec6d2a97dcb84ac9f8171f not found: ID does not exist"
Nov 03 20:06:20.698989 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:20.698915    4921 scope.go:122] "RemoveContainer" containerID="f15b475c18fb67278f8a04cec8e777f2c319a4f7942bb052509323f7c2ccce1a"
Nov 03 20:06:20.710472 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:20.710450    4921 scope.go:122] "RemoveContainer" containerID="f15b475c18fb67278f8a04cec8e777f2c319a4f7942bb052509323f7c2ccce1a"
Nov 03 20:06:20.710784 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:06:20.710758    4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f15b475c18fb67278f8a04cec8e777f2c319a4f7942bb052509323f7c2ccce1a\": container with ID starting with f15b475c18fb67278f8a04cec8e777f2c319a4f7942bb052509323f7c2ccce1a not found: ID does not exist" containerID="f15b475c18fb67278f8a04cec8e777f2c319a4f7942bb052509323f7c2ccce1a"
Nov 03 20:06:20.710897 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:20.710792    4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f15b475c18fb67278f8a04cec8e777f2c319a4f7942bb052509323f7c2ccce1a"} err="failed to get container status \"f15b475c18fb67278f8a04cec8e777f2c319a4f7942bb052509323f7c2ccce1a\": rpc error: code = NotFound desc = could not find container \"f15b475c18fb67278f8a04cec8e777f2c319a4f7942bb052509323f7c2ccce1a\": container with ID starting with f15b475c18fb67278f8a04cec8e777f2c319a4f7942bb052509323f7c2ccce1a not found: ID does not exist"
Nov 03 20:06:20.715329 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:20.715306    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j"]
Nov 03 20:06:20.720321 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:20.720294    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-dk95j"]
Nov 03 20:06:20.728265 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:20.728239    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45"]
Nov 03 20:06:20.731450 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:20.731426    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a35658-openshift-default-77fbcb7647-4rn45"]
Nov 03 20:06:20.888907 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:20.888875    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e78a74a9-c32a-4047-a64f-b344f8ebaf22" path="/var/lib/kubelet/pods/e78a74a9-c32a-4047-a64f-b344f8ebaf22/volumes"
Nov 03 20:06:20.889431 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:20.889415    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa3f8ec3-6ad5-4433-8dbb-71f8038dffce" path="/var/lib/kubelet/pods/fa3f8ec3-6ad5-4433-8dbb-71f8038dffce/volumes"
Nov 03 20:06:25.533424 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:25.533378    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8d4127-6fb576bf67-qr2v6" podUID="4bda8b9f-49ce-45d8-9c50-3e56a865b849" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:06:26.236184 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:26.236153    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8d4127-6fb576bf67-qr2v6"
Nov 03 20:06:26.391850 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:26.391214    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bda8b9f-49ce-45d8-9c50-3e56a865b849-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bda8b9f-49ce-45d8-9c50-3e56a865b849-plugins-config-volume\") pod \"4bda8b9f-49ce-45d8-9c50-3e56a865b849\" (UID: \"4bda8b9f-49ce-45d8-9c50-3e56a865b849\") "
Nov 03 20:06:26.391850 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:26.391292    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bda8b9f-49ce-45d8-9c50-3e56a865b849-kube-api-access-xwg9q\" (UniqueName: \"kubernetes.io/projected/4bda8b9f-49ce-45d8-9c50-3e56a865b849-kube-api-access-xwg9q\") pod \"4bda8b9f-49ce-45d8-9c50-3e56a865b849\" (UID: \"4bda8b9f-49ce-45d8-9c50-3e56a865b849\") "
Nov 03 20:06:26.394780 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:26.394736    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bda8b9f-49ce-45d8-9c50-3e56a865b849-plugins-config-volume" pod "4bda8b9f-49ce-45d8-9c50-3e56a865b849" (UID: "4bda8b9f-49ce-45d8-9c50-3e56a865b849"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:06:26.397943 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:26.397899    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bda8b9f-49ce-45d8-9c50-3e56a865b849-kube-api-access-xwg9q" pod "4bda8b9f-49ce-45d8-9c50-3e56a865b849" (UID: "4bda8b9f-49ce-45d8-9c50-3e56a865b849"). InnerVolumeSpecName "kube-api-access-xwg9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:06:26.493032 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:26.492998    4921 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bda8b9f-49ce-45d8-9c50-3e56a865b849-plugins-config-volume\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:26.493032 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:26.493029    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwg9q\" (UniqueName: \"kubernetes.io/projected/4bda8b9f-49ce-45d8-9c50-3e56a865b849-kube-api-access-xwg9q\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:26.732133 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:26.732099    4921 generic.go:356] "Generic (PLEG): container finished" podID="4bda8b9f-49ce-45d8-9c50-3e56a865b849" containerID="716c5ffad050785d674c0827a94fc0c872735446fa401b34607eadfa83ddb46c" exitCode=0
Nov 03 20:06:26.732612 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:26.732146    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8d4127-6fb576bf67-qr2v6"
Nov 03 20:06:26.732612 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:26.732180    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8d4127-6fb576bf67-qr2v6" event={"ID":"4bda8b9f-49ce-45d8-9c50-3e56a865b849","Type":"ContainerDied","Data":"716c5ffad050785d674c0827a94fc0c872735446fa401b34607eadfa83ddb46c"}
Nov 03 20:06:26.732612 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:26.732231    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8d4127-6fb576bf67-qr2v6" event={"ID":"4bda8b9f-49ce-45d8-9c50-3e56a865b849","Type":"ContainerDied","Data":"e4532155e908aef9d53551e331490ef3604e48c0ea70e2fc67d3bb8a5519f2cd"}
Nov 03 20:06:26.732612 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:26.732254    4921 scope.go:122] "RemoveContainer" containerID="716c5ffad050785d674c0827a94fc0c872735446fa401b34607eadfa83ddb46c"
Nov 03 20:06:26.743568 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:26.743552    4921 scope.go:122] "RemoveContainer" containerID="716c5ffad050785d674c0827a94fc0c872735446fa401b34607eadfa83ddb46c"
Nov 03 20:06:26.743852 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:06:26.743829    4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"716c5ffad050785d674c0827a94fc0c872735446fa401b34607eadfa83ddb46c\": container with ID starting with 716c5ffad050785d674c0827a94fc0c872735446fa401b34607eadfa83ddb46c not found: ID does not exist" containerID="716c5ffad050785d674c0827a94fc0c872735446fa401b34607eadfa83ddb46c"
Nov 03 20:06:26.743958 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:26.743856    4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"716c5ffad050785d674c0827a94fc0c872735446fa401b34607eadfa83ddb46c"} err="failed to get container status \"716c5ffad050785d674c0827a94fc0c872735446fa401b34607eadfa83ddb46c\": rpc error: code = NotFound desc = could not find container \"716c5ffad050785d674c0827a94fc0c872735446fa401b34607eadfa83ddb46c\": container with ID starting with 716c5ffad050785d674c0827a94fc0c872735446fa401b34607eadfa83ddb46c not found: ID does not exist"
Nov 03 20:06:26.754053 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:26.754030    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-8d4127-6fb576bf67-qr2v6"]
Nov 03 20:06:26.757377 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:26.757345    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-8d4127-6fb576bf67-qr2v6"]
Nov 03 20:06:26.887704 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:26.887659    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bda8b9f-49ce-45d8-9c50-3e56a865b849" path="/var/lib/kubelet/pods/4bda8b9f-49ce-45d8-9c50-3e56a865b849/volumes"
Nov 03 20:06:31.877748 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:31.877106    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gzf8k_6f215b42-0f09-46a0-bc98-392c0fa909c9/console-operator/2.log"
Nov 03 20:06:31.901688 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:31.890585    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gzf8k_6f215b42-0f09-46a0-bc98-392c0fa909c9/console-operator/2.log"
Nov 03 20:06:32.037691 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:32.030496    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5f6acf3-45g45"]
Nov 03 20:06:32.037691 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:32.035310    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5f6acf3-45g45"]
Nov 03 20:06:32.899694 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:32.896236    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97249a07-bd8b-404c-a0e8-d416fa948542" path="/var/lib/kubelet/pods/97249a07-bd8b-404c-a0e8-d416fa948542/volumes"
Nov 03 20:06:32.939640 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:06:32.939602    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 20:06:34.670616 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:34.669494    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs"]
Nov 03 20:06:34.670616 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:34.669832    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs" podUID="f838f4d0-1e63-4b98-8cfe-bd1b96db7162" containerName="istio-proxy" containerID="cri-o://ac430cd619e6eac1717ec6dd03d93a6c78e1bb5104c473fb54212cbf72062599" gracePeriod=30
Nov 03 20:06:39.720387 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:06:39.720357    4921 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf838f4d0_1e63_4b98_8cfe_bd1b96db7162.slice/crio-conmon-ac430cd619e6eac1717ec6dd03d93a6c78e1bb5104c473fb54212cbf72062599.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 20:06:39.900271 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:39.900142    4921 generic.go:356] "Generic (PLEG): container finished" podID="f838f4d0-1e63-4b98-8cfe-bd1b96db7162" containerID="ac430cd619e6eac1717ec6dd03d93a6c78e1bb5104c473fb54212cbf72062599" exitCode=0
Nov 03 20:06:39.900271 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:39.900190    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs" event={"ID":"f838f4d0-1e63-4b98-8cfe-bd1b96db7162","Type":"ContainerDied","Data":"ac430cd619e6eac1717ec6dd03d93a6c78e1bb5104c473fb54212cbf72062599"}
Nov 03 20:06:39.966539 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:39.966476    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs"
Nov 03 20:06:40.004526 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:40.004487    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f838f4d0-1e63-4b98-8cfe-bd1b96db7162-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f838f4d0-1e63-4b98-8cfe-bd1b96db7162-workload-socket\") pod \"f838f4d0-1e63-4b98-8cfe-bd1b96db7162\" (UID: \"f838f4d0-1e63-4b98-8cfe-bd1b96db7162\") "
Nov 03 20:06:40.004722 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:40.004553    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f838f4d0-1e63-4b98-8cfe-bd1b96db7162-istio-token\" (UniqueName: \"kubernetes.io/projected/f838f4d0-1e63-4b98-8cfe-bd1b96db7162-istio-token\") pod \"f838f4d0-1e63-4b98-8cfe-bd1b96db7162\" (UID: \"f838f4d0-1e63-4b98-8cfe-bd1b96db7162\") "
Nov 03 20:06:40.004722 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:40.004711    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f838f4d0-1e63-4b98-8cfe-bd1b96db7162-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f838f4d0-1e63-4b98-8cfe-bd1b96db7162-istiod-ca-cert\") pod \"f838f4d0-1e63-4b98-8cfe-bd1b96db7162\" (UID: \"f838f4d0-1e63-4b98-8cfe-bd1b96db7162\") "
Nov 03 20:06:40.004865 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:40.004751    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f838f4d0-1e63-4b98-8cfe-bd1b96db7162-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f838f4d0-1e63-4b98-8cfe-bd1b96db7162-istio-envoy\") pod \"f838f4d0-1e63-4b98-8cfe-bd1b96db7162\" (UID: \"f838f4d0-1e63-4b98-8cfe-bd1b96db7162\") "
Nov 03 20:06:40.004865 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:40.004783    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f838f4d0-1e63-4b98-8cfe-bd1b96db7162-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f838f4d0-1e63-4b98-8cfe-bd1b96db7162-istio-data\") pod \"f838f4d0-1e63-4b98-8cfe-bd1b96db7162\" (UID: \"f838f4d0-1e63-4b98-8cfe-bd1b96db7162\") "
Nov 03 20:06:40.004987 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:40.004883    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f838f4d0-1e63-4b98-8cfe-bd1b96db7162-kube-api-access-d27st\" (UniqueName: \"kubernetes.io/projected/f838f4d0-1e63-4b98-8cfe-bd1b96db7162-kube-api-access-d27st\") pod \"f838f4d0-1e63-4b98-8cfe-bd1b96db7162\" (UID: \"f838f4d0-1e63-4b98-8cfe-bd1b96db7162\") "
Nov 03 20:06:40.004987 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:40.004934    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f838f4d0-1e63-4b98-8cfe-bd1b96db7162-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f838f4d0-1e63-4b98-8cfe-bd1b96db7162-istio-podinfo\") pod \"f838f4d0-1e63-4b98-8cfe-bd1b96db7162\" (UID: \"f838f4d0-1e63-4b98-8cfe-bd1b96db7162\") "
Nov 03 20:06:40.005529 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:40.004989    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f838f4d0-1e63-4b98-8cfe-bd1b96db7162-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f838f4d0-1e63-4b98-8cfe-bd1b96db7162-workload-certs\") pod \"f838f4d0-1e63-4b98-8cfe-bd1b96db7162\" (UID: \"f838f4d0-1e63-4b98-8cfe-bd1b96db7162\") "
Nov 03 20:06:40.005529 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:40.005029    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f838f4d0-1e63-4b98-8cfe-bd1b96db7162-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f838f4d0-1e63-4b98-8cfe-bd1b96db7162-credential-socket\") pod \"f838f4d0-1e63-4b98-8cfe-bd1b96db7162\" (UID: \"f838f4d0-1e63-4b98-8cfe-bd1b96db7162\") "
Nov 03 20:06:40.006426 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:40.006257    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f838f4d0-1e63-4b98-8cfe-bd1b96db7162-istio-data" pod "f838f4d0-1e63-4b98-8cfe-bd1b96db7162" (UID: "f838f4d0-1e63-4b98-8cfe-bd1b96db7162"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:40.006426 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:40.006386    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f838f4d0-1e63-4b98-8cfe-bd1b96db7162-workload-certs" pod "f838f4d0-1e63-4b98-8cfe-bd1b96db7162" (UID: "f838f4d0-1e63-4b98-8cfe-bd1b96db7162"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:40.006622 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:40.006480    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f838f4d0-1e63-4b98-8cfe-bd1b96db7162-istiod-ca-cert" pod "f838f4d0-1e63-4b98-8cfe-bd1b96db7162" (UID: "f838f4d0-1e63-4b98-8cfe-bd1b96db7162"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:06:40.006712 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:40.006644    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f838f4d0-1e63-4b98-8cfe-bd1b96db7162-workload-socket" pod "f838f4d0-1e63-4b98-8cfe-bd1b96db7162" (UID: "f838f4d0-1e63-4b98-8cfe-bd1b96db7162"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:40.006950 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:40.006835    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f838f4d0-1e63-4b98-8cfe-bd1b96db7162-credential-socket" pod "f838f4d0-1e63-4b98-8cfe-bd1b96db7162" (UID: "f838f4d0-1e63-4b98-8cfe-bd1b96db7162"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:40.008720 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:40.008698    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f838f4d0-1e63-4b98-8cfe-bd1b96db7162-istio-podinfo" pod "f838f4d0-1e63-4b98-8cfe-bd1b96db7162" (UID: "f838f4d0-1e63-4b98-8cfe-bd1b96db7162"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:06:40.008720 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:40.008704    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f838f4d0-1e63-4b98-8cfe-bd1b96db7162-istio-token" pod "f838f4d0-1e63-4b98-8cfe-bd1b96db7162" (UID: "f838f4d0-1e63-4b98-8cfe-bd1b96db7162"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:06:40.008998 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:40.008968    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f838f4d0-1e63-4b98-8cfe-bd1b96db7162-kube-api-access-d27st" pod "f838f4d0-1e63-4b98-8cfe-bd1b96db7162" (UID: "f838f4d0-1e63-4b98-8cfe-bd1b96db7162"). InnerVolumeSpecName "kube-api-access-d27st". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:06:40.009069 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:40.009027    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f838f4d0-1e63-4b98-8cfe-bd1b96db7162-istio-envoy" pod "f838f4d0-1e63-4b98-8cfe-bd1b96db7162" (UID: "f838f4d0-1e63-4b98-8cfe-bd1b96db7162"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:40.036055 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:40.036029    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh"]
Nov 03 20:06:40.056105 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:40.056067    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv"]
Nov 03 20:06:40.106571 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:40.106538    4921 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f838f4d0-1e63-4b98-8cfe-bd1b96db7162-workload-certs\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:40.106571 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:40.106568    4921 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f838f4d0-1e63-4b98-8cfe-bd1b96db7162-credential-socket\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:40.106858 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:40.106585    4921 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f838f4d0-1e63-4b98-8cfe-bd1b96db7162-workload-socket\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:40.106858 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:40.106599    4921 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f838f4d0-1e63-4b98-8cfe-bd1b96db7162-istio-token\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:40.106858 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:40.106612    4921 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f838f4d0-1e63-4b98-8cfe-bd1b96db7162-istiod-ca-cert\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:40.106858 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:40.106626    4921 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f838f4d0-1e63-4b98-8cfe-bd1b96db7162-istio-envoy\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:40.106858 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:40.106638    4921 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f838f4d0-1e63-4b98-8cfe-bd1b96db7162-istio-data\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:40.106858 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:40.106650    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d27st\" (UniqueName: \"kubernetes.io/projected/f838f4d0-1e63-4b98-8cfe-bd1b96db7162-kube-api-access-d27st\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:40.106858 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:40.106662    4921 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f838f4d0-1e63-4b98-8cfe-bd1b96db7162-istio-podinfo\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:40.908701 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:40.908650    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs"
Nov 03 20:06:40.909161 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:40.908695    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs" event={"ID":"f838f4d0-1e63-4b98-8cfe-bd1b96db7162","Type":"ContainerDied","Data":"af2739dfc89639e6d5d4812608808752f05b046b0f7b2abf07f402f095f4ea9f"}
Nov 03 20:06:40.909161 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:40.908770    4921 scope.go:122] "RemoveContainer" containerID="ac430cd619e6eac1717ec6dd03d93a6c78e1bb5104c473fb54212cbf72062599"
Nov 03 20:06:40.928985 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:40.928961    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs"]
Nov 03 20:06:40.935359 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:40.935334    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d4127-openshift-default-bf8f7445c-d7rfs"]
Nov 03 20:06:42.888813 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:42.888780    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f838f4d0-1e63-4b98-8cfe-bd1b96db7162" path="/var/lib/kubelet/pods/f838f4d0-1e63-4b98-8cfe-bd1b96db7162/volumes"
Nov 03 20:06:55.061183 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:55.061141    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh" podUID="eac6c47d-8ae5-4e4b-b691-98d177887767" containerName="main" containerID="cri-o://8a18fbce0741dd69abce8924aac85d552641c9371e318c16b68052cdfe928ce4" gracePeriod=45
Nov 03 20:06:55.082214 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:55.082147    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv" podUID="01c749b9-d188-4673-9a3d-99b9c4ae0793" containerName="main" containerID="cri-o://ab440e7a2ef9ea255636d7dffb0c5e02ea286d89a6519ae4e020dc0365c055e8" gracePeriod=45
Nov 03 20:06:55.179138 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:55.179098    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh" podUID="eac6c47d-8ae5-4e4b-b691-98d177887767" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": dial tcp 10.132.0.92:8000: connect: connection refused"
Nov 03 20:06:55.391165 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:55.391133    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh"
Nov 03 20:06:55.402538 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:55.402515    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv"
Nov 03 20:06:55.490952 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:55.490919    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eac6c47d-8ae5-4e4b-b691-98d177887767-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eac6c47d-8ae5-4e4b-b691-98d177887767-tmp-dir\") pod \"eac6c47d-8ae5-4e4b-b691-98d177887767\" (UID: \"eac6c47d-8ae5-4e4b-b691-98d177887767\") "
Nov 03 20:06:55.491103 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:55.490987    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eac6c47d-8ae5-4e4b-b691-98d177887767-tls-certs\" (UniqueName: \"kubernetes.io/secret/eac6c47d-8ae5-4e4b-b691-98d177887767-tls-certs\") pod \"eac6c47d-8ae5-4e4b-b691-98d177887767\" (UID: \"eac6c47d-8ae5-4e4b-b691-98d177887767\") "
Nov 03 20:06:55.491158 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:55.491118    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eac6c47d-8ae5-4e4b-b691-98d177887767-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eac6c47d-8ae5-4e4b-b691-98d177887767-model-cache\") pod \"eac6c47d-8ae5-4e4b-b691-98d177887767\" (UID: \"eac6c47d-8ae5-4e4b-b691-98d177887767\") "
Nov 03 20:06:55.491201 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:55.491189    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eac6c47d-8ae5-4e4b-b691-98d177887767-home\" (UniqueName: \"kubernetes.io/empty-dir/eac6c47d-8ae5-4e4b-b691-98d177887767-home\") pod \"eac6c47d-8ae5-4e4b-b691-98d177887767\" (UID: \"eac6c47d-8ae5-4e4b-b691-98d177887767\") "
Nov 03 20:06:55.491257 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:55.491238    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eac6c47d-8ae5-4e4b-b691-98d177887767-kube-api-access-l6tf8\" (UniqueName: \"kubernetes.io/projected/eac6c47d-8ae5-4e4b-b691-98d177887767-kube-api-access-l6tf8\") pod \"eac6c47d-8ae5-4e4b-b691-98d177887767\" (UID: \"eac6c47d-8ae5-4e4b-b691-98d177887767\") "
Nov 03 20:06:55.491300 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:55.491266    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eac6c47d-8ae5-4e4b-b691-98d177887767-tmp-dir" pod "eac6c47d-8ae5-4e4b-b691-98d177887767" (UID: "eac6c47d-8ae5-4e4b-b691-98d177887767"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:55.491342 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:55.491326    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eac6c47d-8ae5-4e4b-b691-98d177887767-dshm\" (UniqueName: \"kubernetes.io/empty-dir/eac6c47d-8ae5-4e4b-b691-98d177887767-dshm\") pod \"eac6c47d-8ae5-4e4b-b691-98d177887767\" (UID: \"eac6c47d-8ae5-4e4b-b691-98d177887767\") "
Nov 03 20:06:55.491539 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:55.491511    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eac6c47d-8ae5-4e4b-b691-98d177887767-home" pod "eac6c47d-8ae5-4e4b-b691-98d177887767" (UID: "eac6c47d-8ae5-4e4b-b691-98d177887767"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:55.491744 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:55.491723    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eac6c47d-8ae5-4e4b-b691-98d177887767-model-cache" pod "eac6c47d-8ae5-4e4b-b691-98d177887767" (UID: "eac6c47d-8ae5-4e4b-b691-98d177887767"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:55.491836 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:55.491804    4921 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eac6c47d-8ae5-4e4b-b691-98d177887767-model-cache\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:55.491836 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:55.491817    4921 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eac6c47d-8ae5-4e4b-b691-98d177887767-home\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:55.492010 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:55.491850    4921 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eac6c47d-8ae5-4e4b-b691-98d177887767-tmp-dir\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:55.493617 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:55.493591    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eac6c47d-8ae5-4e4b-b691-98d177887767-kube-api-access-l6tf8" pod "eac6c47d-8ae5-4e4b-b691-98d177887767" (UID: "eac6c47d-8ae5-4e4b-b691-98d177887767"). InnerVolumeSpecName "kube-api-access-l6tf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:06:55.493800 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:55.493772    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eac6c47d-8ae5-4e4b-b691-98d177887767-dshm" pod "eac6c47d-8ae5-4e4b-b691-98d177887767" (UID: "eac6c47d-8ae5-4e4b-b691-98d177887767"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:55.494710 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:55.494024    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eac6c47d-8ae5-4e4b-b691-98d177887767-tls-certs" pod "eac6c47d-8ae5-4e4b-b691-98d177887767" (UID: "eac6c47d-8ae5-4e4b-b691-98d177887767"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:06:55.592508 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:55.592470    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01c749b9-d188-4673-9a3d-99b9c4ae0793-tls-certs\" (UniqueName: \"kubernetes.io/secret/01c749b9-d188-4673-9a3d-99b9c4ae0793-tls-certs\") pod \"01c749b9-d188-4673-9a3d-99b9c4ae0793\" (UID: \"01c749b9-d188-4673-9a3d-99b9c4ae0793\") "
Nov 03 20:06:55.592786 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:55.592559    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01c749b9-d188-4673-9a3d-99b9c4ae0793-kube-api-access-46s9s\" (UniqueName: \"kubernetes.io/projected/01c749b9-d188-4673-9a3d-99b9c4ae0793-kube-api-access-46s9s\") pod \"01c749b9-d188-4673-9a3d-99b9c4ae0793\" (UID: \"01c749b9-d188-4673-9a3d-99b9c4ae0793\") "
Nov 03 20:06:55.592786 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:55.592626    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01c749b9-d188-4673-9a3d-99b9c4ae0793-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01c749b9-d188-4673-9a3d-99b9c4ae0793-model-cache\") pod \"01c749b9-d188-4673-9a3d-99b9c4ae0793\" (UID: \"01c749b9-d188-4673-9a3d-99b9c4ae0793\") "
Nov 03 20:06:55.592786 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:55.592662    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01c749b9-d188-4673-9a3d-99b9c4ae0793-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01c749b9-d188-4673-9a3d-99b9c4ae0793-tmp-dir\") pod \"01c749b9-d188-4673-9a3d-99b9c4ae0793\" (UID: \"01c749b9-d188-4673-9a3d-99b9c4ae0793\") "
Nov 03 20:06:55.592786 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:55.592726    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01c749b9-d188-4673-9a3d-99b9c4ae0793-home\" (UniqueName: \"kubernetes.io/empty-dir/01c749b9-d188-4673-9a3d-99b9c4ae0793-home\") pod \"01c749b9-d188-4673-9a3d-99b9c4ae0793\" (UID: \"01c749b9-d188-4673-9a3d-99b9c4ae0793\") "
Nov 03 20:06:55.592786 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:55.592757    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01c749b9-d188-4673-9a3d-99b9c4ae0793-dshm\" (UniqueName: \"kubernetes.io/empty-dir/01c749b9-d188-4673-9a3d-99b9c4ae0793-dshm\") pod \"01c749b9-d188-4673-9a3d-99b9c4ae0793\" (UID: \"01c749b9-d188-4673-9a3d-99b9c4ae0793\") "
Nov 03 20:06:55.593097 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:55.592987    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01c749b9-d188-4673-9a3d-99b9c4ae0793-model-cache" pod "01c749b9-d188-4673-9a3d-99b9c4ae0793" (UID: "01c749b9-d188-4673-9a3d-99b9c4ae0793"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:55.593097 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:55.593057    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01c749b9-d188-4673-9a3d-99b9c4ae0793-home" pod "01c749b9-d188-4673-9a3d-99b9c4ae0793" (UID: "01c749b9-d188-4673-9a3d-99b9c4ae0793"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:55.593193 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:55.593089    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01c749b9-d188-4673-9a3d-99b9c4ae0793-tmp-dir" pod "01c749b9-d188-4673-9a3d-99b9c4ae0793" (UID: "01c749b9-d188-4673-9a3d-99b9c4ae0793"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:55.593265 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:55.593249    4921 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01c749b9-d188-4673-9a3d-99b9c4ae0793-model-cache\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:55.593307 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:55.593267    4921 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01c749b9-d188-4673-9a3d-99b9c4ae0793-tmp-dir\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:55.593307 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:55.593277    4921 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01c749b9-d188-4673-9a3d-99b9c4ae0793-home\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:55.593307 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:55.593287    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6tf8\" (UniqueName: \"kubernetes.io/projected/eac6c47d-8ae5-4e4b-b691-98d177887767-kube-api-access-l6tf8\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:55.593307 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:55.593296    4921 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eac6c47d-8ae5-4e4b-b691-98d177887767-dshm\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:55.593448 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:55.593307    4921 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eac6c47d-8ae5-4e4b-b691-98d177887767-tls-certs\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:55.594845 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:55.594822    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01c749b9-d188-4673-9a3d-99b9c4ae0793-tls-certs" pod "01c749b9-d188-4673-9a3d-99b9c4ae0793" (UID: "01c749b9-d188-4673-9a3d-99b9c4ae0793"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:06:55.595247 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:55.595229    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01c749b9-d188-4673-9a3d-99b9c4ae0793-dshm" pod "01c749b9-d188-4673-9a3d-99b9c4ae0793" (UID: "01c749b9-d188-4673-9a3d-99b9c4ae0793"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:55.595290 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:55.595253    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01c749b9-d188-4673-9a3d-99b9c4ae0793-kube-api-access-46s9s" pod "01c749b9-d188-4673-9a3d-99b9c4ae0793" (UID: "01c749b9-d188-4673-9a3d-99b9c4ae0793"). InnerVolumeSpecName "kube-api-access-46s9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:06:55.694742 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:55.694628    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46s9s\" (UniqueName: \"kubernetes.io/projected/01c749b9-d188-4673-9a3d-99b9c4ae0793-kube-api-access-46s9s\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:55.694742 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:55.694657    4921 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01c749b9-d188-4673-9a3d-99b9c4ae0793-dshm\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:55.694742 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:55.694700    4921 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01c749b9-d188-4673-9a3d-99b9c4ae0793-tls-certs\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:56.018722 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:56.018683    4921 generic.go:356] "Generic (PLEG): container finished" podID="01c749b9-d188-4673-9a3d-99b9c4ae0793" containerID="ab440e7a2ef9ea255636d7dffb0c5e02ea286d89a6519ae4e020dc0365c055e8" exitCode=0
Nov 03 20:06:56.018919 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:56.018713    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv" event={"ID":"01c749b9-d188-4673-9a3d-99b9c4ae0793","Type":"ContainerDied","Data":"ab440e7a2ef9ea255636d7dffb0c5e02ea286d89a6519ae4e020dc0365c055e8"}
Nov 03 20:06:56.018919 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:56.018733    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv"
Nov 03 20:06:56.018919 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:56.018764    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv" event={"ID":"01c749b9-d188-4673-9a3d-99b9c4ae0793","Type":"ContainerDied","Data":"ff6dd056f65745ffaaee00a3d549d9a7e7f2d9ea90e99bb31c73b1b96209a625"}
Nov 03 20:06:56.018919 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:56.018793    4921 scope.go:122] "RemoveContainer" containerID="ab440e7a2ef9ea255636d7dffb0c5e02ea286d89a6519ae4e020dc0365c055e8"
Nov 03 20:06:56.020414 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:56.020394    4921 generic.go:356] "Generic (PLEG): container finished" podID="eac6c47d-8ae5-4e4b-b691-98d177887767" containerID="8a18fbce0741dd69abce8924aac85d552641c9371e318c16b68052cdfe928ce4" exitCode=0
Nov 03 20:06:56.020533 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:56.020439    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh"
Nov 03 20:06:56.020533 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:56.020492    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh" event={"ID":"eac6c47d-8ae5-4e4b-b691-98d177887767","Type":"ContainerDied","Data":"8a18fbce0741dd69abce8924aac85d552641c9371e318c16b68052cdfe928ce4"}
Nov 03 20:06:56.020629 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:56.020533    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh" event={"ID":"eac6c47d-8ae5-4e4b-b691-98d177887767","Type":"ContainerDied","Data":"3c4912c5d61f1b17f31a9b3d04e47185495254546a25c44c4b07e3b22053df77"}
Nov 03 20:06:56.030885 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:56.030864    4921 scope.go:122] "RemoveContainer" containerID="ab440e7a2ef9ea255636d7dffb0c5e02ea286d89a6519ae4e020dc0365c055e8"
Nov 03 20:06:56.031263 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:06:56.031243    4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab440e7a2ef9ea255636d7dffb0c5e02ea286d89a6519ae4e020dc0365c055e8\": container with ID starting with ab440e7a2ef9ea255636d7dffb0c5e02ea286d89a6519ae4e020dc0365c055e8 not found: ID does not exist" containerID="ab440e7a2ef9ea255636d7dffb0c5e02ea286d89a6519ae4e020dc0365c055e8"
Nov 03 20:06:56.031369 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:56.031275    4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab440e7a2ef9ea255636d7dffb0c5e02ea286d89a6519ae4e020dc0365c055e8"} err="failed to get container status \"ab440e7a2ef9ea255636d7dffb0c5e02ea286d89a6519ae4e020dc0365c055e8\": rpc error: code = NotFound desc = could not find container \"ab440e7a2ef9ea255636d7dffb0c5e02ea286d89a6519ae4e020dc0365c055e8\": container with ID starting with ab440e7a2ef9ea255636d7dffb0c5e02ea286d89a6519ae4e020dc0365c055e8 not found: ID does not exist"
Nov 03 20:06:56.031369 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:56.031297    4921 scope.go:122] "RemoveContainer" containerID="8a18fbce0741dd69abce8924aac85d552641c9371e318c16b68052cdfe928ce4"
Nov 03 20:06:56.040909 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:56.040857    4921 scope.go:122] "RemoveContainer" containerID="8a18fbce0741dd69abce8924aac85d552641c9371e318c16b68052cdfe928ce4"
Nov 03 20:06:56.041257 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:06:56.041231    4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a18fbce0741dd69abce8924aac85d552641c9371e318c16b68052cdfe928ce4\": container with ID starting with 8a18fbce0741dd69abce8924aac85d552641c9371e318c16b68052cdfe928ce4 not found: ID does not exist" containerID="8a18fbce0741dd69abce8924aac85d552641c9371e318c16b68052cdfe928ce4"
Nov 03 20:06:56.041351 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:56.041265    4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a18fbce0741dd69abce8924aac85d552641c9371e318c16b68052cdfe928ce4"} err="failed to get container status \"8a18fbce0741dd69abce8924aac85d552641c9371e318c16b68052cdfe928ce4\": rpc error: code = NotFound desc = could not find container \"8a18fbce0741dd69abce8924aac85d552641c9371e318c16b68052cdfe928ce4\": container with ID starting with 8a18fbce0741dd69abce8924aac85d552641c9371e318c16b68052cdfe928ce4 not found: ID does not exist"
Nov 03 20:06:56.044084 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:56.044063    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv"]
Nov 03 20:06:56.046935 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:56.046918    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-f4e123/premium-w4-f4e123-kserve-67fdcf4b67-bwmkv"]
Nov 03 20:06:56.054356 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:56.054335    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh"]
Nov 03 20:06:56.057427 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:56.057407    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-f4e123/distinct-2-w4-f4e123-kserve-86d4f8dc9c-557lh"]
Nov 03 20:06:56.889755 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:56.889719    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01c749b9-d188-4673-9a3d-99b9c4ae0793" path="/var/lib/kubelet/pods/01c749b9-d188-4673-9a3d-99b9c4ae0793/volumes"
Nov 03 20:06:56.890425 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:06:56.890404    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eac6c47d-8ae5-4e4b-b691-98d177887767" path="/var/lib/kubelet/pods/eac6c47d-8ae5-4e4b-b691-98d177887767/volumes"
Nov 03 20:07:02.920302 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:07:02.920265    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 20:07:07.102950 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:07:07.102915    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt"]
Nov 03 20:07:07.105655 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:07:07.103640    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt" podUID="ffb30cac-fb9f-4a3f-b954-ecd050b8cb41" containerName="istio-proxy" containerID="cri-o://8197b39e92a354ea25c97f74a91165cc703aa711b45383df2ee9f05f04a8dd79" gracePeriod=30
Nov 03 20:07:11.128461 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:07:11.128421    4921 patch_prober.go:29] interesting pod/e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:07:11.128985 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:07:11.128487    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt" podUID="ffb30cac-fb9f-4a3f-b954-ecd050b8cb41" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:07:12.388758 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:07:12.388696    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt"
Nov 03 20:07:12.489827 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:07:12.489798    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffb30cac-fb9f-4a3f-b954-ecd050b8cb41-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffb30cac-fb9f-4a3f-b954-ecd050b8cb41-istio-data\") pod \"ffb30cac-fb9f-4a3f-b954-ecd050b8cb41\" (UID: \"ffb30cac-fb9f-4a3f-b954-ecd050b8cb41\") "
Nov 03 20:07:12.489991 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:07:12.489841    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffb30cac-fb9f-4a3f-b954-ecd050b8cb41-istio-token\" (UniqueName: \"kubernetes.io/projected/ffb30cac-fb9f-4a3f-b954-ecd050b8cb41-istio-token\") pod \"ffb30cac-fb9f-4a3f-b954-ecd050b8cb41\" (UID: \"ffb30cac-fb9f-4a3f-b954-ecd050b8cb41\") "
Nov 03 20:07:12.489991 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:07:12.489887    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffb30cac-fb9f-4a3f-b954-ecd050b8cb41-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffb30cac-fb9f-4a3f-b954-ecd050b8cb41-istiod-ca-cert\") pod \"ffb30cac-fb9f-4a3f-b954-ecd050b8cb41\" (UID: \"ffb30cac-fb9f-4a3f-b954-ecd050b8cb41\") "
Nov 03 20:07:12.489991 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:07:12.489923    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffb30cac-fb9f-4a3f-b954-ecd050b8cb41-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffb30cac-fb9f-4a3f-b954-ecd050b8cb41-istio-envoy\") pod \"ffb30cac-fb9f-4a3f-b954-ecd050b8cb41\" (UID: \"ffb30cac-fb9f-4a3f-b954-ecd050b8cb41\") "
Nov 03 20:07:12.489991 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:07:12.489955    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ffb30cac-fb9f-4a3f-b954-ecd050b8cb41-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffb30cac-fb9f-4a3f-b954-ecd050b8cb41-istio-podinfo\") pod \"ffb30cac-fb9f-4a3f-b954-ecd050b8cb41\" (UID: \"ffb30cac-fb9f-4a3f-b954-ecd050b8cb41\") "
Nov 03 20:07:12.489991 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:07:12.489986    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffb30cac-fb9f-4a3f-b954-ecd050b8cb41-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffb30cac-fb9f-4a3f-b954-ecd050b8cb41-credential-socket\") pod \"ffb30cac-fb9f-4a3f-b954-ecd050b8cb41\" (UID: \"ffb30cac-fb9f-4a3f-b954-ecd050b8cb41\") "
Nov 03 20:07:12.490326 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:07:12.490014    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffb30cac-fb9f-4a3f-b954-ecd050b8cb41-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffb30cac-fb9f-4a3f-b954-ecd050b8cb41-workload-socket\") pod \"ffb30cac-fb9f-4a3f-b954-ecd050b8cb41\" (UID: \"ffb30cac-fb9f-4a3f-b954-ecd050b8cb41\") "
Nov 03 20:07:12.490326 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:07:12.490079    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffb30cac-fb9f-4a3f-b954-ecd050b8cb41-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffb30cac-fb9f-4a3f-b954-ecd050b8cb41-workload-certs\") pod \"ffb30cac-fb9f-4a3f-b954-ecd050b8cb41\" (UID: \"ffb30cac-fb9f-4a3f-b954-ecd050b8cb41\") "
Nov 03 20:07:12.490326 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:07:12.490092    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffb30cac-fb9f-4a3f-b954-ecd050b8cb41-istio-data" pod "ffb30cac-fb9f-4a3f-b954-ecd050b8cb41" (UID: "ffb30cac-fb9f-4a3f-b954-ecd050b8cb41"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:07:12.490326 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:07:12.490178    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffb30cac-fb9f-4a3f-b954-ecd050b8cb41-kube-api-access-jqrmk\" (UniqueName: \"kubernetes.io/projected/ffb30cac-fb9f-4a3f-b954-ecd050b8cb41-kube-api-access-jqrmk\") pod \"ffb30cac-fb9f-4a3f-b954-ecd050b8cb41\" (UID: \"ffb30cac-fb9f-4a3f-b954-ecd050b8cb41\") "
Nov 03 20:07:12.490326 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:07:12.490283    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffb30cac-fb9f-4a3f-b954-ecd050b8cb41-credential-socket" pod "ffb30cac-fb9f-4a3f-b954-ecd050b8cb41" (UID: "ffb30cac-fb9f-4a3f-b954-ecd050b8cb41"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:07:12.490584 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:07:12.490377    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffb30cac-fb9f-4a3f-b954-ecd050b8cb41-workload-socket" pod "ffb30cac-fb9f-4a3f-b954-ecd050b8cb41" (UID: "ffb30cac-fb9f-4a3f-b954-ecd050b8cb41"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:07:12.490584 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:07:12.490423    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffb30cac-fb9f-4a3f-b954-ecd050b8cb41-workload-certs" pod "ffb30cac-fb9f-4a3f-b954-ecd050b8cb41" (UID: "ffb30cac-fb9f-4a3f-b954-ecd050b8cb41"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:07:12.490756 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:07:12.490735    4921 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffb30cac-fb9f-4a3f-b954-ecd050b8cb41-workload-certs\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:12.490812 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:07:12.490761    4921 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffb30cac-fb9f-4a3f-b954-ecd050b8cb41-istio-data\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:12.490812 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:07:12.490777    4921 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffb30cac-fb9f-4a3f-b954-ecd050b8cb41-credential-socket\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:12.490812 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:07:12.490791    4921 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffb30cac-fb9f-4a3f-b954-ecd050b8cb41-workload-socket\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:12.491082 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:07:12.491060    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffb30cac-fb9f-4a3f-b954-ecd050b8cb41-istiod-ca-cert" pod "ffb30cac-fb9f-4a3f-b954-ecd050b8cb41" (UID: "ffb30cac-fb9f-4a3f-b954-ecd050b8cb41"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:07:12.492308 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:07:12.492282    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ffb30cac-fb9f-4a3f-b954-ecd050b8cb41-istio-podinfo" pod "ffb30cac-fb9f-4a3f-b954-ecd050b8cb41" (UID: "ffb30cac-fb9f-4a3f-b954-ecd050b8cb41"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:07:12.492420 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:07:12.492306    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffb30cac-fb9f-4a3f-b954-ecd050b8cb41-istio-token" pod "ffb30cac-fb9f-4a3f-b954-ecd050b8cb41" (UID: "ffb30cac-fb9f-4a3f-b954-ecd050b8cb41"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:07:12.492625 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:07:12.492602    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffb30cac-fb9f-4a3f-b954-ecd050b8cb41-istio-envoy" pod "ffb30cac-fb9f-4a3f-b954-ecd050b8cb41" (UID: "ffb30cac-fb9f-4a3f-b954-ecd050b8cb41"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:07:12.492712 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:07:12.492621    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffb30cac-fb9f-4a3f-b954-ecd050b8cb41-kube-api-access-jqrmk" pod "ffb30cac-fb9f-4a3f-b954-ecd050b8cb41" (UID: "ffb30cac-fb9f-4a3f-b954-ecd050b8cb41"). InnerVolumeSpecName "kube-api-access-jqrmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:07:12.591938 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:07:12.591907    4921 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffb30cac-fb9f-4a3f-b954-ecd050b8cb41-istio-token\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:12.591938 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:07:12.591934    4921 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffb30cac-fb9f-4a3f-b954-ecd050b8cb41-istiod-ca-cert\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:12.591938 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:07:12.591944    4921 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffb30cac-fb9f-4a3f-b954-ecd050b8cb41-istio-envoy\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:12.592155 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:07:12.591954    4921 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffb30cac-fb9f-4a3f-b954-ecd050b8cb41-istio-podinfo\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:12.592155 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:07:12.591963    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqrmk\" (UniqueName: \"kubernetes.io/projected/ffb30cac-fb9f-4a3f-b954-ecd050b8cb41-kube-api-access-jqrmk\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:13.151258 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:07:13.151177    4921 generic.go:356] "Generic (PLEG): container finished" podID="ffb30cac-fb9f-4a3f-b954-ecd050b8cb41" containerID="8197b39e92a354ea25c97f74a91165cc703aa711b45383df2ee9f05f04a8dd79" exitCode=0
Nov 03 20:07:13.151258 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:07:13.151224    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt"
Nov 03 20:07:13.151451 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:07:13.151261    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt" event={"ID":"ffb30cac-fb9f-4a3f-b954-ecd050b8cb41","Type":"ContainerDied","Data":"8197b39e92a354ea25c97f74a91165cc703aa711b45383df2ee9f05f04a8dd79"}
Nov 03 20:07:13.151451 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:07:13.151316    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt" event={"ID":"ffb30cac-fb9f-4a3f-b954-ecd050b8cb41","Type":"ContainerDied","Data":"71b35644987cc18d4a7056f19783b214f4626681eaa4fc9fdc91f36d1b3ca599"}
Nov 03 20:07:13.151451 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:07:13.151347    4921 scope.go:122] "RemoveContainer" containerID="8197b39e92a354ea25c97f74a91165cc703aa711b45383df2ee9f05f04a8dd79"
Nov 03 20:07:13.162609 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:07:13.162580    4921 scope.go:122] "RemoveContainer" containerID="8197b39e92a354ea25c97f74a91165cc703aa711b45383df2ee9f05f04a8dd79"
Nov 03 20:07:13.162925 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:07:13.162905    4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8197b39e92a354ea25c97f74a91165cc703aa711b45383df2ee9f05f04a8dd79\": container with ID starting with 8197b39e92a354ea25c97f74a91165cc703aa711b45383df2ee9f05f04a8dd79 not found: ID does not exist" containerID="8197b39e92a354ea25c97f74a91165cc703aa711b45383df2ee9f05f04a8dd79"
Nov 03 20:07:13.162981 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:07:13.162938    4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8197b39e92a354ea25c97f74a91165cc703aa711b45383df2ee9f05f04a8dd79"} err="failed to get container status \"8197b39e92a354ea25c97f74a91165cc703aa711b45383df2ee9f05f04a8dd79\": rpc error: code = NotFound desc = could not find container \"8197b39e92a354ea25c97f74a91165cc703aa711b45383df2ee9f05f04a8dd79\": container with ID starting with 8197b39e92a354ea25c97f74a91165cc703aa711b45383df2ee9f05f04a8dd79 not found: ID does not exist"
Nov 03 20:07:13.167496 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:07:13.167474    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt"]
Nov 03 20:07:13.171153 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:07:13.171133    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-f4e123-openshift-default-77f874d747-zv5wt"]
Nov 03 20:07:14.890255 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:07:14.890218    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffb30cac-fb9f-4a3f-b954-ecd050b8cb41" path="/var/lib/kubelet/pods/ffb30cac-fb9f-4a3f-b954-ecd050b8cb41/volumes"
Nov 03 20:07:31.669134 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:07:31.669102    4921 scope.go:122] "RemoveContainer" containerID="09364d1871f49d52685b16b5c35681673f68553431f9b98300c0323f5aa599ff"
Nov 03 20:07:31.679541 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:07:31.679522    4921 scope.go:122] "RemoveContainer" containerID="8464e2562ac52ee9e1b79c576735ff42e63ff6e81a1fda86a5bb26c1888b30c7"
Nov 03 20:07:32.922893 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:07:32.922851    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 20:08:01.558136 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:01.558103    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-801162-74759cf659-bdg8m"]
Nov 03 20:08:01.570568 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:01.570532    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-801162-74c864f4c-kwqd4"]
Nov 03 20:08:01.867454 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:01.867376    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-801162-695dd8cc6c-hl47x"]
Nov 03 20:08:01.867694 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:01.867638    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-801162-695dd8cc6c-hl47x" podUID="28288186-9960-4f98-8506-95c3dbf53cfd" containerName="maas-api" containerID="cri-o://40d43bcefc6d10b4f36bb9063d6a6b70877fca16125196b3add451cd63bc5248" gracePeriod=30
Nov 03 20:08:02.341407 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:02.341374    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-801162-695dd8cc6c-hl47x"
Nov 03 20:08:02.463552 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:02.463420    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28288186-9960-4f98-8506-95c3dbf53cfd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28288186-9960-4f98-8506-95c3dbf53cfd-openshift-service-ca\") pod \"28288186-9960-4f98-8506-95c3dbf53cfd\" (UID: \"28288186-9960-4f98-8506-95c3dbf53cfd\") "
Nov 03 20:08:02.464826 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:02.463623    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28288186-9960-4f98-8506-95c3dbf53cfd-kube-api-access-8q54g\" (UniqueName: \"kubernetes.io/projected/28288186-9960-4f98-8506-95c3dbf53cfd-kube-api-access-8q54g\") pod \"28288186-9960-4f98-8506-95c3dbf53cfd\" (UID: \"28288186-9960-4f98-8506-95c3dbf53cfd\") "
Nov 03 20:08:02.464826 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:02.463744    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28288186-9960-4f98-8506-95c3dbf53cfd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28288186-9960-4f98-8506-95c3dbf53cfd-maas-api-tls\") pod \"28288186-9960-4f98-8506-95c3dbf53cfd\" (UID: \"28288186-9960-4f98-8506-95c3dbf53cfd\") "
Nov 03 20:08:02.464826 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:02.463801    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28288186-9960-4f98-8506-95c3dbf53cfd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/28288186-9960-4f98-8506-95c3dbf53cfd-metrics-certs\") pod \"28288186-9960-4f98-8506-95c3dbf53cfd\" (UID: \"28288186-9960-4f98-8506-95c3dbf53cfd\") "
Nov 03 20:08:02.464826 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:02.463938    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28288186-9960-4f98-8506-95c3dbf53cfd-openshift-service-ca" pod "28288186-9960-4f98-8506-95c3dbf53cfd" (UID: "28288186-9960-4f98-8506-95c3dbf53cfd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:08:02.464826 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:02.464297    4921 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28288186-9960-4f98-8506-95c3dbf53cfd-openshift-service-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:02.467500 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:02.467474    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28288186-9960-4f98-8506-95c3dbf53cfd-kube-api-access-8q54g" pod "28288186-9960-4f98-8506-95c3dbf53cfd" (UID: "28288186-9960-4f98-8506-95c3dbf53cfd"). InnerVolumeSpecName "kube-api-access-8q54g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:08:02.467649 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:02.467508    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28288186-9960-4f98-8506-95c3dbf53cfd-metrics-certs" pod "28288186-9960-4f98-8506-95c3dbf53cfd" (UID: "28288186-9960-4f98-8506-95c3dbf53cfd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:08:02.467748 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:02.467719    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28288186-9960-4f98-8506-95c3dbf53cfd-maas-api-tls" pod "28288186-9960-4f98-8506-95c3dbf53cfd" (UID: "28288186-9960-4f98-8506-95c3dbf53cfd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:08:02.492550 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:02.492517    4921 generic.go:356] "Generic (PLEG): container finished" podID="28288186-9960-4f98-8506-95c3dbf53cfd" containerID="40d43bcefc6d10b4f36bb9063d6a6b70877fca16125196b3add451cd63bc5248" exitCode=0
Nov 03 20:08:02.492701 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:02.492566    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-801162-695dd8cc6c-hl47x"
Nov 03 20:08:02.492701 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:02.492593    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-801162-695dd8cc6c-hl47x" event={"ID":"28288186-9960-4f98-8506-95c3dbf53cfd","Type":"ContainerDied","Data":"40d43bcefc6d10b4f36bb9063d6a6b70877fca16125196b3add451cd63bc5248"}
Nov 03 20:08:02.492701 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:02.492649    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-801162-695dd8cc6c-hl47x" event={"ID":"28288186-9960-4f98-8506-95c3dbf53cfd","Type":"ContainerDied","Data":"31265a8bdb5825f26711f89f15772e3dd891e58e69881352e261370712aa9cde"}
Nov 03 20:08:02.492701 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:02.492685    4921 scope.go:122] "RemoveContainer" containerID="40d43bcefc6d10b4f36bb9063d6a6b70877fca16125196b3add451cd63bc5248"
Nov 03 20:08:02.504603 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:02.504586    4921 scope.go:122] "RemoveContainer" containerID="40d43bcefc6d10b4f36bb9063d6a6b70877fca16125196b3add451cd63bc5248"
Nov 03 20:08:02.504954 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:08:02.504931    4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40d43bcefc6d10b4f36bb9063d6a6b70877fca16125196b3add451cd63bc5248\": container with ID starting with 40d43bcefc6d10b4f36bb9063d6a6b70877fca16125196b3add451cd63bc5248 not found: ID does not exist" containerID="40d43bcefc6d10b4f36bb9063d6a6b70877fca16125196b3add451cd63bc5248"
Nov 03 20:08:02.505050 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:02.504961    4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40d43bcefc6d10b4f36bb9063d6a6b70877fca16125196b3add451cd63bc5248"} err="failed to get container status \"40d43bcefc6d10b4f36bb9063d6a6b70877fca16125196b3add451cd63bc5248\": rpc error: code = NotFound desc = could not find container \"40d43bcefc6d10b4f36bb9063d6a6b70877fca16125196b3add451cd63bc5248\": container with ID starting with 40d43bcefc6d10b4f36bb9063d6a6b70877fca16125196b3add451cd63bc5248 not found: ID does not exist"
Nov 03 20:08:02.515353 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:02.515330    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-801162-695dd8cc6c-hl47x"]
Nov 03 20:08:02.517915 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:02.517896    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-801162-695dd8cc6c-hl47x"]
Nov 03 20:08:02.565304 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:02.565275    4921 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28288186-9960-4f98-8506-95c3dbf53cfd-metrics-certs\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:02.565304 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:02.565302    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8q54g\" (UniqueName: \"kubernetes.io/projected/28288186-9960-4f98-8506-95c3dbf53cfd-kube-api-access-8q54g\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:02.565738 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:02.565314    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28288186-9960-4f98-8506-95c3dbf53cfd-maas-api-tls\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:02.887940 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:02.887906    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28288186-9960-4f98-8506-95c3dbf53cfd" path="/var/lib/kubelet/pods/28288186-9960-4f98-8506-95c3dbf53cfd/volumes"
Nov 03 20:08:02.920527 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:08:02.920490    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 20:08:04.298413 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:04.298376    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-dce558-566ddc874c-ctw65"]
Nov 03 20:08:04.311460 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:04.311425    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dce558-77fccb9bff-qgtj8"]
Nov 03 20:08:04.514713 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:04.514636    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dce558-7bcc44d6c-s92vv"]
Nov 03 20:08:04.514946 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:04.514916    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dce558-7bcc44d6c-s92vv" podUID="6cb2ad5f-35b4-4c92-9499-d7ac8152a72c" containerName="maas-api" containerID="cri-o://c794af3dc9751f62d719af8054db882f33d445e48fc32054cd4f774e40b29d2f" gracePeriod=30
Nov 03 20:08:04.863511 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:04.863485    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dce558-7bcc44d6c-s92vv"
Nov 03 20:08:04.896090 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:04.896020    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6cb2ad5f-35b4-4c92-9499-d7ac8152a72c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cb2ad5f-35b4-4c92-9499-d7ac8152a72c-metrics-certs\") pod \"6cb2ad5f-35b4-4c92-9499-d7ac8152a72c\" (UID: \"6cb2ad5f-35b4-4c92-9499-d7ac8152a72c\") "
Nov 03 20:08:04.896245 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:04.896104    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6cb2ad5f-35b4-4c92-9499-d7ac8152a72c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cb2ad5f-35b4-4c92-9499-d7ac8152a72c-maas-api-tls\") pod \"6cb2ad5f-35b4-4c92-9499-d7ac8152a72c\" (UID: \"6cb2ad5f-35b4-4c92-9499-d7ac8152a72c\") "
Nov 03 20:08:04.896245 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:04.896197    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cb2ad5f-35b4-4c92-9499-d7ac8152a72c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cb2ad5f-35b4-4c92-9499-d7ac8152a72c-openshift-service-ca\") pod \"6cb2ad5f-35b4-4c92-9499-d7ac8152a72c\" (UID: \"6cb2ad5f-35b4-4c92-9499-d7ac8152a72c\") "
Nov 03 20:08:04.896579 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:04.896547    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cb2ad5f-35b4-4c92-9499-d7ac8152a72c-openshift-service-ca" pod "6cb2ad5f-35b4-4c92-9499-d7ac8152a72c" (UID: "6cb2ad5f-35b4-4c92-9499-d7ac8152a72c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:08:04.898268 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:04.898238    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cb2ad5f-35b4-4c92-9499-d7ac8152a72c-maas-api-tls" pod "6cb2ad5f-35b4-4c92-9499-d7ac8152a72c" (UID: "6cb2ad5f-35b4-4c92-9499-d7ac8152a72c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:08:04.898368 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:04.898318    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cb2ad5f-35b4-4c92-9499-d7ac8152a72c-metrics-certs" pod "6cb2ad5f-35b4-4c92-9499-d7ac8152a72c" (UID: "6cb2ad5f-35b4-4c92-9499-d7ac8152a72c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:08:04.997040 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:04.997004    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cb2ad5f-35b4-4c92-9499-d7ac8152a72c-kube-api-access-mcgvh\" (UniqueName: \"kubernetes.io/projected/6cb2ad5f-35b4-4c92-9499-d7ac8152a72c-kube-api-access-mcgvh\") pod \"6cb2ad5f-35b4-4c92-9499-d7ac8152a72c\" (UID: \"6cb2ad5f-35b4-4c92-9499-d7ac8152a72c\") "
Nov 03 20:08:04.997302 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:04.997289    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cb2ad5f-35b4-4c92-9499-d7ac8152a72c-maas-api-tls\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:04.997350 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:04.997308    4921 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cb2ad5f-35b4-4c92-9499-d7ac8152a72c-openshift-service-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:04.997350 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:04.997321    4921 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cb2ad5f-35b4-4c92-9499-d7ac8152a72c-metrics-certs\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:04.999083 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:04.999060    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cb2ad5f-35b4-4c92-9499-d7ac8152a72c-kube-api-access-mcgvh" pod "6cb2ad5f-35b4-4c92-9499-d7ac8152a72c" (UID: "6cb2ad5f-35b4-4c92-9499-d7ac8152a72c"). InnerVolumeSpecName "kube-api-access-mcgvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:08:05.098165 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:05.098127    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcgvh\" (UniqueName: \"kubernetes.io/projected/6cb2ad5f-35b4-4c92-9499-d7ac8152a72c-kube-api-access-mcgvh\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:05.523406 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:05.523364    4921 generic.go:356] "Generic (PLEG): container finished" podID="6cb2ad5f-35b4-4c92-9499-d7ac8152a72c" containerID="c794af3dc9751f62d719af8054db882f33d445e48fc32054cd4f774e40b29d2f" exitCode=0
Nov 03 20:08:05.524007 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:05.523430    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dce558-7bcc44d6c-s92vv" event={"ID":"6cb2ad5f-35b4-4c92-9499-d7ac8152a72c","Type":"ContainerDied","Data":"c794af3dc9751f62d719af8054db882f33d445e48fc32054cd4f774e40b29d2f"}
Nov 03 20:08:05.524007 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:05.523527    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dce558-7bcc44d6c-s92vv" event={"ID":"6cb2ad5f-35b4-4c92-9499-d7ac8152a72c","Type":"ContainerDied","Data":"7ccfffc1db40d26dca2d10ce6dda5ec22d1849744484ae1122cdd427285fa124"}
Nov 03 20:08:05.524007 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:05.523581    4921 scope.go:122] "RemoveContainer" containerID="c794af3dc9751f62d719af8054db882f33d445e48fc32054cd4f774e40b29d2f"
Nov 03 20:08:05.524537 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:05.524503    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dce558-7bcc44d6c-s92vv"
Nov 03 20:08:05.543520 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:05.543498    4921 scope.go:122] "RemoveContainer" containerID="c794af3dc9751f62d719af8054db882f33d445e48fc32054cd4f774e40b29d2f"
Nov 03 20:08:05.543899 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:08:05.543878    4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c794af3dc9751f62d719af8054db882f33d445e48fc32054cd4f774e40b29d2f\": container with ID starting with c794af3dc9751f62d719af8054db882f33d445e48fc32054cd4f774e40b29d2f not found: ID does not exist" containerID="c794af3dc9751f62d719af8054db882f33d445e48fc32054cd4f774e40b29d2f"
Nov 03 20:08:05.543979 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:05.543908    4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c794af3dc9751f62d719af8054db882f33d445e48fc32054cd4f774e40b29d2f"} err="failed to get container status \"c794af3dc9751f62d719af8054db882f33d445e48fc32054cd4f774e40b29d2f\": rpc error: code = NotFound desc = could not find container \"c794af3dc9751f62d719af8054db882f33d445e48fc32054cd4f774e40b29d2f\": container with ID starting with c794af3dc9751f62d719af8054db882f33d445e48fc32054cd4f774e40b29d2f not found: ID does not exist"
Nov 03 20:08:05.558624 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:05.558598    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dce558-7bcc44d6c-s92vv"]
Nov 03 20:08:05.562162 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:05.562140    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-dce558-7bcc44d6c-s92vv"]
Nov 03 20:08:06.559400 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:06.559353    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-801162-74759cf659-bdg8m" podUID="0df8b636-6a8f-4a9d-ab89-2dabedc2bc21" containerName="payload-processing" containerID="cri-o://df4113ff6e49262a34805e323c91ceb7c186c5b1c36dccae631fd70e206eb4c0" gracePeriod=25
Nov 03 20:08:06.571548 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:06.571484    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-801162-74c864f4c-kwqd4" podUID="cbdab358-70c7-4c98-a2c4-4dc4a924e8d9" containerName="payload-pre-processing" containerID="cri-o://10feec450e6673f6a0b644e19d8bdc6ea0a6994747257faeba9230a38a598ba3" gracePeriod=25
Nov 03 20:08:06.855591 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:06.855561    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-801162-74759cf659-bdg8m"
Nov 03 20:08:06.877624 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:06.877596    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-801162-74c864f4c-kwqd4"
Nov 03 20:08:06.887618 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:06.887586    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cb2ad5f-35b4-4c92-9499-d7ac8152a72c" path="/var/lib/kubelet/pods/6cb2ad5f-35b4-4c92-9499-d7ac8152a72c/volumes"
Nov 03 20:08:06.914551 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:06.914520    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0df8b636-6a8f-4a9d-ab89-2dabedc2bc21-kube-api-access-tvsxx\" (UniqueName: \"kubernetes.io/projected/0df8b636-6a8f-4a9d-ab89-2dabedc2bc21-kube-api-access-tvsxx\") pod \"0df8b636-6a8f-4a9d-ab89-2dabedc2bc21\" (UID: \"0df8b636-6a8f-4a9d-ab89-2dabedc2bc21\") "
Nov 03 20:08:06.914759 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:06.914606    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0df8b636-6a8f-4a9d-ab89-2dabedc2bc21-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0df8b636-6a8f-4a9d-ab89-2dabedc2bc21-plugins-config-volume\") pod \"0df8b636-6a8f-4a9d-ab89-2dabedc2bc21\" (UID: \"0df8b636-6a8f-4a9d-ab89-2dabedc2bc21\") "
Nov 03 20:08:06.914759 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:06.914646    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbdab358-70c7-4c98-a2c4-4dc4a924e8d9-kube-api-access-86ww2\" (UniqueName: \"kubernetes.io/projected/cbdab358-70c7-4c98-a2c4-4dc4a924e8d9-kube-api-access-86ww2\") pod \"cbdab358-70c7-4c98-a2c4-4dc4a924e8d9\" (UID: \"cbdab358-70c7-4c98-a2c4-4dc4a924e8d9\") "
Nov 03 20:08:06.914759 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:06.914726    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbdab358-70c7-4c98-a2c4-4dc4a924e8d9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbdab358-70c7-4c98-a2c4-4dc4a924e8d9-plugins-config-volume\") pod \"cbdab358-70c7-4c98-a2c4-4dc4a924e8d9\" (UID: \"cbdab358-70c7-4c98-a2c4-4dc4a924e8d9\") "
Nov 03 20:08:06.915130 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:06.915100    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0df8b636-6a8f-4a9d-ab89-2dabedc2bc21-plugins-config-volume" pod "0df8b636-6a8f-4a9d-ab89-2dabedc2bc21" (UID: "0df8b636-6a8f-4a9d-ab89-2dabedc2bc21"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:08:06.915395 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:06.915368    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbdab358-70c7-4c98-a2c4-4dc4a924e8d9-plugins-config-volume" pod "cbdab358-70c7-4c98-a2c4-4dc4a924e8d9" (UID: "cbdab358-70c7-4c98-a2c4-4dc4a924e8d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:08:06.917545 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:06.917522    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbdab358-70c7-4c98-a2c4-4dc4a924e8d9-kube-api-access-86ww2" pod "cbdab358-70c7-4c98-a2c4-4dc4a924e8d9" (UID: "cbdab358-70c7-4c98-a2c4-4dc4a924e8d9"). InnerVolumeSpecName "kube-api-access-86ww2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:08:06.917638 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:06.917550    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0df8b636-6a8f-4a9d-ab89-2dabedc2bc21-kube-api-access-tvsxx" pod "0df8b636-6a8f-4a9d-ab89-2dabedc2bc21" (UID: "0df8b636-6a8f-4a9d-ab89-2dabedc2bc21"). InnerVolumeSpecName "kube-api-access-tvsxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:08:07.016011 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:07.015974    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvsxx\" (UniqueName: \"kubernetes.io/projected/0df8b636-6a8f-4a9d-ab89-2dabedc2bc21-kube-api-access-tvsxx\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:07.016011 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:07.016005    4921 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0df8b636-6a8f-4a9d-ab89-2dabedc2bc21-plugins-config-volume\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:07.016011 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:07.016015    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86ww2\" (UniqueName: \"kubernetes.io/projected/cbdab358-70c7-4c98-a2c4-4dc4a924e8d9-kube-api-access-86ww2\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:07.016255 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:07.016026    4921 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbdab358-70c7-4c98-a2c4-4dc4a924e8d9-plugins-config-volume\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:07.538519 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:07.538484    4921 generic.go:356] "Generic (PLEG): container finished" podID="0df8b636-6a8f-4a9d-ab89-2dabedc2bc21" containerID="df4113ff6e49262a34805e323c91ceb7c186c5b1c36dccae631fd70e206eb4c0" exitCode=0
Nov 03 20:08:07.538773 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:07.538552    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-801162-74759cf659-bdg8m"
Nov 03 20:08:07.538773 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:07.538574    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-801162-74759cf659-bdg8m" event={"ID":"0df8b636-6a8f-4a9d-ab89-2dabedc2bc21","Type":"ContainerDied","Data":"df4113ff6e49262a34805e323c91ceb7c186c5b1c36dccae631fd70e206eb4c0"}
Nov 03 20:08:07.538773 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:07.538643    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-801162-74759cf659-bdg8m" event={"ID":"0df8b636-6a8f-4a9d-ab89-2dabedc2bc21","Type":"ContainerDied","Data":"eda212e404d652116073cf1a1f0b72a1d707e24b5b2ef4df5db5e220cefa2895"}
Nov 03 20:08:07.538773 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:07.538703    4921 scope.go:122] "RemoveContainer" containerID="df4113ff6e49262a34805e323c91ceb7c186c5b1c36dccae631fd70e206eb4c0"
Nov 03 20:08:07.540376 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:07.540351    4921 generic.go:356] "Generic (PLEG): container finished" podID="cbdab358-70c7-4c98-a2c4-4dc4a924e8d9" containerID="10feec450e6673f6a0b644e19d8bdc6ea0a6994747257faeba9230a38a598ba3" exitCode=0
Nov 03 20:08:07.540509 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:07.540460    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-801162-74c864f4c-kwqd4" event={"ID":"cbdab358-70c7-4c98-a2c4-4dc4a924e8d9","Type":"ContainerDied","Data":"10feec450e6673f6a0b644e19d8bdc6ea0a6994747257faeba9230a38a598ba3"}
Nov 03 20:08:07.540509 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:07.540474    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-801162-74c864f4c-kwqd4"
Nov 03 20:08:07.540509 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:07.540500    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-801162-74c864f4c-kwqd4" event={"ID":"cbdab358-70c7-4c98-a2c4-4dc4a924e8d9","Type":"ContainerDied","Data":"5691f25b141e55994421ade9c42e2c8ed473dc77a13d335caf6b575588605632"}
Nov 03 20:08:07.551065 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:07.551041    4921 scope.go:122] "RemoveContainer" containerID="df4113ff6e49262a34805e323c91ceb7c186c5b1c36dccae631fd70e206eb4c0"
Nov 03 20:08:07.551378 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:08:07.551358    4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df4113ff6e49262a34805e323c91ceb7c186c5b1c36dccae631fd70e206eb4c0\": container with ID starting with df4113ff6e49262a34805e323c91ceb7c186c5b1c36dccae631fd70e206eb4c0 not found: ID does not exist" containerID="df4113ff6e49262a34805e323c91ceb7c186c5b1c36dccae631fd70e206eb4c0"
Nov 03 20:08:07.551453 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:07.551386    4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df4113ff6e49262a34805e323c91ceb7c186c5b1c36dccae631fd70e206eb4c0"} err="failed to get container status \"df4113ff6e49262a34805e323c91ceb7c186c5b1c36dccae631fd70e206eb4c0\": rpc error: code = NotFound desc = could not find container \"df4113ff6e49262a34805e323c91ceb7c186c5b1c36dccae631fd70e206eb4c0\": container with ID starting with df4113ff6e49262a34805e323c91ceb7c186c5b1c36dccae631fd70e206eb4c0 not found: ID does not exist"
Nov 03 20:08:07.551453 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:07.551402    4921 scope.go:122] "RemoveContainer" containerID="10feec450e6673f6a0b644e19d8bdc6ea0a6994747257faeba9230a38a598ba3"
Nov 03 20:08:07.561461 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:07.561255    4921 scope.go:122] "RemoveContainer" containerID="10feec450e6673f6a0b644e19d8bdc6ea0a6994747257faeba9230a38a598ba3"
Nov 03 20:08:07.561741 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:08:07.561530    4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10feec450e6673f6a0b644e19d8bdc6ea0a6994747257faeba9230a38a598ba3\": container with ID starting with 10feec450e6673f6a0b644e19d8bdc6ea0a6994747257faeba9230a38a598ba3 not found: ID does not exist" containerID="10feec450e6673f6a0b644e19d8bdc6ea0a6994747257faeba9230a38a598ba3"
Nov 03 20:08:07.561741 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:07.561554    4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10feec450e6673f6a0b644e19d8bdc6ea0a6994747257faeba9230a38a598ba3"} err="failed to get container status \"10feec450e6673f6a0b644e19d8bdc6ea0a6994747257faeba9230a38a598ba3\": rpc error: code = NotFound desc = could not find container \"10feec450e6673f6a0b644e19d8bdc6ea0a6994747257faeba9230a38a598ba3\": container with ID starting with 10feec450e6673f6a0b644e19d8bdc6ea0a6994747257faeba9230a38a598ba3 not found: ID does not exist"
Nov 03 20:08:07.567307 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:07.567286    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-801162-74759cf659-bdg8m"]
Nov 03 20:08:07.571068 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:07.571047    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-801162-74759cf659-bdg8m"]
Nov 03 20:08:07.581766 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:07.581741    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-801162-74c864f4c-kwqd4"]
Nov 03 20:08:07.587048 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:07.587027    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-801162-74c864f4c-kwqd4"]
Nov 03 20:08:07.816420 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:07.816321    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-801162-74759cf659-bdg8m" podUID="0df8b636-6a8f-4a9d-ab89-2dabedc2bc21" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.182:50052\" within 1s: context deadline exceeded"
Nov 03 20:08:07.816420 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:07.816324    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-801162-74c864f4c-kwqd4" podUID="cbdab358-70c7-4c98-a2c4-4dc4a924e8d9" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.181:50052\" within 1s: context deadline exceeded"
Nov 03 20:08:07.816621 ip-10-0-134-222 kubenswrapper[4921]: W1103 20:08:07.816434    4921 logging.go:55] [core] [Channel #4091 SubChannel #4093]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.181:50052", ServerName: "10.132.0.181:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.181:50052: operation was canceled"
Nov 03 20:08:07.816621 ip-10-0-134-222 kubenswrapper[4921]: W1103 20:08:07.816434    4921 logging.go:55] [core] [Channel #4092 SubChannel #4094]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.182:50052", ServerName: "10.132.0.182:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.182:50052: operation was canceled"
Nov 03 20:08:08.888101 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:08.888059    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0df8b636-6a8f-4a9d-ab89-2dabedc2bc21" path="/var/lib/kubelet/pods/0df8b636-6a8f-4a9d-ab89-2dabedc2bc21/volumes"
Nov 03 20:08:08.888690 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:08.888655    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbdab358-70c7-4c98-a2c4-4dc4a924e8d9" path="/var/lib/kubelet/pods/cbdab358-70c7-4c98-a2c4-4dc4a924e8d9/volumes"
Nov 03 20:08:09.299721 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:09.299627    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-dce558-566ddc874c-ctw65" podUID="8d974919-e441-47b7-9044-9bb57982bdc1" containerName="payload-processing" containerID="cri-o://5ce027893323fce2895f873269ccdb3dfd0e039f5e1fd9827bd6f3e13ba85436" gracePeriod=25
Nov 03 20:08:09.312626 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:09.312585    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dce558-77fccb9bff-qgtj8" podUID="6c554c44-d20a-4efe-9faf-5c30ec3a3f5e" containerName="payload-pre-processing" containerID="cri-o://8eec240a6663240347160fa2c893eb54a925408b75bee523c5313a91310a1a37" gracePeriod=25
Nov 03 20:08:09.567651 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:09.567621    4921 generic.go:356] "Generic (PLEG): container finished" podID="8d974919-e441-47b7-9044-9bb57982bdc1" containerID="5ce027893323fce2895f873269ccdb3dfd0e039f5e1fd9827bd6f3e13ba85436" exitCode=0
Nov 03 20:08:09.567796 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:09.567767    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-dce558-566ddc874c-ctw65" event={"ID":"8d974919-e441-47b7-9044-9bb57982bdc1","Type":"ContainerDied","Data":"5ce027893323fce2895f873269ccdb3dfd0e039f5e1fd9827bd6f3e13ba85436"}
Nov 03 20:08:09.569641 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:09.569620    4921 generic.go:356] "Generic (PLEG): container finished" podID="6c554c44-d20a-4efe-9faf-5c30ec3a3f5e" containerID="8eec240a6663240347160fa2c893eb54a925408b75bee523c5313a91310a1a37" exitCode=0
Nov 03 20:08:09.569764 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:09.569741    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dce558-77fccb9bff-qgtj8" event={"ID":"6c554c44-d20a-4efe-9faf-5c30ec3a3f5e","Type":"ContainerDied","Data":"8eec240a6663240347160fa2c893eb54a925408b75bee523c5313a91310a1a37"}
Nov 03 20:08:09.598500 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:09.598476    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-dce558-566ddc874c-ctw65"
Nov 03 20:08:09.633797 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:09.633697    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dce558-77fccb9bff-qgtj8"
Nov 03 20:08:09.760042 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:09.760003    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c554c44-d20a-4efe-9faf-5c30ec3a3f5e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c554c44-d20a-4efe-9faf-5c30ec3a3f5e-plugins-config-volume\") pod \"6c554c44-d20a-4efe-9faf-5c30ec3a3f5e\" (UID: \"6c554c44-d20a-4efe-9faf-5c30ec3a3f5e\") "
Nov 03 20:08:09.760247 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:09.760112    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d974919-e441-47b7-9044-9bb57982bdc1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d974919-e441-47b7-9044-9bb57982bdc1-plugins-config-volume\") pod \"8d974919-e441-47b7-9044-9bb57982bdc1\" (UID: \"8d974919-e441-47b7-9044-9bb57982bdc1\") "
Nov 03 20:08:09.760247 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:09.760163    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c554c44-d20a-4efe-9faf-5c30ec3a3f5e-kube-api-access-wz4t5\" (UniqueName: \"kubernetes.io/projected/6c554c44-d20a-4efe-9faf-5c30ec3a3f5e-kube-api-access-wz4t5\") pod \"6c554c44-d20a-4efe-9faf-5c30ec3a3f5e\" (UID: \"6c554c44-d20a-4efe-9faf-5c30ec3a3f5e\") "
Nov 03 20:08:09.760247 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:09.760197    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d974919-e441-47b7-9044-9bb57982bdc1-kube-api-access-rr2w7\" (UniqueName: \"kubernetes.io/projected/8d974919-e441-47b7-9044-9bb57982bdc1-kube-api-access-rr2w7\") pod \"8d974919-e441-47b7-9044-9bb57982bdc1\" (UID: \"8d974919-e441-47b7-9044-9bb57982bdc1\") "
Nov 03 20:08:09.761100 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:09.761070    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d974919-e441-47b7-9044-9bb57982bdc1-plugins-config-volume" pod "8d974919-e441-47b7-9044-9bb57982bdc1" (UID: "8d974919-e441-47b7-9044-9bb57982bdc1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:08:09.762054 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:09.762026    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c554c44-d20a-4efe-9faf-5c30ec3a3f5e-plugins-config-volume" pod "6c554c44-d20a-4efe-9faf-5c30ec3a3f5e" (UID: "6c554c44-d20a-4efe-9faf-5c30ec3a3f5e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:08:09.764017 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:09.763993    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d974919-e441-47b7-9044-9bb57982bdc1-kube-api-access-rr2w7" pod "8d974919-e441-47b7-9044-9bb57982bdc1" (UID: "8d974919-e441-47b7-9044-9bb57982bdc1"). InnerVolumeSpecName "kube-api-access-rr2w7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:08:09.764124 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:09.764037    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c554c44-d20a-4efe-9faf-5c30ec3a3f5e-kube-api-access-wz4t5" pod "6c554c44-d20a-4efe-9faf-5c30ec3a3f5e" (UID: "6c554c44-d20a-4efe-9faf-5c30ec3a3f5e"). InnerVolumeSpecName "kube-api-access-wz4t5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:08:09.861776 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:09.861626    4921 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c554c44-d20a-4efe-9faf-5c30ec3a3f5e-plugins-config-volume\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:09.861776 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:09.861720    4921 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d974919-e441-47b7-9044-9bb57982bdc1-plugins-config-volume\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:09.861776 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:09.861740    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wz4t5\" (UniqueName: \"kubernetes.io/projected/6c554c44-d20a-4efe-9faf-5c30ec3a3f5e-kube-api-access-wz4t5\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:09.861776 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:09.861754    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rr2w7\" (UniqueName: \"kubernetes.io/projected/8d974919-e441-47b7-9044-9bb57982bdc1-kube-api-access-rr2w7\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:10.577453 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:10.577375    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-dce558-566ddc874c-ctw65"
Nov 03 20:08:10.577453 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:10.577404    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-dce558-566ddc874c-ctw65" event={"ID":"8d974919-e441-47b7-9044-9bb57982bdc1","Type":"ContainerDied","Data":"80f5dc380d2fa29c7166db1a47139fe512fe93779e898695ec53d36ff5a3b413"}
Nov 03 20:08:10.577979 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:10.577469    4921 scope.go:122] "RemoveContainer" containerID="5ce027893323fce2895f873269ccdb3dfd0e039f5e1fd9827bd6f3e13ba85436"
Nov 03 20:08:10.578973 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:10.578954    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dce558-77fccb9bff-qgtj8"
Nov 03 20:08:10.579077 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:10.578973    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dce558-77fccb9bff-qgtj8" event={"ID":"6c554c44-d20a-4efe-9faf-5c30ec3a3f5e","Type":"ContainerDied","Data":"d8d13f45ce6cafa28158ba3203a5ace7d0b789ad37bb85056aa3517c58132e29"}
Nov 03 20:08:10.588653 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:10.588636    4921 scope.go:122] "RemoveContainer" containerID="8eec240a6663240347160fa2c893eb54a925408b75bee523c5313a91310a1a37"
Nov 03 20:08:10.602031 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:10.602009    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-dce558-566ddc874c-ctw65"]
Nov 03 20:08:10.613657 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:10.613625    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-dce558-566ddc874c-ctw65"]
Nov 03 20:08:10.617255 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:10.617228    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dce558-77fccb9bff-qgtj8"]
Nov 03 20:08:10.620084 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:10.620060    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dce558-77fccb9bff-qgtj8"]
Nov 03 20:08:10.887912 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:10.887822    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c554c44-d20a-4efe-9faf-5c30ec3a3f5e" path="/var/lib/kubelet/pods/6c554c44-d20a-4efe-9faf-5c30ec3a3f5e/volumes"
Nov 03 20:08:10.888333 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:10.888312    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d974919-e441-47b7-9044-9bb57982bdc1" path="/var/lib/kubelet/pods/8d974919-e441-47b7-9044-9bb57982bdc1/volumes"
Nov 03 20:08:14.897768 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:14.897736    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c"]
Nov 03 20:08:14.898274 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:14.897930    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c" podUID="1b9000a2-e11e-4afb-9ea7-4e4f1446d918" containerName="istio-proxy" containerID="cri-o://7ccd826447701663e2f015beb307be7f4751b6292b68aba7f43b41b959fcbe92" gracePeriod=30
Nov 03 20:08:16.317796 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:16.317761    4921 patch_prober.go:29] interesting pod/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:08:16.318196 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:16.317825    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c" podUID="1b9000a2-e11e-4afb-9ea7-4e4f1446d918" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:08:20.162282 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:20.162253    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c"
Nov 03 20:08:20.259042 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:20.259013    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b9000a2-e11e-4afb-9ea7-4e4f1446d918-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b9000a2-e11e-4afb-9ea7-4e4f1446d918-workload-certs\") pod \"1b9000a2-e11e-4afb-9ea7-4e4f1446d918\" (UID: \"1b9000a2-e11e-4afb-9ea7-4e4f1446d918\") "
Nov 03 20:08:20.259249 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:20.259065    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b9000a2-e11e-4afb-9ea7-4e4f1446d918-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b9000a2-e11e-4afb-9ea7-4e4f1446d918-istiod-ca-cert\") pod \"1b9000a2-e11e-4afb-9ea7-4e4f1446d918\" (UID: \"1b9000a2-e11e-4afb-9ea7-4e4f1446d918\") "
Nov 03 20:08:20.259249 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:20.259098    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b9000a2-e11e-4afb-9ea7-4e4f1446d918-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b9000a2-e11e-4afb-9ea7-4e4f1446d918-workload-socket\") pod \"1b9000a2-e11e-4afb-9ea7-4e4f1446d918\" (UID: \"1b9000a2-e11e-4afb-9ea7-4e4f1446d918\") "
Nov 03 20:08:20.259249 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:20.259182    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b9000a2-e11e-4afb-9ea7-4e4f1446d918-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b9000a2-e11e-4afb-9ea7-4e4f1446d918-istio-envoy\") pod \"1b9000a2-e11e-4afb-9ea7-4e4f1446d918\" (UID: \"1b9000a2-e11e-4afb-9ea7-4e4f1446d918\") "
Nov 03 20:08:20.259249 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:20.259236    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b9000a2-e11e-4afb-9ea7-4e4f1446d918-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b9000a2-e11e-4afb-9ea7-4e4f1446d918-credential-socket\") pod \"1b9000a2-e11e-4afb-9ea7-4e4f1446d918\" (UID: \"1b9000a2-e11e-4afb-9ea7-4e4f1446d918\") "
Nov 03 20:08:20.259482 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:20.259290    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b9000a2-e11e-4afb-9ea7-4e4f1446d918-istio-token\" (UniqueName: \"kubernetes.io/projected/1b9000a2-e11e-4afb-9ea7-4e4f1446d918-istio-token\") pod \"1b9000a2-e11e-4afb-9ea7-4e4f1446d918\" (UID: \"1b9000a2-e11e-4afb-9ea7-4e4f1446d918\") "
Nov 03 20:08:20.259482 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:20.259315    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1b9000a2-e11e-4afb-9ea7-4e4f1446d918-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b9000a2-e11e-4afb-9ea7-4e4f1446d918-istio-podinfo\") pod \"1b9000a2-e11e-4afb-9ea7-4e4f1446d918\" (UID: \"1b9000a2-e11e-4afb-9ea7-4e4f1446d918\") "
Nov 03 20:08:20.259482 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:20.259342    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b9000a2-e11e-4afb-9ea7-4e4f1446d918-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b9000a2-e11e-4afb-9ea7-4e4f1446d918-istio-data\") pod \"1b9000a2-e11e-4afb-9ea7-4e4f1446d918\" (UID: \"1b9000a2-e11e-4afb-9ea7-4e4f1446d918\") "
Nov 03 20:08:20.259482 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:20.259370    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b9000a2-e11e-4afb-9ea7-4e4f1446d918-kube-api-access-57pth\" (UniqueName: \"kubernetes.io/projected/1b9000a2-e11e-4afb-9ea7-4e4f1446d918-kube-api-access-57pth\") pod \"1b9000a2-e11e-4afb-9ea7-4e4f1446d918\" (UID: \"1b9000a2-e11e-4afb-9ea7-4e4f1446d918\") "
Nov 03 20:08:20.259482 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:20.259367    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b9000a2-e11e-4afb-9ea7-4e4f1446d918-workload-certs" pod "1b9000a2-e11e-4afb-9ea7-4e4f1446d918" (UID: "1b9000a2-e11e-4afb-9ea7-4e4f1446d918"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:20.259778 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:20.259496    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b9000a2-e11e-4afb-9ea7-4e4f1446d918-workload-socket" pod "1b9000a2-e11e-4afb-9ea7-4e4f1446d918" (UID: "1b9000a2-e11e-4afb-9ea7-4e4f1446d918"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:20.259778 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:20.259708    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b9000a2-e11e-4afb-9ea7-4e4f1446d918-istiod-ca-cert" pod "1b9000a2-e11e-4afb-9ea7-4e4f1446d918" (UID: "1b9000a2-e11e-4afb-9ea7-4e4f1446d918"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:08:20.259778 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:20.259718    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b9000a2-e11e-4afb-9ea7-4e4f1446d918-istio-data" pod "1b9000a2-e11e-4afb-9ea7-4e4f1446d918" (UID: "1b9000a2-e11e-4afb-9ea7-4e4f1446d918"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:20.259918 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:20.259804    4921 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b9000a2-e11e-4afb-9ea7-4e4f1446d918-workload-certs\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:20.259918 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:20.259825    4921 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b9000a2-e11e-4afb-9ea7-4e4f1446d918-workload-socket\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:20.259918 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:20.259826    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b9000a2-e11e-4afb-9ea7-4e4f1446d918-credential-socket" pod "1b9000a2-e11e-4afb-9ea7-4e4f1446d918" (UID: "1b9000a2-e11e-4afb-9ea7-4e4f1446d918"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:20.261538 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:20.261505    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b9000a2-e11e-4afb-9ea7-4e4f1446d918-istio-token" pod "1b9000a2-e11e-4afb-9ea7-4e4f1446d918" (UID: "1b9000a2-e11e-4afb-9ea7-4e4f1446d918"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:08:20.261643 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:20.261600    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1b9000a2-e11e-4afb-9ea7-4e4f1446d918-istio-podinfo" pod "1b9000a2-e11e-4afb-9ea7-4e4f1446d918" (UID: "1b9000a2-e11e-4afb-9ea7-4e4f1446d918"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:08:20.261643 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:20.261626    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b9000a2-e11e-4afb-9ea7-4e4f1446d918-istio-envoy" pod "1b9000a2-e11e-4afb-9ea7-4e4f1446d918" (UID: "1b9000a2-e11e-4afb-9ea7-4e4f1446d918"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:20.261788 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:20.261755    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b9000a2-e11e-4afb-9ea7-4e4f1446d918-kube-api-access-57pth" pod "1b9000a2-e11e-4afb-9ea7-4e4f1446d918" (UID: "1b9000a2-e11e-4afb-9ea7-4e4f1446d918"). InnerVolumeSpecName "kube-api-access-57pth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:08:20.346840 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:20.346800    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk"]
Nov 03 20:08:20.360385 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:20.360350    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv"]
Nov 03 20:08:20.360575 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:20.360451    4921 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b9000a2-e11e-4afb-9ea7-4e4f1446d918-istio-envoy\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:20.360575 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:20.360473    4921 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b9000a2-e11e-4afb-9ea7-4e4f1446d918-credential-socket\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:20.360575 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:20.360483    4921 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b9000a2-e11e-4afb-9ea7-4e4f1446d918-istio-token\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:20.360575 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:20.360492    4921 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b9000a2-e11e-4afb-9ea7-4e4f1446d918-istio-podinfo\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:20.360575 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:20.360501    4921 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b9000a2-e11e-4afb-9ea7-4e4f1446d918-istio-data\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:20.360575 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:20.360510    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57pth\" (UniqueName: \"kubernetes.io/projected/1b9000a2-e11e-4afb-9ea7-4e4f1446d918-kube-api-access-57pth\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:20.360575 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:20.360519    4921 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b9000a2-e11e-4afb-9ea7-4e4f1446d918-istiod-ca-cert\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:20.374788 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:20.374743    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-b30676-5b9668d5b5-xgdqw"]
Nov 03 20:08:20.396331 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:20.396299    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-b30676-7bd74677b7-vw8sk"]
Nov 03 20:08:20.611427 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:20.611335    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-crz4q"]
Nov 03 20:08:20.611593 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:20.611532    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-crz4q" podUID="4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc" containerName="discovery" containerID="cri-o://16f51738c532765a9301416e1c41576c198001166cc12ebb3e00745e487c96a4" gracePeriod=30
Nov 03 20:08:20.644996 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:20.644964    4921 generic.go:356] "Generic (PLEG): container finished" podID="1b9000a2-e11e-4afb-9ea7-4e4f1446d918" containerID="7ccd826447701663e2f015beb307be7f4751b6292b68aba7f43b41b959fcbe92" exitCode=0
Nov 03 20:08:20.645173 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:20.645012    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c"
Nov 03 20:08:20.645173 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:20.645011    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c" event={"ID":"1b9000a2-e11e-4afb-9ea7-4e4f1446d918","Type":"ContainerDied","Data":"7ccd826447701663e2f015beb307be7f4751b6292b68aba7f43b41b959fcbe92"}
Nov 03 20:08:20.645173 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:20.645144    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c" event={"ID":"1b9000a2-e11e-4afb-9ea7-4e4f1446d918","Type":"ContainerDied","Data":"74656a60a921f88ff54820ddced059200af6a8ff5568952ecbac63e0f4ddc638"}
Nov 03 20:08:20.645359 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:20.645176    4921 scope.go:122] "RemoveContainer" containerID="7ccd826447701663e2f015beb307be7f4751b6292b68aba7f43b41b959fcbe92"
Nov 03 20:08:20.677474 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:20.677436    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c"]
Nov 03 20:08:20.681000 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:20.680957    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-dce558-openshift-default-595cb8c97-ffr5c"]
Nov 03 20:08:20.760691 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:20.737918    4921 scope.go:122] "RemoveContainer" containerID="7ccd826447701663e2f015beb307be7f4751b6292b68aba7f43b41b959fcbe92"
Nov 03 20:08:20.760691 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:08:20.752405    4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ccd826447701663e2f015beb307be7f4751b6292b68aba7f43b41b959fcbe92\": container with ID starting with 7ccd826447701663e2f015beb307be7f4751b6292b68aba7f43b41b959fcbe92 not found: ID does not exist" containerID="7ccd826447701663e2f015beb307be7f4751b6292b68aba7f43b41b959fcbe92"
Nov 03 20:08:20.760691 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:20.752455    4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ccd826447701663e2f015beb307be7f4751b6292b68aba7f43b41b959fcbe92"} err="failed to get container status \"7ccd826447701663e2f015beb307be7f4751b6292b68aba7f43b41b959fcbe92\": rpc error: code = NotFound desc = could not find container \"7ccd826447701663e2f015beb307be7f4751b6292b68aba7f43b41b959fcbe92\": container with ID starting with 7ccd826447701663e2f015beb307be7f4751b6292b68aba7f43b41b959fcbe92 not found: ID does not exist"
Nov 03 20:08:20.889234 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:20.889142    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b9000a2-e11e-4afb-9ea7-4e4f1446d918" path="/var/lib/kubelet/pods/1b9000a2-e11e-4afb-9ea7-4e4f1446d918/volumes"
Nov 03 20:08:20.912904 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:20.912881    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-crz4q"
Nov 03 20:08:20.969118 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:20.969081    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc-istio-csr-dns-cert\") pod \"4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc\" (UID: \"4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc\") "
Nov 03 20:08:20.969324 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:20.969144    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc-cacerts\" (UniqueName: \"kubernetes.io/secret/4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc-cacerts\") pod \"4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc\" (UID: \"4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc\") "
Nov 03 20:08:20.969324 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:20.969169    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc-istio-token\" (UniqueName: \"kubernetes.io/projected/4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc-istio-token\") pod \"4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc\" (UID: \"4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc\") "
Nov 03 20:08:20.969324 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:20.969252    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc-local-certs\") pod \"4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc\" (UID: \"4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc\") "
Nov 03 20:08:20.969324 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:20.969294    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc-istio-csr-ca-configmap\") pod \"4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc\" (UID: \"4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc\") "
Nov 03 20:08:20.969571 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:20.969324    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc-istio-kubeconfig\") pod \"4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc\" (UID: \"4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc\") "
Nov 03 20:08:20.969571 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:20.969353    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc-kube-api-access-tc4pf\" (UniqueName: \"kubernetes.io/projected/4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc-kube-api-access-tc4pf\") pod \"4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc\" (UID: \"4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc\") "
Nov 03 20:08:20.971072 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:20.970792    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc-istio-csr-ca-configmap" pod "4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc" (UID: "4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:08:20.972422 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:20.972395    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-b30676-57f49c45c6-pkbfh"]
Nov 03 20:08:20.973903 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:20.972638    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-b30676-57f49c45c6-pkbfh" podUID="817f547d-db14-409a-a162-9dc0a49be5af" containerName="maas-api" containerID="cri-o://30915a3149e65d675c0b262bd72bb982608e93eddd6a0e6a4c590e8fd2822c26" gracePeriod=30
Nov 03 20:08:20.982824 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:20.982561    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc-istio-csr-dns-cert" pod "4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc" (UID: "4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:08:20.983389 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:20.983326    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc-local-certs" pod "4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc" (UID: "4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:20.986300 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:20.986252    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc-cacerts" pod "4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc" (UID: "4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:08:20.987255 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:20.986509    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc-kube-api-access-tc4pf" pod "4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc" (UID: "4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc"). InnerVolumeSpecName "kube-api-access-tc4pf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:08:20.987422 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:20.987392    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc-istio-kubeconfig" pod "4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc" (UID: "4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:08:20.987640 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:20.987618    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc-istio-token" pod "4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc" (UID: "4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:08:21.071243 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:21.071199    4921 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc-cacerts\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:21.071243 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:21.071230    4921 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc-istio-token\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:21.071243 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:21.071242    4921 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc-local-certs\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:21.071243 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:21.071251    4921 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc-istio-csr-ca-configmap\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:21.071609 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:21.071260    4921 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc-istio-kubeconfig\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:21.071609 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:21.071268    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tc4pf\" (UniqueName: \"kubernetes.io/projected/4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc-kube-api-access-tc4pf\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:21.071609 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:21.071277    4921 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc-istio-csr-dns-cert\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:21.289999 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:21.289972    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-b30676-57f49c45c6-pkbfh"
Nov 03 20:08:21.375225 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:21.375172    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/817f547d-db14-409a-a162-9dc0a49be5af-kube-api-access-fg547\" (UniqueName: \"kubernetes.io/projected/817f547d-db14-409a-a162-9dc0a49be5af-kube-api-access-fg547\") pod \"817f547d-db14-409a-a162-9dc0a49be5af\" (UID: \"817f547d-db14-409a-a162-9dc0a49be5af\") "
Nov 03 20:08:21.375412 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:21.375274    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/817f547d-db14-409a-a162-9dc0a49be5af-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/817f547d-db14-409a-a162-9dc0a49be5af-openshift-service-ca\") pod \"817f547d-db14-409a-a162-9dc0a49be5af\" (UID: \"817f547d-db14-409a-a162-9dc0a49be5af\") "
Nov 03 20:08:21.375412 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:21.375318    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/817f547d-db14-409a-a162-9dc0a49be5af-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/817f547d-db14-409a-a162-9dc0a49be5af-maas-api-tls\") pod \"817f547d-db14-409a-a162-9dc0a49be5af\" (UID: \"817f547d-db14-409a-a162-9dc0a49be5af\") "
Nov 03 20:08:21.375412 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:21.375375    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/817f547d-db14-409a-a162-9dc0a49be5af-metrics-certs\" (UniqueName: \"kubernetes.io/secret/817f547d-db14-409a-a162-9dc0a49be5af-metrics-certs\") pod \"817f547d-db14-409a-a162-9dc0a49be5af\" (UID: \"817f547d-db14-409a-a162-9dc0a49be5af\") "
Nov 03 20:08:21.375706 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:21.375641    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/817f547d-db14-409a-a162-9dc0a49be5af-openshift-service-ca" pod "817f547d-db14-409a-a162-9dc0a49be5af" (UID: "817f547d-db14-409a-a162-9dc0a49be5af"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:08:21.377900 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:21.377869    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/817f547d-db14-409a-a162-9dc0a49be5af-kube-api-access-fg547" pod "817f547d-db14-409a-a162-9dc0a49be5af" (UID: "817f547d-db14-409a-a162-9dc0a49be5af"). InnerVolumeSpecName "kube-api-access-fg547". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:08:21.378021 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:21.378000    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/817f547d-db14-409a-a162-9dc0a49be5af-maas-api-tls" pod "817f547d-db14-409a-a162-9dc0a49be5af" (UID: "817f547d-db14-409a-a162-9dc0a49be5af"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:08:21.378078 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:21.378028    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/817f547d-db14-409a-a162-9dc0a49be5af-metrics-certs" pod "817f547d-db14-409a-a162-9dc0a49be5af" (UID: "817f547d-db14-409a-a162-9dc0a49be5af"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:08:21.479324 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:21.478121    4921 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/817f547d-db14-409a-a162-9dc0a49be5af-openshift-service-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:21.479324 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:21.478166    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/817f547d-db14-409a-a162-9dc0a49be5af-maas-api-tls\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:21.483485 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:21.483414    4921 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/817f547d-db14-409a-a162-9dc0a49be5af-metrics-certs\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:21.483485 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:21.483456    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fg547\" (UniqueName: \"kubernetes.io/projected/817f547d-db14-409a-a162-9dc0a49be5af-kube-api-access-fg547\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:21.654541 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:21.654501    4921 generic.go:356] "Generic (PLEG): container finished" podID="4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc" containerID="16f51738c532765a9301416e1c41576c198001166cc12ebb3e00745e487c96a4" exitCode=0
Nov 03 20:08:21.654755 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:21.654565    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-crz4q"
Nov 03 20:08:21.654755 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:21.654590    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-crz4q" event={"ID":"4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc","Type":"ContainerDied","Data":"16f51738c532765a9301416e1c41576c198001166cc12ebb3e00745e487c96a4"}
Nov 03 20:08:21.654755 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:21.654701    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-crz4q" event={"ID":"4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc","Type":"ContainerDied","Data":"9db66fe81c82c8f74c4401805c45e7d9b1952f9bf45ac5a82983b49f2863f08d"}
Nov 03 20:08:21.654755 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:21.654728    4921 scope.go:122] "RemoveContainer" containerID="16f51738c532765a9301416e1c41576c198001166cc12ebb3e00745e487c96a4"
Nov 03 20:08:21.657210 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:21.657181    4921 generic.go:356] "Generic (PLEG): container finished" podID="817f547d-db14-409a-a162-9dc0a49be5af" containerID="30915a3149e65d675c0b262bd72bb982608e93eddd6a0e6a4c590e8fd2822c26" exitCode=0
Nov 03 20:08:21.657358 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:21.657222    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-b30676-57f49c45c6-pkbfh" event={"ID":"817f547d-db14-409a-a162-9dc0a49be5af","Type":"ContainerDied","Data":"30915a3149e65d675c0b262bd72bb982608e93eddd6a0e6a4c590e8fd2822c26"}
Nov 03 20:08:21.657358 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:21.657279    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-b30676-57f49c45c6-pkbfh"
Nov 03 20:08:21.657573 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:21.657285    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-b30676-57f49c45c6-pkbfh" event={"ID":"817f547d-db14-409a-a162-9dc0a49be5af","Type":"ContainerDied","Data":"21c480f12462f48b1d568a2202082e4dad1c289d2b3b835e6bac74027b345cbc"}
Nov 03 20:08:21.666706 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:21.666658    4921 scope.go:122] "RemoveContainer" containerID="16f51738c532765a9301416e1c41576c198001166cc12ebb3e00745e487c96a4"
Nov 03 20:08:21.666992 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:08:21.666971    4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16f51738c532765a9301416e1c41576c198001166cc12ebb3e00745e487c96a4\": container with ID starting with 16f51738c532765a9301416e1c41576c198001166cc12ebb3e00745e487c96a4 not found: ID does not exist" containerID="16f51738c532765a9301416e1c41576c198001166cc12ebb3e00745e487c96a4"
Nov 03 20:08:21.667093 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:21.666999    4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16f51738c532765a9301416e1c41576c198001166cc12ebb3e00745e487c96a4"} err="failed to get container status \"16f51738c532765a9301416e1c41576c198001166cc12ebb3e00745e487c96a4\": rpc error: code = NotFound desc = could not find container \"16f51738c532765a9301416e1c41576c198001166cc12ebb3e00745e487c96a4\": container with ID starting with 16f51738c532765a9301416e1c41576c198001166cc12ebb3e00745e487c96a4 not found: ID does not exist"
Nov 03 20:08:21.667093 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:21.667016    4921 scope.go:122] "RemoveContainer" containerID="30915a3149e65d675c0b262bd72bb982608e93eddd6a0e6a4c590e8fd2822c26"
Nov 03 20:08:21.677247 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:21.677227    4921 scope.go:122] "RemoveContainer" containerID="30915a3149e65d675c0b262bd72bb982608e93eddd6a0e6a4c590e8fd2822c26"
Nov 03 20:08:21.677625 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:08:21.677600    4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30915a3149e65d675c0b262bd72bb982608e93eddd6a0e6a4c590e8fd2822c26\": container with ID starting with 30915a3149e65d675c0b262bd72bb982608e93eddd6a0e6a4c590e8fd2822c26 not found: ID does not exist" containerID="30915a3149e65d675c0b262bd72bb982608e93eddd6a0e6a4c590e8fd2822c26"
Nov 03 20:08:21.677760 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:21.677631    4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30915a3149e65d675c0b262bd72bb982608e93eddd6a0e6a4c590e8fd2822c26"} err="failed to get container status \"30915a3149e65d675c0b262bd72bb982608e93eddd6a0e6a4c590e8fd2822c26\": rpc error: code = NotFound desc = could not find container \"30915a3149e65d675c0b262bd72bb982608e93eddd6a0e6a4c590e8fd2822c26\": container with ID starting with 30915a3149e65d675c0b262bd72bb982608e93eddd6a0e6a4c590e8fd2822c26 not found: ID does not exist"
Nov 03 20:08:21.680730 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:21.680705    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-crz4q"]
Nov 03 20:08:21.686694 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:21.686660    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-crz4q"]
Nov 03 20:08:21.695526 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:21.695505    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-b30676-57f49c45c6-pkbfh"]
Nov 03 20:08:21.697904 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:21.697881    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-b30676-57f49c45c6-pkbfh"]
Nov 03 20:08:22.887535 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:22.887500    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc" path="/var/lib/kubelet/pods/4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc/volumes"
Nov 03 20:08:22.887985 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:22.887970    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="817f547d-db14-409a-a162-9dc0a49be5af" path="/var/lib/kubelet/pods/817f547d-db14-409a-a162-9dc0a49be5af/volumes"
Nov 03 20:08:25.375206 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:25.375142    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-b30676-5b9668d5b5-xgdqw" podUID="c51313a1-0e14-4824-89a8-c715b2f7ba9b" containerName="payload-processing" containerID="cri-o://cf2e4aa5ef68419a1ca9e8299bb1f69a3644e57a5f54c019bba6bfd38f171b8e" gracePeriod=25
Nov 03 20:08:25.396989 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:25.396952    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-b30676-7bd74677b7-vw8sk" podUID="b677e48e-572f-4c27-86de-dd30566c8acb" containerName="payload-pre-processing" containerID="cri-o://d38dadd6324eece9309158ff441fd5324ae5df23a8d80cb2eaf17b3fcbc6c7f1" gracePeriod=25
Nov 03 20:08:25.651656 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:25.651625    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-b30676-5b9668d5b5-xgdqw"
Nov 03 20:08:25.689087 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:25.689055    4921 generic.go:356] "Generic (PLEG): container finished" podID="b677e48e-572f-4c27-86de-dd30566c8acb" containerID="d38dadd6324eece9309158ff441fd5324ae5df23a8d80cb2eaf17b3fcbc6c7f1" exitCode=0
Nov 03 20:08:25.689256 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:25.689118    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-b30676-7bd74677b7-vw8sk" event={"ID":"b677e48e-572f-4c27-86de-dd30566c8acb","Type":"ContainerDied","Data":"d38dadd6324eece9309158ff441fd5324ae5df23a8d80cb2eaf17b3fcbc6c7f1"}
Nov 03 20:08:25.690814 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:25.690710    4921 generic.go:356] "Generic (PLEG): container finished" podID="c51313a1-0e14-4824-89a8-c715b2f7ba9b" containerID="cf2e4aa5ef68419a1ca9e8299bb1f69a3644e57a5f54c019bba6bfd38f171b8e" exitCode=0
Nov 03 20:08:25.690955 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:25.690817    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-b30676-5b9668d5b5-xgdqw" event={"ID":"c51313a1-0e14-4824-89a8-c715b2f7ba9b","Type":"ContainerDied","Data":"cf2e4aa5ef68419a1ca9e8299bb1f69a3644e57a5f54c019bba6bfd38f171b8e"}
Nov 03 20:08:25.690955 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:25.690881    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-b30676-5b9668d5b5-xgdqw" event={"ID":"c51313a1-0e14-4824-89a8-c715b2f7ba9b","Type":"ContainerDied","Data":"a10dae68691aa4f5b4d23323bf97bba56956da646ff9a2bd348a7ecb1c66c7fd"}
Nov 03 20:08:25.690955 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:25.690908    4921 scope.go:122] "RemoveContainer" containerID="cf2e4aa5ef68419a1ca9e8299bb1f69a3644e57a5f54c019bba6bfd38f171b8e"
Nov 03 20:08:25.690955 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:25.690823    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-b30676-5b9668d5b5-xgdqw"
Nov 03 20:08:25.705426 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:25.705398    4921 scope.go:122] "RemoveContainer" containerID="cf2e4aa5ef68419a1ca9e8299bb1f69a3644e57a5f54c019bba6bfd38f171b8e"
Nov 03 20:08:25.705878 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:08:25.705844    4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf2e4aa5ef68419a1ca9e8299bb1f69a3644e57a5f54c019bba6bfd38f171b8e\": container with ID starting with cf2e4aa5ef68419a1ca9e8299bb1f69a3644e57a5f54c019bba6bfd38f171b8e not found: ID does not exist" containerID="cf2e4aa5ef68419a1ca9e8299bb1f69a3644e57a5f54c019bba6bfd38f171b8e"
Nov 03 20:08:25.705992 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:25.705883    4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf2e4aa5ef68419a1ca9e8299bb1f69a3644e57a5f54c019bba6bfd38f171b8e"} err="failed to get container status \"cf2e4aa5ef68419a1ca9e8299bb1f69a3644e57a5f54c019bba6bfd38f171b8e\": rpc error: code = NotFound desc = could not find container \"cf2e4aa5ef68419a1ca9e8299bb1f69a3644e57a5f54c019bba6bfd38f171b8e\": container with ID starting with cf2e4aa5ef68419a1ca9e8299bb1f69a3644e57a5f54c019bba6bfd38f171b8e not found: ID does not exist"
Nov 03 20:08:25.729510 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:25.728586    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c51313a1-0e14-4824-89a8-c715b2f7ba9b-kube-api-access-ffcjs\" (UniqueName: \"kubernetes.io/projected/c51313a1-0e14-4824-89a8-c715b2f7ba9b-kube-api-access-ffcjs\") pod \"c51313a1-0e14-4824-89a8-c715b2f7ba9b\" (UID: \"c51313a1-0e14-4824-89a8-c715b2f7ba9b\") "
Nov 03 20:08:25.729510 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:25.728777    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c51313a1-0e14-4824-89a8-c715b2f7ba9b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c51313a1-0e14-4824-89a8-c715b2f7ba9b-plugins-config-volume\") pod \"c51313a1-0e14-4824-89a8-c715b2f7ba9b\" (UID: \"c51313a1-0e14-4824-89a8-c715b2f7ba9b\") "
Nov 03 20:08:25.729510 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:25.729466    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c51313a1-0e14-4824-89a8-c715b2f7ba9b-plugins-config-volume" pod "c51313a1-0e14-4824-89a8-c715b2f7ba9b" (UID: "c51313a1-0e14-4824-89a8-c715b2f7ba9b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:08:25.731231 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:25.730756    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-b30676-7bd74677b7-vw8sk"
Nov 03 20:08:25.731502 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:25.731477    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c51313a1-0e14-4824-89a8-c715b2f7ba9b-kube-api-access-ffcjs" pod "c51313a1-0e14-4824-89a8-c715b2f7ba9b" (UID: "c51313a1-0e14-4824-89a8-c715b2f7ba9b"). InnerVolumeSpecName "kube-api-access-ffcjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:08:25.829526 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:25.829490    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b677e48e-572f-4c27-86de-dd30566c8acb-kube-api-access-kf4gm\" (UniqueName: \"kubernetes.io/projected/b677e48e-572f-4c27-86de-dd30566c8acb-kube-api-access-kf4gm\") pod \"b677e48e-572f-4c27-86de-dd30566c8acb\" (UID: \"b677e48e-572f-4c27-86de-dd30566c8acb\") "
Nov 03 20:08:25.829765 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:25.829743    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b677e48e-572f-4c27-86de-dd30566c8acb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b677e48e-572f-4c27-86de-dd30566c8acb-plugins-config-volume\") pod \"b677e48e-572f-4c27-86de-dd30566c8acb\" (UID: \"b677e48e-572f-4c27-86de-dd30566c8acb\") "
Nov 03 20:08:25.830120 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:25.830096    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffcjs\" (UniqueName: \"kubernetes.io/projected/c51313a1-0e14-4824-89a8-c715b2f7ba9b-kube-api-access-ffcjs\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:25.830120 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:25.830103    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b677e48e-572f-4c27-86de-dd30566c8acb-plugins-config-volume" pod "b677e48e-572f-4c27-86de-dd30566c8acb" (UID: "b677e48e-572f-4c27-86de-dd30566c8acb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:08:25.830120 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:25.830121    4921 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c51313a1-0e14-4824-89a8-c715b2f7ba9b-plugins-config-volume\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:25.831765 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:25.831739    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b677e48e-572f-4c27-86de-dd30566c8acb-kube-api-access-kf4gm" pod "b677e48e-572f-4c27-86de-dd30566c8acb" (UID: "b677e48e-572f-4c27-86de-dd30566c8acb"). InnerVolumeSpecName "kube-api-access-kf4gm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:08:25.934805 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:25.933061    4921 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b677e48e-572f-4c27-86de-dd30566c8acb-plugins-config-volume\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:25.934805 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:25.933111    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kf4gm\" (UniqueName: \"kubernetes.io/projected/b677e48e-572f-4c27-86de-dd30566c8acb-kube-api-access-kf4gm\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:26.013492 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:26.013454    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-b30676-5b9668d5b5-xgdqw"]
Nov 03 20:08:26.017471 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:26.017432    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-b30676-5b9668d5b5-xgdqw"]
Nov 03 20:08:26.699149 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:26.699062    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-b30676-7bd74677b7-vw8sk" event={"ID":"b677e48e-572f-4c27-86de-dd30566c8acb","Type":"ContainerDied","Data":"f1b641fabaeee87c050ecfc05cc3b473ab087a419e5773b9e77ff781a0b2c397"}
Nov 03 20:08:26.699149 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:26.699097    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-b30676-7bd74677b7-vw8sk"
Nov 03 20:08:26.699149 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:26.699127    4921 scope.go:122] "RemoveContainer" containerID="d38dadd6324eece9309158ff441fd5324ae5df23a8d80cb2eaf17b3fcbc6c7f1"
Nov 03 20:08:26.722138 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:26.722097    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-b30676-7bd74677b7-vw8sk"]
Nov 03 20:08:26.725177 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:26.725155    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-b30676-7bd74677b7-vw8sk"]
Nov 03 20:08:26.894224 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:26.894190    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b677e48e-572f-4c27-86de-dd30566c8acb" path="/var/lib/kubelet/pods/b677e48e-572f-4c27-86de-dd30566c8acb/volumes"
Nov 03 20:08:26.894584 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:26.894572    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c51313a1-0e14-4824-89a8-c715b2f7ba9b" path="/var/lib/kubelet/pods/c51313a1-0e14-4824-89a8-c715b2f7ba9b/volumes"
Nov 03 20:08:29.116887 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.116845    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0486e8-p49hl"]
Nov 03 20:08:29.117492 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.117475    4921 cpu_manager.go:414] "containerMap: removing container" podUID="d4de0090-d775-4bd7-9ee2-058b5d380638" containerName="main"
Nov 03 20:08:29.117576 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.117495    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4de0090-d775-4bd7-9ee2-058b5d380638" containerName="main"
Nov 03 20:08:29.117576 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.117515    4921 cpu_manager.go:414] "containerMap: removing container" podUID="fa3f8ec3-6ad5-4433-8dbb-71f8038dffce" containerName="istio-proxy"
Nov 03 20:08:29.117576 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.117523    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa3f8ec3-6ad5-4433-8dbb-71f8038dffce" containerName="istio-proxy"
Nov 03 20:08:29.117576 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.117534    4921 cpu_manager.go:414] "containerMap: removing container" podUID="c51313a1-0e14-4824-89a8-c715b2f7ba9b" containerName="payload-processing"
Nov 03 20:08:29.117576 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.117542    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c51313a1-0e14-4824-89a8-c715b2f7ba9b" containerName="payload-processing"
Nov 03 20:08:29.117576 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.117557    4921 cpu_manager.go:414] "containerMap: removing container" podUID="26379d10-3da7-4549-8b8c-41730a41792b" containerName="payload-pre-processing"
Nov 03 20:08:29.117576 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.117566    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26379d10-3da7-4549-8b8c-41730a41792b" containerName="payload-pre-processing"
Nov 03 20:08:29.117987 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.117588    4921 cpu_manager.go:414] "containerMap: removing container" podUID="28288186-9960-4f98-8506-95c3dbf53cfd" containerName="maas-api"
Nov 03 20:08:29.117987 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.117596    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28288186-9960-4f98-8506-95c3dbf53cfd" containerName="maas-api"
Nov 03 20:08:29.117987 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.117609    4921 cpu_manager.go:414] "containerMap: removing container" podUID="1aaee61d-b036-4c2b-bee7-a692b8bed612" containerName="main"
Nov 03 20:08:29.117987 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.117616    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1aaee61d-b036-4c2b-bee7-a692b8bed612" containerName="main"
Nov 03 20:08:29.117987 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.117627    4921 cpu_manager.go:414] "containerMap: removing container" podUID="d1fd32d8-1484-4622-86a3-24b16903084b" containerName="maas-api"
Nov 03 20:08:29.117987 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.117634    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1fd32d8-1484-4622-86a3-24b16903084b" containerName="maas-api"
Nov 03 20:08:29.117987 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.117645    4921 cpu_manager.go:414] "containerMap: removing container" podUID="1b9000a2-e11e-4afb-9ea7-4e4f1446d918" containerName="istio-proxy"
Nov 03 20:08:29.117987 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.117652    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b9000a2-e11e-4afb-9ea7-4e4f1446d918" containerName="istio-proxy"
Nov 03 20:08:29.117987 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.117662    4921 cpu_manager.go:414] "containerMap: removing container" podUID="0df8b636-6a8f-4a9d-ab89-2dabedc2bc21" containerName="payload-processing"
Nov 03 20:08:29.117987 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.117686    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0df8b636-6a8f-4a9d-ab89-2dabedc2bc21" containerName="payload-processing"
Nov 03 20:08:29.117987 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.117701    4921 cpu_manager.go:414] "containerMap: removing container" podUID="72d96abc-f3b4-4eda-b3a1-b93ba658e1d5" containerName="main"
Nov 03 20:08:29.117987 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.117708    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72d96abc-f3b4-4eda-b3a1-b93ba658e1d5" containerName="main"
Nov 03 20:08:29.117987 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.117727    4921 cpu_manager.go:414] "containerMap: removing container" podUID="01c749b9-d188-4673-9a3d-99b9c4ae0793" containerName="main"
Nov 03 20:08:29.117987 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.117734    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01c749b9-d188-4673-9a3d-99b9c4ae0793" containerName="main"
Nov 03 20:08:29.117987 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.117746    4921 cpu_manager.go:414] "containerMap: removing container" podUID="817f547d-db14-409a-a162-9dc0a49be5af" containerName="maas-api"
Nov 03 20:08:29.117987 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.117753    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="817f547d-db14-409a-a162-9dc0a49be5af" containerName="maas-api"
Nov 03 20:08:29.117987 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.117763    4921 cpu_manager.go:414] "containerMap: removing container" podUID="4aa9c292-f06c-45d9-9a58-8aeae3b7b6ab" containerName="payload-processing"
Nov 03 20:08:29.117987 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.117771    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4aa9c292-f06c-45d9-9a58-8aeae3b7b6ab" containerName="payload-processing"
Nov 03 20:08:29.117987 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.117782    4921 cpu_manager.go:414] "containerMap: removing container" podUID="b677e48e-572f-4c27-86de-dd30566c8acb" containerName="payload-pre-processing"
Nov 03 20:08:29.117987 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.117789    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b677e48e-572f-4c27-86de-dd30566c8acb" containerName="payload-pre-processing"
Nov 03 20:08:29.117987 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.117807    4921 cpu_manager.go:414] "containerMap: removing container" podUID="0f48e7cf-0a2d-44ff-8949-77d277a40138" containerName="maas-api"
Nov 03 20:08:29.117987 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.117813    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f48e7cf-0a2d-44ff-8949-77d277a40138" containerName="maas-api"
Nov 03 20:08:29.117987 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.117826    4921 cpu_manager.go:414] "containerMap: removing container" podUID="48ab742a-e2bf-4751-bb31-a962f0996078" containerName="payload-pre-processing"
Nov 03 20:08:29.117987 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.117833    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48ab742a-e2bf-4751-bb31-a962f0996078" containerName="payload-pre-processing"
Nov 03 20:08:29.117987 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.117845    4921 cpu_manager.go:414] "containerMap: removing container" podUID="cbdab358-70c7-4c98-a2c4-4dc4a924e8d9" containerName="payload-pre-processing"
Nov 03 20:08:29.117987 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.117853    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbdab358-70c7-4c98-a2c4-4dc4a924e8d9" containerName="payload-pre-processing"
Nov 03 20:08:29.117987 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.117869    4921 cpu_manager.go:414] "containerMap: removing container" podUID="4bda8b9f-49ce-45d8-9c50-3e56a865b849" containerName="payload-pre-processing"
Nov 03 20:08:29.117987 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.117876    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bda8b9f-49ce-45d8-9c50-3e56a865b849" containerName="payload-pre-processing"
Nov 03 20:08:29.117987 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.117887    4921 cpu_manager.go:414] "containerMap: removing container" podUID="6cb2ad5f-35b4-4c92-9499-d7ac8152a72c" containerName="maas-api"
Nov 03 20:08:29.117987 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.117895    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cb2ad5f-35b4-4c92-9499-d7ac8152a72c" containerName="maas-api"
Nov 03 20:08:29.117987 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.117913    4921 cpu_manager.go:414] "containerMap: removing container" podUID="f838f4d0-1e63-4b98-8cfe-bd1b96db7162" containerName="istio-proxy"
Nov 03 20:08:29.117987 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.117920    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f838f4d0-1e63-4b98-8cfe-bd1b96db7162" containerName="istio-proxy"
Nov 03 20:08:29.117987 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.117930    4921 cpu_manager.go:414] "containerMap: removing container" podUID="8d974919-e441-47b7-9044-9bb57982bdc1" containerName="payload-processing"
Nov 03 20:08:29.117987 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.117937    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d974919-e441-47b7-9044-9bb57982bdc1" containerName="payload-processing"
Nov 03 20:08:29.117987 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.117949    4921 cpu_manager.go:414] "containerMap: removing container" podUID="62b4fa76-8b78-4a67-af03-b126b71aa8c6" containerName="payload-processing"
Nov 03 20:08:29.117987 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.117957    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62b4fa76-8b78-4a67-af03-b126b71aa8c6" containerName="payload-processing"
Nov 03 20:08:29.117987 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.117969    4921 cpu_manager.go:414] "containerMap: removing container" podUID="17dc0ed2-5164-449d-be12-a38bdfeb946d" containerName="main"
Nov 03 20:08:29.117987 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.117975    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17dc0ed2-5164-449d-be12-a38bdfeb946d" containerName="main"
Nov 03 20:08:29.117987 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.117986    4921 cpu_manager.go:414] "containerMap: removing container" podUID="e78a74a9-c32a-4047-a64f-b344f8ebaf22" containerName="istio-proxy"
Nov 03 20:08:29.117987 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.117992    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e78a74a9-c32a-4047-a64f-b344f8ebaf22" containerName="istio-proxy"
Nov 03 20:08:29.117987 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.118003    4921 cpu_manager.go:414] "containerMap: removing container" podUID="ffb30cac-fb9f-4a3f-b954-ecd050b8cb41" containerName="istio-proxy"
Nov 03 20:08:29.120072 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.118011    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffb30cac-fb9f-4a3f-b954-ecd050b8cb41" containerName="istio-proxy"
Nov 03 20:08:29.120072 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.118021    4921 cpu_manager.go:414] "containerMap: removing container" podUID="4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc" containerName="discovery"
Nov 03 20:08:29.120072 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.118029    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc" containerName="discovery"
Nov 03 20:08:29.120072 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.118043    4921 cpu_manager.go:414] "containerMap: removing container" podUID="3f347075-1266-4c50-81ff-a8d5ab87b1b3" containerName="maas-api"
Nov 03 20:08:29.120072 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.118050    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f347075-1266-4c50-81ff-a8d5ab87b1b3" containerName="maas-api"
Nov 03 20:08:29.120072 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.118061    4921 cpu_manager.go:414] "containerMap: removing container" podUID="6c554c44-d20a-4efe-9faf-5c30ec3a3f5e" containerName="payload-pre-processing"
Nov 03 20:08:29.120072 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.118068    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c554c44-d20a-4efe-9faf-5c30ec3a3f5e" containerName="payload-pre-processing"
Nov 03 20:08:29.120072 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.118080    4921 cpu_manager.go:414] "containerMap: removing container" podUID="eac6c47d-8ae5-4e4b-b691-98d177887767" containerName="main"
Nov 03 20:08:29.120072 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.118087    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eac6c47d-8ae5-4e4b-b691-98d177887767" containerName="main"
Nov 03 20:08:29.120072 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.118198    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="f838f4d0-1e63-4b98-8cfe-bd1b96db7162" containerName="istio-proxy"
Nov 03 20:08:29.120072 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.118209    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d974919-e441-47b7-9044-9bb57982bdc1" containerName="payload-processing"
Nov 03 20:08:29.120072 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.118222    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="62b4fa76-8b78-4a67-af03-b126b71aa8c6" containerName="payload-processing"
Nov 03 20:08:29.120072 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.118232    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="17dc0ed2-5164-449d-be12-a38bdfeb946d" containerName="main"
Nov 03 20:08:29.120072 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.118242    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="e78a74a9-c32a-4047-a64f-b344f8ebaf22" containerName="istio-proxy"
Nov 03 20:08:29.120072 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.118253    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffb30cac-fb9f-4a3f-b954-ecd050b8cb41" containerName="istio-proxy"
Nov 03 20:08:29.120072 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.118262    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cf5fa96-a4ca-465b-90bb-92e0db5fb2bc" containerName="discovery"
Nov 03 20:08:29.120072 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.118272    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f347075-1266-4c50-81ff-a8d5ab87b1b3" containerName="maas-api"
Nov 03 20:08:29.120072 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.118281    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c554c44-d20a-4efe-9faf-5c30ec3a3f5e" containerName="payload-pre-processing"
Nov 03 20:08:29.120072 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.118291    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="eac6c47d-8ae5-4e4b-b691-98d177887767" containerName="main"
Nov 03 20:08:29.120072 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.118300    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4de0090-d775-4bd7-9ee2-058b5d380638" containerName="main"
Nov 03 20:08:29.120072 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.118309    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa3f8ec3-6ad5-4433-8dbb-71f8038dffce" containerName="istio-proxy"
Nov 03 20:08:29.120072 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.118318    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="c51313a1-0e14-4824-89a8-c715b2f7ba9b" containerName="payload-processing"
Nov 03 20:08:29.120072 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.118329    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="26379d10-3da7-4549-8b8c-41730a41792b" containerName="payload-pre-processing"
Nov 03 20:08:29.120072 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.118339    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="28288186-9960-4f98-8506-95c3dbf53cfd" containerName="maas-api"
Nov 03 20:08:29.120072 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.118348    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="1aaee61d-b036-4c2b-bee7-a692b8bed612" containerName="main"
Nov 03 20:08:29.120072 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.118358    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1fd32d8-1484-4622-86a3-24b16903084b" containerName="maas-api"
Nov 03 20:08:29.120072 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.118367    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b9000a2-e11e-4afb-9ea7-4e4f1446d918" containerName="istio-proxy"
Nov 03 20:08:29.120072 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.118375    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="0df8b636-6a8f-4a9d-ab89-2dabedc2bc21" containerName="payload-processing"
Nov 03 20:08:29.120072 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.118388    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="01c749b9-d188-4673-9a3d-99b9c4ae0793" containerName="main"
Nov 03 20:08:29.120072 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.118396    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="817f547d-db14-409a-a162-9dc0a49be5af" containerName="maas-api"
Nov 03 20:08:29.120072 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.118405    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="72d96abc-f3b4-4eda-b3a1-b93ba658e1d5" containerName="main"
Nov 03 20:08:29.120072 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.118414    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="4aa9c292-f06c-45d9-9a58-8aeae3b7b6ab" containerName="payload-processing"
Nov 03 20:08:29.120072 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.118423    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="b677e48e-572f-4c27-86de-dd30566c8acb" containerName="payload-pre-processing"
Nov 03 20:08:29.120072 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.118432    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f48e7cf-0a2d-44ff-8949-77d277a40138" containerName="maas-api"
Nov 03 20:08:29.120072 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.118440    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="48ab742a-e2bf-4751-bb31-a962f0996078" containerName="payload-pre-processing"
Nov 03 20:08:29.120072 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.118449    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbdab358-70c7-4c98-a2c4-4dc4a924e8d9" containerName="payload-pre-processing"
Nov 03 20:08:29.120072 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.118460    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bda8b9f-49ce-45d8-9c50-3e56a865b849" containerName="payload-pre-processing"
Nov 03 20:08:29.120072 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.118469    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cb2ad5f-35b4-4c92-9499-d7ac8152a72c" containerName="maas-api"
Nov 03 20:08:29.123738 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.123717    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0486e8-p49hl"
Nov 03 20:08:29.128734 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.127120    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:08:29.128734 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.127384    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 20:08:29.128734 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.128352    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0486e8-p49hl"]
Nov 03 20:08:29.267664 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.267629    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/994d46ff-d6e1-4563-9f0e-3a3d767c8b68-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-0486e8-p49hl\" (UID: \"994d46ff-d6e1-4563-9f0e-3a3d767c8b68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0486e8-p49hl"
Nov 03 20:08:29.267860 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.267755    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxmxw\" (UniqueName: \"kubernetes.io/projected/994d46ff-d6e1-4563-9f0e-3a3d767c8b68-kube-api-access-kxmxw\") pod \"maas-api-revoke-keys-e2e-worker-w5-0486e8-p49hl\" (UID: \"994d46ff-d6e1-4563-9f0e-3a3d767c8b68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0486e8-p49hl"
Nov 03 20:08:29.267860 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.267818    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/994d46ff-d6e1-4563-9f0e-3a3d767c8b68-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-0486e8-p49hl\" (UID: \"994d46ff-d6e1-4563-9f0e-3a3d767c8b68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0486e8-p49hl"
Nov 03 20:08:29.368947 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.368872    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/994d46ff-d6e1-4563-9f0e-3a3d767c8b68-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-0486e8-p49hl\" (UID: \"994d46ff-d6e1-4563-9f0e-3a3d767c8b68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0486e8-p49hl"
Nov 03 20:08:29.369075 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.368985    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/994d46ff-d6e1-4563-9f0e-3a3d767c8b68-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-0486e8-p49hl\" (UID: \"994d46ff-d6e1-4563-9f0e-3a3d767c8b68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0486e8-p49hl"
Nov 03 20:08:29.369075 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.369038    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxmxw\" (UniqueName: \"kubernetes.io/projected/994d46ff-d6e1-4563-9f0e-3a3d767c8b68-kube-api-access-kxmxw\") pod \"maas-api-revoke-keys-e2e-worker-w5-0486e8-p49hl\" (UID: \"994d46ff-d6e1-4563-9f0e-3a3d767c8b68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0486e8-p49hl"
Nov 03 20:08:29.369682 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.369652    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/994d46ff-d6e1-4563-9f0e-3a3d767c8b68-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-0486e8-p49hl\" (UID: \"994d46ff-d6e1-4563-9f0e-3a3d767c8b68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0486e8-p49hl"
Nov 03 20:08:29.371379 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.371358    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/994d46ff-d6e1-4563-9f0e-3a3d767c8b68-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-0486e8-p49hl\" (UID: \"994d46ff-d6e1-4563-9f0e-3a3d767c8b68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0486e8-p49hl"
Nov 03 20:08:29.376215 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.376186    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxmxw\" (UniqueName: \"kubernetes.io/projected/994d46ff-d6e1-4563-9f0e-3a3d767c8b68-kube-api-access-kxmxw\") pod \"maas-api-revoke-keys-e2e-worker-w5-0486e8-p49hl\" (UID: \"994d46ff-d6e1-4563-9f0e-3a3d767c8b68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0486e8-p49hl"
Nov 03 20:08:29.438611 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.438587    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0486e8-p49hl"
Nov 03 20:08:29.711125 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:29.711098    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0486e8-p49hl"]
Nov 03 20:08:29.713756 ip-10-0-134-222 kubenswrapper[4921]: W1103 20:08:29.713727    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod994d46ff_d6e1_4563_9f0e_3a3d767c8b68.slice/crio-f50d7d9b6e4bd81652f62df009ed591a3f46b958e98f713b2d86905959d396fe WatchSource:0}: Error finding container f50d7d9b6e4bd81652f62df009ed591a3f46b958e98f713b2d86905959d396fe: Status 404 returned error can't find the container with id f50d7d9b6e4bd81652f62df009ed591a3f46b958e98f713b2d86905959d396fe
Nov 03 20:08:30.734703 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:30.734149    4921 generic.go:356] "Generic (PLEG): container finished" podID="994d46ff-d6e1-4563-9f0e-3a3d767c8b68" containerID="e0d3ab73da2b20800ab4956bc261bb36fccd2df76de73f31720cc5764e66999c" exitCode=0
Nov 03 20:08:30.734703 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:30.734233    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0486e8-p49hl" event={"ID":"994d46ff-d6e1-4563-9f0e-3a3d767c8b68","Type":"ContainerDied","Data":"e0d3ab73da2b20800ab4956bc261bb36fccd2df76de73f31720cc5764e66999c"}
Nov 03 20:08:30.734703 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:30.734285    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0486e8-p49hl" event={"ID":"994d46ff-d6e1-4563-9f0e-3a3d767c8b68","Type":"ContainerStarted","Data":"f50d7d9b6e4bd81652f62df009ed591a3f46b958e98f713b2d86905959d396fe"}
Nov 03 20:08:31.732423 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:31.732385    4921 scope.go:122] "RemoveContainer" containerID="961f0731683dd96ac13f5d500e8b444ec3daf371310bdebfd246fb2d73050e86"
Nov 03 20:08:31.742162 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:31.742142    4921 scope.go:122] "RemoveContainer" containerID="8419e40ba0382609488be7d78f20a38389d1869cf12496d568ce054898a5873a"
Nov 03 20:08:31.899605 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:31.899569    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0486e8-p49hl"
Nov 03 20:08:31.999715 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:31.999594    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/994d46ff-d6e1-4563-9f0e-3a3d767c8b68-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/994d46ff-d6e1-4563-9f0e-3a3d767c8b68-maas-api-xks-ca\") pod \"994d46ff-d6e1-4563-9f0e-3a3d767c8b68\" (UID: \"994d46ff-d6e1-4563-9f0e-3a3d767c8b68\") "
Nov 03 20:08:31.999899 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:31.999794    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/994d46ff-d6e1-4563-9f0e-3a3d767c8b68-kube-api-access-kxmxw\" (UniqueName: \"kubernetes.io/projected/994d46ff-d6e1-4563-9f0e-3a3d767c8b68-kube-api-access-kxmxw\") pod \"994d46ff-d6e1-4563-9f0e-3a3d767c8b68\" (UID: \"994d46ff-d6e1-4563-9f0e-3a3d767c8b68\") "
Nov 03 20:08:31.999899 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:31.999858    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/994d46ff-d6e1-4563-9f0e-3a3d767c8b68-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/994d46ff-d6e1-4563-9f0e-3a3d767c8b68-maas-api-openshift-ca\") pod \"994d46ff-d6e1-4563-9f0e-3a3d767c8b68\" (UID: \"994d46ff-d6e1-4563-9f0e-3a3d767c8b68\") "
Nov 03 20:08:32.001069 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:32.001038    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/994d46ff-d6e1-4563-9f0e-3a3d767c8b68-maas-api-openshift-ca" pod "994d46ff-d6e1-4563-9f0e-3a3d767c8b68" (UID: "994d46ff-d6e1-4563-9f0e-3a3d767c8b68"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:08:32.002630 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:32.002606    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/994d46ff-d6e1-4563-9f0e-3a3d767c8b68-maas-api-xks-ca" pod "994d46ff-d6e1-4563-9f0e-3a3d767c8b68" (UID: "994d46ff-d6e1-4563-9f0e-3a3d767c8b68"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:08:32.003528 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:32.003504    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/994d46ff-d6e1-4563-9f0e-3a3d767c8b68-kube-api-access-kxmxw" pod "994d46ff-d6e1-4563-9f0e-3a3d767c8b68" (UID: "994d46ff-d6e1-4563-9f0e-3a3d767c8b68"). InnerVolumeSpecName "kube-api-access-kxmxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:08:32.100891 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:32.100853    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxmxw\" (UniqueName: \"kubernetes.io/projected/994d46ff-d6e1-4563-9f0e-3a3d767c8b68-kube-api-access-kxmxw\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:32.100891 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:32.100885    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/994d46ff-d6e1-4563-9f0e-3a3d767c8b68-maas-api-openshift-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:32.100891 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:32.100898    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/994d46ff-d6e1-4563-9f0e-3a3d767c8b68-maas-api-xks-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:32.749422 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:32.749392    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0486e8-p49hl"
Nov 03 20:08:32.749862 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:32.749426    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0486e8-p49hl" event={"ID":"994d46ff-d6e1-4563-9f0e-3a3d767c8b68","Type":"ContainerDied","Data":"f50d7d9b6e4bd81652f62df009ed591a3f46b958e98f713b2d86905959d396fe"}
Nov 03 20:08:32.749862 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:32.749501    4921 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f50d7d9b6e4bd81652f62df009ed591a3f46b958e98f713b2d86905959d396fe"
Nov 03 20:08:32.920855 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:08:32.920823    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 20:08:35.369841 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:35.369794    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk" podUID="2d0a934e-4816-44d2-9bb5-e8b4e95dd73c" containerName="main" containerID="cri-o://952987dd0924fcc54c37caca6862e3f205ef1dc433eec329287564fcf713af06" gracePeriod=45
Nov 03 20:08:35.382563 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:35.382524    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv" podUID="96b73342-0079-4709-8570-3dbce57e5243" containerName="main" containerID="cri-o://61708f049c22b2df2a4dd0624f8354a4e37d3a2e7d2d44a41c051e5812287ff5" gracePeriod=45
Nov 03 20:08:35.541491 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:35.541229    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv" podUID="96b73342-0079-4709-8570-3dbce57e5243" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": dial tcp 10.132.0.68:8000: connect: connection refused"
Nov 03 20:08:35.639147 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:35.639113    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk"
Nov 03 20:08:35.681590 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:35.681562    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv"
Nov 03 20:08:35.739084 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:35.739053    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d0a934e-4816-44d2-9bb5-e8b4e95dd73c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d0a934e-4816-44d2-9bb5-e8b4e95dd73c-dshm\") pod \"2d0a934e-4816-44d2-9bb5-e8b4e95dd73c\" (UID: \"2d0a934e-4816-44d2-9bb5-e8b4e95dd73c\") "
Nov 03 20:08:35.739273 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:35.739180    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d0a934e-4816-44d2-9bb5-e8b4e95dd73c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d0a934e-4816-44d2-9bb5-e8b4e95dd73c-tmp-dir\") pod \"2d0a934e-4816-44d2-9bb5-e8b4e95dd73c\" (UID: \"2d0a934e-4816-44d2-9bb5-e8b4e95dd73c\") "
Nov 03 20:08:35.739273 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:35.739210    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d0a934e-4816-44d2-9bb5-e8b4e95dd73c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d0a934e-4816-44d2-9bb5-e8b4e95dd73c-model-cache\") pod \"2d0a934e-4816-44d2-9bb5-e8b4e95dd73c\" (UID: \"2d0a934e-4816-44d2-9bb5-e8b4e95dd73c\") "
Nov 03 20:08:35.739273 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:35.739250    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d0a934e-4816-44d2-9bb5-e8b4e95dd73c-tls-certs\" (UniqueName: \"kubernetes.io/secret/2d0a934e-4816-44d2-9bb5-e8b4e95dd73c-tls-certs\") pod \"2d0a934e-4816-44d2-9bb5-e8b4e95dd73c\" (UID: \"2d0a934e-4816-44d2-9bb5-e8b4e95dd73c\") "
Nov 03 20:08:35.739439 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:35.739306    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d0a934e-4816-44d2-9bb5-e8b4e95dd73c-home\" (UniqueName: \"kubernetes.io/empty-dir/2d0a934e-4816-44d2-9bb5-e8b4e95dd73c-home\") pod \"2d0a934e-4816-44d2-9bb5-e8b4e95dd73c\" (UID: \"2d0a934e-4816-44d2-9bb5-e8b4e95dd73c\") "
Nov 03 20:08:35.739439 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:35.739371    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d0a934e-4816-44d2-9bb5-e8b4e95dd73c-kube-api-access-6cwrh\" (UniqueName: \"kubernetes.io/projected/2d0a934e-4816-44d2-9bb5-e8b4e95dd73c-kube-api-access-6cwrh\") pod \"2d0a934e-4816-44d2-9bb5-e8b4e95dd73c\" (UID: \"2d0a934e-4816-44d2-9bb5-e8b4e95dd73c\") "
Nov 03 20:08:35.739527 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:35.739499    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d0a934e-4816-44d2-9bb5-e8b4e95dd73c-tmp-dir" pod "2d0a934e-4816-44d2-9bb5-e8b4e95dd73c" (UID: "2d0a934e-4816-44d2-9bb5-e8b4e95dd73c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:35.739829 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:35.739745    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d0a934e-4816-44d2-9bb5-e8b4e95dd73c-model-cache" pod "2d0a934e-4816-44d2-9bb5-e8b4e95dd73c" (UID: "2d0a934e-4816-44d2-9bb5-e8b4e95dd73c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:35.740054 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:35.739973    4921 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d0a934e-4816-44d2-9bb5-e8b4e95dd73c-tmp-dir\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:35.740054 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:35.739989    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d0a934e-4816-44d2-9bb5-e8b4e95dd73c-home" pod "2d0a934e-4816-44d2-9bb5-e8b4e95dd73c" (UID: "2d0a934e-4816-44d2-9bb5-e8b4e95dd73c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:35.740054 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:35.739997    4921 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d0a934e-4816-44d2-9bb5-e8b4e95dd73c-model-cache\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:35.741806 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:35.741734    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d0a934e-4816-44d2-9bb5-e8b4e95dd73c-dshm" pod "2d0a934e-4816-44d2-9bb5-e8b4e95dd73c" (UID: "2d0a934e-4816-44d2-9bb5-e8b4e95dd73c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:35.742154 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:35.742132    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d0a934e-4816-44d2-9bb5-e8b4e95dd73c-tls-certs" pod "2d0a934e-4816-44d2-9bb5-e8b4e95dd73c" (UID: "2d0a934e-4816-44d2-9bb5-e8b4e95dd73c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:08:35.742154 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:35.742144    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d0a934e-4816-44d2-9bb5-e8b4e95dd73c-kube-api-access-6cwrh" pod "2d0a934e-4816-44d2-9bb5-e8b4e95dd73c" (UID: "2d0a934e-4816-44d2-9bb5-e8b4e95dd73c"). InnerVolumeSpecName "kube-api-access-6cwrh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:08:35.770964 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:35.770936    4921 generic.go:356] "Generic (PLEG): container finished" podID="2d0a934e-4816-44d2-9bb5-e8b4e95dd73c" containerID="952987dd0924fcc54c37caca6862e3f205ef1dc433eec329287564fcf713af06" exitCode=0
Nov 03 20:08:35.771113 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:35.770995    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk"
Nov 03 20:08:35.771113 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:35.771021    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk" event={"ID":"2d0a934e-4816-44d2-9bb5-e8b4e95dd73c","Type":"ContainerDied","Data":"952987dd0924fcc54c37caca6862e3f205ef1dc433eec329287564fcf713af06"}
Nov 03 20:08:35.771113 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:35.771065    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk" event={"ID":"2d0a934e-4816-44d2-9bb5-e8b4e95dd73c","Type":"ContainerDied","Data":"47bdf4de52f7e540e4b2b513cdf4b8f923cd41286422bf2db25506f36c620aef"}
Nov 03 20:08:35.771113 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:35.771086    4921 scope.go:122] "RemoveContainer" containerID="952987dd0924fcc54c37caca6862e3f205ef1dc433eec329287564fcf713af06"
Nov 03 20:08:35.772720 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:35.772698    4921 generic.go:356] "Generic (PLEG): container finished" podID="96b73342-0079-4709-8570-3dbce57e5243" containerID="61708f049c22b2df2a4dd0624f8354a4e37d3a2e7d2d44a41c051e5812287ff5" exitCode=0
Nov 03 20:08:35.772833 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:35.772800    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv"
Nov 03 20:08:35.772833 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:35.772817    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv" event={"ID":"96b73342-0079-4709-8570-3dbce57e5243","Type":"ContainerDied","Data":"61708f049c22b2df2a4dd0624f8354a4e37d3a2e7d2d44a41c051e5812287ff5"}
Nov 03 20:08:35.772930 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:35.772847    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv" event={"ID":"96b73342-0079-4709-8570-3dbce57e5243","Type":"ContainerDied","Data":"29c8e1be5b53076fb3e866ceebb8c136104e8ce2f4ed963b54a65d26e19a0035"}
Nov 03 20:08:35.781818 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:35.781799    4921 scope.go:122] "RemoveContainer" containerID="952987dd0924fcc54c37caca6862e3f205ef1dc433eec329287564fcf713af06"
Nov 03 20:08:35.782124 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:08:35.782101    4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"952987dd0924fcc54c37caca6862e3f205ef1dc433eec329287564fcf713af06\": container with ID starting with 952987dd0924fcc54c37caca6862e3f205ef1dc433eec329287564fcf713af06 not found: ID does not exist" containerID="952987dd0924fcc54c37caca6862e3f205ef1dc433eec329287564fcf713af06"
Nov 03 20:08:35.782219 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:35.782128    4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"952987dd0924fcc54c37caca6862e3f205ef1dc433eec329287564fcf713af06"} err="failed to get container status \"952987dd0924fcc54c37caca6862e3f205ef1dc433eec329287564fcf713af06\": rpc error: code = NotFound desc = could not find container \"952987dd0924fcc54c37caca6862e3f205ef1dc433eec329287564fcf713af06\": container with ID starting with 952987dd0924fcc54c37caca6862e3f205ef1dc433eec329287564fcf713af06 not found: ID does not exist"
Nov 03 20:08:35.782219 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:35.782144    4921 scope.go:122] "RemoveContainer" containerID="61708f049c22b2df2a4dd0624f8354a4e37d3a2e7d2d44a41c051e5812287ff5"
Nov 03 20:08:35.793480 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:35.793425    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk"]
Nov 03 20:08:35.796056 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:35.796036    4921 scope.go:122] "RemoveContainer" containerID="61708f049c22b2df2a4dd0624f8354a4e37d3a2e7d2d44a41c051e5812287ff5"
Nov 03 20:08:35.797197 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:35.797176    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-82dbbf/distinct-2-w0-82dbbf-kserve-66c67fdc78-gb5jk"]
Nov 03 20:08:35.797293 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:08:35.797236    4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61708f049c22b2df2a4dd0624f8354a4e37d3a2e7d2d44a41c051e5812287ff5\": container with ID starting with 61708f049c22b2df2a4dd0624f8354a4e37d3a2e7d2d44a41c051e5812287ff5 not found: ID does not exist" containerID="61708f049c22b2df2a4dd0624f8354a4e37d3a2e7d2d44a41c051e5812287ff5"
Nov 03 20:08:35.797293 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:35.797264    4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61708f049c22b2df2a4dd0624f8354a4e37d3a2e7d2d44a41c051e5812287ff5"} err="failed to get container status \"61708f049c22b2df2a4dd0624f8354a4e37d3a2e7d2d44a41c051e5812287ff5\": rpc error: code = NotFound desc = could not find container \"61708f049c22b2df2a4dd0624f8354a4e37d3a2e7d2d44a41c051e5812287ff5\": container with ID starting with 61708f049c22b2df2a4dd0624f8354a4e37d3a2e7d2d44a41c051e5812287ff5 not found: ID does not exist"
Nov 03 20:08:35.840456 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:35.840429    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96b73342-0079-4709-8570-3dbce57e5243-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96b73342-0079-4709-8570-3dbce57e5243-tmp-dir\") pod \"96b73342-0079-4709-8570-3dbce57e5243\" (UID: \"96b73342-0079-4709-8570-3dbce57e5243\") "
Nov 03 20:08:35.840612 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:35.840526    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96b73342-0079-4709-8570-3dbce57e5243-tls-certs\" (UniqueName: \"kubernetes.io/secret/96b73342-0079-4709-8570-3dbce57e5243-tls-certs\") pod \"96b73342-0079-4709-8570-3dbce57e5243\" (UID: \"96b73342-0079-4709-8570-3dbce57e5243\") "
Nov 03 20:08:35.840612 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:35.840550    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96b73342-0079-4709-8570-3dbce57e5243-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96b73342-0079-4709-8570-3dbce57e5243-model-cache\") pod \"96b73342-0079-4709-8570-3dbce57e5243\" (UID: \"96b73342-0079-4709-8570-3dbce57e5243\") "
Nov 03 20:08:35.840612 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:35.840577    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96b73342-0079-4709-8570-3dbce57e5243-home\" (UniqueName: \"kubernetes.io/empty-dir/96b73342-0079-4709-8570-3dbce57e5243-home\") pod \"96b73342-0079-4709-8570-3dbce57e5243\" (UID: \"96b73342-0079-4709-8570-3dbce57e5243\") "
Nov 03 20:08:35.840860 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:35.840618    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96b73342-0079-4709-8570-3dbce57e5243-dshm\" (UniqueName: \"kubernetes.io/empty-dir/96b73342-0079-4709-8570-3dbce57e5243-dshm\") pod \"96b73342-0079-4709-8570-3dbce57e5243\" (UID: \"96b73342-0079-4709-8570-3dbce57e5243\") "
Nov 03 20:08:35.840860 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:35.840680    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96b73342-0079-4709-8570-3dbce57e5243-kube-api-access-lxsdr\" (UniqueName: \"kubernetes.io/projected/96b73342-0079-4709-8570-3dbce57e5243-kube-api-access-lxsdr\") pod \"96b73342-0079-4709-8570-3dbce57e5243\" (UID: \"96b73342-0079-4709-8570-3dbce57e5243\") "
Nov 03 20:08:35.840860 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:35.840748    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96b73342-0079-4709-8570-3dbce57e5243-tmp-dir" pod "96b73342-0079-4709-8570-3dbce57e5243" (UID: "96b73342-0079-4709-8570-3dbce57e5243"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:35.841000 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:35.840868    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96b73342-0079-4709-8570-3dbce57e5243-model-cache" pod "96b73342-0079-4709-8570-3dbce57e5243" (UID: "96b73342-0079-4709-8570-3dbce57e5243"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:35.841000 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:35.840885    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96b73342-0079-4709-8570-3dbce57e5243-home" pod "96b73342-0079-4709-8570-3dbce57e5243" (UID: "96b73342-0079-4709-8570-3dbce57e5243"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:35.841115 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:35.841057    4921 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d0a934e-4816-44d2-9bb5-e8b4e95dd73c-tls-certs\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:35.841115 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:35.841070    4921 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96b73342-0079-4709-8570-3dbce57e5243-model-cache\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:35.841115 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:35.841080    4921 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d0a934e-4816-44d2-9bb5-e8b4e95dd73c-home\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:35.841115 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:35.841089    4921 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96b73342-0079-4709-8570-3dbce57e5243-home\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:35.841115 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:35.841099    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cwrh\" (UniqueName: \"kubernetes.io/projected/2d0a934e-4816-44d2-9bb5-e8b4e95dd73c-kube-api-access-6cwrh\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:35.841115 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:35.841107    4921 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d0a934e-4816-44d2-9bb5-e8b4e95dd73c-dshm\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:35.841115 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:35.841117    4921 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96b73342-0079-4709-8570-3dbce57e5243-tmp-dir\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:35.842795 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:35.842773    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96b73342-0079-4709-8570-3dbce57e5243-dshm" pod "96b73342-0079-4709-8570-3dbce57e5243" (UID: "96b73342-0079-4709-8570-3dbce57e5243"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:35.842893 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:35.842810    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96b73342-0079-4709-8570-3dbce57e5243-kube-api-access-lxsdr" pod "96b73342-0079-4709-8570-3dbce57e5243" (UID: "96b73342-0079-4709-8570-3dbce57e5243"). InnerVolumeSpecName "kube-api-access-lxsdr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:08:35.842893 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:35.842826    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96b73342-0079-4709-8570-3dbce57e5243-tls-certs" pod "96b73342-0079-4709-8570-3dbce57e5243" (UID: "96b73342-0079-4709-8570-3dbce57e5243"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:08:35.942526 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:35.942466    4921 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96b73342-0079-4709-8570-3dbce57e5243-tls-certs\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:35.942526 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:35.942491    4921 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96b73342-0079-4709-8570-3dbce57e5243-dshm\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:35.942526 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:35.942507    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxsdr\" (UniqueName: \"kubernetes.io/projected/96b73342-0079-4709-8570-3dbce57e5243-kube-api-access-lxsdr\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:36.101031 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:36.100995    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv"]
Nov 03 20:08:36.104306 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:36.104239    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-82dbbf/premium-w0-82dbbf-kserve-7cb8b9765d-s64gv"]
Nov 03 20:08:36.887329 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:36.887298    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d0a934e-4816-44d2-9bb5-e8b4e95dd73c" path="/var/lib/kubelet/pods/2d0a934e-4816-44d2-9bb5-e8b4e95dd73c/volumes"
Nov 03 20:08:36.887745 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:36.887731    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96b73342-0079-4709-8570-3dbce57e5243" path="/var/lib/kubelet/pods/96b73342-0079-4709-8570-3dbce57e5243/volumes"
Nov 03 20:08:47.581569 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:47.581527    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww"]
Nov 03 20:08:47.582123 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:47.581827    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww" podUID="a52b2e9b-9ca0-4897-a071-a79c80e40d14" containerName="istio-proxy" containerID="cri-o://3e157dd7b9d6cfe501547f12915c5cd99acd3f6e3b27711109c13f06e3a31875" gracePeriod=30
Nov 03 20:08:51.262312 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:51.262030    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-0486e8-5b894565b7-76jhb"]
Nov 03 20:08:51.262884 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:51.262297    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-0486e8-5b894565b7-76jhb" podUID="dfb9124e-3670-46b5-aa88-702ec75bda1f" containerName="maas-api" containerID="cri-o://de96b32c357cbf61dd527cdcfac6f654bb1f7cb0ee63f0a38157ec5274b72f08" gracePeriod=30
Nov 03 20:08:51.458292 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:51.458218    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0486e8-p49hl"]
Nov 03 20:08:51.460516 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:51.460489    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-0486e8-p49hl"]
Nov 03 20:08:51.550406 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:51.550098    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-0486e8-5b894565b7-76jhb"
Nov 03 20:08:51.625642 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:51.625605    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dfb9124e-3670-46b5-aa88-702ec75bda1f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfb9124e-3670-46b5-aa88-702ec75bda1f-maas-api-tls\") pod \"dfb9124e-3670-46b5-aa88-702ec75bda1f\" (UID: \"dfb9124e-3670-46b5-aa88-702ec75bda1f\") "
Nov 03 20:08:51.625871 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:51.625655    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfb9124e-3670-46b5-aa88-702ec75bda1f-kube-api-access-cvxhh\" (UniqueName: \"kubernetes.io/projected/dfb9124e-3670-46b5-aa88-702ec75bda1f-kube-api-access-cvxhh\") pod \"dfb9124e-3670-46b5-aa88-702ec75bda1f\" (UID: \"dfb9124e-3670-46b5-aa88-702ec75bda1f\") "
Nov 03 20:08:51.625871 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:51.625699    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dfb9124e-3670-46b5-aa88-702ec75bda1f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfb9124e-3670-46b5-aa88-702ec75bda1f-metrics-certs\") pod \"dfb9124e-3670-46b5-aa88-702ec75bda1f\" (UID: \"dfb9124e-3670-46b5-aa88-702ec75bda1f\") "
Nov 03 20:08:51.625871 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:51.625757    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfb9124e-3670-46b5-aa88-702ec75bda1f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfb9124e-3670-46b5-aa88-702ec75bda1f-openshift-service-ca\") pod \"dfb9124e-3670-46b5-aa88-702ec75bda1f\" (UID: \"dfb9124e-3670-46b5-aa88-702ec75bda1f\") "
Nov 03 20:08:51.626163 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:51.626136    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfb9124e-3670-46b5-aa88-702ec75bda1f-openshift-service-ca" pod "dfb9124e-3670-46b5-aa88-702ec75bda1f" (UID: "dfb9124e-3670-46b5-aa88-702ec75bda1f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:08:51.627801 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:51.627771    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfb9124e-3670-46b5-aa88-702ec75bda1f-kube-api-access-cvxhh" pod "dfb9124e-3670-46b5-aa88-702ec75bda1f" (UID: "dfb9124e-3670-46b5-aa88-702ec75bda1f"). InnerVolumeSpecName "kube-api-access-cvxhh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:08:51.627916 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:51.627817    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfb9124e-3670-46b5-aa88-702ec75bda1f-metrics-certs" pod "dfb9124e-3670-46b5-aa88-702ec75bda1f" (UID: "dfb9124e-3670-46b5-aa88-702ec75bda1f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:08:51.627916 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:51.627831    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfb9124e-3670-46b5-aa88-702ec75bda1f-maas-api-tls" pod "dfb9124e-3670-46b5-aa88-702ec75bda1f" (UID: "dfb9124e-3670-46b5-aa88-702ec75bda1f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:08:51.726687 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:51.726636    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfb9124e-3670-46b5-aa88-702ec75bda1f-maas-api-tls\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:51.726877 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:51.726706    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvxhh\" (UniqueName: \"kubernetes.io/projected/dfb9124e-3670-46b5-aa88-702ec75bda1f-kube-api-access-cvxhh\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:51.726877 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:51.726722    4921 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfb9124e-3670-46b5-aa88-702ec75bda1f-metrics-certs\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:51.726877 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:51.726735    4921 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfb9124e-3670-46b5-aa88-702ec75bda1f-openshift-service-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:51.886794 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:51.886701    4921 generic.go:356] "Generic (PLEG): container finished" podID="dfb9124e-3670-46b5-aa88-702ec75bda1f" containerID="de96b32c357cbf61dd527cdcfac6f654bb1f7cb0ee63f0a38157ec5274b72f08" exitCode=0
Nov 03 20:08:51.886794 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:51.886748    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-0486e8-5b894565b7-76jhb"
Nov 03 20:08:51.887044 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:51.886754    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-0486e8-5b894565b7-76jhb" event={"ID":"dfb9124e-3670-46b5-aa88-702ec75bda1f","Type":"ContainerDied","Data":"de96b32c357cbf61dd527cdcfac6f654bb1f7cb0ee63f0a38157ec5274b72f08"}
Nov 03 20:08:51.887044 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:51.886864    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-0486e8-5b894565b7-76jhb" event={"ID":"dfb9124e-3670-46b5-aa88-702ec75bda1f","Type":"ContainerDied","Data":"f4199046d869189ce8a67f83c86f269631db22c2ebe053c6a9025e1c7c3bd3b9"}
Nov 03 20:08:51.887044 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:51.886892    4921 scope.go:122] "RemoveContainer" containerID="de96b32c357cbf61dd527cdcfac6f654bb1f7cb0ee63f0a38157ec5274b72f08"
Nov 03 20:08:51.898422 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:51.898399    4921 scope.go:122] "RemoveContainer" containerID="de96b32c357cbf61dd527cdcfac6f654bb1f7cb0ee63f0a38157ec5274b72f08"
Nov 03 20:08:51.898757 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:08:51.898733    4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de96b32c357cbf61dd527cdcfac6f654bb1f7cb0ee63f0a38157ec5274b72f08\": container with ID starting with de96b32c357cbf61dd527cdcfac6f654bb1f7cb0ee63f0a38157ec5274b72f08 not found: ID does not exist" containerID="de96b32c357cbf61dd527cdcfac6f654bb1f7cb0ee63f0a38157ec5274b72f08"
Nov 03 20:08:51.898904 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:51.898763    4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de96b32c357cbf61dd527cdcfac6f654bb1f7cb0ee63f0a38157ec5274b72f08"} err="failed to get container status \"de96b32c357cbf61dd527cdcfac6f654bb1f7cb0ee63f0a38157ec5274b72f08\": rpc error: code = NotFound desc = could not find container \"de96b32c357cbf61dd527cdcfac6f654bb1f7cb0ee63f0a38157ec5274b72f08\": container with ID starting with de96b32c357cbf61dd527cdcfac6f654bb1f7cb0ee63f0a38157ec5274b72f08 not found: ID does not exist"
Nov 03 20:08:51.911133 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:51.911109    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-0486e8-5b894565b7-76jhb"]
Nov 03 20:08:51.915024 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:51.914998    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-0486e8-5b894565b7-76jhb"]
Nov 03 20:08:51.916502 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:51.916481    4921 patch_prober.go:29] interesting pod/e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:08:51.916592 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:51.916532    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww" podUID="a52b2e9b-9ca0-4897-a071-a79c80e40d14" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:08:52.849596 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:52.849571    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww"
Nov 03 20:08:52.894232 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:52.894142    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="994d46ff-d6e1-4563-9f0e-3a3d767c8b68" path="/var/lib/kubelet/pods/994d46ff-d6e1-4563-9f0e-3a3d767c8b68/volumes"
Nov 03 20:08:52.894645 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:52.894625    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfb9124e-3670-46b5-aa88-702ec75bda1f" path="/var/lib/kubelet/pods/dfb9124e-3670-46b5-aa88-702ec75bda1f/volumes"
Nov 03 20:08:52.895297 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:52.895269    4921 generic.go:356] "Generic (PLEG): container finished" podID="a52b2e9b-9ca0-4897-a071-a79c80e40d14" containerID="3e157dd7b9d6cfe501547f12915c5cd99acd3f6e3b27711109c13f06e3a31875" exitCode=0
Nov 03 20:08:52.895415 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:52.895388    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww"
Nov 03 20:08:52.896934 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:52.896915    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww" event={"ID":"a52b2e9b-9ca0-4897-a071-a79c80e40d14","Type":"ContainerDied","Data":"3e157dd7b9d6cfe501547f12915c5cd99acd3f6e3b27711109c13f06e3a31875"}
Nov 03 20:08:52.897017 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:52.896956    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww" event={"ID":"a52b2e9b-9ca0-4897-a071-a79c80e40d14","Type":"ContainerDied","Data":"6a752ad6e84dbf3274e4fb4f05284d8e57ed47fb1f57383bb5bd76c8909f4c3d"}
Nov 03 20:08:52.897017 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:52.896979    4921 scope.go:122] "RemoveContainer" containerID="3e157dd7b9d6cfe501547f12915c5cd99acd3f6e3b27711109c13f06e3a31875"
Nov 03 20:08:52.907056 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:52.907034    4921 scope.go:122] "RemoveContainer" containerID="3e157dd7b9d6cfe501547f12915c5cd99acd3f6e3b27711109c13f06e3a31875"
Nov 03 20:08:52.907323 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:08:52.907304    4921 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e157dd7b9d6cfe501547f12915c5cd99acd3f6e3b27711109c13f06e3a31875\": container with ID starting with 3e157dd7b9d6cfe501547f12915c5cd99acd3f6e3b27711109c13f06e3a31875 not found: ID does not exist" containerID="3e157dd7b9d6cfe501547f12915c5cd99acd3f6e3b27711109c13f06e3a31875"
Nov 03 20:08:52.907379 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:52.907330    4921 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e157dd7b9d6cfe501547f12915c5cd99acd3f6e3b27711109c13f06e3a31875"} err="failed to get container status \"3e157dd7b9d6cfe501547f12915c5cd99acd3f6e3b27711109c13f06e3a31875\": rpc error: code = NotFound desc = could not find container \"3e157dd7b9d6cfe501547f12915c5cd99acd3f6e3b27711109c13f06e3a31875\": container with ID starting with 3e157dd7b9d6cfe501547f12915c5cd99acd3f6e3b27711109c13f06e3a31875 not found: ID does not exist"
Nov 03 20:08:53.037114 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:53.037083    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a52b2e9b-9ca0-4897-a071-a79c80e40d14-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a52b2e9b-9ca0-4897-a071-a79c80e40d14-istio-data\") pod \"a52b2e9b-9ca0-4897-a071-a79c80e40d14\" (UID: \"a52b2e9b-9ca0-4897-a071-a79c80e40d14\") "
Nov 03 20:08:53.037114 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:53.037119    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a52b2e9b-9ca0-4897-a071-a79c80e40d14-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a52b2e9b-9ca0-4897-a071-a79c80e40d14-credential-socket\") pod \"a52b2e9b-9ca0-4897-a071-a79c80e40d14\" (UID: \"a52b2e9b-9ca0-4897-a071-a79c80e40d14\") "
Nov 03 20:08:53.037355 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:53.037181    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a52b2e9b-9ca0-4897-a071-a79c80e40d14-istio-token\" (UniqueName: \"kubernetes.io/projected/a52b2e9b-9ca0-4897-a071-a79c80e40d14-istio-token\") pod \"a52b2e9b-9ca0-4897-a071-a79c80e40d14\" (UID: \"a52b2e9b-9ca0-4897-a071-a79c80e40d14\") "
Nov 03 20:08:53.037355 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:53.037227    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a52b2e9b-9ca0-4897-a071-a79c80e40d14-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a52b2e9b-9ca0-4897-a071-a79c80e40d14-istiod-ca-cert\") pod \"a52b2e9b-9ca0-4897-a071-a79c80e40d14\" (UID: \"a52b2e9b-9ca0-4897-a071-a79c80e40d14\") "
Nov 03 20:08:53.037355 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:53.037254    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a52b2e9b-9ca0-4897-a071-a79c80e40d14-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a52b2e9b-9ca0-4897-a071-a79c80e40d14-workload-certs\") pod \"a52b2e9b-9ca0-4897-a071-a79c80e40d14\" (UID: \"a52b2e9b-9ca0-4897-a071-a79c80e40d14\") "
Nov 03 20:08:53.037355 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:53.037273    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a52b2e9b-9ca0-4897-a071-a79c80e40d14-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a52b2e9b-9ca0-4897-a071-a79c80e40d14-istio-envoy\") pod \"a52b2e9b-9ca0-4897-a071-a79c80e40d14\" (UID: \"a52b2e9b-9ca0-4897-a071-a79c80e40d14\") "
Nov 03 20:08:53.037355 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:53.037329    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a52b2e9b-9ca0-4897-a071-a79c80e40d14-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a52b2e9b-9ca0-4897-a071-a79c80e40d14-workload-socket\") pod \"a52b2e9b-9ca0-4897-a071-a79c80e40d14\" (UID: \"a52b2e9b-9ca0-4897-a071-a79c80e40d14\") "
Nov 03 20:08:53.037626 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:53.037404    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a52b2e9b-9ca0-4897-a071-a79c80e40d14-istio-data" pod "a52b2e9b-9ca0-4897-a071-a79c80e40d14" (UID: "a52b2e9b-9ca0-4897-a071-a79c80e40d14"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:53.037626 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:53.037408    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a52b2e9b-9ca0-4897-a071-a79c80e40d14-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a52b2e9b-9ca0-4897-a071-a79c80e40d14-istio-podinfo\") pod \"a52b2e9b-9ca0-4897-a071-a79c80e40d14\" (UID: \"a52b2e9b-9ca0-4897-a071-a79c80e40d14\") "
Nov 03 20:08:53.037626 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:53.037553    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a52b2e9b-9ca0-4897-a071-a79c80e40d14-kube-api-access-lrnhr\" (UniqueName: \"kubernetes.io/projected/a52b2e9b-9ca0-4897-a071-a79c80e40d14-kube-api-access-lrnhr\") pod \"a52b2e9b-9ca0-4897-a071-a79c80e40d14\" (UID: \"a52b2e9b-9ca0-4897-a071-a79c80e40d14\") "
Nov 03 20:08:53.037626 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:53.037537    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a52b2e9b-9ca0-4897-a071-a79c80e40d14-credential-socket" pod "a52b2e9b-9ca0-4897-a071-a79c80e40d14" (UID: "a52b2e9b-9ca0-4897-a071-a79c80e40d14"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:53.038059 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:53.037723    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a52b2e9b-9ca0-4897-a071-a79c80e40d14-istiod-ca-cert" pod "a52b2e9b-9ca0-4897-a071-a79c80e40d14" (UID: "a52b2e9b-9ca0-4897-a071-a79c80e40d14"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:08:53.038059 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:53.037819    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a52b2e9b-9ca0-4897-a071-a79c80e40d14-workload-certs" pod "a52b2e9b-9ca0-4897-a071-a79c80e40d14" (UID: "a52b2e9b-9ca0-4897-a071-a79c80e40d14"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:53.038059 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:53.037975    4921 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a52b2e9b-9ca0-4897-a071-a79c80e40d14-istio-data\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:53.038059 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:53.037996    4921 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a52b2e9b-9ca0-4897-a071-a79c80e40d14-credential-socket\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:53.038059 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:53.038011    4921 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a52b2e9b-9ca0-4897-a071-a79c80e40d14-istiod-ca-cert\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:53.038059 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:53.038025    4921 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a52b2e9b-9ca0-4897-a071-a79c80e40d14-workload-certs\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:53.038356 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:53.038265    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a52b2e9b-9ca0-4897-a071-a79c80e40d14-workload-socket" pod "a52b2e9b-9ca0-4897-a071-a79c80e40d14" (UID: "a52b2e9b-9ca0-4897-a071-a79c80e40d14"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:53.039519 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:53.039484    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a52b2e9b-9ca0-4897-a071-a79c80e40d14-istio-envoy" pod "a52b2e9b-9ca0-4897-a071-a79c80e40d14" (UID: "a52b2e9b-9ca0-4897-a071-a79c80e40d14"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:53.039627 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:53.039605    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a52b2e9b-9ca0-4897-a071-a79c80e40d14-kube-api-access-lrnhr" pod "a52b2e9b-9ca0-4897-a071-a79c80e40d14" (UID: "a52b2e9b-9ca0-4897-a071-a79c80e40d14"). InnerVolumeSpecName "kube-api-access-lrnhr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:08:53.039873 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:53.039843    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a52b2e9b-9ca0-4897-a071-a79c80e40d14-istio-podinfo" pod "a52b2e9b-9ca0-4897-a071-a79c80e40d14" (UID: "a52b2e9b-9ca0-4897-a071-a79c80e40d14"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:08:53.039966 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:53.039898    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a52b2e9b-9ca0-4897-a071-a79c80e40d14-istio-token" pod "a52b2e9b-9ca0-4897-a071-a79c80e40d14" (UID: "a52b2e9b-9ca0-4897-a071-a79c80e40d14"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:08:53.139216 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:53.139182    4921 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a52b2e9b-9ca0-4897-a071-a79c80e40d14-istio-podinfo\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:53.139216 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:53.139210    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrnhr\" (UniqueName: \"kubernetes.io/projected/a52b2e9b-9ca0-4897-a071-a79c80e40d14-kube-api-access-lrnhr\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:53.139216 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:53.139221    4921 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a52b2e9b-9ca0-4897-a071-a79c80e40d14-istio-token\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:53.139446 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:53.139230    4921 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a52b2e9b-9ca0-4897-a071-a79c80e40d14-istio-envoy\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:53.139446 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:53.139239    4921 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a52b2e9b-9ca0-4897-a071-a79c80e40d14-workload-socket\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:53.217046 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:53.217014    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww"]
Nov 03 20:08:53.222661 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:53.222633    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-82dbbf-openshift-default-7886794857-tzzww"]
Nov 03 20:08:54.887873 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:54.887834    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a52b2e9b-9ca0-4897-a071-a79c80e40d14" path="/var/lib/kubelet/pods/a52b2e9b-9ca0-4897-a071-a79c80e40d14/volumes"
Nov 03 20:08:58.773688 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:58.773350    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9"]
Nov 03 20:08:58.774183 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:08:58.773694    4921 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9" podUID="902fa980-da03-4354-8484-ad392278c1d9" containerName="istio-proxy" containerID="cri-o://a51393deb579f21fb040fc87d3911e67269bcd5d8200a4ba9644e71a01bfc0dc" gracePeriod=30
Nov 03 20:09:02.921984 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:09:02.921940    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 20:09:03.968328 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:03.968057    4921 generic.go:356] "Generic (PLEG): container finished" podID="902fa980-da03-4354-8484-ad392278c1d9" containerID="a51393deb579f21fb040fc87d3911e67269bcd5d8200a4ba9644e71a01bfc0dc" exitCode=0
Nov 03 20:09:03.969347 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:03.968156    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9" event={"ID":"902fa980-da03-4354-8484-ad392278c1d9","Type":"ContainerDied","Data":"a51393deb579f21fb040fc87d3911e67269bcd5d8200a4ba9644e71a01bfc0dc"}
Nov 03 20:09:03.969691 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:03.969626    4921 scope.go:122] "RemoveContainer" containerID="00ffdf26993b52e3e6c8035e50c8b0e14c3f4ffbf8f617d1e7d2886d27a289fa"
Nov 03 20:09:04.133035 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:04.133006    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9"
Nov 03 20:09:04.242171 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:04.242129    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/902fa980-da03-4354-8484-ad392278c1d9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/902fa980-da03-4354-8484-ad392278c1d9-istio-podinfo\") pod \"902fa980-da03-4354-8484-ad392278c1d9\" (UID: \"902fa980-da03-4354-8484-ad392278c1d9\") "
Nov 03 20:09:04.242373 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:04.242184    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/902fa980-da03-4354-8484-ad392278c1d9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/902fa980-da03-4354-8484-ad392278c1d9-istio-envoy\") pod \"902fa980-da03-4354-8484-ad392278c1d9\" (UID: \"902fa980-da03-4354-8484-ad392278c1d9\") "
Nov 03 20:09:04.242373 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:04.242221    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/902fa980-da03-4354-8484-ad392278c1d9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/902fa980-da03-4354-8484-ad392278c1d9-workload-certs\") pod \"902fa980-da03-4354-8484-ad392278c1d9\" (UID: \"902fa980-da03-4354-8484-ad392278c1d9\") "
Nov 03 20:09:04.242373 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:04.242267    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/902fa980-da03-4354-8484-ad392278c1d9-istio-token\" (UniqueName: \"kubernetes.io/projected/902fa980-da03-4354-8484-ad392278c1d9-istio-token\") pod \"902fa980-da03-4354-8484-ad392278c1d9\" (UID: \"902fa980-da03-4354-8484-ad392278c1d9\") "
Nov 03 20:09:04.242373 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:04.242343    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/902fa980-da03-4354-8484-ad392278c1d9-kube-api-access-jf7hp\" (UniqueName: \"kubernetes.io/projected/902fa980-da03-4354-8484-ad392278c1d9-kube-api-access-jf7hp\") pod \"902fa980-da03-4354-8484-ad392278c1d9\" (UID: \"902fa980-da03-4354-8484-ad392278c1d9\") "
Nov 03 20:09:04.242590 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:04.242391    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/902fa980-da03-4354-8484-ad392278c1d9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/902fa980-da03-4354-8484-ad392278c1d9-istiod-ca-cert\") pod \"902fa980-da03-4354-8484-ad392278c1d9\" (UID: \"902fa980-da03-4354-8484-ad392278c1d9\") "
Nov 03 20:09:04.242590 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:04.242425    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/902fa980-da03-4354-8484-ad392278c1d9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/902fa980-da03-4354-8484-ad392278c1d9-workload-socket\") pod \"902fa980-da03-4354-8484-ad392278c1d9\" (UID: \"902fa980-da03-4354-8484-ad392278c1d9\") "
Nov 03 20:09:04.242590 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:04.242569    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/902fa980-da03-4354-8484-ad392278c1d9-workload-certs" pod "902fa980-da03-4354-8484-ad392278c1d9" (UID: "902fa980-da03-4354-8484-ad392278c1d9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:09:04.242796 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:04.242695    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/902fa980-da03-4354-8484-ad392278c1d9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/902fa980-da03-4354-8484-ad392278c1d9-istio-data\") pod \"902fa980-da03-4354-8484-ad392278c1d9\" (UID: \"902fa980-da03-4354-8484-ad392278c1d9\") "
Nov 03 20:09:04.242796 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:04.242703    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/902fa980-da03-4354-8484-ad392278c1d9-workload-socket" pod "902fa980-da03-4354-8484-ad392278c1d9" (UID: "902fa980-da03-4354-8484-ad392278c1d9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:09:04.242796 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:04.242735    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/902fa980-da03-4354-8484-ad392278c1d9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/902fa980-da03-4354-8484-ad392278c1d9-credential-socket\") pod \"902fa980-da03-4354-8484-ad392278c1d9\" (UID: \"902fa980-da03-4354-8484-ad392278c1d9\") "
Nov 03 20:09:04.242958 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:04.242908    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/902fa980-da03-4354-8484-ad392278c1d9-istio-data" pod "902fa980-da03-4354-8484-ad392278c1d9" (UID: "902fa980-da03-4354-8484-ad392278c1d9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:09:04.242958 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:04.242938    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/902fa980-da03-4354-8484-ad392278c1d9-istiod-ca-cert" pod "902fa980-da03-4354-8484-ad392278c1d9" (UID: "902fa980-da03-4354-8484-ad392278c1d9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:09:04.243101 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:04.243081    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/902fa980-da03-4354-8484-ad392278c1d9-credential-socket" pod "902fa980-da03-4354-8484-ad392278c1d9" (UID: "902fa980-da03-4354-8484-ad392278c1d9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:09:04.243156 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:04.243123    4921 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/902fa980-da03-4354-8484-ad392278c1d9-istiod-ca-cert\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:04.243156 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:04.243139    4921 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/902fa980-da03-4354-8484-ad392278c1d9-workload-socket\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:04.243156 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:04.243150    4921 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/902fa980-da03-4354-8484-ad392278c1d9-istio-data\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:04.243270 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:04.243164    4921 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/902fa980-da03-4354-8484-ad392278c1d9-workload-certs\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:04.245107 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:04.245070    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/902fa980-da03-4354-8484-ad392278c1d9-istio-envoy" pod "902fa980-da03-4354-8484-ad392278c1d9" (UID: "902fa980-da03-4354-8484-ad392278c1d9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:09:04.245241 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:04.245128    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/902fa980-da03-4354-8484-ad392278c1d9-istio-podinfo" pod "902fa980-da03-4354-8484-ad392278c1d9" (UID: "902fa980-da03-4354-8484-ad392278c1d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:09:04.245241 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:04.245144    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/902fa980-da03-4354-8484-ad392278c1d9-kube-api-access-jf7hp" pod "902fa980-da03-4354-8484-ad392278c1d9" (UID: "902fa980-da03-4354-8484-ad392278c1d9"). InnerVolumeSpecName "kube-api-access-jf7hp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:09:04.245241 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:04.245152    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/902fa980-da03-4354-8484-ad392278c1d9-istio-token" pod "902fa980-da03-4354-8484-ad392278c1d9" (UID: "902fa980-da03-4354-8484-ad392278c1d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:09:04.344287 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:04.344182    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jf7hp\" (UniqueName: \"kubernetes.io/projected/902fa980-da03-4354-8484-ad392278c1d9-kube-api-access-jf7hp\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:04.344287 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:04.344223    4921 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/902fa980-da03-4354-8484-ad392278c1d9-credential-socket\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:04.344287 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:04.344234    4921 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/902fa980-da03-4354-8484-ad392278c1d9-istio-podinfo\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:04.344287 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:04.344244    4921 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/902fa980-da03-4354-8484-ad392278c1d9-istio-envoy\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:04.344287 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:04.344254    4921 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/902fa980-da03-4354-8484-ad392278c1d9-istio-token\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:04.976923 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:04.976891    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9"
Nov 03 20:09:04.977406 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:04.976939    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9" event={"ID":"902fa980-da03-4354-8484-ad392278c1d9","Type":"ContainerDied","Data":"5177993d834aedf1282ea91a6ed07b2f6debdf5a42d57e1b262ae2640c7c963f"}
Nov 03 20:09:04.977406 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:04.977006    4921 scope.go:122] "RemoveContainer" containerID="a51393deb579f21fb040fc87d3911e67269bcd5d8200a4ba9644e71a01bfc0dc"
Nov 03 20:09:04.992539 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:04.992498    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9"]
Nov 03 20:09:04.997629 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:04.997593    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-0486e8-openshift-default-5dbcd76fb5-49sq9"]
Nov 03 20:09:06.887266 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:06.887229    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="902fa980-da03-4354-8484-ad392278c1d9" path="/var/lib/kubelet/pods/902fa980-da03-4354-8484-ad392278c1d9/volumes"
Nov 03 20:09:11.028783 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:11.028741    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-km8l8"]
Nov 03 20:09:11.029407 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:11.029386    4921 cpu_manager.go:414] "containerMap: removing container" podUID="a52b2e9b-9ca0-4897-a071-a79c80e40d14" containerName="istio-proxy"
Nov 03 20:09:11.029506 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:11.029410    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a52b2e9b-9ca0-4897-a071-a79c80e40d14" containerName="istio-proxy"
Nov 03 20:09:11.029506 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:11.029434    4921 cpu_manager.go:414] "containerMap: removing container" podUID="902fa980-da03-4354-8484-ad392278c1d9" containerName="istio-proxy"
Nov 03 20:09:11.029506 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:11.029442    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="902fa980-da03-4354-8484-ad392278c1d9" containerName="istio-proxy"
Nov 03 20:09:11.029506 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:11.029460    4921 cpu_manager.go:414] "containerMap: removing container" podUID="96b73342-0079-4709-8570-3dbce57e5243" containerName="main"
Nov 03 20:09:11.029506 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:11.029467    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96b73342-0079-4709-8570-3dbce57e5243" containerName="main"
Nov 03 20:09:11.029506 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:11.029487    4921 cpu_manager.go:414] "containerMap: removing container" podUID="dfb9124e-3670-46b5-aa88-702ec75bda1f" containerName="maas-api"
Nov 03 20:09:11.029506 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:11.029494    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfb9124e-3670-46b5-aa88-702ec75bda1f" containerName="maas-api"
Nov 03 20:09:11.029937 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:11.029512    4921 cpu_manager.go:414] "containerMap: removing container" podUID="2d0a934e-4816-44d2-9bb5-e8b4e95dd73c" containerName="main"
Nov 03 20:09:11.029937 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:11.029520    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d0a934e-4816-44d2-9bb5-e8b4e95dd73c" containerName="main"
Nov 03 20:09:11.029937 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:11.029534    4921 cpu_manager.go:414] "containerMap: removing container" podUID="902fa980-da03-4354-8484-ad392278c1d9" containerName="istio-proxy"
Nov 03 20:09:11.029937 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:11.029541    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="902fa980-da03-4354-8484-ad392278c1d9" containerName="istio-proxy"
Nov 03 20:09:11.029937 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:11.029563    4921 cpu_manager.go:414] "containerMap: removing container" podUID="994d46ff-d6e1-4563-9f0e-3a3d767c8b68" containerName="revoke-keys"
Nov 03 20:09:11.029937 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:11.029571    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="994d46ff-d6e1-4563-9f0e-3a3d767c8b68" containerName="revoke-keys"
Nov 03 20:09:11.029937 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:11.029687    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfb9124e-3670-46b5-aa88-702ec75bda1f" containerName="maas-api"
Nov 03 20:09:11.029937 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:11.029701    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d0a934e-4816-44d2-9bb5-e8b4e95dd73c" containerName="main"
Nov 03 20:09:11.029937 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:11.029711    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="902fa980-da03-4354-8484-ad392278c1d9" containerName="istio-proxy"
Nov 03 20:09:11.029937 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:11.029724    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="994d46ff-d6e1-4563-9f0e-3a3d767c8b68" containerName="revoke-keys"
Nov 03 20:09:11.029937 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:11.029735    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="a52b2e9b-9ca0-4897-a071-a79c80e40d14" containerName="istio-proxy"
Nov 03 20:09:11.029937 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:11.029746    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="902fa980-da03-4354-8484-ad392278c1d9" containerName="istio-proxy"
Nov 03 20:09:11.029937 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:11.029758    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="96b73342-0079-4709-8570-3dbce57e5243" containerName="main"
Nov 03 20:09:11.036031 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:11.036008    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-km8l8"
Nov 03 20:09:11.047448 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:11.046360    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-km8l8"]
Nov 03 20:09:11.110178 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:11.110141    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b858864-6582-48ad-9728-5e5e8c5aa818-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-km8l8\" (UID: \"4b858864-6582-48ad-9728-5e5e8c5aa818\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-km8l8"
Nov 03 20:09:11.110373 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:11.110181    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67c5d\" (UniqueName: \"kubernetes.io/projected/4b858864-6582-48ad-9728-5e5e8c5aa818-kube-api-access-67c5d\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-km8l8\" (UID: \"4b858864-6582-48ad-9728-5e5e8c5aa818\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-km8l8"
Nov 03 20:09:11.110373 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:11.110339    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b858864-6582-48ad-9728-5e5e8c5aa818-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-km8l8\" (UID: \"4b858864-6582-48ad-9728-5e5e8c5aa818\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-km8l8"
Nov 03 20:09:11.110490 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:11.110394    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b858864-6582-48ad-9728-5e5e8c5aa818-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-km8l8\" (UID: \"4b858864-6582-48ad-9728-5e5e8c5aa818\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-km8l8"
Nov 03 20:09:11.110490 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:11.110483    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b858864-6582-48ad-9728-5e5e8c5aa818-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-km8l8\" (UID: \"4b858864-6582-48ad-9728-5e5e8c5aa818\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-km8l8"
Nov 03 20:09:11.110583 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:11.110518    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b858864-6582-48ad-9728-5e5e8c5aa818-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-km8l8\" (UID: \"4b858864-6582-48ad-9728-5e5e8c5aa818\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-km8l8"
Nov 03 20:09:11.110583 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:11.110537    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b858864-6582-48ad-9728-5e5e8c5aa818-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-km8l8\" (UID: \"4b858864-6582-48ad-9728-5e5e8c5aa818\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-km8l8"
Nov 03 20:09:11.110728 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:11.110607    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b858864-6582-48ad-9728-5e5e8c5aa818-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-km8l8\" (UID: \"4b858864-6582-48ad-9728-5e5e8c5aa818\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-km8l8"
Nov 03 20:09:11.110728 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:11.110682    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b858864-6582-48ad-9728-5e5e8c5aa818-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-km8l8\" (UID: \"4b858864-6582-48ad-9728-5e5e8c5aa818\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-km8l8"
Nov 03 20:09:11.211567 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:11.211532    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b858864-6582-48ad-9728-5e5e8c5aa818-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-km8l8\" (UID: \"4b858864-6582-48ad-9728-5e5e8c5aa818\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-km8l8"
Nov 03 20:09:11.211756 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:11.211584    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b858864-6582-48ad-9728-5e5e8c5aa818-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-km8l8\" (UID: \"4b858864-6582-48ad-9728-5e5e8c5aa818\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-km8l8"
Nov 03 20:09:11.211756 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:11.211622    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b858864-6582-48ad-9728-5e5e8c5aa818-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-km8l8\" (UID: \"4b858864-6582-48ad-9728-5e5e8c5aa818\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-km8l8"
Nov 03 20:09:11.211756 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:11.211653    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b858864-6582-48ad-9728-5e5e8c5aa818-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-km8l8\" (UID: \"4b858864-6582-48ad-9728-5e5e8c5aa818\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-km8l8"
Nov 03 20:09:11.211756 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:11.211701    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b858864-6582-48ad-9728-5e5e8c5aa818-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-km8l8\" (UID: \"4b858864-6582-48ad-9728-5e5e8c5aa818\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-km8l8"
Nov 03 20:09:11.211756 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:11.211746    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b858864-6582-48ad-9728-5e5e8c5aa818-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-km8l8\" (UID: \"4b858864-6582-48ad-9728-5e5e8c5aa818\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-km8l8"
Nov 03 20:09:11.211941 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:11.211806    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b858864-6582-48ad-9728-5e5e8c5aa818-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-km8l8\" (UID: \"4b858864-6582-48ad-9728-5e5e8c5aa818\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-km8l8"
Nov 03 20:09:11.211941 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:11.211835    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b858864-6582-48ad-9728-5e5e8c5aa818-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-km8l8\" (UID: \"4b858864-6582-48ad-9728-5e5e8c5aa818\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-km8l8"
Nov 03 20:09:11.211941 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:11.211866    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67c5d\" (UniqueName: \"kubernetes.io/projected/4b858864-6582-48ad-9728-5e5e8c5aa818-kube-api-access-67c5d\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-km8l8\" (UID: \"4b858864-6582-48ad-9728-5e5e8c5aa818\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-km8l8"
Nov 03 20:09:11.212041 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:11.211960    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b858864-6582-48ad-9728-5e5e8c5aa818-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-km8l8\" (UID: \"4b858864-6582-48ad-9728-5e5e8c5aa818\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-km8l8"
Nov 03 20:09:11.212317 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:11.212294    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b858864-6582-48ad-9728-5e5e8c5aa818-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-km8l8\" (UID: \"4b858864-6582-48ad-9728-5e5e8c5aa818\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-km8l8"
Nov 03 20:09:11.212394 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:11.212373    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b858864-6582-48ad-9728-5e5e8c5aa818-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-km8l8\" (UID: \"4b858864-6582-48ad-9728-5e5e8c5aa818\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-km8l8"
Nov 03 20:09:11.212510 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:11.212492    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b858864-6582-48ad-9728-5e5e8c5aa818-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-km8l8\" (UID: \"4b858864-6582-48ad-9728-5e5e8c5aa818\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-km8l8"
Nov 03 20:09:11.212772 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:11.212754    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b858864-6582-48ad-9728-5e5e8c5aa818-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-km8l8\" (UID: \"4b858864-6582-48ad-9728-5e5e8c5aa818\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-km8l8"
Nov 03 20:09:11.214647 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:11.214625    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b858864-6582-48ad-9728-5e5e8c5aa818-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-km8l8\" (UID: \"4b858864-6582-48ad-9728-5e5e8c5aa818\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-km8l8"
Nov 03 20:09:11.214647 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:11.214631    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b858864-6582-48ad-9728-5e5e8c5aa818-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-km8l8\" (UID: \"4b858864-6582-48ad-9728-5e5e8c5aa818\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-km8l8"
Nov 03 20:09:11.220712 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:11.220646    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b858864-6582-48ad-9728-5e5e8c5aa818-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-km8l8\" (UID: \"4b858864-6582-48ad-9728-5e5e8c5aa818\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-km8l8"
Nov 03 20:09:11.221438 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:11.221415    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67c5d\" (UniqueName: \"kubernetes.io/projected/4b858864-6582-48ad-9728-5e5e8c5aa818-kube-api-access-67c5d\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-km8l8\" (UID: \"4b858864-6582-48ad-9728-5e5e8c5aa818\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-km8l8"
Nov 03 20:09:11.356688 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:11.356577    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-km8l8"
Nov 03 20:09:11.641604 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:11.641403    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-km8l8"]
Nov 03 20:09:11.644071 ip-10-0-134-222 kubenswrapper[4921]: W1103 20:09:11.644040    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b858864_6582_48ad_9728_5e5e8c5aa818.slice/crio-6007c631f6eb786053c8388669e9a32442cf67c3d78ef0b7015b61a3adae8d87 WatchSource:0}: Error finding container 6007c631f6eb786053c8388669e9a32442cf67c3d78ef0b7015b61a3adae8d87: Status 404 returned error can't find the container with id 6007c631f6eb786053c8388669e9a32442cf67c3d78ef0b7015b61a3adae8d87
Nov 03 20:09:11.646411 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:11.646377    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:09:11.646494 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:11.646444    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:09:11.646494 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:11.646480    4921 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:09:12.026945 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:12.026902    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-km8l8" event={"ID":"4b858864-6582-48ad-9728-5e5e8c5aa818","Type":"ContainerStarted","Data":"4fa9004a2fb565ae135dadc495bbef723c1b8671f9149aa9c858c36faca59a71"}
Nov 03 20:09:12.027128 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:12.026966    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-km8l8" event={"ID":"4b858864-6582-48ad-9728-5e5e8c5aa818","Type":"ContainerStarted","Data":"6007c631f6eb786053c8388669e9a32442cf67c3d78ef0b7015b61a3adae8d87"}
Nov 03 20:09:12.045030 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:12.044973    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-km8l8" podStartSLOduration=1.044956976 podStartE2EDuration="1.044956976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:09:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:09:12.043843914 +0000 UTC m=+2265.287809379" watchObservedRunningTime="2026-11-03 20:09:12.044956976 +0000 UTC m=+2265.288922419"
Nov 03 20:09:12.370739 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:12.369749    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-km8l8"
Nov 03 20:09:12.373833 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:12.373800    4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-km8l8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body=
Nov 03 20:09:12.373997 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:12.373863    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-km8l8" podUID="4b858864-6582-48ad-9728-5e5e8c5aa818" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused"
Nov 03 20:09:13.371692 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:13.364157    4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-km8l8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body=
Nov 03 20:09:13.371692 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:13.364224    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-km8l8" podUID="4b858864-6582-48ad-9728-5e5e8c5aa818" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused"
Nov 03 20:09:14.363128 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:14.363087    4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-km8l8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body=
Nov 03 20:09:14.363477 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:14.363435    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-km8l8" podUID="4b858864-6582-48ad-9728-5e5e8c5aa818" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused"
Nov 03 20:09:15.369694 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:15.367078    4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-km8l8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body=
Nov 03 20:09:15.369694 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:15.367142    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-km8l8" podUID="4b858864-6582-48ad-9728-5e5e8c5aa818" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused"
Nov 03 20:09:16.364199 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:16.364155    4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-km8l8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body=
Nov 03 20:09:16.364513 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:16.364478    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-km8l8" podUID="4b858864-6582-48ad-9728-5e5e8c5aa818" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused"
Nov 03 20:09:17.371700 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:17.371022    4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-km8l8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body=
Nov 03 20:09:17.371700 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:17.371083    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-km8l8" podUID="4b858864-6582-48ad-9728-5e5e8c5aa818" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused"
Nov 03 20:09:18.357182 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:18.357138    4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-km8l8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body=
Nov 03 20:09:18.357382 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:18.357190    4921 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-km8l8" podUID="4b858864-6582-48ad-9728-5e5e8c5aa818" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused"
Nov 03 20:09:19.880564 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:19.880517    4921 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-km8l8"
Nov 03 20:09:19.881368 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:19.881340    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-km8l8"
Nov 03 20:09:19.881826 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:09:19.881803    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-km8l8"
Nov 03 20:09:32.920746 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:09:32.920711    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 20:10:02.919550 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:10:02.919513    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 20:10:32.920767 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:10:32.920731    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 20:10:50.925929 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:10:50.925861    4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-km8l8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:10:50.926585 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:10:50.925940    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-km8l8" podUID="4b858864-6582-48ad-9728-5e5e8c5aa818" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:11:02.920247 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:11:02.920210    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 20:11:31.932971 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:11:31.932893    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gzf8k_6f215b42-0f09-46a0-bc98-392c0fa909c9/console-operator/2.log"
Nov 03 20:11:31.948569 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:11:31.948547    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gzf8k_6f215b42-0f09-46a0-bc98-392c0fa909c9/console-operator/2.log"
Nov 03 20:11:32.920144 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:11:32.920066    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 20:12:02.920800 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:12:02.920759    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 20:12:32.923322 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:12:32.923284    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 20:12:35.924952 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:12:35.924833    4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-km8l8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:12:35.925700 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:12:35.925626    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-km8l8" podUID="4b858864-6582-48ad-9728-5e5e8c5aa818" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:13:02.929342 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:13:02.929296    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 20:13:32.920836 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:13:32.920798    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 20:14:02.920222 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:14:02.920181    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 20:14:31.946574 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:14:31.946476    4921 scope.go:122] "RemoveContainer" containerID="e0d3ab73da2b20800ab4956bc261bb36fccd2df76de73f31720cc5764e66999c"
Nov 03 20:14:32.927376 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:14:32.927337    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 20:15:00.130173 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:15:00.130135    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-030599-29895616d7qp"]
Nov 03 20:15:00.133822 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:15:00.133800    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-030599-29895616d7qp"
Nov 03 20:15:00.141953 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:15:00.141918    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-030599-29895616d7qp"]
Nov 03 20:15:00.143804 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:15:00.143784    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 20:15:00.176980 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:15:00.176941    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzqcs\" (UniqueName: \"kubernetes.io/projected/159f95e2-29a1-4f95-8fbb-eff99dba6b80-kube-api-access-bzqcs\") pod \"maas-api-key-cleanup-e2e-worker-single-main-030599-29895616d7qp\" (UID: \"159f95e2-29a1-4f95-8fbb-eff99dba6b80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-030599-29895616d7qp"
Nov 03 20:15:00.177183 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:15:00.176999    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/159f95e2-29a1-4f95-8fbb-eff99dba6b80-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-single-main-030599-29895616d7qp\" (UID: \"159f95e2-29a1-4f95-8fbb-eff99dba6b80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-030599-29895616d7qp"
Nov 03 20:15:00.277661 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:15:00.277621    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/159f95e2-29a1-4f95-8fbb-eff99dba6b80-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-single-main-030599-29895616d7qp\" (UID: \"159f95e2-29a1-4f95-8fbb-eff99dba6b80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-030599-29895616d7qp"
Nov 03 20:15:00.277898 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:15:00.277804    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzqcs\" (UniqueName: \"kubernetes.io/projected/159f95e2-29a1-4f95-8fbb-eff99dba6b80-kube-api-access-bzqcs\") pod \"maas-api-key-cleanup-e2e-worker-single-main-030599-29895616d7qp\" (UID: \"159f95e2-29a1-4f95-8fbb-eff99dba6b80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-030599-29895616d7qp"
Nov 03 20:15:00.278278 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:15:00.278257    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/159f95e2-29a1-4f95-8fbb-eff99dba6b80-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-single-main-030599-29895616d7qp\" (UID: \"159f95e2-29a1-4f95-8fbb-eff99dba6b80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-030599-29895616d7qp"
Nov 03 20:15:00.285041 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:15:00.285011    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzqcs\" (UniqueName: \"kubernetes.io/projected/159f95e2-29a1-4f95-8fbb-eff99dba6b80-kube-api-access-bzqcs\") pod \"maas-api-key-cleanup-e2e-worker-single-main-030599-29895616d7qp\" (UID: \"159f95e2-29a1-4f95-8fbb-eff99dba6b80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-030599-29895616d7qp"
Nov 03 20:15:00.449459 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:15:00.449373    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-030599-29895616d7qp"
Nov 03 20:15:00.724482 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:15:00.724454    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-030599-29895616d7qp"]
Nov 03 20:15:00.726817 ip-10-0-134-222 kubenswrapper[4921]: W1103 20:15:00.726781    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod159f95e2_29a1_4f95_8fbb_eff99dba6b80.slice/crio-0c54d8cec224adbe77fadbf293d50e6694080bc4951e06614c34eb128525d082 WatchSource:0}: Error finding container 0c54d8cec224adbe77fadbf293d50e6694080bc4951e06614c34eb128525d082: Status 404 returned error can't find the container with id 0c54d8cec224adbe77fadbf293d50e6694080bc4951e06614c34eb128525d082
Nov 03 20:15:01.167048 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:15:01.167009    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-030599-29895616d7qp" event={"ID":"159f95e2-29a1-4f95-8fbb-eff99dba6b80","Type":"ContainerStarted","Data":"53c9561343732e3d4ea1cc5578492715e87c8c22fc9ddea4fba88f435e9a9b59"}
Nov 03 20:15:01.167472 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:15:01.167074    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-030599-29895616d7qp" event={"ID":"159f95e2-29a1-4f95-8fbb-eff99dba6b80","Type":"ContainerStarted","Data":"0c54d8cec224adbe77fadbf293d50e6694080bc4951e06614c34eb128525d082"}
Nov 03 20:15:01.183734 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:15:01.183663    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-030599-29895616d7qp" podStartSLOduration=1.183649023 podStartE2EDuration="1.183649023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:01.180592522 +0000 UTC m=+2614.424557988" watchObservedRunningTime="2026-11-03 20:15:01.183649023 +0000 UTC m=+2614.427614487"
Nov 03 20:15:02.174495 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:15:02.174453    4921 generic.go:356] "Generic (PLEG): container finished" podID="159f95e2-29a1-4f95-8fbb-eff99dba6b80" containerID="53c9561343732e3d4ea1cc5578492715e87c8c22fc9ddea4fba88f435e9a9b59" exitCode=60
Nov 03 20:15:02.174946 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:15:02.174542    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-030599-29895616d7qp" event={"ID":"159f95e2-29a1-4f95-8fbb-eff99dba6b80","Type":"ContainerDied","Data":"53c9561343732e3d4ea1cc5578492715e87c8c22fc9ddea4fba88f435e9a9b59"}
Nov 03 20:15:02.174946 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:15:02.174776    4921 scope.go:122] "RemoveContainer" containerID="53c9561343732e3d4ea1cc5578492715e87c8c22fc9ddea4fba88f435e9a9b59"
Nov 03 20:15:02.921956 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:15:02.921922    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 20:15:03.182196 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:15:03.182108    4921 generic.go:356] "Generic (PLEG): container finished" podID="159f95e2-29a1-4f95-8fbb-eff99dba6b80" containerID="b29978e9840bb16d361e6a45c73718443831db1db1749c946706d35c8923fd5d" exitCode=60
Nov 03 20:15:03.182196 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:15:03.182176    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-030599-29895616d7qp" event={"ID":"159f95e2-29a1-4f95-8fbb-eff99dba6b80","Type":"ContainerDied","Data":"b29978e9840bb16d361e6a45c73718443831db1db1749c946706d35c8923fd5d"}
Nov 03 20:15:03.182759 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:15:03.182236    4921 scope.go:122] "RemoveContainer" containerID="53c9561343732e3d4ea1cc5578492715e87c8c22fc9ddea4fba88f435e9a9b59"
Nov 03 20:15:03.182759 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:15:03.182495    4921 scope.go:122] "RemoveContainer" containerID="b29978e9840bb16d361e6a45c73718443831db1db1749c946706d35c8923fd5d"
Nov 03 20:15:03.182874 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:15:03.182830    4921 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-single-main-030599-29895616d7qp_odh-ai-gateway-infra(159f95e2-29a1-4f95-8fbb-eff99dba6b80)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-030599-29895616d7qp" podUID="159f95e2-29a1-4f95-8fbb-eff99dba6b80"
Nov 03 20:15:04.189120 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:15:04.189088    4921 scope.go:122] "RemoveContainer" containerID="b29978e9840bb16d361e6a45c73718443831db1db1749c946706d35c8923fd5d"
Nov 03 20:15:04.189547 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:15:04.189328    4921 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-single-main-030599-29895616d7qp_odh-ai-gateway-infra(159f95e2-29a1-4f95-8fbb-eff99dba6b80)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-030599-29895616d7qp" podUID="159f95e2-29a1-4f95-8fbb-eff99dba6b80"
Nov 03 20:15:14.881350 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:15:14.881309    4921 scope.go:122] "RemoveContainer" containerID="b29978e9840bb16d361e6a45c73718443831db1db1749c946706d35c8923fd5d"
Nov 03 20:15:15.266799 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:15:15.266764    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-030599-29895616d7qp" event={"ID":"159f95e2-29a1-4f95-8fbb-eff99dba6b80","Type":"ContainerStarted","Data":"85bf3d182ad8e4d0cbac2b2ed15c1ac389ac5c1e95a87a95683c8bea9ca7fd31"}
Nov 03 20:15:16.275398 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:15:16.275360    4921 generic.go:356] "Generic (PLEG): container finished" podID="159f95e2-29a1-4f95-8fbb-eff99dba6b80" containerID="85bf3d182ad8e4d0cbac2b2ed15c1ac389ac5c1e95a87a95683c8bea9ca7fd31" exitCode=60
Nov 03 20:15:16.275934 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:15:16.275403    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-030599-29895616d7qp" event={"ID":"159f95e2-29a1-4f95-8fbb-eff99dba6b80","Type":"ContainerDied","Data":"85bf3d182ad8e4d0cbac2b2ed15c1ac389ac5c1e95a87a95683c8bea9ca7fd31"}
Nov 03 20:15:16.275934 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:15:16.275468    4921 scope.go:122] "RemoveContainer" containerID="b29978e9840bb16d361e6a45c73718443831db1db1749c946706d35c8923fd5d"
Nov 03 20:15:16.275934 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:15:16.275851    4921 scope.go:122] "RemoveContainer" containerID="85bf3d182ad8e4d0cbac2b2ed15c1ac389ac5c1e95a87a95683c8bea9ca7fd31"
Nov 03 20:15:16.276136 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:15:16.276115    4921 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-single-main-030599-29895616d7qp_odh-ai-gateway-infra(159f95e2-29a1-4f95-8fbb-eff99dba6b80)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-030599-29895616d7qp" podUID="159f95e2-29a1-4f95-8fbb-eff99dba6b80"
Nov 03 20:15:16.295240 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:15:16.295217    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-030599-29895616d7qp"]
Nov 03 20:15:17.442690 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:15:17.439238    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-030599-29895616d7qp"
Nov 03 20:15:17.583777 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:15:17.583690    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/159f95e2-29a1-4f95-8fbb-eff99dba6b80-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/159f95e2-29a1-4f95-8fbb-eff99dba6b80-maas-api-service-ca\") pod \"159f95e2-29a1-4f95-8fbb-eff99dba6b80\" (UID: \"159f95e2-29a1-4f95-8fbb-eff99dba6b80\") "
Nov 03 20:15:17.583959 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:15:17.583846    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/159f95e2-29a1-4f95-8fbb-eff99dba6b80-kube-api-access-bzqcs\" (UniqueName: \"kubernetes.io/projected/159f95e2-29a1-4f95-8fbb-eff99dba6b80-kube-api-access-bzqcs\") pod \"159f95e2-29a1-4f95-8fbb-eff99dba6b80\" (UID: \"159f95e2-29a1-4f95-8fbb-eff99dba6b80\") "
Nov 03 20:15:17.584042 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:15:17.583967    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/159f95e2-29a1-4f95-8fbb-eff99dba6b80-maas-api-service-ca" pod "159f95e2-29a1-4f95-8fbb-eff99dba6b80" (UID: "159f95e2-29a1-4f95-8fbb-eff99dba6b80"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:15:17.584517 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:15:17.584496    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/159f95e2-29a1-4f95-8fbb-eff99dba6b80-maas-api-service-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:17.586076 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:15:17.586046    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/159f95e2-29a1-4f95-8fbb-eff99dba6b80-kube-api-access-bzqcs" pod "159f95e2-29a1-4f95-8fbb-eff99dba6b80" (UID: "159f95e2-29a1-4f95-8fbb-eff99dba6b80"). InnerVolumeSpecName "kube-api-access-bzqcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:15:17.685861 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:15:17.685823    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzqcs\" (UniqueName: \"kubernetes.io/projected/159f95e2-29a1-4f95-8fbb-eff99dba6b80-kube-api-access-bzqcs\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:18.290711 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:15:18.290678    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-030599-29895616d7qp"
Nov 03 20:15:18.290912 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:15:18.290706    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-030599-29895616d7qp" event={"ID":"159f95e2-29a1-4f95-8fbb-eff99dba6b80","Type":"ContainerDied","Data":"0c54d8cec224adbe77fadbf293d50e6694080bc4951e06614c34eb128525d082"}
Nov 03 20:15:18.290912 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:15:18.290776    4921 scope.go:122] "RemoveContainer" containerID="85bf3d182ad8e4d0cbac2b2ed15c1ac389ac5c1e95a87a95683c8bea9ca7fd31"
Nov 03 20:15:18.321201 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:15:18.321164    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-030599-29895616d7qp"]
Nov 03 20:15:18.330398 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:15:18.330371    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-030599-29895616d7qp"]
Nov 03 20:15:18.887003 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:15:18.886971    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="159f95e2-29a1-4f95-8fbb-eff99dba6b80" path="/var/lib/kubelet/pods/159f95e2-29a1-4f95-8fbb-eff99dba6b80/volumes"
Nov 03 20:15:32.920974 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:15:32.920937    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 20:16:02.921237 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:16:02.921197    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 20:16:31.970523 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:16:31.970494    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gzf8k_6f215b42-0f09-46a0-bc98-392c0fa909c9/console-operator/2.log"
Nov 03 20:16:31.987397 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:16:31.987371    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gzf8k_6f215b42-0f09-46a0-bc98-392c0fa909c9/console-operator/2.log"
Nov 03 20:16:32.920302 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:16:32.920267    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 20:17:02.921405 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:17:02.921364    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 20:17:32.920858 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:17:32.920816    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 20:18:02.923036 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:18:02.922996    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 20:18:32.844236 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:18:32.844200    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e696f37-2dzls"]
Nov 03 20:18:32.844640 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:18:32.844605    4921 cpu_manager.go:414] "containerMap: removing container" podUID="159f95e2-29a1-4f95-8fbb-eff99dba6b80" containerName="cleanup"
Nov 03 20:18:32.844640 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:18:32.844617    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="159f95e2-29a1-4f95-8fbb-eff99dba6b80" containerName="cleanup"
Nov 03 20:18:32.844640 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:18:32.844630    4921 cpu_manager.go:414] "containerMap: removing container" podUID="159f95e2-29a1-4f95-8fbb-eff99dba6b80" containerName="cleanup"
Nov 03 20:18:32.844640 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:18:32.844635    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="159f95e2-29a1-4f95-8fbb-eff99dba6b80" containerName="cleanup"
Nov 03 20:18:32.844819 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:18:32.844750    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="159f95e2-29a1-4f95-8fbb-eff99dba6b80" containerName="cleanup"
Nov 03 20:18:32.844819 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:18:32.844759    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="159f95e2-29a1-4f95-8fbb-eff99dba6b80" containerName="cleanup"
Nov 03 20:18:32.844819 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:18:32.844765    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="159f95e2-29a1-4f95-8fbb-eff99dba6b80" containerName="cleanup"
Nov 03 20:18:32.847436 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:18:32.847417    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e696f37-2dzls"
Nov 03 20:18:32.850351 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:18:32.850317    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:18:32.850771 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:18:32.850752    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 20:18:32.856830 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:18:32.856801    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e696f37-2dzls"]
Nov 03 20:18:32.894044 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:18:32.894003    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8207a8b4-aab5-43e9-8967-c4ada60ed838-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e696f37-2dzls\" (UID: \"8207a8b4-aab5-43e9-8967-c4ada60ed838\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e696f37-2dzls"
Nov 03 20:18:32.894241 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:18:32.894058    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8207a8b4-aab5-43e9-8967-c4ada60ed838-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e696f37-2dzls\" (UID: \"8207a8b4-aab5-43e9-8967-c4ada60ed838\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e696f37-2dzls"
Nov 03 20:18:32.894241 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:18:32.894196    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkc9b\" (UniqueName: \"kubernetes.io/projected/8207a8b4-aab5-43e9-8967-c4ada60ed838-kube-api-access-qkc9b\") pod \"maas-api-revoke-sub-9e696f37-2dzls\" (UID: \"8207a8b4-aab5-43e9-8967-c4ada60ed838\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e696f37-2dzls"
Nov 03 20:18:32.921091 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:18:32.921057    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 20:18:32.994758 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:18:32.994727    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8207a8b4-aab5-43e9-8967-c4ada60ed838-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e696f37-2dzls\" (UID: \"8207a8b4-aab5-43e9-8967-c4ada60ed838\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e696f37-2dzls"
Nov 03 20:18:32.994974 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:18:32.994846    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkc9b\" (UniqueName: \"kubernetes.io/projected/8207a8b4-aab5-43e9-8967-c4ada60ed838-kube-api-access-qkc9b\") pod \"maas-api-revoke-sub-9e696f37-2dzls\" (UID: \"8207a8b4-aab5-43e9-8967-c4ada60ed838\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e696f37-2dzls"
Nov 03 20:18:32.994974 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:18:32.994960    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8207a8b4-aab5-43e9-8967-c4ada60ed838-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e696f37-2dzls\" (UID: \"8207a8b4-aab5-43e9-8967-c4ada60ed838\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e696f37-2dzls"
Nov 03 20:18:32.995587 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:18:32.995566    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8207a8b4-aab5-43e9-8967-c4ada60ed838-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e696f37-2dzls\" (UID: \"8207a8b4-aab5-43e9-8967-c4ada60ed838\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e696f37-2dzls"
Nov 03 20:18:32.997189 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:18:32.997161    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8207a8b4-aab5-43e9-8967-c4ada60ed838-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e696f37-2dzls\" (UID: \"8207a8b4-aab5-43e9-8967-c4ada60ed838\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e696f37-2dzls"
Nov 03 20:18:33.002810 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:18:33.002781    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkc9b\" (UniqueName: \"kubernetes.io/projected/8207a8b4-aab5-43e9-8967-c4ada60ed838-kube-api-access-qkc9b\") pod \"maas-api-revoke-sub-9e696f37-2dzls\" (UID: \"8207a8b4-aab5-43e9-8967-c4ada60ed838\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e696f37-2dzls"
Nov 03 20:18:33.163328 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:18:33.163245    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e696f37-2dzls"
Nov 03 20:18:33.445526 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:18:33.445498    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e696f37-2dzls"]
Nov 03 20:18:33.446939 ip-10-0-134-222 kubenswrapper[4921]: W1103 20:18:33.446907    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8207a8b4_aab5_43e9_8967_c4ada60ed838.slice/crio-6fed1c0afb62a59e2f588588826f6e32e9c32f468deee3baa7a35f956d7dda63 WatchSource:0}: Error finding container 6fed1c0afb62a59e2f588588826f6e32e9c32f468deee3baa7a35f956d7dda63: Status 404 returned error can't find the container with id 6fed1c0afb62a59e2f588588826f6e32e9c32f468deee3baa7a35f956d7dda63
Nov 03 20:18:34.435582 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:18:34.435550    4921 generic.go:356] "Generic (PLEG): container finished" podID="8207a8b4-aab5-43e9-8967-c4ada60ed838" containerID="18af5b245b2b704afc78e08fed36c23fb4305e83f858f129dba71f63c79f9d0f" exitCode=0
Nov 03 20:18:34.436057 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:18:34.435594    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e696f37-2dzls" event={"ID":"8207a8b4-aab5-43e9-8967-c4ada60ed838","Type":"ContainerDied","Data":"18af5b245b2b704afc78e08fed36c23fb4305e83f858f129dba71f63c79f9d0f"}
Nov 03 20:18:34.436057 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:18:34.435635    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e696f37-2dzls" event={"ID":"8207a8b4-aab5-43e9-8967-c4ada60ed838","Type":"ContainerStarted","Data":"6fed1c0afb62a59e2f588588826f6e32e9c32f468deee3baa7a35f956d7dda63"}
Nov 03 20:18:35.569353 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:18:35.569327    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e696f37-2dzls"
Nov 03 20:18:35.629437 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:18:35.629408    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8207a8b4-aab5-43e9-8967-c4ada60ed838-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8207a8b4-aab5-43e9-8967-c4ada60ed838-maas-api-openshift-ca\") pod \"8207a8b4-aab5-43e9-8967-c4ada60ed838\" (UID: \"8207a8b4-aab5-43e9-8967-c4ada60ed838\") "
Nov 03 20:18:35.629633 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:18:35.629458    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8207a8b4-aab5-43e9-8967-c4ada60ed838-kube-api-access-qkc9b\" (UniqueName: \"kubernetes.io/projected/8207a8b4-aab5-43e9-8967-c4ada60ed838-kube-api-access-qkc9b\") pod \"8207a8b4-aab5-43e9-8967-c4ada60ed838\" (UID: \"8207a8b4-aab5-43e9-8967-c4ada60ed838\") "
Nov 03 20:18:35.629633 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:18:35.629565    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8207a8b4-aab5-43e9-8967-c4ada60ed838-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8207a8b4-aab5-43e9-8967-c4ada60ed838-maas-api-xks-ca\") pod \"8207a8b4-aab5-43e9-8967-c4ada60ed838\" (UID: \"8207a8b4-aab5-43e9-8967-c4ada60ed838\") "
Nov 03 20:18:35.629900 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:18:35.629874    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8207a8b4-aab5-43e9-8967-c4ada60ed838-maas-api-openshift-ca" pod "8207a8b4-aab5-43e9-8967-c4ada60ed838" (UID: "8207a8b4-aab5-43e9-8967-c4ada60ed838"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:18:35.630092 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:18:35.629963    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8207a8b4-aab5-43e9-8967-c4ada60ed838-maas-api-openshift-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:35.631829 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:18:35.631803    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8207a8b4-aab5-43e9-8967-c4ada60ed838-maas-api-xks-ca" pod "8207a8b4-aab5-43e9-8967-c4ada60ed838" (UID: "8207a8b4-aab5-43e9-8967-c4ada60ed838"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:18:35.632006 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:18:35.631977    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8207a8b4-aab5-43e9-8967-c4ada60ed838-kube-api-access-qkc9b" pod "8207a8b4-aab5-43e9-8967-c4ada60ed838" (UID: "8207a8b4-aab5-43e9-8967-c4ada60ed838"). InnerVolumeSpecName "kube-api-access-qkc9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:18:35.730723 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:18:35.730698    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkc9b\" (UniqueName: \"kubernetes.io/projected/8207a8b4-aab5-43e9-8967-c4ada60ed838-kube-api-access-qkc9b\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:35.730723 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:18:35.730724    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8207a8b4-aab5-43e9-8967-c4ada60ed838-maas-api-xks-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:36.453587 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:18:36.453550    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e696f37-2dzls" event={"ID":"8207a8b4-aab5-43e9-8967-c4ada60ed838","Type":"ContainerDied","Data":"6fed1c0afb62a59e2f588588826f6e32e9c32f468deee3baa7a35f956d7dda63"}
Nov 03 20:18:36.453587 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:18:36.453580    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e696f37-2dzls"
Nov 03 20:18:36.453837 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:18:36.453634    4921 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6fed1c0afb62a59e2f588588826f6e32e9c32f468deee3baa7a35f956d7dda63"
Nov 03 20:18:43.706461 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:18:43.706423    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-794e97-klssb"]
Nov 03 20:18:43.706881 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:18:43.706868    4921 cpu_manager.go:414] "containerMap: removing container" podUID="159f95e2-29a1-4f95-8fbb-eff99dba6b80" containerName="cleanup"
Nov 03 20:18:43.706925 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:18:43.706881    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="159f95e2-29a1-4f95-8fbb-eff99dba6b80" containerName="cleanup"
Nov 03 20:18:43.706925 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:18:43.706916    4921 cpu_manager.go:414] "containerMap: removing container" podUID="8207a8b4-aab5-43e9-8967-c4ada60ed838" containerName="revoke-keys"
Nov 03 20:18:43.706925 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:18:43.706922    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8207a8b4-aab5-43e9-8967-c4ada60ed838" containerName="revoke-keys"
Nov 03 20:18:43.707027 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:18:43.706999    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="8207a8b4-aab5-43e9-8967-c4ada60ed838" containerName="revoke-keys"
Nov 03 20:18:43.710389 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:18:43.710370    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-794e97-klssb"
Nov 03 20:18:43.714935 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:18:43.714910    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:18:43.715244 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:18:43.715227    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 20:18:43.719145 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:18:43.719123    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-794e97-klssb"]
Nov 03 20:18:43.809141 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:18:43.809097    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6dvj\" (UniqueName: \"kubernetes.io/projected/871bfcb6-5fcf-43e8-b531-da586421f1c2-kube-api-access-r6dvj\") pod \"maas-api-revoke-keys-e2e-worker-single-main-794e97-klssb\" (UID: \"871bfcb6-5fcf-43e8-b531-da586421f1c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-794e97-klssb"
Nov 03 20:18:43.809141 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:18:43.809160    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/871bfcb6-5fcf-43e8-b531-da586421f1c2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-794e97-klssb\" (UID: \"871bfcb6-5fcf-43e8-b531-da586421f1c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-794e97-klssb"
Nov 03 20:18:43.809365 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:18:43.809237    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/871bfcb6-5fcf-43e8-b531-da586421f1c2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-794e97-klssb\" (UID: \"871bfcb6-5fcf-43e8-b531-da586421f1c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-794e97-klssb"
Nov 03 20:18:43.910794 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:18:43.910751    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6dvj\" (UniqueName: \"kubernetes.io/projected/871bfcb6-5fcf-43e8-b531-da586421f1c2-kube-api-access-r6dvj\") pod \"maas-api-revoke-keys-e2e-worker-single-main-794e97-klssb\" (UID: \"871bfcb6-5fcf-43e8-b531-da586421f1c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-794e97-klssb"
Nov 03 20:18:43.910975 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:18:43.910821    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/871bfcb6-5fcf-43e8-b531-da586421f1c2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-794e97-klssb\" (UID: \"871bfcb6-5fcf-43e8-b531-da586421f1c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-794e97-klssb"
Nov 03 20:18:43.910975 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:18:43.910853    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/871bfcb6-5fcf-43e8-b531-da586421f1c2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-794e97-klssb\" (UID: \"871bfcb6-5fcf-43e8-b531-da586421f1c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-794e97-klssb"
Nov 03 20:18:43.911811 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:18:43.911773    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/871bfcb6-5fcf-43e8-b531-da586421f1c2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-794e97-klssb\" (UID: \"871bfcb6-5fcf-43e8-b531-da586421f1c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-794e97-klssb"
Nov 03 20:18:43.913732 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:18:43.913705    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/871bfcb6-5fcf-43e8-b531-da586421f1c2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-794e97-klssb\" (UID: \"871bfcb6-5fcf-43e8-b531-da586421f1c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-794e97-klssb"
Nov 03 20:18:43.917993 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:18:43.917968    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6dvj\" (UniqueName: \"kubernetes.io/projected/871bfcb6-5fcf-43e8-b531-da586421f1c2-kube-api-access-r6dvj\") pod \"maas-api-revoke-keys-e2e-worker-single-main-794e97-klssb\" (UID: \"871bfcb6-5fcf-43e8-b531-da586421f1c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-794e97-klssb"
Nov 03 20:18:44.024591 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:18:44.024560    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-794e97-klssb"
Nov 03 20:18:44.301019 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:18:44.300989    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-794e97-klssb"]
Nov 03 20:18:44.306995 ip-10-0-134-222 kubenswrapper[4921]: W1103 20:18:44.303776    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod871bfcb6_5fcf_43e8_b531_da586421f1c2.slice/crio-11c0e7354ba0fe6b888122bebb64ce1e4105e3a32563e8933d15d81d4c26fdc5 WatchSource:0}: Error finding container 11c0e7354ba0fe6b888122bebb64ce1e4105e3a32563e8933d15d81d4c26fdc5: Status 404 returned error can't find the container with id 11c0e7354ba0fe6b888122bebb64ce1e4105e3a32563e8933d15d81d4c26fdc5
Nov 03 20:18:44.503523 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:18:44.503490    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-794e97-klssb" event={"ID":"871bfcb6-5fcf-43e8-b531-da586421f1c2","Type":"ContainerStarted","Data":"8bddceb14c3234f42e5984a10e50ecc6c103a278533cfd4ff278b7b590acac6d"}
Nov 03 20:18:44.503637 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:18:44.503545    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-794e97-klssb" event={"ID":"871bfcb6-5fcf-43e8-b531-da586421f1c2","Type":"ContainerStarted","Data":"11c0e7354ba0fe6b888122bebb64ce1e4105e3a32563e8933d15d81d4c26fdc5"}
Nov 03 20:18:44.519244 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:18:44.519198    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-794e97-klssb" podStartSLOduration=1.5191858059999999 podStartE2EDuration="1.519185806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:44.516295882 +0000 UTC m=+2837.760261347" watchObservedRunningTime="2026-11-03 20:18:44.519185806 +0000 UTC m=+2837.763151265"
Nov 03 20:18:45.510870 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:18:45.510836    4921 generic.go:356] "Generic (PLEG): container finished" podID="871bfcb6-5fcf-43e8-b531-da586421f1c2" containerID="8bddceb14c3234f42e5984a10e50ecc6c103a278533cfd4ff278b7b590acac6d" exitCode=0
Nov 03 20:18:45.511273 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:18:45.510945    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-794e97-klssb" event={"ID":"871bfcb6-5fcf-43e8-b531-da586421f1c2","Type":"ContainerDied","Data":"8bddceb14c3234f42e5984a10e50ecc6c103a278533cfd4ff278b7b590acac6d"}
Nov 03 20:18:46.650308 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:18:46.650280    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-794e97-klssb"
Nov 03 20:18:46.748965 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:18:46.748935    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/871bfcb6-5fcf-43e8-b531-da586421f1c2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/871bfcb6-5fcf-43e8-b531-da586421f1c2-maas-api-xks-ca\") pod \"871bfcb6-5fcf-43e8-b531-da586421f1c2\" (UID: \"871bfcb6-5fcf-43e8-b531-da586421f1c2\") "
Nov 03 20:18:46.749162 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:18:46.748983    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/871bfcb6-5fcf-43e8-b531-da586421f1c2-kube-api-access-r6dvj\" (UniqueName: \"kubernetes.io/projected/871bfcb6-5fcf-43e8-b531-da586421f1c2-kube-api-access-r6dvj\") pod \"871bfcb6-5fcf-43e8-b531-da586421f1c2\" (UID: \"871bfcb6-5fcf-43e8-b531-da586421f1c2\") "
Nov 03 20:18:46.749162 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:18:46.749037    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/871bfcb6-5fcf-43e8-b531-da586421f1c2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/871bfcb6-5fcf-43e8-b531-da586421f1c2-maas-api-openshift-ca\") pod \"871bfcb6-5fcf-43e8-b531-da586421f1c2\" (UID: \"871bfcb6-5fcf-43e8-b531-da586421f1c2\") "
Nov 03 20:18:46.749472 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:18:46.749448    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/871bfcb6-5fcf-43e8-b531-da586421f1c2-maas-api-openshift-ca" pod "871bfcb6-5fcf-43e8-b531-da586421f1c2" (UID: "871bfcb6-5fcf-43e8-b531-da586421f1c2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:18:46.751280 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:18:46.751254    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/871bfcb6-5fcf-43e8-b531-da586421f1c2-kube-api-access-r6dvj" pod "871bfcb6-5fcf-43e8-b531-da586421f1c2" (UID: "871bfcb6-5fcf-43e8-b531-da586421f1c2"). InnerVolumeSpecName "kube-api-access-r6dvj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:18:46.751408 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:18:46.751386    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/871bfcb6-5fcf-43e8-b531-da586421f1c2-maas-api-xks-ca" pod "871bfcb6-5fcf-43e8-b531-da586421f1c2" (UID: "871bfcb6-5fcf-43e8-b531-da586421f1c2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:18:46.850106 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:18:46.850034    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/871bfcb6-5fcf-43e8-b531-da586421f1c2-maas-api-xks-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:46.850106 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:18:46.850061    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6dvj\" (UniqueName: \"kubernetes.io/projected/871bfcb6-5fcf-43e8-b531-da586421f1c2-kube-api-access-r6dvj\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:46.850106 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:18:46.850071    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/871bfcb6-5fcf-43e8-b531-da586421f1c2-maas-api-openshift-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:47.522840 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:18:47.522808    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-794e97-klssb"
Nov 03 20:18:47.523021 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:18:47.522808    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-794e97-klssb" event={"ID":"871bfcb6-5fcf-43e8-b531-da586421f1c2","Type":"ContainerDied","Data":"11c0e7354ba0fe6b888122bebb64ce1e4105e3a32563e8933d15d81d4c26fdc5"}
Nov 03 20:18:47.523021 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:18:47.522934    4921 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11c0e7354ba0fe6b888122bebb64ce1e4105e3a32563e8933d15d81d4c26fdc5"
Nov 03 20:18:55.034078 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:18:55.034037    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-794e97-klssb"]
Nov 03 20:18:55.036741 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:18:55.036696    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-794e97-klssb"]
Nov 03 20:18:56.888628 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:18:56.888593    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="871bfcb6-5fcf-43e8-b531-da586421f1c2" path="/var/lib/kubelet/pods/871bfcb6-5fcf-43e8-b531-da586421f1c2/volumes"
Nov 03 20:19:02.921626 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:19:02.921587    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 20:19:32.920003 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:19:32.919969    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 20:20:02.920341 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:20:02.920304    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 20:20:32.920766 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:20:32.920729    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 20:20:50.665300 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:20:50.665263    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-57cdc37c-jjs7v"]
Nov 03 20:20:50.665837 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:20:50.665770    4921 cpu_manager.go:414] "containerMap: removing container" podUID="871bfcb6-5fcf-43e8-b531-da586421f1c2" containerName="revoke-keys"
Nov 03 20:20:50.665837 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:20:50.665785    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="871bfcb6-5fcf-43e8-b531-da586421f1c2" containerName="revoke-keys"
Nov 03 20:20:50.665917 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:20:50.665864    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="871bfcb6-5fcf-43e8-b531-da586421f1c2" containerName="revoke-keys"
Nov 03 20:20:50.669141 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:20:50.669115    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-57cdc37c-jjs7v"
Nov 03 20:20:50.674420 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:20:50.674391    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z92ps\""
Nov 03 20:20:50.674580 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:20:50.674430    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:20:50.677840 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:20:50.677818    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-57cdc37c-jjs7v"]
Nov 03 20:20:50.796866 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:20:50.796832    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5bebcb9-2bad-401f-800d-dcd7e5ce658e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-57cdc37c-jjs7v\" (UID: \"b5bebcb9-2bad-401f-800d-dcd7e5ce658e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-57cdc37c-jjs7v"
Nov 03 20:20:50.797050 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:20:50.796897    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfxpv\" (UniqueName: \"kubernetes.io/projected/b5bebcb9-2bad-401f-800d-dcd7e5ce658e-kube-api-access-bfxpv\") pod \"maas-api-revoke-keys-e2e-ait-57cdc37c-jjs7v\" (UID: \"b5bebcb9-2bad-401f-800d-dcd7e5ce658e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-57cdc37c-jjs7v"
Nov 03 20:20:50.797050 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:20:50.796982    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5bebcb9-2bad-401f-800d-dcd7e5ce658e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-57cdc37c-jjs7v\" (UID: \"b5bebcb9-2bad-401f-800d-dcd7e5ce658e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-57cdc37c-jjs7v"
Nov 03 20:20:50.898440 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:20:50.898407    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfxpv\" (UniqueName: \"kubernetes.io/projected/b5bebcb9-2bad-401f-800d-dcd7e5ce658e-kube-api-access-bfxpv\") pod \"maas-api-revoke-keys-e2e-ait-57cdc37c-jjs7v\" (UID: \"b5bebcb9-2bad-401f-800d-dcd7e5ce658e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-57cdc37c-jjs7v"
Nov 03 20:20:50.898440 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:20:50.898443    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5bebcb9-2bad-401f-800d-dcd7e5ce658e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-57cdc37c-jjs7v\" (UID: \"b5bebcb9-2bad-401f-800d-dcd7e5ce658e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-57cdc37c-jjs7v"
Nov 03 20:20:50.898744 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:20:50.898530    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5bebcb9-2bad-401f-800d-dcd7e5ce658e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-57cdc37c-jjs7v\" (UID: \"b5bebcb9-2bad-401f-800d-dcd7e5ce658e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-57cdc37c-jjs7v"
Nov 03 20:20:50.899193 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:20:50.899170    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5bebcb9-2bad-401f-800d-dcd7e5ce658e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-57cdc37c-jjs7v\" (UID: \"b5bebcb9-2bad-401f-800d-dcd7e5ce658e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-57cdc37c-jjs7v"
Nov 03 20:20:50.900964 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:20:50.900943    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5bebcb9-2bad-401f-800d-dcd7e5ce658e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-57cdc37c-jjs7v\" (UID: \"b5bebcb9-2bad-401f-800d-dcd7e5ce658e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-57cdc37c-jjs7v"
Nov 03 20:20:50.902504 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:20:50.902484    4921 projected.go:196] Error preparing data for projected volume kube-api-access-bfxpv for pod odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-57cdc37c-jjs7v: failed to fetch token: serviceaccounts "maas-api-cleanup" not found
Nov 03 20:20:50.902600 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:20:50.902559    4921 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5bebcb9-2bad-401f-800d-dcd7e5ce658e-kube-api-access-bfxpv podName:b5bebcb9-2bad-401f-800d-dcd7e5ce658e nodeName:}" failed. No retries permitted until 2026-11-03 20:20:51.402535756 +0000 UTC m=+2964.646501198 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-bfxpv" (UniqueName: "kubernetes.io/projected/b5bebcb9-2bad-401f-800d-dcd7e5ce658e-kube-api-access-bfxpv") pod "maas-api-revoke-keys-e2e-ait-57cdc37c-jjs7v" (UID: "b5bebcb9-2bad-401f-800d-dcd7e5ce658e") : failed to fetch token: serviceaccounts "maas-api-cleanup" not found
Nov 03 20:20:51.403381 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:20:51.403344    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfxpv\" (UniqueName: \"kubernetes.io/projected/b5bebcb9-2bad-401f-800d-dcd7e5ce658e-kube-api-access-bfxpv\") pod \"maas-api-revoke-keys-e2e-ait-57cdc37c-jjs7v\" (UID: \"b5bebcb9-2bad-401f-800d-dcd7e5ce658e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-57cdc37c-jjs7v"
Nov 03 20:20:51.413502 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:20:51.413471    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfxpv\" (UniqueName: \"kubernetes.io/projected/b5bebcb9-2bad-401f-800d-dcd7e5ce658e-kube-api-access-bfxpv\") pod \"maas-api-revoke-keys-e2e-ait-57cdc37c-jjs7v\" (UID: \"b5bebcb9-2bad-401f-800d-dcd7e5ce658e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-57cdc37c-jjs7v"
Nov 03 20:20:51.588056 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:20:51.588013    4921 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-57cdc37c-jjs7v" secret="" err="secret \"maas-api-cleanup-dockercfg-z92ps\" not found"
Nov 03 20:20:51.588242 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:20:51.588075    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-57cdc37c-jjs7v"
Nov 03 20:20:51.860662 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:20:51.860584    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-57cdc37c-jjs7v"]
Nov 03 20:20:51.863243 ip-10-0-134-222 kubenswrapper[4921]: W1103 20:20:51.863219    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5bebcb9_2bad_401f_800d_dcd7e5ce658e.slice/crio-02920839e5b6b6852ed1bd9634dfe6d00cddbca1d4e8780c3fa0495830aa403b WatchSource:0}: Error finding container 02920839e5b6b6852ed1bd9634dfe6d00cddbca1d4e8780c3fa0495830aa403b: Status 404 returned error can't find the container with id 02920839e5b6b6852ed1bd9634dfe6d00cddbca1d4e8780c3fa0495830aa403b
Nov 03 20:20:52.393060 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:20:52.393024    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-57cdc37c-jjs7v" event={"ID":"b5bebcb9-2bad-401f-800d-dcd7e5ce658e","Type":"ContainerStarted","Data":"2e3d2930789e8999f67094abfc49b1969522e4e21940909b875e89f4ed91e4b1"}
Nov 03 20:20:52.393286 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:20:52.393086    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-57cdc37c-jjs7v" event={"ID":"b5bebcb9-2bad-401f-800d-dcd7e5ce658e","Type":"ContainerStarted","Data":"02920839e5b6b6852ed1bd9634dfe6d00cddbca1d4e8780c3fa0495830aa403b"}
Nov 03 20:20:52.393358 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:20:52.393340    4921 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-57cdc37c-jjs7v" secret="" err="secret \"maas-api-cleanup-dockercfg-z92ps\" not found"
Nov 03 20:20:52.408332 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:20:52.408243    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-57cdc37c-jjs7v" podStartSLOduration=2.4082297759999998 podStartE2EDuration="2.408229776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:52.405618485 +0000 UTC m=+2965.649583949" watchObservedRunningTime="2026-11-03 20:20:52.408229776 +0000 UTC m=+2965.652195240"
Nov 03 20:20:53.400205 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:20:53.400174    4921 generic.go:356] "Generic (PLEG): container finished" podID="b5bebcb9-2bad-401f-800d-dcd7e5ce658e" containerID="2e3d2930789e8999f67094abfc49b1969522e4e21940909b875e89f4ed91e4b1" exitCode=0
Nov 03 20:20:53.400614 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:20:53.400263    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-57cdc37c-jjs7v" event={"ID":"b5bebcb9-2bad-401f-800d-dcd7e5ce658e","Type":"ContainerDied","Data":"2e3d2930789e8999f67094abfc49b1969522e4e21940909b875e89f4ed91e4b1"}
Nov 03 20:20:54.547647 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:20:54.547621    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-57cdc37c-jjs7v"
Nov 03 20:20:54.637439 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:20:54.637402    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5bebcb9-2bad-401f-800d-dcd7e5ce658e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5bebcb9-2bad-401f-800d-dcd7e5ce658e-maas-api-xks-ca\") pod \"b5bebcb9-2bad-401f-800d-dcd7e5ce658e\" (UID: \"b5bebcb9-2bad-401f-800d-dcd7e5ce658e\") "
Nov 03 20:20:54.637624 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:20:54.637449    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5bebcb9-2bad-401f-800d-dcd7e5ce658e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5bebcb9-2bad-401f-800d-dcd7e5ce658e-maas-api-openshift-ca\") pod \"b5bebcb9-2bad-401f-800d-dcd7e5ce658e\" (UID: \"b5bebcb9-2bad-401f-800d-dcd7e5ce658e\") "
Nov 03 20:20:54.637624 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:20:54.637485    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5bebcb9-2bad-401f-800d-dcd7e5ce658e-kube-api-access-bfxpv\" (UniqueName: \"kubernetes.io/projected/b5bebcb9-2bad-401f-800d-dcd7e5ce658e-kube-api-access-bfxpv\") pod \"b5bebcb9-2bad-401f-800d-dcd7e5ce658e\" (UID: \"b5bebcb9-2bad-401f-800d-dcd7e5ce658e\") "
Nov 03 20:20:54.637936 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:20:54.637908    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5bebcb9-2bad-401f-800d-dcd7e5ce658e-maas-api-openshift-ca" pod "b5bebcb9-2bad-401f-800d-dcd7e5ce658e" (UID: "b5bebcb9-2bad-401f-800d-dcd7e5ce658e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:20:54.639774 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:20:54.639747    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5bebcb9-2bad-401f-800d-dcd7e5ce658e-kube-api-access-bfxpv" pod "b5bebcb9-2bad-401f-800d-dcd7e5ce658e" (UID: "b5bebcb9-2bad-401f-800d-dcd7e5ce658e"). InnerVolumeSpecName "kube-api-access-bfxpv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:20:54.639855 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:20:54.639791    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5bebcb9-2bad-401f-800d-dcd7e5ce658e-maas-api-xks-ca" pod "b5bebcb9-2bad-401f-800d-dcd7e5ce658e" (UID: "b5bebcb9-2bad-401f-800d-dcd7e5ce658e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:20:54.738385 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:20:54.738344    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5bebcb9-2bad-401f-800d-dcd7e5ce658e-maas-api-xks-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:54.738385 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:20:54.738374    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5bebcb9-2bad-401f-800d-dcd7e5ce658e-maas-api-openshift-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:54.738609 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:20:54.738390    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfxpv\" (UniqueName: \"kubernetes.io/projected/b5bebcb9-2bad-401f-800d-dcd7e5ce658e-kube-api-access-bfxpv\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:55.412623 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:20:55.412588    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-57cdc37c-jjs7v"
Nov 03 20:20:55.412925 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:20:55.412629    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-57cdc37c-jjs7v" event={"ID":"b5bebcb9-2bad-401f-800d-dcd7e5ce658e","Type":"ContainerDied","Data":"02920839e5b6b6852ed1bd9634dfe6d00cddbca1d4e8780c3fa0495830aa403b"}
Nov 03 20:20:55.412925 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:20:55.412725    4921 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02920839e5b6b6852ed1bd9634dfe6d00cddbca1d4e8780c3fa0495830aa403b"
Nov 03 20:21:01.756122 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:21:01.756084    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-57cdc37c-jjs7v"]
Nov 03 20:21:01.758412 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:21:01.758385    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-57cdc37c-jjs7v"]
Nov 03 20:21:02.886951 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:21:02.886916    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5bebcb9-2bad-401f-800d-dcd7e5ce658e" path="/var/lib/kubelet/pods/b5bebcb9-2bad-401f-800d-dcd7e5ce658e/volumes"
Nov 03 20:21:02.920145 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:21:02.920109    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 20:21:32.010860 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:21:32.010828    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gzf8k_6f215b42-0f09-46a0-bc98-392c0fa909c9/console-operator/2.log"
Nov 03 20:21:32.040899 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:21:32.040871    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gzf8k_6f215b42-0f09-46a0-bc98-392c0fa909c9/console-operator/2.log"
Nov 03 20:21:32.920711 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:21:32.920650    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 20:22:02.922563 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:22:02.922521    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 20:22:32.921215 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:22:32.921145    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 20:22:49.474151 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:22:49.474120    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-17f6e3-9d76l"]
Nov 03 20:22:49.474649 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:22:49.474537    4921 cpu_manager.go:414] "containerMap: removing container" podUID="b5bebcb9-2bad-401f-800d-dcd7e5ce658e" containerName="revoke-keys"
Nov 03 20:22:49.474649 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:22:49.474547    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5bebcb9-2bad-401f-800d-dcd7e5ce658e" containerName="revoke-keys"
Nov 03 20:22:49.474764 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:22:49.474681    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5bebcb9-2bad-401f-800d-dcd7e5ce658e" containerName="revoke-keys"
Nov 03 20:22:49.477832 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:22:49.477816    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-17f6e3-9d76l"
Nov 03 20:22:49.483498 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:22:49.483476    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:22:49.483836 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:22:49.483813    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w2ggk\""
Nov 03 20:22:49.486848 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:22:49.486817    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-17f6e3-9d76l"]
Nov 03 20:22:49.594763 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:22:49.594724    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3a4a0cd-f846-409b-b7af-8ee554155bc1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-17f6e3-9d76l\" (UID: \"a3a4a0cd-f846-409b-b7af-8ee554155bc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-17f6e3-9d76l"
Nov 03 20:22:49.594970 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:22:49.594776    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3a4a0cd-f846-409b-b7af-8ee554155bc1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-17f6e3-9d76l\" (UID: \"a3a4a0cd-f846-409b-b7af-8ee554155bc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-17f6e3-9d76l"
Nov 03 20:22:49.594970 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:22:49.594803    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scgvb\" (UniqueName: \"kubernetes.io/projected/a3a4a0cd-f846-409b-b7af-8ee554155bc1-kube-api-access-scgvb\") pod \"maas-api-revoke-keys-e2e-authcfg-http-17f6e3-9d76l\" (UID: \"a3a4a0cd-f846-409b-b7af-8ee554155bc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-17f6e3-9d76l"
Nov 03 20:22:49.695535 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:22:49.695496    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3a4a0cd-f846-409b-b7af-8ee554155bc1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-17f6e3-9d76l\" (UID: \"a3a4a0cd-f846-409b-b7af-8ee554155bc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-17f6e3-9d76l"
Nov 03 20:22:49.695721 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:22:49.695550    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3a4a0cd-f846-409b-b7af-8ee554155bc1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-17f6e3-9d76l\" (UID: \"a3a4a0cd-f846-409b-b7af-8ee554155bc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-17f6e3-9d76l"
Nov 03 20:22:49.695721 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:22:49.695588    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scgvb\" (UniqueName: \"kubernetes.io/projected/a3a4a0cd-f846-409b-b7af-8ee554155bc1-kube-api-access-scgvb\") pod \"maas-api-revoke-keys-e2e-authcfg-http-17f6e3-9d76l\" (UID: \"a3a4a0cd-f846-409b-b7af-8ee554155bc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-17f6e3-9d76l"
Nov 03 20:22:49.696272 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:22:49.696249    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3a4a0cd-f846-409b-b7af-8ee554155bc1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-17f6e3-9d76l\" (UID: \"a3a4a0cd-f846-409b-b7af-8ee554155bc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-17f6e3-9d76l"
Nov 03 20:22:49.697962 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:22:49.697938    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3a4a0cd-f846-409b-b7af-8ee554155bc1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-17f6e3-9d76l\" (UID: \"a3a4a0cd-f846-409b-b7af-8ee554155bc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-17f6e3-9d76l"
Nov 03 20:22:49.703456 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:22:49.703430    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scgvb\" (UniqueName: \"kubernetes.io/projected/a3a4a0cd-f846-409b-b7af-8ee554155bc1-kube-api-access-scgvb\") pod \"maas-api-revoke-keys-e2e-authcfg-http-17f6e3-9d76l\" (UID: \"a3a4a0cd-f846-409b-b7af-8ee554155bc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-17f6e3-9d76l"
Nov 03 20:22:49.792147 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:22:49.792116    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-17f6e3-9d76l"
Nov 03 20:22:50.063498 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:22:50.063471    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-17f6e3-9d76l"]
Nov 03 20:22:50.065599 ip-10-0-134-222 kubenswrapper[4921]: W1103 20:22:50.065567    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3a4a0cd_f846_409b_b7af_8ee554155bc1.slice/crio-e647be80c8243a29250d85a5930eadc5b1f086d7c5748e20b4def96c8e811b68 WatchSource:0}: Error finding container e647be80c8243a29250d85a5930eadc5b1f086d7c5748e20b4def96c8e811b68: Status 404 returned error can't find the container with id e647be80c8243a29250d85a5930eadc5b1f086d7c5748e20b4def96c8e811b68
Nov 03 20:22:50.118631 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:22:50.118593    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-17f6e3-9d76l" event={"ID":"a3a4a0cd-f846-409b-b7af-8ee554155bc1","Type":"ContainerStarted","Data":"e647be80c8243a29250d85a5930eadc5b1f086d7c5748e20b4def96c8e811b68"}
Nov 03 20:22:51.125793 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:22:51.125759    4921 generic.go:356] "Generic (PLEG): container finished" podID="a3a4a0cd-f846-409b-b7af-8ee554155bc1" containerID="848a3824615e2ebc1d1afc20c8b1b96bbbecaca1f1e9aa28dcaf659566d0e01b" exitCode=0
Nov 03 20:22:51.126246 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:22:51.125841    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-17f6e3-9d76l" event={"ID":"a3a4a0cd-f846-409b-b7af-8ee554155bc1","Type":"ContainerDied","Data":"848a3824615e2ebc1d1afc20c8b1b96bbbecaca1f1e9aa28dcaf659566d0e01b"}
Nov 03 20:22:52.275444 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:22:52.275415    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-17f6e3-9d76l"
Nov 03 20:22:52.330050 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:22:52.330017    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3a4a0cd-f846-409b-b7af-8ee554155bc1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3a4a0cd-f846-409b-b7af-8ee554155bc1-maas-api-xks-ca\") pod \"a3a4a0cd-f846-409b-b7af-8ee554155bc1\" (UID: \"a3a4a0cd-f846-409b-b7af-8ee554155bc1\") "
Nov 03 20:22:52.330240 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:22:52.330102    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3a4a0cd-f846-409b-b7af-8ee554155bc1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3a4a0cd-f846-409b-b7af-8ee554155bc1-maas-api-openshift-ca\") pod \"a3a4a0cd-f846-409b-b7af-8ee554155bc1\" (UID: \"a3a4a0cd-f846-409b-b7af-8ee554155bc1\") "
Nov 03 20:22:52.330240 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:22:52.330171    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3a4a0cd-f846-409b-b7af-8ee554155bc1-kube-api-access-scgvb\" (UniqueName: \"kubernetes.io/projected/a3a4a0cd-f846-409b-b7af-8ee554155bc1-kube-api-access-scgvb\") pod \"a3a4a0cd-f846-409b-b7af-8ee554155bc1\" (UID: \"a3a4a0cd-f846-409b-b7af-8ee554155bc1\") "
Nov 03 20:22:52.330499 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:22:52.330476    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3a4a0cd-f846-409b-b7af-8ee554155bc1-maas-api-openshift-ca" pod "a3a4a0cd-f846-409b-b7af-8ee554155bc1" (UID: "a3a4a0cd-f846-409b-b7af-8ee554155bc1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:22:52.332179 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:22:52.332143    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3a4a0cd-f846-409b-b7af-8ee554155bc1-maas-api-xks-ca" pod "a3a4a0cd-f846-409b-b7af-8ee554155bc1" (UID: "a3a4a0cd-f846-409b-b7af-8ee554155bc1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:22:52.332310 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:22:52.332239    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3a4a0cd-f846-409b-b7af-8ee554155bc1-kube-api-access-scgvb" pod "a3a4a0cd-f846-409b-b7af-8ee554155bc1" (UID: "a3a4a0cd-f846-409b-b7af-8ee554155bc1"). InnerVolumeSpecName "kube-api-access-scgvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:22:52.430883 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:22:52.430799    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scgvb\" (UniqueName: \"kubernetes.io/projected/a3a4a0cd-f846-409b-b7af-8ee554155bc1-kube-api-access-scgvb\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:52.430883 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:22:52.430829    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3a4a0cd-f846-409b-b7af-8ee554155bc1-maas-api-xks-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:52.430883 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:22:52.430839    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3a4a0cd-f846-409b-b7af-8ee554155bc1-maas-api-openshift-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:53.142631 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:22:53.142601    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-17f6e3-9d76l"
Nov 03 20:22:53.142826 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:22:53.142643    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-17f6e3-9d76l" event={"ID":"a3a4a0cd-f846-409b-b7af-8ee554155bc1","Type":"ContainerDied","Data":"e647be80c8243a29250d85a5930eadc5b1f086d7c5748e20b4def96c8e811b68"}
Nov 03 20:22:53.142826 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:22:53.142703    4921 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e647be80c8243a29250d85a5930eadc5b1f086d7c5748e20b4def96c8e811b68"
Nov 03 20:23:02.923712 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:23:02.923579    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 20:23:06.119065 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:06.118954    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-17f6e3-9d76l"]
Nov 03 20:23:06.122365 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:06.122338    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-17f6e3-9d76l"]
Nov 03 20:23:06.888335 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:06.888299    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3a4a0cd-f846-409b-b7af-8ee554155bc1" path="/var/lib/kubelet/pods/a3a4a0cd-f846-409b-b7af-8ee554155bc1/volumes"
Nov 03 20:23:24.072545 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:24.072500    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-9kwv8"]
Nov 03 20:23:24.073245 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:24.073117    4921 cpu_manager.go:414] "containerMap: removing container" podUID="a3a4a0cd-f846-409b-b7af-8ee554155bc1" containerName="revoke-keys"
Nov 03 20:23:24.073245 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:24.073140    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3a4a0cd-f846-409b-b7af-8ee554155bc1" containerName="revoke-keys"
Nov 03 20:23:24.073390 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:24.073291    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3a4a0cd-f846-409b-b7af-8ee554155bc1" containerName="revoke-keys"
Nov 03 20:23:24.076650 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:24.076628    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-9kwv8"
Nov 03 20:23:24.082870 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:24.082841    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w2ggk\""
Nov 03 20:23:24.082870 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:24.082860    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:23:24.086244 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:24.086218    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-9kwv8"]
Nov 03 20:23:24.203201 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:24.203151    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbbcb8b1-b3b0-4b2c-aa58-2e34358e2561-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-9kwv8\" (UID: \"cbbcb8b1-b3b0-4b2c-aa58-2e34358e2561\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-9kwv8"
Nov 03 20:23:24.203427 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:24.203214    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvbcx\" (UniqueName: \"kubernetes.io/projected/cbbcb8b1-b3b0-4b2c-aa58-2e34358e2561-kube-api-access-zvbcx\") pod \"maas-api-revoke-sub-f03629db-9kwv8\" (UID: \"cbbcb8b1-b3b0-4b2c-aa58-2e34358e2561\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-9kwv8"
Nov 03 20:23:24.203427 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:24.203348    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbbcb8b1-b3b0-4b2c-aa58-2e34358e2561-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-9kwv8\" (UID: \"cbbcb8b1-b3b0-4b2c-aa58-2e34358e2561\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-9kwv8"
Nov 03 20:23:24.304603 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:24.304546    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbbcb8b1-b3b0-4b2c-aa58-2e34358e2561-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-9kwv8\" (UID: \"cbbcb8b1-b3b0-4b2c-aa58-2e34358e2561\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-9kwv8"
Nov 03 20:23:24.304603 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:24.304618    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvbcx\" (UniqueName: \"kubernetes.io/projected/cbbcb8b1-b3b0-4b2c-aa58-2e34358e2561-kube-api-access-zvbcx\") pod \"maas-api-revoke-sub-f03629db-9kwv8\" (UID: \"cbbcb8b1-b3b0-4b2c-aa58-2e34358e2561\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-9kwv8"
Nov 03 20:23:24.304914 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:24.304729    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbbcb8b1-b3b0-4b2c-aa58-2e34358e2561-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-9kwv8\" (UID: \"cbbcb8b1-b3b0-4b2c-aa58-2e34358e2561\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-9kwv8"
Nov 03 20:23:24.305407 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:24.305385    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbbcb8b1-b3b0-4b2c-aa58-2e34358e2561-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-9kwv8\" (UID: \"cbbcb8b1-b3b0-4b2c-aa58-2e34358e2561\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-9kwv8"
Nov 03 20:23:24.307074 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:24.307053    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbbcb8b1-b3b0-4b2c-aa58-2e34358e2561-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-9kwv8\" (UID: \"cbbcb8b1-b3b0-4b2c-aa58-2e34358e2561\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-9kwv8"
Nov 03 20:23:24.311730 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:24.311711    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvbcx\" (UniqueName: \"kubernetes.io/projected/cbbcb8b1-b3b0-4b2c-aa58-2e34358e2561-kube-api-access-zvbcx\") pod \"maas-api-revoke-sub-f03629db-9kwv8\" (UID: \"cbbcb8b1-b3b0-4b2c-aa58-2e34358e2561\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-9kwv8"
Nov 03 20:23:24.391795 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:24.391710    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-9kwv8"
Nov 03 20:23:24.664636 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:24.664608    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-9kwv8"]
Nov 03 20:23:24.667052 ip-10-0-134-222 kubenswrapper[4921]: W1103 20:23:24.667023    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbbcb8b1_b3b0_4b2c_aa58_2e34358e2561.slice/crio-8cd9d3e1b8f329343fb8b77bb8b624aa1b4719debff3504bb31267d8ac991f75 WatchSource:0}: Error finding container 8cd9d3e1b8f329343fb8b77bb8b624aa1b4719debff3504bb31267d8ac991f75: Status 404 returned error can't find the container with id 8cd9d3e1b8f329343fb8b77bb8b624aa1b4719debff3504bb31267d8ac991f75
Nov 03 20:23:25.369688 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:25.369618    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-9kwv8" event={"ID":"cbbcb8b1-b3b0-4b2c-aa58-2e34358e2561","Type":"ContainerStarted","Data":"1246d826b3e1740516bbe814da53773d4bd094e371a55e4f0190b4fd601e6750"}
Nov 03 20:23:25.370112 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:25.369706    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-9kwv8" event={"ID":"cbbcb8b1-b3b0-4b2c-aa58-2e34358e2561","Type":"ContainerStarted","Data":"8cd9d3e1b8f329343fb8b77bb8b624aa1b4719debff3504bb31267d8ac991f75"}
Nov 03 20:23:25.383537 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:25.383483    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-9kwv8" podStartSLOduration=1.383467745 podStartE2EDuration="1.383467745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:23:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:23:25.381928607 +0000 UTC m=+3118.625894085" watchObservedRunningTime="2026-11-03 20:23:25.383467745 +0000 UTC m=+3118.627433209"
Nov 03 20:23:26.376979 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:26.376947    4921 generic.go:356] "Generic (PLEG): container finished" podID="cbbcb8b1-b3b0-4b2c-aa58-2e34358e2561" containerID="1246d826b3e1740516bbe814da53773d4bd094e371a55e4f0190b4fd601e6750" exitCode=0
Nov 03 20:23:26.377405 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:26.377033    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-9kwv8" event={"ID":"cbbcb8b1-b3b0-4b2c-aa58-2e34358e2561","Type":"ContainerDied","Data":"1246d826b3e1740516bbe814da53773d4bd094e371a55e4f0190b4fd601e6750"}
Nov 03 20:23:27.517246 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:27.517215    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-9kwv8"
Nov 03 20:23:27.636580 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:27.636489    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbbcb8b1-b3b0-4b2c-aa58-2e34358e2561-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbbcb8b1-b3b0-4b2c-aa58-2e34358e2561-maas-api-xks-ca\") pod \"cbbcb8b1-b3b0-4b2c-aa58-2e34358e2561\" (UID: \"cbbcb8b1-b3b0-4b2c-aa58-2e34358e2561\") "
Nov 03 20:23:27.636580 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:27.636537    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbbcb8b1-b3b0-4b2c-aa58-2e34358e2561-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbbcb8b1-b3b0-4b2c-aa58-2e34358e2561-maas-api-openshift-ca\") pod \"cbbcb8b1-b3b0-4b2c-aa58-2e34358e2561\" (UID: \"cbbcb8b1-b3b0-4b2c-aa58-2e34358e2561\") "
Nov 03 20:23:27.636868 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:27.636634    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbbcb8b1-b3b0-4b2c-aa58-2e34358e2561-kube-api-access-zvbcx\" (UniqueName: \"kubernetes.io/projected/cbbcb8b1-b3b0-4b2c-aa58-2e34358e2561-kube-api-access-zvbcx\") pod \"cbbcb8b1-b3b0-4b2c-aa58-2e34358e2561\" (UID: \"cbbcb8b1-b3b0-4b2c-aa58-2e34358e2561\") "
Nov 03 20:23:27.637030 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:27.637004    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbbcb8b1-b3b0-4b2c-aa58-2e34358e2561-maas-api-openshift-ca" pod "cbbcb8b1-b3b0-4b2c-aa58-2e34358e2561" (UID: "cbbcb8b1-b3b0-4b2c-aa58-2e34358e2561"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:23:27.638706 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:27.638663    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbbcb8b1-b3b0-4b2c-aa58-2e34358e2561-maas-api-xks-ca" pod "cbbcb8b1-b3b0-4b2c-aa58-2e34358e2561" (UID: "cbbcb8b1-b3b0-4b2c-aa58-2e34358e2561"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:23:27.638807 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:27.638755    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbbcb8b1-b3b0-4b2c-aa58-2e34358e2561-kube-api-access-zvbcx" pod "cbbcb8b1-b3b0-4b2c-aa58-2e34358e2561" (UID: "cbbcb8b1-b3b0-4b2c-aa58-2e34358e2561"). InnerVolumeSpecName "kube-api-access-zvbcx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:23:27.737959 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:27.737926    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbbcb8b1-b3b0-4b2c-aa58-2e34358e2561-maas-api-xks-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:27.737959 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:27.737954    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbbcb8b1-b3b0-4b2c-aa58-2e34358e2561-maas-api-openshift-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:27.737959 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:27.737966    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvbcx\" (UniqueName: \"kubernetes.io/projected/cbbcb8b1-b3b0-4b2c-aa58-2e34358e2561-kube-api-access-zvbcx\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:28.390139 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:28.390107    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-9kwv8"
Nov 03 20:23:28.390139 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:28.390116    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-9kwv8" event={"ID":"cbbcb8b1-b3b0-4b2c-aa58-2e34358e2561","Type":"ContainerDied","Data":"8cd9d3e1b8f329343fb8b77bb8b624aa1b4719debff3504bb31267d8ac991f75"}
Nov 03 20:23:28.390417 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:28.390159    4921 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8cd9d3e1b8f329343fb8b77bb8b624aa1b4719debff3504bb31267d8ac991f75"
Nov 03 20:23:32.920852 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:23:32.920815    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 20:23:36.022122 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:36.022088    4921 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e696f37-2dzls"]
Nov 03 20:23:36.025896 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:36.025862    4921 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e696f37-2dzls"]
Nov 03 20:23:36.892951 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:36.892912    4921 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8207a8b4-aab5-43e9-8967-c4ada60ed838" path="/var/lib/kubelet/pods/8207a8b4-aab5-43e9-8967-c4ada60ed838/volumes"
Nov 03 20:23:39.943385 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:39.943345    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-wkn28"]
Nov 03 20:23:39.943808 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:39.943774    4921 cpu_manager.go:414] "containerMap: removing container" podUID="cbbcb8b1-b3b0-4b2c-aa58-2e34358e2561" containerName="revoke-keys"
Nov 03 20:23:39.943808 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:39.943786    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbbcb8b1-b3b0-4b2c-aa58-2e34358e2561" containerName="revoke-keys"
Nov 03 20:23:39.943890 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:39.943875    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbbcb8b1-b3b0-4b2c-aa58-2e34358e2561" containerName="revoke-keys"
Nov 03 20:23:39.948312 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:39.948291    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-wkn28"
Nov 03 20:23:39.952870 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:39.952845    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:23:39.953000 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:39.952923    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w2ggk\""
Nov 03 20:23:39.956617 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:39.956246    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-wkn28"]
Nov 03 20:23:39.977785 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:39.977750    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc3358ba-263a-467a-b5f6-6f07313a494d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-wkn28\" (UID: \"dc3358ba-263a-467a-b5f6-6f07313a494d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-wkn28"
Nov 03 20:23:39.977963 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:39.977804    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc3358ba-263a-467a-b5f6-6f07313a494d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-wkn28\" (UID: \"dc3358ba-263a-467a-b5f6-6f07313a494d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-wkn28"
Nov 03 20:23:39.977963 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:39.977860    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfqzd\" (UniqueName: \"kubernetes.io/projected/dc3358ba-263a-467a-b5f6-6f07313a494d-kube-api-access-mfqzd\") pod \"maas-api-revoke-sub-0165d7d2-wkn28\" (UID: \"dc3358ba-263a-467a-b5f6-6f07313a494d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-wkn28"
Nov 03 20:23:40.079083 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:40.079046    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc3358ba-263a-467a-b5f6-6f07313a494d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-wkn28\" (UID: \"dc3358ba-263a-467a-b5f6-6f07313a494d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-wkn28"
Nov 03 20:23:40.079270 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:40.079095    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc3358ba-263a-467a-b5f6-6f07313a494d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-wkn28\" (UID: \"dc3358ba-263a-467a-b5f6-6f07313a494d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-wkn28"
Nov 03 20:23:40.079270 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:40.079127    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfqzd\" (UniqueName: \"kubernetes.io/projected/dc3358ba-263a-467a-b5f6-6f07313a494d-kube-api-access-mfqzd\") pod \"maas-api-revoke-sub-0165d7d2-wkn28\" (UID: \"dc3358ba-263a-467a-b5f6-6f07313a494d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-wkn28"
Nov 03 20:23:40.079884 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:40.079859    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc3358ba-263a-467a-b5f6-6f07313a494d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-wkn28\" (UID: \"dc3358ba-263a-467a-b5f6-6f07313a494d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-wkn28"
Nov 03 20:23:40.081615 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:40.081591    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc3358ba-263a-467a-b5f6-6f07313a494d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-wkn28\" (UID: \"dc3358ba-263a-467a-b5f6-6f07313a494d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-wkn28"
Nov 03 20:23:40.086251 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:40.086225    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfqzd\" (UniqueName: \"kubernetes.io/projected/dc3358ba-263a-467a-b5f6-6f07313a494d-kube-api-access-mfqzd\") pod \"maas-api-revoke-sub-0165d7d2-wkn28\" (UID: \"dc3358ba-263a-467a-b5f6-6f07313a494d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-wkn28"
Nov 03 20:23:40.266961 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:40.266931    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-wkn28"
Nov 03 20:23:40.540107 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:40.540083    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-wkn28"]
Nov 03 20:23:40.542098 ip-10-0-134-222 kubenswrapper[4921]: W1103 20:23:40.542068    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc3358ba_263a_467a_b5f6_6f07313a494d.slice/crio-ede4db1c9698ad6de475df08bd1241c2fb437f857bc17a95b1e995599359e351 WatchSource:0}: Error finding container ede4db1c9698ad6de475df08bd1241c2fb437f857bc17a95b1e995599359e351: Status 404 returned error can't find the container with id ede4db1c9698ad6de475df08bd1241c2fb437f857bc17a95b1e995599359e351
Nov 03 20:23:41.466955 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:41.466923    4921 generic.go:356] "Generic (PLEG): container finished" podID="dc3358ba-263a-467a-b5f6-6f07313a494d" containerID="e3a6f15a87106425e37fd3131fcaf69d48d2a9341cfe95aabba6540f1c5aa609" exitCode=0
Nov 03 20:23:41.467346 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:41.466981    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-wkn28" event={"ID":"dc3358ba-263a-467a-b5f6-6f07313a494d","Type":"ContainerDied","Data":"e3a6f15a87106425e37fd3131fcaf69d48d2a9341cfe95aabba6540f1c5aa609"}
Nov 03 20:23:41.467346 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:41.467021    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-wkn28" event={"ID":"dc3358ba-263a-467a-b5f6-6f07313a494d","Type":"ContainerStarted","Data":"ede4db1c9698ad6de475df08bd1241c2fb437f857bc17a95b1e995599359e351"}
Nov 03 20:23:42.601091 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:42.601065    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-wkn28"
Nov 03 20:23:42.701266 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:42.701230    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc3358ba-263a-467a-b5f6-6f07313a494d-kube-api-access-mfqzd\" (UniqueName: \"kubernetes.io/projected/dc3358ba-263a-467a-b5f6-6f07313a494d-kube-api-access-mfqzd\") pod \"dc3358ba-263a-467a-b5f6-6f07313a494d\" (UID: \"dc3358ba-263a-467a-b5f6-6f07313a494d\") "
Nov 03 20:23:42.701445 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:42.701349    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc3358ba-263a-467a-b5f6-6f07313a494d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc3358ba-263a-467a-b5f6-6f07313a494d-maas-api-xks-ca\") pod \"dc3358ba-263a-467a-b5f6-6f07313a494d\" (UID: \"dc3358ba-263a-467a-b5f6-6f07313a494d\") "
Nov 03 20:23:42.701445 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:42.701379    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc3358ba-263a-467a-b5f6-6f07313a494d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc3358ba-263a-467a-b5f6-6f07313a494d-maas-api-openshift-ca\") pod \"dc3358ba-263a-467a-b5f6-6f07313a494d\" (UID: \"dc3358ba-263a-467a-b5f6-6f07313a494d\") "
Nov 03 20:23:42.701810 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:42.701785    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc3358ba-263a-467a-b5f6-6f07313a494d-maas-api-openshift-ca" pod "dc3358ba-263a-467a-b5f6-6f07313a494d" (UID: "dc3358ba-263a-467a-b5f6-6f07313a494d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:23:42.703300 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:42.703281    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc3358ba-263a-467a-b5f6-6f07313a494d-maas-api-xks-ca" pod "dc3358ba-263a-467a-b5f6-6f07313a494d" (UID: "dc3358ba-263a-467a-b5f6-6f07313a494d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:23:42.703364 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:42.703340    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc3358ba-263a-467a-b5f6-6f07313a494d-kube-api-access-mfqzd" pod "dc3358ba-263a-467a-b5f6-6f07313a494d" (UID: "dc3358ba-263a-467a-b5f6-6f07313a494d"). InnerVolumeSpecName "kube-api-access-mfqzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:23:42.802834 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:42.802800    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfqzd\" (UniqueName: \"kubernetes.io/projected/dc3358ba-263a-467a-b5f6-6f07313a494d-kube-api-access-mfqzd\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:42.802834 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:42.802830    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc3358ba-263a-467a-b5f6-6f07313a494d-maas-api-xks-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:42.802834 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:42.802840    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc3358ba-263a-467a-b5f6-6f07313a494d-maas-api-openshift-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:43.479267 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:43.479234    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-wkn28" event={"ID":"dc3358ba-263a-467a-b5f6-6f07313a494d","Type":"ContainerDied","Data":"ede4db1c9698ad6de475df08bd1241c2fb437f857bc17a95b1e995599359e351"}
Nov 03 20:23:43.479267 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:43.479256    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-wkn28"
Nov 03 20:23:43.479482 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:43.479285    4921 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ede4db1c9698ad6de475df08bd1241c2fb437f857bc17a95b1e995599359e351"
Nov 03 20:23:50.754519 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:50.754484    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-d858r"]
Nov 03 20:23:50.754934 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:50.754919    4921 cpu_manager.go:414] "containerMap: removing container" podUID="dc3358ba-263a-467a-b5f6-6f07313a494d" containerName="revoke-keys"
Nov 03 20:23:50.754980 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:50.754935    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc3358ba-263a-467a-b5f6-6f07313a494d" containerName="revoke-keys"
Nov 03 20:23:50.755049 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:50.755037    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc3358ba-263a-467a-b5f6-6f07313a494d" containerName="revoke-keys"
Nov 03 20:23:50.758419 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:50.758400    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-d858r"
Nov 03 20:23:50.765019 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:50.764995    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:23:50.765177 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:50.765099    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w2ggk\""
Nov 03 20:23:50.768776 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:50.768749    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-d858r"]
Nov 03 20:23:50.873564 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:50.873529    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65157bb6-230a-4d47-86e9-f3518ac4d77d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-d858r\" (UID: \"65157bb6-230a-4d47-86e9-f3518ac4d77d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-d858r"
Nov 03 20:23:50.873777 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:50.873574    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65157bb6-230a-4d47-86e9-f3518ac4d77d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-d858r\" (UID: \"65157bb6-230a-4d47-86e9-f3518ac4d77d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-d858r"
Nov 03 20:23:50.873777 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:50.873691    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbbgf\" (UniqueName: \"kubernetes.io/projected/65157bb6-230a-4d47-86e9-f3518ac4d77d-kube-api-access-lbbgf\") pod \"maas-api-revoke-sub-6b19338f-d858r\" (UID: \"65157bb6-230a-4d47-86e9-f3518ac4d77d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-d858r"
Nov 03 20:23:50.974473 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:50.974441    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65157bb6-230a-4d47-86e9-f3518ac4d77d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-d858r\" (UID: \"65157bb6-230a-4d47-86e9-f3518ac4d77d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-d858r"
Nov 03 20:23:50.974473 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:50.974476    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65157bb6-230a-4d47-86e9-f3518ac4d77d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-d858r\" (UID: \"65157bb6-230a-4d47-86e9-f3518ac4d77d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-d858r"
Nov 03 20:23:50.974734 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:50.974522    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbbgf\" (UniqueName: \"kubernetes.io/projected/65157bb6-230a-4d47-86e9-f3518ac4d77d-kube-api-access-lbbgf\") pod \"maas-api-revoke-sub-6b19338f-d858r\" (UID: \"65157bb6-230a-4d47-86e9-f3518ac4d77d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-d858r"
Nov 03 20:23:50.975217 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:50.975190    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65157bb6-230a-4d47-86e9-f3518ac4d77d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-d858r\" (UID: \"65157bb6-230a-4d47-86e9-f3518ac4d77d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-d858r"
Nov 03 20:23:50.976783 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:50.976762    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65157bb6-230a-4d47-86e9-f3518ac4d77d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-d858r\" (UID: \"65157bb6-230a-4d47-86e9-f3518ac4d77d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-d858r"
Nov 03 20:23:50.981168 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:50.981140    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbbgf\" (UniqueName: \"kubernetes.io/projected/65157bb6-230a-4d47-86e9-f3518ac4d77d-kube-api-access-lbbgf\") pod \"maas-api-revoke-sub-6b19338f-d858r\" (UID: \"65157bb6-230a-4d47-86e9-f3518ac4d77d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-d858r"
Nov 03 20:23:51.076014 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:51.075932    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-d858r"
Nov 03 20:23:51.350551 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:51.350518    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-d858r"]
Nov 03 20:23:51.352409 ip-10-0-134-222 kubenswrapper[4921]: W1103 20:23:51.352377    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65157bb6_230a_4d47_86e9_f3518ac4d77d.slice/crio-bbe14471ec0d89811ca00a1d0898cf22fb1af240c1fed07be9f19aa888e9e2ba WatchSource:0}: Error finding container bbe14471ec0d89811ca00a1d0898cf22fb1af240c1fed07be9f19aa888e9e2ba: Status 404 returned error can't find the container with id bbe14471ec0d89811ca00a1d0898cf22fb1af240c1fed07be9f19aa888e9e2ba
Nov 03 20:23:51.525238 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:51.525200    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-d858r" event={"ID":"65157bb6-230a-4d47-86e9-f3518ac4d77d","Type":"ContainerStarted","Data":"bbe14471ec0d89811ca00a1d0898cf22fb1af240c1fed07be9f19aa888e9e2ba"}
Nov 03 20:23:52.533130 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:52.533096    4921 generic.go:356] "Generic (PLEG): container finished" podID="65157bb6-230a-4d47-86e9-f3518ac4d77d" containerID="934a79dde6b1f771573ea843016f33ae1cdcc7c1709913b9033cec938e5cbdff" exitCode=0
Nov 03 20:23:52.533561 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:52.533190    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-d858r" event={"ID":"65157bb6-230a-4d47-86e9-f3518ac4d77d","Type":"ContainerDied","Data":"934a79dde6b1f771573ea843016f33ae1cdcc7c1709913b9033cec938e5cbdff"}
Nov 03 20:23:53.682307 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:53.682284    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-d858r"
Nov 03 20:23:53.805177 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:53.805084    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65157bb6-230a-4d47-86e9-f3518ac4d77d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65157bb6-230a-4d47-86e9-f3518ac4d77d-maas-api-openshift-ca\") pod \"65157bb6-230a-4d47-86e9-f3518ac4d77d\" (UID: \"65157bb6-230a-4d47-86e9-f3518ac4d77d\") "
Nov 03 20:23:53.805177 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:53.805126    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65157bb6-230a-4d47-86e9-f3518ac4d77d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65157bb6-230a-4d47-86e9-f3518ac4d77d-maas-api-xks-ca\") pod \"65157bb6-230a-4d47-86e9-f3518ac4d77d\" (UID: \"65157bb6-230a-4d47-86e9-f3518ac4d77d\") "
Nov 03 20:23:53.805177 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:53.805171    4921 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65157bb6-230a-4d47-86e9-f3518ac4d77d-kube-api-access-lbbgf\" (UniqueName: \"kubernetes.io/projected/65157bb6-230a-4d47-86e9-f3518ac4d77d-kube-api-access-lbbgf\") pod \"65157bb6-230a-4d47-86e9-f3518ac4d77d\" (UID: \"65157bb6-230a-4d47-86e9-f3518ac4d77d\") "
Nov 03 20:23:53.805536 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:53.805511    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65157bb6-230a-4d47-86e9-f3518ac4d77d-maas-api-openshift-ca" pod "65157bb6-230a-4d47-86e9-f3518ac4d77d" (UID: "65157bb6-230a-4d47-86e9-f3518ac4d77d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:23:53.807259 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:53.807235    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65157bb6-230a-4d47-86e9-f3518ac4d77d-maas-api-xks-ca" pod "65157bb6-230a-4d47-86e9-f3518ac4d77d" (UID: "65157bb6-230a-4d47-86e9-f3518ac4d77d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:23:53.807359 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:53.807295    4921 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65157bb6-230a-4d47-86e9-f3518ac4d77d-kube-api-access-lbbgf" pod "65157bb6-230a-4d47-86e9-f3518ac4d77d" (UID: "65157bb6-230a-4d47-86e9-f3518ac4d77d"). InnerVolumeSpecName "kube-api-access-lbbgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:23:53.906324 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:53.906291    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65157bb6-230a-4d47-86e9-f3518ac4d77d-maas-api-openshift-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:53.906324 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:53.906319    4921 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65157bb6-230a-4d47-86e9-f3518ac4d77d-maas-api-xks-ca\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:53.906324 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:53.906329    4921 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbbgf\" (UniqueName: \"kubernetes.io/projected/65157bb6-230a-4d47-86e9-f3518ac4d77d-kube-api-access-lbbgf\") on node \"ip-10-0-134-222.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:54.545958 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:54.545930    4921 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-d858r"
Nov 03 20:23:54.546146 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:54.545956    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-d858r" event={"ID":"65157bb6-230a-4d47-86e9-f3518ac4d77d","Type":"ContainerDied","Data":"bbe14471ec0d89811ca00a1d0898cf22fb1af240c1fed07be9f19aa888e9e2ba"}
Nov 03 20:23:54.546146 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:23:54.546000    4921 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bbe14471ec0d89811ca00a1d0898cf22fb1af240c1fed07be9f19aa888e9e2ba"
Nov 03 20:24:02.010012 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:24:02.009975    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6ccf875c55-rz2d8_7e77f59c-fa04-4fef-80f6-6da7f2fbf046/authorino/0.log"
Nov 03 20:24:02.943494 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:24:02.943457    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 20:24:03.740506 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:24:03.740464    4921 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-cz4nm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:24:03.740964 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:24:03.740530    4921 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-cz4nm" podUID="5053431a-0bcc-4e99-8776-c226339e3bde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:24:16.750009 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:24:16.749976    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-c5ldk_b45584be-2f2e-4d38-9954-00befca7e68f/manager/0.log"
Nov 03 20:24:16.927988 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:24:16.927955    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-9dhcg_4a679f4b-2bbf-42b8-96a1-a3b6469a43ce/manager/0.log"
Nov 03 20:24:17.124007 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:24:17.123927    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-gp6bm_9c518498-21b1-4e86-9e9c-4f67855afa5f/manager/0.log"
Nov 03 20:24:17.597008 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:24:17.596979    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-6r9xw_21d707aa-f4eb-4b58-b122-86a668ba2b6c/server/0.log"
Nov 03 20:24:19.064305 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:24:19.064273    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-wkn28_dc3358ba-263a-467a-b5f6-6f07313a494d/revoke-keys/0.log"
Nov 03 20:24:19.570312 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:24:19.570288    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-d858r_65157bb6-230a-4d47-86e9-f3518ac4d77d/revoke-keys/0.log"
Nov 03 20:24:19.897285 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:24:19.897217    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-9kwv8_cbbcb8b1-b3b0-4b2c-aa58-2e34358e2561/revoke-keys/0.log"
Nov 03 20:24:20.064356 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:24:20.064324    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-jhps4_2dc0536d-df3d-4cf9-829f-83a55b746a05/postgres/0.log"
Nov 03 20:24:21.508211 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:24:21.508178    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6ccf875c55-rz2d8_7e77f59c-fa04-4fef-80f6-6da7f2fbf046/authorino/0.log"
Nov 03 20:24:21.732009 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:24:21.731978    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-4nlfd_6c22cc72-92e3-42a2-9a82-ecc3973562f8/manager/0.log"
Nov 03 20:24:21.914065 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:24:21.913991    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-5gsgr_8ac562db-3ea3-4b98-8189-d758dd0f500a/manager/0.log"
Nov 03 20:24:22.667316 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:24:22.667287    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-6sfqz_0eff05fe-81c6-4ced-adfe-e426333985a8/manager/0.log"
Nov 03 20:24:23.375123 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:24:23.375080    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6c786f9b57rcpnr_36946052-eb8d-448d-8907-db1ab6888bf3/istio-proxy/0.log"
Nov 03 20:24:23.534902 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:24:23.534870    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-rfxvm_effc08c9-9b29-4cd0-ac6b-3e52d748ab34/discovery/0.log"
Nov 03 20:24:24.406983 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:24:24.406952    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-cz4nm_5053431a-0bcc-4e99-8776-c226339e3bde/istio-proxy/0.log"
Nov 03 20:24:24.738498 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:24:24.738471    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-km8l8_4b858864-6582-48ad-9728-5e5e8c5aa818/istio-proxy/0.log"
Nov 03 20:24:25.583843 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:24:25.583815    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-zl5bx_d0e23fd0-6cce-402f-9e00-0f7da02ed53d/istio-proxy/1.log"
Nov 03 20:24:25.768583 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:24:25.768548    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-7454465567-vr74f_788abbb1-9316-46b4-8b47-a75066db292b/istio-proxy/0.log"
Nov 03 20:24:26.282012 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:24:26.281980    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-569d4c45f-7pszr_a023911d-3669-43e1-ac77-8a986f3715d6/router/0.log"
Nov 03 20:24:26.732500 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:24:26.732469    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-zl5bx_d0e23fd0-6cce-402f-9e00-0f7da02ed53d/istio-proxy/0.log"
Nov 03 20:24:29.184544 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:24:29.184508    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6ccf875c55-rz2d8_7e77f59c-fa04-4fef-80f6-6da7f2fbf046/authorino/0.log"
Nov 03 20:24:32.920164 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:24:32.920119    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 20:24:37.881783 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:24:37.881685    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-c5ldk_b45584be-2f2e-4d38-9954-00befca7e68f/manager/0.log"
Nov 03 20:24:38.060466 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:24:38.060438    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-9dhcg_4a679f4b-2bbf-42b8-96a1-a3b6469a43ce/manager/0.log"
Nov 03 20:24:38.249964 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:24:38.249939    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-gp6bm_9c518498-21b1-4e86-9e9c-4f67855afa5f/manager/0.log"
Nov 03 20:24:38.658273 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:24:38.658198    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-6r9xw_21d707aa-f4eb-4b58-b122-86a668ba2b6c/server/0.log"
Nov 03 20:24:39.883926 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:24:39.883894    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-wkn28_dc3358ba-263a-467a-b5f6-6f07313a494d/revoke-keys/0.log"
Nov 03 20:24:40.372859 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:24:40.372829    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-d858r_65157bb6-230a-4d47-86e9-f3518ac4d77d/revoke-keys/0.log"
Nov 03 20:24:40.695313 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:24:40.695235    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-9kwv8_cbbcb8b1-b3b0-4b2c-aa58-2e34358e2561/revoke-keys/0.log"
Nov 03 20:24:40.863715 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:24:40.863686    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-jhps4_2dc0536d-df3d-4cf9-829f-83a55b746a05/postgres/0.log"
Nov 03 20:24:41.887544 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:24:41.887513    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6ccf875c55-rz2d8_7e77f59c-fa04-4fef-80f6-6da7f2fbf046/authorino/0.log"
Nov 03 20:24:42.099146 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:24:42.099114    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-4nlfd_6c22cc72-92e3-42a2-9a82-ecc3973562f8/manager/0.log"
Nov 03 20:24:42.282973 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:24:42.282943    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-5gsgr_8ac562db-3ea3-4b98-8189-d758dd0f500a/manager/0.log"
Nov 03 20:24:42.973279 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:24:42.973244    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-6sfqz_0eff05fe-81c6-4ced-adfe-e426333985a8/manager/0.log"
Nov 03 20:24:43.494070 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:24:43.494040    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6c786f9b57rcpnr_36946052-eb8d-448d-8907-db1ab6888bf3/istio-proxy/0.log"
Nov 03 20:24:43.660221 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:24:43.660186    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-rfxvm_effc08c9-9b29-4cd0-ac6b-3e52d748ab34/discovery/0.log"
Nov 03 20:24:44.470767 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:24:44.470732    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-cz4nm_5053431a-0bcc-4e99-8776-c226339e3bde/istio-proxy/0.log"
Nov 03 20:24:44.816903 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:24:44.816873    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-km8l8_4b858864-6582-48ad-9728-5e5e8c5aa818/istio-proxy/0.log"
Nov 03 20:24:45.663571 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:24:45.663540    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-zl5bx_d0e23fd0-6cce-402f-9e00-0f7da02ed53d/istio-proxy/1.log"
Nov 03 20:24:45.846176 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:24:45.846144    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-7454465567-vr74f_788abbb1-9316-46b4-8b47-a75066db292b/istio-proxy/0.log"
Nov 03 20:24:46.343687 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:24:46.343650    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-569d4c45f-7pszr_a023911d-3669-43e1-ac77-8a986f3715d6/router/0.log"
Nov 03 20:25:02.919855 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:25:02.919820    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 20:25:32.077185 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:25:32.077147    4921 scope.go:122] "RemoveContainer" containerID="18af5b245b2b704afc78e08fed36c23fb4305e83f858f129dba71f63c79f9d0f"
Nov 03 20:25:32.089762 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:25:32.089742    4921 scope.go:122] "RemoveContainer" containerID="8bddceb14c3234f42e5984a10e50ecc6c103a278533cfd4ff278b7b590acac6d"
Nov 03 20:25:32.921233 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:25:32.921198    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 20:26:02.920487 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:26:02.920440    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 20:26:10.423708 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:10.423623    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-hrxdt_eee821c4-6fb0-4f84-a58d-23dc2ff77720/global-pull-secret-syncer/0.log"
Nov 03 20:26:10.514723 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:10.514694    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-42xn2_8fa315db-e48d-4ac5-960a-23750974d431/connection-checker/0.log"
Nov 03 20:26:10.545009 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:10.544981    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-wtgvw_f97b063e-8403-46bd-9716-72411589a494/connection-checker/0.log"
Nov 03 20:26:10.574986 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:10.574956    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-xgbkd_df9f9c4a-d054-42cf-b17c-0d23acd0459b/connection-checker/0.log"
Nov 03 20:26:10.729648 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:10.729619    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-jwrp9_1c00e787-a35d-4ac4-95a4-2546e29602aa/konnectivity-agent/0.log"
Nov 03 20:26:10.822946 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:10.822916    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-222.us-east-2.compute.internal_66cbb2b4d2abe69f7cf665c408777b87/haproxy/0.log"
Nov 03 20:26:18.197938 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:18.197902    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6ccf875c55-rz2d8_7e77f59c-fa04-4fef-80f6-6da7f2fbf046/authorino/0.log"
Nov 03 20:26:18.232537 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:18.232507    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-4nlfd_6c22cc72-92e3-42a2-9a82-ecc3973562f8/manager/0.log"
Nov 03 20:26:18.264254 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:18.264232    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-5gsgr_8ac562db-3ea3-4b98-8189-d758dd0f500a/manager/0.log"
Nov 03 20:26:18.490289 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:18.490255    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-6sfqz_0eff05fe-81c6-4ced-adfe-e426333985a8/manager/0.log"
Nov 03 20:26:19.958037 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:19.958004    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-kfdsx_e2b752d9-88c4-4dee-8b0c-7e9ac252adb3/cluster-monitoring-operator/0.log"
Nov 03 20:26:20.120961 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:20.120926    4921 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fg8c7/perf-node-gather-daemonset-lkmmk"]
Nov 03 20:26:20.121354 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:20.121340    4921 cpu_manager.go:414] "containerMap: removing container" podUID="65157bb6-230a-4d47-86e9-f3518ac4d77d" containerName="revoke-keys"
Nov 03 20:26:20.121402 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:20.121355    4921 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65157bb6-230a-4d47-86e9-f3518ac4d77d" containerName="revoke-keys"
Nov 03 20:26:20.121451 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:20.121442    4921 memory_manager.go:365] "RemoveStaleState removing state" podUID="65157bb6-230a-4d47-86e9-f3518ac4d77d" containerName="revoke-keys"
Nov 03 20:26:20.127392 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:20.127373    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fg8c7/perf-node-gather-daemonset-lkmmk"
Nov 03 20:26:20.130825 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:20.130794    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fg8c7\"/\"openshift-service-ca.crt\""
Nov 03 20:26:20.131389 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:20.131182    4921 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fg8c7\"/\"kube-root-ca.crt\""
Nov 03 20:26:20.131389 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:20.131250    4921 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-fg8c7\"/\"default-dockercfg-8dkgq\""
Nov 03 20:26:20.135198 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:20.135172    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fg8c7/perf-node-gather-daemonset-lkmmk"]
Nov 03 20:26:20.159285 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:20.159250    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2bg7x_131f2758-bb6e-4e73-979c-2db443bf6b05/node-exporter/0.log"
Nov 03 20:26:20.197807 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:20.197781    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2bg7x_131f2758-bb6e-4e73-979c-2db443bf6b05/kube-rbac-proxy/0.log"
Nov 03 20:26:20.217777 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:20.217708    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d0887431-231c-4cd4-93df-6e7f7ab4a62f-podres\") pod \"perf-node-gather-daemonset-lkmmk\" (UID: \"d0887431-231c-4cd4-93df-6e7f7ab4a62f\") " pod="openshift-must-gather-fg8c7/perf-node-gather-daemonset-lkmmk"
Nov 03 20:26:20.217777 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:20.217758    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d0887431-231c-4cd4-93df-6e7f7ab4a62f-proc\") pod \"perf-node-gather-daemonset-lkmmk\" (UID: \"d0887431-231c-4cd4-93df-6e7f7ab4a62f\") " pod="openshift-must-gather-fg8c7/perf-node-gather-daemonset-lkmmk"
Nov 03 20:26:20.217975 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:20.217832    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d0887431-231c-4cd4-93df-6e7f7ab4a62f-lib-modules\") pod \"perf-node-gather-daemonset-lkmmk\" (UID: \"d0887431-231c-4cd4-93df-6e7f7ab4a62f\") " pod="openshift-must-gather-fg8c7/perf-node-gather-daemonset-lkmmk"
Nov 03 20:26:20.217975 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:20.217860    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d0887431-231c-4cd4-93df-6e7f7ab4a62f-sys\") pod \"perf-node-gather-daemonset-lkmmk\" (UID: \"d0887431-231c-4cd4-93df-6e7f7ab4a62f\") " pod="openshift-must-gather-fg8c7/perf-node-gather-daemonset-lkmmk"
Nov 03 20:26:20.218114 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:20.217970    4921 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzm4q\" (UniqueName: \"kubernetes.io/projected/d0887431-231c-4cd4-93df-6e7f7ab4a62f-kube-api-access-gzm4q\") pod \"perf-node-gather-daemonset-lkmmk\" (UID: \"d0887431-231c-4cd4-93df-6e7f7ab4a62f\") " pod="openshift-must-gather-fg8c7/perf-node-gather-daemonset-lkmmk"
Nov 03 20:26:20.232578 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:20.232554    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2bg7x_131f2758-bb6e-4e73-979c-2db443bf6b05/init-textfile/0.log"
Nov 03 20:26:20.318624 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:20.318588    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d0887431-231c-4cd4-93df-6e7f7ab4a62f-proc\") pod \"perf-node-gather-daemonset-lkmmk\" (UID: \"d0887431-231c-4cd4-93df-6e7f7ab4a62f\") " pod="openshift-must-gather-fg8c7/perf-node-gather-daemonset-lkmmk"
Nov 03 20:26:20.318624 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:20.318629    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d0887431-231c-4cd4-93df-6e7f7ab4a62f-lib-modules\") pod \"perf-node-gather-daemonset-lkmmk\" (UID: \"d0887431-231c-4cd4-93df-6e7f7ab4a62f\") " pod="openshift-must-gather-fg8c7/perf-node-gather-daemonset-lkmmk"
Nov 03 20:26:20.318889 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:20.318762    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d0887431-231c-4cd4-93df-6e7f7ab4a62f-lib-modules\") pod \"perf-node-gather-daemonset-lkmmk\" (UID: \"d0887431-231c-4cd4-93df-6e7f7ab4a62f\") " pod="openshift-must-gather-fg8c7/perf-node-gather-daemonset-lkmmk"
Nov 03 20:26:20.318889 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:20.318764    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d0887431-231c-4cd4-93df-6e7f7ab4a62f-sys\") pod \"perf-node-gather-daemonset-lkmmk\" (UID: \"d0887431-231c-4cd4-93df-6e7f7ab4a62f\") " pod="openshift-must-gather-fg8c7/perf-node-gather-daemonset-lkmmk"
Nov 03 20:26:20.318889 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:20.318759    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d0887431-231c-4cd4-93df-6e7f7ab4a62f-proc\") pod \"perf-node-gather-daemonset-lkmmk\" (UID: \"d0887431-231c-4cd4-93df-6e7f7ab4a62f\") " pod="openshift-must-gather-fg8c7/perf-node-gather-daemonset-lkmmk"
Nov 03 20:26:20.318889 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:20.318830    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzm4q\" (UniqueName: \"kubernetes.io/projected/d0887431-231c-4cd4-93df-6e7f7ab4a62f-kube-api-access-gzm4q\") pod \"perf-node-gather-daemonset-lkmmk\" (UID: \"d0887431-231c-4cd4-93df-6e7f7ab4a62f\") " pod="openshift-must-gather-fg8c7/perf-node-gather-daemonset-lkmmk"
Nov 03 20:26:20.318889 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:20.318879    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d0887431-231c-4cd4-93df-6e7f7ab4a62f-sys\") pod \"perf-node-gather-daemonset-lkmmk\" (UID: \"d0887431-231c-4cd4-93df-6e7f7ab4a62f\") " pod="openshift-must-gather-fg8c7/perf-node-gather-daemonset-lkmmk"
Nov 03 20:26:20.319096 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:20.318886    4921 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d0887431-231c-4cd4-93df-6e7f7ab4a62f-podres\") pod \"perf-node-gather-daemonset-lkmmk\" (UID: \"d0887431-231c-4cd4-93df-6e7f7ab4a62f\") " pod="openshift-must-gather-fg8c7/perf-node-gather-daemonset-lkmmk"
Nov 03 20:26:20.319096 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:20.318936    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d0887431-231c-4cd4-93df-6e7f7ab4a62f-podres\") pod \"perf-node-gather-daemonset-lkmmk\" (UID: \"d0887431-231c-4cd4-93df-6e7f7ab4a62f\") " pod="openshift-must-gather-fg8c7/perf-node-gather-daemonset-lkmmk"
Nov 03 20:26:20.326257 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:20.326232    4921 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzm4q\" (UniqueName: \"kubernetes.io/projected/d0887431-231c-4cd4-93df-6e7f7ab4a62f-kube-api-access-gzm4q\") pod \"perf-node-gather-daemonset-lkmmk\" (UID: \"d0887431-231c-4cd4-93df-6e7f7ab4a62f\") " pod="openshift-must-gather-fg8c7/perf-node-gather-daemonset-lkmmk"
Nov 03 20:26:20.442204 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:20.442175    4921 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fg8c7/perf-node-gather-daemonset-lkmmk"
Nov 03 20:26:20.716712 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:20.716654    4921 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fg8c7/perf-node-gather-daemonset-lkmmk"]
Nov 03 20:26:20.719577 ip-10-0-134-222 kubenswrapper[4921]: W1103 20:26:20.719544    4921 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd0887431_231c_4cd4_93df_6e7f7ab4a62f.slice/crio-23487df98d78e52b9bca73f3083dd6376c39fdd44393be8763d9c4f99a0f9910 WatchSource:0}: Error finding container 23487df98d78e52b9bca73f3083dd6376c39fdd44393be8763d9c4f99a0f9910: Status 404 returned error can't find the container with id 23487df98d78e52b9bca73f3083dd6376c39fdd44393be8763d9c4f99a0f9910
Nov 03 20:26:21.408214 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:21.408177    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fg8c7/perf-node-gather-daemonset-lkmmk" event={"ID":"d0887431-231c-4cd4-93df-6e7f7ab4a62f","Type":"ContainerStarted","Data":"23b1d71f221589bea1336da23d660f1602977241645355b25376bd4530319fc3"}
Nov 03 20:26:21.408635 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:21.408238    4921 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fg8c7/perf-node-gather-daemonset-lkmmk" event={"ID":"d0887431-231c-4cd4-93df-6e7f7ab4a62f","Type":"ContainerStarted","Data":"23487df98d78e52b9bca73f3083dd6376c39fdd44393be8763d9c4f99a0f9910"}
Nov 03 20:26:21.408635 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:21.408265    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-fg8c7/perf-node-gather-daemonset-lkmmk"
Nov 03 20:26:21.424158 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:21.424093    4921 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-fg8c7/perf-node-gather-daemonset-lkmmk" podStartSLOduration=1.424073459 podStartE2EDuration="1.424073459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:26:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:26:21.420913171 +0000 UTC m=+3294.664878637" watchObservedRunningTime="2026-11-03 20:26:21.424073459 +0000 UTC m=+3294.668038925"
Nov 03 20:26:22.490431 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:22.490404    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-zpgl6_eb488ece-2ac3-499f-84c1-73534fcbb20b/networking-console-plugin/0.log"
Nov 03 20:26:23.071332 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:23.071306    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gzf8k_6f215b42-0f09-46a0-bc98-392c0fa909c9/console-operator/2.log"
Nov 03 20:26:23.076345 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:23.076312    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gzf8k_6f215b42-0f09-46a0-bc98-392c0fa909c9/console-operator/3.log"
Nov 03 20:26:24.246815 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:24.246778    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-tx285_f89ef12d-9c60-4538-af9c-f97453bbc855/volume-data-source-validator/0.log"
Nov 03 20:26:25.582335 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:25.582303    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wj8mp_38a6add1-b051-450b-bb96-8ace8a0ccff3/dns/0.log"
Nov 03 20:26:25.609314 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:25.609280    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wj8mp_38a6add1-b051-450b-bb96-8ace8a0ccff3/kube-rbac-proxy/0.log"
Nov 03 20:26:25.668919 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:25.668893    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-p2p8d_56e21184-e568-46bc-8e90-204684d6c366/dns-node-resolver/0.log"
Nov 03 20:26:26.431328 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:26.431289    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vv2s2_545f3c9a-2ac6-44a3-be0b-cf71b1592d42/node-ca/0.log"
Nov 03 20:26:27.423995 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:27.423964    4921 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-fg8c7/perf-node-gather-daemonset-lkmmk"
Nov 03 20:26:27.563251 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:27.563226    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6c786f9b57rcpnr_36946052-eb8d-448d-8907-db1ab6888bf3/istio-proxy/0.log"
Nov 03 20:26:28.157106 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:28.157074    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-rfxvm_effc08c9-9b29-4cd0-ac6b-3e52d748ab34/discovery/0.log"
Nov 03 20:26:29.022962 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:29.022932    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-cz4nm_5053431a-0bcc-4e99-8776-c226339e3bde/istio-proxy/0.log"
Nov 03 20:26:29.093461 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:29.093434    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-km8l8_4b858864-6582-48ad-9728-5e5e8c5aa818/istio-proxy/0.log"
Nov 03 20:26:29.266515 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:29.266485    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-zl5bx_d0e23fd0-6cce-402f-9e00-0f7da02ed53d/istio-proxy/0.log"
Nov 03 20:26:29.267994 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:29.267975    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-zl5bx_d0e23fd0-6cce-402f-9e00-0f7da02ed53d/istio-proxy/1.log"
Nov 03 20:26:29.295042 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:29.294976    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-7454465567-vr74f_788abbb1-9316-46b4-8b47-a75066db292b/istio-proxy/0.log"
Nov 03 20:26:29.395913 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:29.395871    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-569d4c45f-7pszr_a023911d-3669-43e1-ac77-8a986f3715d6/router/0.log"
Nov 03 20:26:30.773053 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:30.773025    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6c786f9b57rcpnr_36946052-eb8d-448d-8907-db1ab6888bf3/istio-proxy/0.log"
Nov 03 20:26:31.360944 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:31.360909    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-rfxvm_effc08c9-9b29-4cd0-ac6b-3e52d748ab34/discovery/0.log"
Nov 03 20:26:31.458342 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:31.458309    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-cz4nm_5053431a-0bcc-4e99-8776-c226339e3bde/istio-proxy/0.log"
Nov 03 20:26:31.500398 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:31.500364    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-km8l8_4b858864-6582-48ad-9728-5e5e8c5aa818/istio-proxy/0.log"
Nov 03 20:26:31.610755 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:31.610724    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-zl5bx_d0e23fd0-6cce-402f-9e00-0f7da02ed53d/istio-proxy/0.log"
Nov 03 20:26:31.612511 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:31.612448    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-zl5bx_d0e23fd0-6cce-402f-9e00-0f7da02ed53d/istio-proxy/1.log"
Nov 03 20:26:31.625891 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:31.625865    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-7454465567-vr74f_788abbb1-9316-46b4-8b47-a75066db292b/istio-proxy/0.log"
Nov 03 20:26:31.685952 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:31.685918    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-569d4c45f-7pszr_a023911d-3669-43e1-ac77-8a986f3715d6/router/0.log"
Nov 03 20:26:31.950225 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:31.950146    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-j7m2z_543b8778-baa4-4e09-b697-68e550e42942/serve-healthcheck-canary/0.log"
Nov 03 20:26:32.052296 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:32.052272    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gzf8k_6f215b42-0f09-46a0-bc98-392c0fa909c9/console-operator/2.log"
Nov 03 20:26:32.084891 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:32.084863    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gzf8k_6f215b42-0f09-46a0-bc98-392c0fa909c9/console-operator/2.log"
Nov 03 20:26:32.689706 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:32.689660    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-8kh6f_3880f3d3-aad1-43c0-b704-01374aad4f78/insights-operator/1.log"
Nov 03 20:26:32.690623 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:32.690606    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-8kh6f_3880f3d3-aad1-43c0-b704-01374aad4f78/insights-operator/2.log"
Nov 03 20:26:32.869079 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:32.869054    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cvll4_10137ba2-4beb-433e-b6df-199738351c61/kube-rbac-proxy/0.log"
Nov 03 20:26:32.909846 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:32.909819    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cvll4_10137ba2-4beb-433e-b6df-199738351c61/exporter/0.log"
Nov 03 20:26:32.921896 ip-10-0-134-222 kubenswrapper[4921]: E1103 20:26:32.921863    4921 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cvll4" containerName="extractor"
Nov 03 20:26:32.950297 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:32.950217    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cvll4_10137ba2-4beb-433e-b6df-199738351c61/extractor/0.log"
Nov 03 20:26:34.939222 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:34.939175    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6c786f9b57rcpnr_36946052-eb8d-448d-8907-db1ab6888bf3/istio-proxy/0.log"
Nov 03 20:26:35.514992 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:35.514961    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-rfxvm_effc08c9-9b29-4cd0-ac6b-3e52d748ab34/discovery/0.log"
Nov 03 20:26:35.620334 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:35.620301    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-cz4nm_5053431a-0bcc-4e99-8776-c226339e3bde/istio-proxy/0.log"
Nov 03 20:26:35.657683 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:35.657643    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-km8l8_4b858864-6582-48ad-9728-5e5e8c5aa818/istio-proxy/0.log"
Nov 03 20:26:35.758583 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:35.758552    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-zl5bx_d0e23fd0-6cce-402f-9e00-0f7da02ed53d/istio-proxy/0.log"
Nov 03 20:26:35.760598 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:35.760576    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-zl5bx_d0e23fd0-6cce-402f-9e00-0f7da02ed53d/istio-proxy/1.log"
Nov 03 20:26:35.776749 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:35.776688    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-7454465567-vr74f_788abbb1-9316-46b4-8b47-a75066db292b/istio-proxy/0.log"
Nov 03 20:26:35.808858 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:35.808833    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-bd9746b54-rxrg2_a1b5e79c-2e6b-44d7-8562-d2e9240be700/manager/0.log"
Nov 03 20:26:35.834826 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:35.834799    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-569d4c45f-7pszr_a023911d-3669-43e1-ac77-8a986f3715d6/router/0.log"
Nov 03 20:26:35.837542 ip-10-0-134-222 kubenswrapper[4921]: I1103 20:26:35.837523    4921 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-bd9746b54-rxrg2_a1b5e79c-2e6b-44d7-8562-d2e9240be700/copy-manifests/0.log"